The Client

U.S Department of Education | Institute of Education Sciences

---

Objective

Upward Bound (UB), launched in 1965, is one of the oldest and largest federal college access programs targeted to low-income students and those who would represent the first-generation of college students in their families. Currently, UB serves more than 60,000 high school students offering an array of academic and college transition support services.

The structure and offerings of UB are prescribed in legislation, but little is known about the focus or delivery of these services or the extent to which they vary. Our report addressed this information gap by identifying approaches that UB projects use to provide core program services—advising, tutoring, academic coursework, college exposure, college entrance exam preparation, college application assistance, and financial aid application assistance.

Data were collected from a survey of UB project directors at institutions that host the projects. The primary goal of the survey was to help identify common—or uncommon, but promising—practices that could inform program improvement studies required of the U.S. Department of Education (ED) by Congress.

---

Methodology

Web Based Survey

Findings

Variation in Focus & Delivery of Services: Variation in the focus and delivery of services among providers appears related to the urbanicity and type of institution (4-year, 2-year, and non-higher education) that hosts the project, but not to other project characteristics examined. There were several differences across projects of different urbanicity and institution type.

Cultures of Giving

---

The Client

W.K Kellog Foundation

---

Objective

DIR conducted a 3-year evaluation of the W.K. Kellogg Foundation’s Cultures of Giving (COG) Cluster. COG was designed to increase the W. K. Kellogg Foundation’s understanding of the gaps in access to leadership, wealth, and influence by and among communities of color.

The clusters sought to change the landscape of the philanthropic community by increasing the quantity and improving the quality of leaders, organizations, networks, and effective practice of philanthropy across communities of color.

The COG cluster was comprised of 28 grantees that were expected to affect three basic outcomes:

• Leaders who were committed to community-led innovation
• New and stronger organizations and networks that promoted the practice
  of cross-race and cross-culture giving
• The development of effective models and practices of cross-race and cross-culture giving

---

Methodology

To accomplish these evaluation objectives and to help structure the data-collection process, DIR developed a conceptual model that postulated the relationship between activities, outputs, and outcomes.

Based on this conceptual model, DIR designed study questions, related indicators, and associated sub-questions to guide the collection of information about how COG grantee activities contributed or did not contribute to the expected cluster outcomes.

---

Findings

• Our findings revealed that effective change in communities, especially change in communities of color, appeared to require a sustained and committed effort. Even though COG was only a five-year effort, at its inception it represented the accumulation of nearly 10 years of knowledge acquisition and tested methods and strategies regarding giving in communities of color. The knowledge acquired in these early periods was instrumental in designing and implementing strategies with very clearly defined outcomes.
• The findings also revealed that the development of a community of practice was considerably more important to this process than originally anticipated. Initially, it was assumed that the direct activities of the grantees represented the primary mechanism by which change would be generated in communities of color.
• Although the activities of grantees were key factors in the change process, the activities of the community of practice played a much more dominant role. In this sense, it was the initial work around the community of practice that drove the work of the grantees, which, in turn, brought about increased giving in communities of color.

Rice University Space and Science Technology Program Teacher Professional Development Initiatives

---

The Client

Rice University

---

Objective

DIR used qualitative and quantitative methodologies to conduct a third-party evaluation of a teacher professional development initiative for the Rice University Space and Science Technology (SST) Program sponsored by Conoco Phillips and the National Science Foundation.

The Applied Math Program (AMP!) is a one-year professional development opportunity for eighth-grade science and mathematics teachers that seeks to accomplish the following objectives:

• Increase teacher content and knowledge
• Improve student engagement and achievement in STEM subjects
• Support and sustain teacher teams
• Create teacher teams that motivate STEM careers and inject a new culture of teacher support in school districts

DIR analyzed the relationship between teachers’ participation in AMP! focused on (1) leadership abilities, (2) mathematics and science efficacy, and (3) mathematics and science content abilities.

---

Methodology

In-depth interview protocols

Focus groups

Classroom observations

Analysis of student achievement data

---

Findings

Both quantitative and qualitative results indicate that the number and types of opportunities provided by AMP! were associated with achievement of some of the project’s goals and objectives.

Goal 1 – Increase mathematics and science content and pedagogical knowledge.

Qualitative findings showed that while mathematics teachers reported a higher level of confidence regarding their content knowledge than did science teachers, science teachers generally felt more confident about developing and facilitating inquiry-based lessons following AMP! implementation.

Goal 2 – Improve student engagement and achievement in STEM Subjects.

The pre-post results showed a) significantly more AMP! mathematics teachers reported that they give tests and/or quizzes that include constructed-response/open-ended items more often since experiencing AMP!; b) significantly more science teachers said that students were asking and answering their own questions and c) significantly more science teachers placed more emphasis on understanding science concepts.

Moreover, there is general evidence of a significant improvement in students’ performance on mathematics and/or science STAAR with an AMP! science or mathematics teacher than students without AMP! mathematics or science teachers, even when scores of the different demographic subgroups are observed.

Goal 3 – Create a supportive and rewarding environment to sustain AMP! teachers in high-needs schools

Qualitative evidence shows that inquiry-based lessons, new approaches to content vocabulary development, and the science-mathematics connection that was emphasized throughout the program assisted program participants in achieving rewarding classroom experiences that benefitted both the teachers and the students.

Goal 4 — Create a community of teachers that can motivate students toward STEM careers.

AMP! participants motivated students toward careers in STEM, primarily through the mathematics-science connection and the real-world applications of mathematics and science.

Goal 5 — Inject a new culture of support in the school districts for accomplished science teachers.

Teachers felt more able to connect mathematics and science, collaborate with their assigned partners as well as with others on their campus, and generally use several strategies to teach students using inquiry methods. Significantly higher post-test scores were posted for mathematics teachers on the needs assessment in three areas: (1) instruction methods, (2) teacher-directed vs. student-centered methods and activities, and (3) inquiry. Science teachers had significantly higher needs assessment scores on the post administration in four areas: (1) instruction methods, (2) understanding science concepts, (3) inquiry, and (4) STEM emphasis.

The Client

U.S Department of Labor | Employment and Training Administration

---

Objective

The Youth Opportunity (YO) Initiative program offered educational, employment, leadership enhancement, and other support services to all youth, ages 14 to 21, in 36 targeted high-poverty areas across the country in order to boost their high school graduation, college enrollment, and employment rates.

YO consisted of a network of large and complex projects intended to build the foundation for community-wide efforts to mobilize resources in helping youths to enter the economic mainstream. The three main objectives of the evaluation were to: 1) document and assess the effectiveness of the delivery of YO-funded services and leveraged services in the target areas; 2) assess the target areas’ sense of well-being before and after receipt of program services; and 3) measure the impact of the program on employment, educational enrollment and attainment, graduation rates, wages, welfare enrollment, and youth involvement in crime in target areas.

---

Methodology

In-depth interviews

Site visits

Focus groups

Area surveys to measure labor market outcomes

Comparisons using propensity-score matching

Analysis of grantee data

Analysis of data collection outcomes

---