Coalition for Evidence-Based Policy

Here's the original study report: mdrc.org/sites/defaul...

Here's our No-Spin report: nospin.evidencebasedpolicy.org/articles/new....

but a 12-year follow-up found sizable earnings gains. Longer-term follow-up of the P-TECH 9-14 study would be valuable to see if similar earnings impacts materialize.

5/ COMMENT: The study found the program to be well implemented - for example, providing students with significant workplace experience during high school. A high-quality RCT of a related model - Career Academies - found initially disappointing results (no impact on high school or college completion)

The study found a possible impact on college degree completion at the 7-year follow-up (13% T vs 8% C). However, this finding is based on only 16% of the sample (the early cohorts with seven years of available data) and not quite statistically significant (p=.08), so is best viewed as preliminary.

The study also found no statistically significant impact on the percent of students either enrolled in college or having completed a degree at the 5-, 6-, and 7-year follow-ups (e.g., 60% T vs 60% C at six years).

4/ FINDINGS: The study found no statistically significant impact on the high school graduation rate at follow-ups 4, 5, and 6 years after study entry (e.g., 76% T vs 77% C had graduated at six years).

Students were randomly assigned via lottery to a treatment group offered admission to a P-TECH 9-14 school (T) or a control group not offered admission (C). Based on careful review, this was a high quality RCT (e.g., baseline balance, negligible sample attrition).

The model has proliferated nationally and internationally since the first P-TECH 9-14 school opened in Brooklyn in 2010.

3/ STUDY DESIGN: The study sample comprised 3,161 rising 9th graders, 86% of whom were Black or Hispanic.

2/ PROGRAM: New York City’s P-TECH 9-14 schools are "a partnership among a high school, a community college, and one or more employer partners that focuses on preparing students for both college and careers within six years."

See No-Spin report on an RCT of P-TECH 9-14 College & Career Prep Schools. In brief: Despite positive portrayal of results in the study's overview, it found no discernible impact on high school grad'n or college progression over 7y, but future earnings impacts are possible.🧵

Beyond the Early Years: The Long-Term Effects of Home Visiting on Mothers, Families, and Children. Results from the Mother and Infant Home Visiting Program Evaluation (MIHOPE) This report presents results from the analysis of data from the kindergarten follow-up of the Mother and Infant Home Visiting Program Evaluation (MIHOPE).

And here's the original study report: acf.gov/opre/report/....

Here's our No-Spin report: nospin.evidencebasedpolicy.org/articles/hom....

8/ The study doesn't report the effect sizes for each of the 4 home visiting models individually. But it appears the model-specific effects are smallish since (as the report shows) few were statistically significant. We've asked the study team for more info on these effects.

7/ However, the statistically significant effects were so small as to be of little *practical* significance for children or families. Their average effect size was 0.03, which is the equivalent of moving the average child or mother from the 50th to the 51st percentile.

6/ 5-7 years after study entry – when most children were in kindergarten – the study found that home visiting produced statistically significant positive effects for 6 of the study’s 8 prespecified research questions. (5 were significant at 0.05; one was significant at 0.10.)

5/ Based on careful review, this is a high-quality RCT (e.g., baseline balance, prespecified outcomes and analyses, use of well-established measures of child and parent outcomes, moderate attrition for most outcomes).

4/ For this latest report on kindergarten-age outcomes, the study organized its 60 prespecified outcome measures into 8 prespecified research questions, and conducted an omnibus test of statistical significance for the set of outcomes within each question.

3/ The programs vary in key features, but all provide home visits (e.g., by a paraprofessional or nurse) to at-risk families that are expecting or have recently given birth. Families entered the study when their children were under six months old or in utero.

2/ This is a large, ongoing RCT, with a sample of 4,102 families across 12 states. The RCT is evaluating four MIECHV-funded home visiting program models - Parents as Teachers, Healthy Families, Early Head Start, and Nurse-Family Partnership.

See our new No-Spin report on a major RCT of federally-funded home visiting programs for at-risk families with young children. In brief: Despite the positive portrayal of results in the study's overview, it found negligible impacts on child & family outcomes at the 5-7yr mark.🧵

YouTube video by Arnold Ventures Arnold Ventures Partners with Maryland to Scale Evidence-Based Programs

AV also produced an excellent video narrated by the Governor: www.youtube.com/watch?v=wD0r...

11/ In short, the Partnership is laser-focused on using strong evidence about "what works" to improve lives at scale. Here's Gov Moore's announcement: governor.maryland.gov/news/press/p....

9/ Finally, the Partnership will build new evidence about what works specifically in MD: 3 of the 4 programs being expanded will include large RCTs, so as to produce definitive evidence about whether the programs deliver meaningful improvements in the lives of Marylanders.

8/ The Partnership's policy impact is also unique: AV’s $20M catalyzed $20M in matched gov't funds for proven programs that previously received ~$0 in public funding in MD. This represents substantial gov't investment in proven programs that wouldn't have otherwise occurred.

7/ These programs are backed by multiple large randomized controlled trials (RCTs) showing sizable gains in K-12 math achievement, college graduation rates, workforce earnings, and youth delinquency and substance use prevention.

6/ The Maryland Partnership represents a fundamentally new approach to social spending, in three respects. First, Maryland and Arnold Ventures (AV) are making a substantial investment - $40M - to scale programs that are truly exceptional in the strength of their evidence.

5/ Previously-awarded Partnership funds are scaling two K-12 programs shown to produce sizable, sustained gains in student math achievement - Saga tutoring ($18M) and ASSISTments ($2M). Total Partnership funding, including the new and earlier awards, is $40M.

4/ It also includes $5.6M to expand Big Brothers Big Sisters (BBBS) Community-Based Mentoring (shown to reduce youth delinquent behavior and substance use by 20-40%).

3/ The Partnership's final round of funding includes $14.4M to replicate the ASAP program for low-income community college students (shown to increase college graduation rates by 11-15 percentage points & long-term earnings by 14%).