A Roaring Success of a Study Arrives at Just the Right Time

As funders and followers of rigorous research, we’re used to playing the long game; it’s the only game we play. We know from years of reviewing reports that while few programs are found effective when rigorously evaluated, there are exceptional programs that really do make a meaningful, measurable difference in people’s lives. Often, it takes time for findings to emerge, but we’re willing to wait for the kind of four-leaf-clover moment when a program we suspected was working (based on promising prior evidence), delivers, step by step, year by year, across populations and across sites, and is ultimately proven effective just when policy is poised to make use of it. It doesn’t happen often, but last month, it did. 

This is a story of evidence-based policy as marathon. 

We begin in 2014. At the time, the Coalition for Evidence-Based Policy (whose leadership and core work were later integrated into what is now Arnold Ventures as the Evidence-Based Policy initiative) worked with members of Congress to develop and advance language that ultimately became law as the Education Innovation and Research (EIR) program under the Every Student Succeeds Act of 2015. 

EIR is a federal ​“tiered-evidence” program. Its purpose is to increase public funding for rigorous evaluations to grow the number of research-proven programs and provide strong incentives for the widespread adoption of such interventions. 

Under EIR, programs meeting the highest ​“top tier” evidence rating — eligible for the largest grants — require ​“a statistically significant effect on improving … relevant outcomes based on strong evidence from at least one well-designed and well-implemented experimental study.” This is the promised land of evidence, and you’d be surprised how few programs reach it.

That same year, we launched our first Low-Cost Randomized Controlled Trial (RCT) Competition and awarded funding to conduct an RCT to evaluate a college access and success program called Bottom Line.

Bottom Line provides one-on-one advising to low-income high school seniors, most of whose parents did not attend college, to help them get into and graduate from college. Bottom Line had been operating since 1997 and had already demonstrated effectiveness in a well-conducted quasi-experimental study. The Bottom Line program providers were eager to prove themselves in a more rigorous study and, in partnership with external researchers Ben Castleman and Andrew Barr, embarked on a low-cost RCT, which we helped launch.

In the years since, we’ve seen the results of the Bottom Line RCT trickle in, tracking college enrollment and persistence across the three study sites (Boston and Worcester, Massachusetts, and New York City). Along the way, those interim study results were looking quite promising. 

Still, earlier this year, we were incredibly excited to see that the six-year impacts on bachelor’s degree receipt were truly blockbuster. The study found that, for the full sample, the program boosted bachelor’s degree receipt by 8 percentage points — from 47% in the control group to 55% in the group that was offered the Bottom Line program. 

Moreover, the effect appears to be growing. For students in the first cohort and for whom an additional year of data is available, Bottom Line boosted bachelor’s degree receipt by 10 percentage points — from 53% in the control group to 63% in the Bottom Line group. Importantly, the effects are similar across each of the three sites in the study, providing confidence that the program will be similarly effective when implemented faithfully in new settings.

These remarkable findings arrive seven years after we launched the study, but just in time to intersect with advances in evidence-based policy in higher education. The College Completion Fund, currently included in the House version of the Build Back Better Act, is specifically designed to fund evidence-based programs that support college completion. It includes a tiered-evidence funding approach, which means that it sets aside approximately 20% of the available funding for programs that meet a top tier as specifically defined by the EIR evidence standard — the same one we helped develop seven years ago. 

And last month, here came Bottom Line, arriving in the promised land of evidence as the only program of its kind proven to increase bachelor’s degree completion rates with ​“a statistically significant effect on improving … relevant outcomes based on strong evidence from at least one well-designed and well-implemented experimental study.” Having been newly inaugurated into the Top Tier, Bottom Line is poised for expansion.

Taken together, the results of these parallel efforts illustrate how our long-held strategy to simultaneously build evidence while advancing evidence-based policy reforms can drive impact and ultimately bring effective programs to those who need them most. 

In the distance, we see more to come. The Bottom Line study is just one of roughly 100 RCTs that we’ve funded — each one designed to inform policy and many of which we hope will be a part of stories like this one. We just have to give them a little time.