Grantee: Mathematica. The full report on the study findings is linked here. [1]
Description of the Intervention: This was a study of the long-term impacts of KIPP middle schools on college enrollment and persistence. KIPP is a national nonprofit network of college-preparatory, public charter schools that serve thousands of predominantly low-income and minority students. KIPP’s goal is to help students develop the knowledge, skills, and character strengths they need to succeed throughout their education and into adulthood. KIPP elementary and middle schools have been found in large, well-conducted randomized controlled trials (RCTs) to produce sizable gains in student reading and math achievement two to three years after random assignment, as summarized here.
Study Design: This was a long-term follow-up of an RCT of 13 KIPP middle schools in five states and Washington, D.C., that were oversubscribed and used randomized lotteries to determine which applying students received admissions offers. The study sample comprised 1,179 students who participated in a lottery for fifth– or sixth-grade admission in the 2008 – 2009 or 2009 – 2010 school years. 55% of sample members were Hispanic, 35% were Black, 63% came from families with annual incomes of less than $35,000, and 48% spoke a language other than English at home. The study has previously reported positive findings on reading and math achievement two to three years after study entry.
With support from Arnold Ventures, the study authors conducted longer-term analyses examining the impacts on enrollment and persistence in college, using administrative data from the National Student Clearinghouse. At the time of this follow-up (10−11 years after random assignment), all students were old enough to have potentially attended college for at least two years.
Impact on the Primary Outcomes: The study found that students who won a KIPP middle school admissions lottery were 6 percentage points more likely to enroll in a four-year college than students who lost the lottery (47% of lottery winners enrolled vs. 41% of lottery losers). We view this finding as highly suggestive but not yet strong evidence of an effect because it did not quite reach statistical significance (p=0.085). The study also found a 4 percentage point increase in the rate of persistence through the first two years of a four-year college (30% vs. 26%), but this finding was not statistically significant and so is preliminary and not reliable (p=0.23). These effects of winning a KIPP lottery (i.e., the “intention-to-treat” effects) are the primary study findings based on the researchers’ pre-registered analysis plan.
However, only 68% of students who won a KIPP lottery actually enrolled in a KIPP school. In an exploratory analysis, the study found that the effect on these 68% (i.e., the “treatment-on-treated” effect) was a 9 percentage point increase in enrollment in a four-year college and a 6 percentage point increase in persistence. The enrollment effect approached statistical significance (p=0.085); the persistence effect did not. [2]
Study Quality: Based on a careful review, we believe this RCT evaluation of KIPP’s impacts on college enrollment and persistence was well-conducted and produced valid findings. [3]
[1]The effects we discuss in this summary are based on the researchers’ pre-registered analysis plan. Prior to conducting the analysis, the researchers made a slight update to the pre-registered approach for handling missing baseline data. In their full study report, the researchers report results based on the updated approach that are marginally more positive than the results we summarize here (and they include the results we summarize here in a report appendix).
[2]This treatment-on-treated estimate is conservative since it does not adjust for the 16% of students who lost a KIPP lottery, but still subsequently enrolled in a KIPP school (i.e., control group “cross-overs”). Adjusting for such cross-overs is challenging in an RCT analysis because they usually enrolled in KIPP schools later than students who won the lottery, and spent less time in KIPP schools, leading to a different pattern of effects for the two groups.
[3] For example, the study had successful random assignment (as evidenced by highly similar treatment and control groups), minimal sample attrition, and valid analyses that were publicly pre-registered.
