Grantee: Abt Associates. The full study report can be found here.
Description of the Intervention: The three RCTs retrospectively evaluated the impact of: (i) four charter high schools; (ii) six charter middle schools; and (iii) four charter elementary schools. Each school in the sample was over-subscribed and used randomized lotteries to determine who was admitted (treatment group) and who was not (control group). These schools represented a diverse array of charter models using varying curricula and instructional approaches. Student outcomes were measured after three to four years of school enrollment (High School RCT), two to three years of enrollment (Middle School RCT), and four years of enrollment (Elementary School RCT).1
Study Design: The High School RCT sample comprised 718 students admitted via lottery in 9th or 10th grade. The Middle School RCT sample comprised 2,181 students admitted via lottery in 6th or 7th grade. The Elementary School RCT sample comprised 646 students admitted via lottery in kindergarten. If a student participated in multiple lotteries, the study appropriately used his or her first lottery to determine placement in the treatment versus control group.
Impact on the Primary Outcomes: The High School RCT found a sizeable increase in the primary outcome, – rate of college enrollment during the fall after scheduled high school graduation (65% for the treatment group versus 50% for the control group). However, this effect did not reach or approach statistical significance and is therefore only suggestive. Neither the Middle School RCT nor the Elementary School RCT found statistically-significant or sizable effects on their primary outcomes – English language and math achievement at the end of 8th grade and at the end of 3rd grade, respectively. However, we believe the results of both the Middle School and Elementary School RCTs are inconclusive due to a key study limitation (discussed below).
Study Quality: Based on a careful review, we believe that the High School RCT was well-conducted but that its sample was likely too small to be able to reliably detect effects on student outcomes.2 The Middle School and Elementary School RCTs each had differential sample attrition at or above What Works Clearinghouse thresholds for unacceptable threat of bias under cautious assumptions, which reduces confidence in their findings.3
-
1
This study also included findings from a quasi-experimental matching design study to evaluate New Mexico’s oversubscribed “Dual Language Immersion” (DLI) elementary charter schools, which provide students with general academic instruction in both English and Spanish.
↩︎ -
2
For example, the High School RCT had successful random assignment (as evidenced by similar treatment and control groups), low sample attrition, and valid analyses that were pre-registered and used an intention-to-treat approach.
↩︎ -
3
Sample attrition in the Middle School RCT was 17% for the treatment group versus 26% for the control group. Sample attrition in the Elementary School RCT was 21% for the treatment group versus 33% for the control group.
↩︎
