Grantee: University of Chicago Education Lab. The full study report is linked here.
Description of the Intervention: Saga Technology is a blended-learning tutoring program, provided to ninth graders at all levels of math proficiency (not just struggling students). Students participate in the program for one class period per day, every day, for the full school year. They alternate, on successive days, between working with an in-person tutor and individually engaging with an education technology platform (ALEKS). Tutors are well-educated individuals (often recent college graduates), without formal teacher training, who are willing to work for one year for a modest stipend as a public service. The average student-tutor ratio in the tutoring sessions is 4:1.
Saga Technology is a lower-cost version of Saga’s traditional math tutoring program, which has been found effective in prior well-conducted RCTs. Saga’s traditional program provides daily in-person tutoring with a 2:1 student-tutor ratio and does not have an educational technology component. Saga Technology’s cost – approximately $2,585 per tutored student – is roughly 30% that of the traditional program.
Study Design: The study sample comprised 2,065 incoming ninth graders in six high-poverty high schools – two in Chicago, and four in New York City – in the 2018 – 2019 school year. 1 These students were randomly assigned to a treatment group that received Saga Technology, versus a control group that received usual school classes and services. 24% of sample members were Black, 57% were Hispanic, and 91% were economically disadvantaged. The study’s pre-registered primary outcome was end-of-year math achievement on the districts’ standardized tests.
Impact on the Primary Outcome: The study found that Saga Technology produced a statistically significant increase in end-of-year math achievement on the district test of 0.19 standard deviations. This effect represents approximately an 86% improvement over the annual gain in math otherwise expected for ninth graders. 2
Study Quality: Based on careful review, we believe this study was well-conducted and produced valid findings. 3
-
1
The study sample originally included a second cohort of incoming ninth graders in the 2019-2020 school year in three Chicago and four New York schools. However, the COVID-19 pandemic disrupted program delivery in the second cohort and caused the cancelation of district tests in spring 2020. Therefore, prior to analyzing the outcome data, the study team revised their pre-registered analysis plan to specify end-of-year test scores for the students in cohort 1 only as the study’s primary outcome measure. The cancelation of the spring 2020 district tests also rendered impossible the study team’s original plan to measure the impact on math scores at the end of tenth grade (one year after program completion), as an exploratory outcome.
↩︎ -
2
The average annual gain in math achievement for U.S. ninth grade students on six nationally normed tests is 0.22 standard deviations (see Bloom, Hill, Black, and Lipsey, 2007). Saga Technology’s impact of 0.19 represents an 86% improvement over this annual gain.
↩︎ -
3
For example, the study had successful random assignment (as evidenced by highly similar treatment and control groups), low and non-differential sample attrition, and valid analyses that were appropriately pre-specified.
↩︎
