Grantee: WestEd. The study report is linked here.
Description of the Intervention: ASSISTments is an online study tool that assists seventh grade students with math homework and produces real-time reports for teachers on student performance. As students work through homework problems and enter their answers into ASSISTments, the system provides immediate feedback on the correctness of answers and offers additional assistance in the form of hints or scaffolds. ASSISTments provides teachers with real-time, easily accessible reports that summarize student work for a particular assignment, which teachers can use to target their homework review in class and tailor instruction to their students’ needs. A prior well-conducted RCT in Maine found that ASSISTments produced a statistically significant positive impact on state test scores in math at the end of seventh grade (effect size 0.18). The program’s cost is less than $100 per student, based on a cost study conducted alongside the RCT.
Study Design: The researchers randomly assigned 63 public schools in North Carolina to a treatment group that implemented ASSISTments in seventh grade math classrooms versus a control group that did not. The student sample comprised all 9,073 seventh graders in these schools in the 2019 – 2020 school year whose sixth grade (baseline) test scores were available to researchers. Sample members were 17% Black, 19% Hispanic, and 56% White, and 53% were economically disadvantaged. The study’s pre-registered primary outcome was math achievement on the North Carolina state test (EOG-MA08) at the end of eighth grade – one year after program completion. 1
Impact on the Primary Outcome: At the end of eighth grade (one year post-program), the study found a statistically significant positive impact on student math achievement on the state test. The impact (0.10 standard deviations, p=0.01) represents approximately a 31% improvement over the annual gain in math otherwise expected for eighth graders. 2 The presence of this impact one year after program completion is evidence of a sustained learning gain. The study also examined, as a primary research question, whether the program’s impact for lower-performing students (those with baseline math scores below the median) differed from its impact for higher-performing students. It found suggestive evidence (near statistically significant, p<0.10) of a larger impact for higher-performing students.
Study Quality. Based on a careful review, we believe the study was well-conducted and produced valid findings. 3
-
1
The researchers originally intended to measure the program’s impact on student math achievement at both the end of seventh and eighth grade, but the state achievement test was canceled during the spring of seventh grade (spring 2020) due to the COVID-19 pandemic.
↩︎ -
2
The average annual gain in math achievement for U.S. eighth grade students on six nationally normed tests is 0.32 standard deviations (see Bloom, Hill, Black, and Lipsey, 2007). ASSISTments’ impact of 0.10 represents a 31% improvement over this annual gain. One caveat regarding this comparison is that while the North Carolina state test used to estimate ASSISTments’ impact was similar to the six tests in Bloom 2007 in measuring broadband math achievement (versus just a specific topic such as fractions), it was – unlike the six tests – not nationally normed. For that reason, the comparison of ASSISTments’ impact against the Bloom 2007 benchmark is best viewed as illustrative rather than exact.
↩︎ -
3
For example, the study had successful random assignment (as evidenced by highly similar treatment and control groups), sample attrition that falls within Works Clearinghouse boundaries for acceptable levels of potential bias under cautious assumptions, and valid analyses that were appropriately pre-specified.
↩︎
