Grantee: The Rector and Visitors of the University of Virginia. The study report is posted here.
Description of the Intervention: This ongoing randomized controlled trial (RCT) evaluated various communications “nudges” to encourage low-income and first-generation students (i.e., those whose parents did not go to college) to complete college financial aid application forms such as the Free Application for Federal Student Aid (FAFSA). The goal was to increase their college attendance and graduation rates. The nudge interventions were targeted primarily to U.S. high school seniors who were low income or first generation, and who had registered with The Common Application — a college application platform widely used nationwide.
Students participating in the treatment groups of the study received one of three nudge interventions: (i) messages about the financial benefits of completing the FAFSA, relative to the costs; (ii) messages reminding them of their motivation for applying to college; and (iii) messages with instructional guidance on how to complete the FAFSA. A random subset of each intervention group was also offered one-on-one advising to help with FAFSA completion. Across all treatment groups, the researchers randomly varied whether the messages were sent via e‑mail, postal mail, or text message.
Study Design: With support from a consortium of funders including Arnold Ventures, the researchers randomly assigned 454,243 students within 20,518 high schools across the United States to one of the interventions described above, or to a control group that did not receive any of these interventions but did receive generic email reminders to submit their FAFSA. Random assignment took place between October and December 2015.
The study’s pre-registered primary outcome is students’ persistence in college, defined as being enrolled in the fall semester of what would be their second year of college (i.e. fall 2017). Due to cost constraints, the researchers measured college persistence for a randomly selected subsample of 271,365 students. All study outcomes were measured with administrative data from the National Student Clearinghouse and other sources (e.g. Federal Student Aid, for FAFSA completion outcomes).
Impact on the Primary Outcome: The study found that none of the nudge interventions, with or without the advising, had a statistically significant effect on students’ persistence into their second year of college, and the effect sizes were close to zero. [1] (The study did find modest positive effects on the intermediate outcome of FAFSA completion [2], but these effects did not lead to an increase in persistence.) In exploratory analyses, the study found no clear pattern of effects among the pre-specified subgroups of interest (e.g., females, first-generation students).
Study Quality: Based on careful review, we believe this was a well-conducted RCT that produced valid findings. [3]
[1] The sample size of students in the advising arm of the study was not large enough to conclusively rule out the possibility that one-on-one advising produced a meaningful effect on persistence.
[2] This estimate comes from a quasi-experimental sample rather than the RCT due to challenges in accessing individual-level data on this outcome.
[3] For example, the study had successful random assignment (as evidenced by highly similar treatment and control groups), no sample attrition, and valid analyses that were publicly pre-registered here.
