Research team: David Levine, University of California-Berkeley; Matthew Johnson, Duke University; and Michael Toffel, Harvard University. The team’s study report to Arnold Ventures is available here and a working paper on the study is available here.
Description of the intervention: This project was a large, multi-site randomized controlled trial (RCT) of the Site-Specific Targeting (SST) program, a major federal workplace health and safety inspections program administered by OSHA. Intended to promote safe workplace practices, SST targets the most hazardous non-construction workplaces in the United States for enforcement inspections, in particular those with the highest injury rates (i.e., 2 – 4 times the U.S. average injury rate). The average cost to OSHA of an SST inspection was approximately $3,400 during the years examined in this study.
Study design: For many business establishments, the SST program has used a randomized lottery to determine which are inspected in a given year. Between 2001 and 2010, 13,251 establishments subject to SST’s randomized lottery were included in the study sample. These establishments were largely in manufacturing (56%), transportation and warehousing (14%), and nursing homes (12%).
With support from Arnold Ventures, the researchers conducted a retrospective RCT, estimating whether an establishment’s being randomly chosen for inspection affected its subsequent rate of serious injuries (i.e., injuries involving days away from work), compared to establishments not chosen for inspection. The study also estimated, as secondary outcomes, the effect of being chosen for inspection on business outcomes, such as sales, employment, and establishment survival. All outcomes were measured with administrative data from OSHA and other sources (e.g., Dun & Bradstreet data on business outcomes).
Impact on the primary outcome: The study found that being randomly chosen for an SST inspection caused a 3% reduction in injuries involving days away from work, on average, during the inspection year and following three years. (This is the “intention-to-treat” effect.) However, only about 80% of establishments chosen for inspection were actually inspected over these four years. The study found that the effect on the 80% actually inspected (i.e., the “treatment-on-treated” effect) was a 4% reduction in such injuries, meaning that each inspection led to 0.8 fewer injuries on average.1 We view these findings (both intention-to-treat and treatment-on-treated) as highly suggestive but not yet strong evidence of an effect because they did not quite reach statistical significance (p=0.07). The study found no significant effects, positive or adverse, on any business outcomes.
The above paragraph summarizes the primary study findings based on the researchers’ pre-specified analysis plan. The researchers present additional, exploratory findings in their study report.
Study quality: Based on a careful review, we believe this study was well-conducted2 and produced valid findings.
-
1
This treatment-on-treated estimate is conservative in that it does not adjust for the 24% of establishments not chosen for inspection in a given year that were inspected at some point in the subsequent three years (i.e., control group “cross-overs”). Adjusting for such cross-overs is challenging in an RCT analysis because cross-over inspections generally take place in a later year than treatment group inspections, leading to a different temporal pattern of effects. In their working paper linked above, the researchers report treatment-on-treated estimates that seek to adjust for such cross-overs.
↩︎ -
2
For example, the study had successful random assignment (as evidenced by highly similar treatment and control groups), low sample attrition, and valid analyses that were publicly pre-registered.
↩︎
