We all learned in science class that an essential part of the scientific method is replication. To quote a typical fifth-grade science lesson, “Scientists can use the scientific method to repeat experiments of other scientists. This helps them make sure that their conclusions are correct.”
Unfortunately, the core scientific value of replication is often poorly rewarded. Everyone from journals to university hiring committees is drawn to researchers who make bold, new discoveries that are going to be highly cited and that will drive the field in new directions. Meanwhile, merely replicating someone else’s work can be seen as derivative.
Indeed, a few years ago, I asked the editor of one of the world’s top science journals whether we could partner to sponsor a replication series in the journal. The response: “Replications aren’t cited as much, and that could hurt our impact factor.”
This isn’t just an anecdote. Large-scale literature reviews have found very few published replications in fields like education or criminal justice.
But in the past few years, the tide seems to have turned.
A significant tipping point was the Reproducibility Project in Psychology, which Arnold Ventures funded, and which was carried out by our grantee Center for Open Science. That project organized more than 200 psychology labs around the world to systematically redo 100 experiments published in top psychology journals. It found that only about 40 percent could be reliably replicated (another 40 percent were inconclusive, and around 20 percent were decisively not replicated). Since those results were published in 2015, the study has already been cited an astounding 2,800 times.
Since 2015, we have seen an explosion in similar efforts. Particularly in social-behavioral science, there are now many replication projects that have organized labs around the world to replicate one or more scientific findings.
- The Social Sciences Replication Project (coordinated by the Center for Open Science) sought to replicate all 21 social science experiments published in Science and Nature between 2010 and 2015. The results were that only 13 could be replicated, and even then, the effects were typically about half of what was seen in the original study.
- The Experimental Economics Replication Project replicated 18 studies published in two of the top economics journals. For 11 out of the 18 studies, the replication effect was in the same direction as the original effect (albeit at two-thirds of the size).
- The Many Labs series of studies (1 through 5) have all sought to systematically replicate psychology studies with many labs around the world. Many Labs 1 came out in 2014 and replicated 13 psychological findings; Many Labs 2 did the same for another 28 classic and newer findings; Many Labs 3 looked at whether psychological effects studied on college students would vary depending on the time of the semester; and Many Labs 4 and 5 are in progress. The fifth Many Labs will be particularly interesting: It is re-examining 10 of the experiments from the Reproducibility Project in Psychology that were contested because the experimental protocol supposedly wasn’t quite on point. Many Labs 5 will do another two replications of each study, one with the supposedly inferior protocol and another with an “improved” experimental protocol that is peer reviewed in advance.
- The Psychological Science Accelerator is a “globally distributed network of psychological science laboratories (currently more than 350),” with the goal of coordinating massive psychology studies across diverse settings.
- The Many Babies projects (of which there are now three) are coordinating multiple labs to study infant cognition (such as how babies develop a theory of mind or how they react to speech).
- In education, Many Numbers is a project to replicate research on how children develop math skills, while Many Classes will study questions such as curriculum across “dozens of contexts, spanning a range of courses, institutions, formats, and student populations.”
- The Many Smiles Project is organizing 18 labs from 17 countries to re-examine a contested question in psychology about whether people actually become happier when they are tricked into smiling (by being asked to hold a pencil between their teeth).
- The “Many” tagline has even reached animal research, including Many Primates (a collaboration to study primate cognition, such as short-term memory, in much larger samples than are typical for the field), and Many Dogs (a collaboration to study dog cognition).
Make no mistake, it is unprecedented for so many replication projects to have emerged in such a short time frame and to have generated so much participation from scholars around the world. If this level of interest continues, we might have found the solution to the problem identified above, i.e., that journals aren’t as likely to publish replications for fear that they aren’t cited enough. That fear may have had some basis as to single-study replications done on a small scale, but these large-scale, multi-author, worldwide projects are attracting so much attention that they are far more publishable.
More fields – from medicine to sociology to biology – should see the value in large replication projects that systematically take stock of whether research stands the test of time. If (or when) such replication projects become the norm, scientific research will be more like the ideal we all learned in fifth grade.
