Laura and John Arnold Foundation Establishes Guidelines to Promote Open, Transparent Research

Houston — Laura and John Arnold Foundation (LJAF) today released Guidelines for Investments in Research, a set of criteria developed to ensure that research funded by LJAF meets the most rigorous standards of quality and transparency. The document, believed to be the most ambitious of its kind for a private foundation, outlines key steps that will help improve the reliability of scientific research.

“As a society, we often rely on published research to guide our decisions on a variety of topics such as what foods to eat, how to treat disease, and how improve education. Although some of the research is rigorous and reliable, some is not. We want to encourage more high-quality research in all fields so that our personal and public policy decisions can be based in accurate science,” LJAF Director of Research Stuart Buck explained.

Leading researchers who study scientific practices have praised the Guidelines. Stanford School of Medicine Professor John Ioannidis said, ​“LJAF is making a very important transformative step towards ensuring that funded research is open, transparent, reproducible, publicly available, and, eventually, credible. Scientific investigation would benefit a lot if other funding agencies, both public and private, could adopt similar standards.”

The Guidelines state that researchers receiving funding from LJAF must follow various requirements including pre-registering their studies by submitting a detailed description of the research design before the statistical analyses are performed and data are collected.

Pre-registration makes it possible for others to recognize whether the study’s ultimate findings were a valid confirmation of a hypothesis, or whether the findings were manipulated after seeing the data. In Dr. Buck’s words, ​“Pre-registration is like drawing a target publicly and then shooting an arrow at it; if you hit the target, it is a meaningful display of your shooting ability. Without pre-registration, however, there is so much flexibility with statistical analysis and dataset manipulation that you might as well shoot an arrow into the air and then later draw a target around wherever the arrow landed.” It is recommended that pre-registration be conducted via the Open Science Framework hosted by the Center for Open Science, an organization dedicated to improving the scientific process.

In addition to pre-registering studies, researchers are expected to make their datasets and computer code publicly available (albeit only to the extent possible under any confidentiality restrictions that apply). They must also share the results of the studies even if they are not published in a peer-reviewed journal. This final step should help mitigate publication bias, the perverse phenomenon in which studies that come to ​“exciting” but possibly false results are ironically the most likely to be published.

“When this information is more widely available, other scholars can make a better judgment as to how valid a study is, or can even try to replicate it to confirm its accuracy,” Buck said. ​“We believe the Guidelines will make research more reliable.”