TRANSPARENCY AND REPRODUCIBILITY IN SCIENCE
The scientific process relies heavily on the peer-review mechanism, which plays a key role in controlling scientific quality, improving performance, and providing credibility. The lack of scientific transparency leads to a peer-review process that is less than ideal and often results in irreproducible science that might have an unpredictable impact on public health or socioeconomic or political factors. To choose just one example, a 2012 report has shown that the results of 47 of 53 published peer-reviewed cancer studies could not be reproduced (Begley and Ioannidis, 2015). Consequently, the last decade has seen increasing awareness and recognition of the weaknesses that have infiltrated the biomedical research peer-review system, and it has been widely acknowledged that this system, under its current implementation, has its limitations.
In terms of journals, the number of signatories of the Transparency and Openness Promotion (TOP) guidelines, as of March 2019, was close to 5000, with an average increase of approximately 120 journals every month over the last three years.
Several impactful studies have detailed the reasons behind the current crisis (Begley and Ioannidis, 2015; Couchman, 2014; Drubin, 2015; Frye et al., 2015; Iorns and Chong, 2014) and identified a spectrum of causes, including inappropriate study design, lack of reagent validation, inadequate documentation of methods and datasets, and insufficient sharing of data and methods – essential for detailed analysis and replication.
Consequently, the replication crisis calls for significant improvements to current practices (McNutt, 2014) that are ultimately meant to restore confidence and facilitate the peer-review process. This implies that the data must be readily available in a comprehensive format that allows the rapid testing of new hypotheses, meaningful meta-analysis, and the development of robust methods. The following points constitute an excellent starting point (Freedman et al., 2017):
- Introduction of blinded analyses to mitigate subconscious biases;
- Repetition of experiments whenever possible;
- Validation of reagents (including cell lines and antibodies);
- Careful determination of appropriate data analysis procedures and statistical tests;
- Sharing of all results, including negative and positive controls.
Beyond experiments and reporting, adequate disclosure of potential conflicts of interest (COI), and proper credit to the researchers involved are two of the critical components of scientific transparency. Although often met with skepticism, we believe that sponsored research must always be properly identified.
Study design principles
There is no shortage of publications suggesting that industry funding increases the likelihood of pro-industry conclusions (Babor and Miller, 2014; Pisinger et al., 2019), and the genre has certainly been of help in identifying many publication biases and cases of misconduct in both industry and non-industry funded research (Allison and Cope, 2010; Boutron et al., 2010; Cope and Allison, 2010; Golder and Loke, 2008). However numerous, these publications do not grant industry-wide generalizations. In 2011, a study found that, over the last few decades, only 3.8% of all misconduct cases that led to the retraction of medical and scientific publications were associated with support from industry (Woolley et al., 2011). In a perspective article, Barton and colleagues reported that they could not find any data showing that financial conflicts of interest lead to a drop in scientific quality (Barton et al., 2014). In fact, on average, industry-funded clinical trials have been found to be qualitatively superior in their reporting and adherence to regulations when compared with non-industry funded clinical trials run by academic institutions (Del Parigi, 2012).
It is noteworthy to point out that scientific transparency is never a concluded process, and there will always be adjustments and more things we can do in order to facilitate innovation and cooperation. We are only at the beginning of an era of big data, high-throughput experiments, and massive parallel calculations, and our scientific tools are in the process of being reshaped for this new reality. Initiatives like INTERVALS spearhead this transformation and encourage transparent sharing of data to allow easy review and understanding, which will facilitate the objective evaluation of the evidence (Carlo et al., 1992).
References
- Allison, D.B., Cope, M.B. Randomized Controlled Trials With Statistically Nonsignificant Results. JAMA 304 (9), 965–965 (2010).
- Babor, T.F., Miller, P.G. McCarthyism, conflict of interest and Addiction’s new transparency declaration procedures. Addiction 109 (3), 341–344 (2014).
- Barton, D. et al. After 20 years, industry critics bury skeptics, despite empirical vacuum. Int. J. Clin. Pract. 68 (6), 666–673 (2014).
- Begley, C.G., Ioannidis, J.P. Reproducibility in science: improving the standard for basic and preclinical research. Circ Res 116 (1), 116–26 (2015).
- Boue, S. et al. Embracing Transparency Through Data Sharing. Int J Toxicol 1091581818803880 (2018).
- Boutron, I. et al. Reporting and Interpretation of Randomized Controlled Trials With Statistically Nonsignificant Results for Primary Outcomes. JAMA 303 (20), 2058–2064 (2010).
- Carlo, G.L. et al. The interplay of science, values, and experiences among scientists asked to evaluate the hazards of dioxin, radon, and environmental tobacco smoke. Risk Anal 12 (1), 37–43 (1992).
- Cope, M.B., Allison, D.B. White hat bias: examples of its presence in obesity research and a call for renewed commitment to faithfulness in research reporting. Int. J. Obes. 34 (1), 84–88 (2010).
- Couchman, J.R. Peer review and reproducibility. Crisis or time for course correction?. J Histochem Cytochem 62 (1), 9–10 (2014).
- Del Parigi, A. Industry funded clinical trials: bias and quality. Curr Med Res Opin 28 (1), 23–5 (2012).
- Drubin, D.G. Great science inspires us to tackle the issue of data reproducibility. Mol Biol Cell 26 (21), 3679–80 (2015).
- Freedman, L.P. et al. Reproducibility2020: Progress and priorities. F1000Res 6, 604 (2017).
- Frye, S.V. et al. Tackling reproducibility in academic preclinical drug discovery. Nat Rev Drug Discov 14 (11), 733–4 (2015).
- Golder, S., Loke, Y.K. Is there evidence for biased reporting of published adverse effects data in pharmaceutical industry-funded studies?. Br. J. Clin. Pharmacol. 66 (6), 767–773 (2008).
- Iorns, E., Chong, C. New forms of checks and balances are needed to improve research integrity. F1000Res 3, 119 (2014).
- McNutt, M. Journals unite for reproducibility. Science 346 (6210), 679 (2014).
- Pisinger, C. et al. A conflict of interest is strongly associated with tobacco industry-favourable results, indicating no harm of e-cigarettes. Prev. Med. 119, 124–131 (2019).
- Woolley, K.L. et al. Lack of involvement of medical writers and the pharmaceutical industry in publications retracted for misconduct: a systematic, controlled, retrospective study. Curr. Med. Res. Opin. 27 (6), 1175–1182 (2011).