Is the reproducibility crisis real? An overview of Nature's survey
Irreproducibility of published results is one of the most discussed topics within scholarly circles and is often referred to as a modern-day crisis that science is facing. The failure to reproduce results is concerning because it is indicative of misjudgment, poor experimental design, selective reporting, or a statistical fluke. However, the lack of a commonly agreed definition of reproducibility, and the various opinions on what underlies the issue and how it can be resolved prompted Nature to conduct a survey.
The survey conducted by Nature quizzed about 1500 scientists about their attitudes towards reproducibility and how results can be made more reproducible. Some of the most significant survey results are listed below:
1. Is the reproducibility crisis a reality?
Of the participants who took the survey, 52% agreed to the view that “there is a significant 'crisis' of reproducibility,” whereas only 7% stated that they did not think reproducibility is a problem.
2. Are any fields free of this problem?
When questioned about reproducibility in their respective fields, 73% believed that at least half of the papers in their field are reliable, revealing contradictory attitudes. Of them, physicists and chemists were most confident about the quality of research published in their disciplines.
3. How challenging is it to publish replication studies?
24% of the respondents reported that they published a successful replication whereas 13% published a failed replication. However, several scientists involved in the survey stated that they found it challenging to get a failed replication published.
4. How often were the participants contacted with regard to their published results?
Remarkably, less than 20% of the participants reported that they had been contacted by another researcher to discuss the irreproducibility of their work.
5. What are the factors that lead to irreproducibility?
Selective reporting and pressure to publish were considered the most responsible factors underlying the reproducibility problem by more than 60% of the respondents. Another reason cited by around 50% of the participants was “low statistical power or poor analysis.”
6. What factors could augment reproducibility?
According to 90% of the respondents, “more robust experimental design,” “better statistics,” and “better mentorship” are the three factors that can ensure reproducibility. “Journals enforcing standards” was rated the lowest with 69% (which is quite high) of respondents voting for it.
The responses of the scientists make it evident that the reproducibility crisis is indeed a reality. While as many as 70% of the surveyed scientists reported their failure at replicating another researcher’s results, only a few were contacted by another researcher to report failed replication. This clearly shows a communication gap among scientists. Moreover, the hesitation of journals to publish failed replications is another indication of a loophole in the publishing system that has a direct impact on the pace of scientific progress.
The roots of this problem run deep, and all the stakeholders of science should work together to create a more transparent and robust system. While funders and institutions are working towards developing policies that improve reproducibility, every researcher should take it as a responsibility to ensure that their findings are free of any flaws and are reported without any omissions. If scientific progress is summed up as looking further by standing on the shoulders of giants, we need to ensure that every giant stands tall and firm.
Recommended reading:
Irreproducibility: The soft underbelly of science
Why are replication studies so rarely published?
How publishing replication studies and negative results helps science
Published on: Jun 03, 2016
Comments
You're looking to give wings to your academic career and publication journey. We like that!
Why don't we give you complete access! Create a free account and get unlimited access to all resources & a vibrant researcher community.
