Quality and trust in peer review: An overview of Sense about Science 2019 survey

To coincide with this year’s Peer Review Week, Sense about Science, an independent charity working to promote honesty and integrity in science, has released the results of its Peer Review Survey 2019. The survey, conducted in partnership with Elsevier, comes a decade after the first survey that analyzed similar themes.

The 2019 survey repeated some questions from the 2009 survey, but included several new ones, taking cognizance of the many developments in science and research in the past decade. More than 3,000 researchers across career stages and disciplines responded to the survey. Going beyond the aspects of process, satisfaction, challenges, and trends, the survey examined perspectives around peer review quality and trust. Incidentally, quality is the theme of this year’s Peer Review Week.

The survey provided compelling insights and also established important action points to secure the future of the system. One key takeaway was a firm validation of the need for the peer review system.

Here is a summary of the results and discussions that emerged from the survey.

• Overall results indicate that researchers have faith in the peer review system. 75% said that they were satisfied with the process (compared with 69% in the 2009 survey).
• 90% believe peer review improves the quality of research papers (similar to the 91% affirmation in the 2009 survey). An equally high percent (85%) felt that without peer review, there would be no control in scientific communication. In the words of Dr Amarachukwu Anyogu, microbiology lecturer at the University of Westminster, UK, “The process is beneficial for your blind spots: when it’s your own research, you think you’ve done an amazing job.”
• The participants also sought to establish what constitutes peer review. 87% said that it is an assessment by at least two researchers, with or without support from an editorial team member.
• 62% respondents shared that they trusted the output they assessed just a week before taking the survey. However, 37% were not sure they could put their trust in the output quality. The respondents talked about several factors that could be affecting trust and also shared ways to build trust among researchers and public alike.

Factors affecting trust

The researchers attributed the following factors for the low level of trust:

• Pressure to publish: The almost-dire need to publish in order to obtain grants and secure career progression is adversely affecting the quality of studies and therefore the papers.
• Low-quality peer review: There is a lack of clarity about the process too, such as inadequate transparency when it comes to the credentials and experience of the reviewer.
• New channels and research outputs: While new channels such as blogs and social media have helped increase the dissemination of science, researchers are skeptical about trusting this content as the owners of these channels typically have no affiliation with the content.
• Decreased focus on innovative or high-quality research: The growth of new scholarly communication channels, such as PLOS ONE and F1000, has resulted in a high acceptance rate for papers but has conversely led to much low-impact research now getting published.
• Growth of data, other material, and research integrity challenges: With the increasing call for open science, which urges that research results and communication be clear and transparent, a large volume of literature is getting published. However, there is concern about the reproducibility of the results of many of these studies. Also, a single study can now be presented in multiple papers for each stage of the study, such as for method and data. This break-down into smaller articles, some researchers feel, can influence the final results.
• Unethical researcher (and reviewer) behavior, errors, and bias: Researchers can fabricate or falsify data to ensure results match their hypotheses. Also, they are often not aware of issues arising from conflicts of interest. Reviewers too can be guilty of errors and bias. Errors can arise from the immense drive to publish, inexperience, researcher culture, or simply, human error. While blind reviews aim to remove bias, some reviewers have taken advantage of loopholes in the system to submit bogus reviews.

Ways to build trust

The respondents also suggested several measures to improve trust, among both researchers and the lay population.

• Introduce quality controls for data and supplementary materials: 76% said that data and supplementary materials shared in a study should be evaluated in some way. To help already-burdened reviewers assess these materials, artificial intelligence (AI) could be a solution.
• Provide clear signals to help assess research: Most researchers affirmed the following ways of assessing research outputs: citations (88%), indicators whether someone else had tried to reproduce the research (82%), and post-publication commentary (79%). A majority (77%) felt that a definite marker of assessing the research output is if the peer-reviewed article clearly states that it has been peer-reviewed.
• Increase transparency and control in delivery of content: A majority (71%) would like to know why a particular article had been recommended or displayed to them in a search result.
• Improve support and recognition for reviewers: 66% said they would benefit from clearer guidance on reviewing criteria. Some ways in which they could be recognized for their work include: employer recognition of the time spent (45%), accreditation (34%), and acknowledgement, such as having their names published along with the article (28%).
• Understand the barriers facing the public: Only 38% believe the public understands the concept of a peer review. However, a bigger cause for concern (49%) is the misinterpretation of findings in media, discussions, and decision-making. However, many (43%) believe this could be deliberate, as some researchers and institutions fake or exaggerate the results of their studies for vested interests.
• Increase and improve communication around science: Some ways suggested to improve the public’s understanding of research are: clearly explaining the findings and their implications in layman terms (70%), empowering non-scientific readers to ask questions of authors (40%), explaining the peer review system (38%), and clearly marking research that has been peer reviewed (35%).

Related Infographic

6 Ways to enhance your academic career in 2019

To help you make meaningful changes to your academic…

5.8k

The study has clearly outlined pathways to take the peer review process forward. Sense about Science has begun working toward various goals and solutions and has invited all interested in research to join hands in the endeavor.

Related reading:

• My peer review wishlist
• Peer reviewers tell all: A Q&A with expert reviewers
• Anyone who has published an article and has gone through a peer review process is a potential reviewer
• How peer review is perceived by authors from emerging research countries

References:

• Peer Review Survey 2019 [About]
• Quality, trust & peer review: researchers’ perspectives 10 years on [Survey results]
• Peer Review Survey 2009: Full Report