How peer review is perceived by authors from emerging research countries
Editage global author survey
A large-scale global author survey conducted by Editage highlighted the perspectives of academic authors on a broad range of topics related to scholarly publishing. Nearly 7,000 authors offered their views on publication processes, journal communication, ethical issues, open access, and the changes they would like to see in journal publishing. This series covers the main report based on the overall findings and several follow-up articles and reports that took a deep dive into specific topics.
Peer review has attracted the lion’s share of attention in recent discussions around whether the academic publishing system is broken. Does peer review still work the way it is meant to? Is it rigorous and objective enough as a gatekeeper of scholarly work? Does it help authors improve the quality of their manuscripts?
Several studies and some large-scale global surveys focusing on peer review have been conducted over the past ten years in attempts to answer these questions, and these have shed light on how different aspects of peer review are perceived by authors and peer reviewers. The Editage Global Author Survey was one such initiative that gathered author perspectives on academic publishing from around the world, with greater representation from emerging research powerhouses such as China, Brazil, Japan, South Korea, and India than seen in most previous surveys (71% of the 6,903 survey respondents belonged to these countries).
Since this year’s theme for Peer Review Week is “Quality in Peer Review,” we would like to share in some detail the author perspectives on peer review we gathered in our survey and how they varied based on region or discipline. Towards the end of this article, we list aspects of peer review that respondents wanted changes in and pose questions to initiate discussions on review quality.
Overview of Editage survey findings
In the Editage survey report, we had noted divided opinions on peer review. While most participants appeared satisfied with the overall quality of peer reviewer comments they had encountered, many reported difficulty in addressing peer reviewer comments and unhappiness with the some aspects of how peer reviews are conducted.
When we asked respondents to rank eight problems in academic publishing according to the urgency with which they need to be addressed, peer review did not feature among the top five. However, it was the second most commonly desired area of change, after publication delay: poor peer review quality and processes featured in 24% of author responses to a question asking them what changes they would like to see in academic publishing.
Demographic variation in author opinions on peer review
Authors were presented with the following positive statements about peer review and asked to rate their level of agreement on a five-point Likert scale (Strongly agree, Agree, Not sure, Disagree, Strongly disagree):
- Peer reviewer comments are generally insightful and help me improve my manuscript.
- Peer reviewer comments are generally clear and easy to understand.
- Peer reviewers usually provide their feedback in a constructive and actionable way.
Regional response patterns
Findings of previous similar surveys, such as the one by the Publishing Research Consortium and the follow-up Sense about Science study, had shown that respondents from non-Anglophone countries tended to have more positive views about peer review than those from Anglophone countries. Since our survey sample had a larger representation from non-Anglophone countries, we wanted to explore whether our sample showed similar geographical patterns. Interestingly, we found that they were different. Barring China, the non-Anglophone countries we examined showed patterns not greatly different from those observed for Anglophone countries.
We analyzed responses from the seven most widely represented countries in the survey: China, Brazil, Japan, the USA, South Korea, India, and the UK. Here are some interesting patterns we found (see also the infographic below):
- More than 50% of participants from all these countries, except South Korea and India, agreed that peer reviewer comments were insightful.
- All countries, except China, showed low agreement rates (32–43%) for the statements saying that reviewer comments were clear and constructive.
- Respondents from China showed the highest agreement rates for each of the statements, with an at least 20-percentage-point higher rate than seen for any of the other countries.
- Respondents from South Korea consistently showed the lowest agreement rate for any of the statements.
- Brazil showed the highest disagreement rates (around 25% of the participants) for the statements that reviewer comments were clear and constructive.
Discipline-specific response patterns
To allow clear comparisons, we included only those participants who chose one of the following four broad disciplines: physical sciences, medicine and allied health sciences, life sciences, and the humanities and social sciences. We excluded those who chose “Other.” For each of the positive statements, the following response patterns were observed:
- Physical sciences showed the highest agreement rates.
- The humanities and social sciences showed the lowest agreement rates (and highest disagreement rates).
The agreement rates for life sciences/medicine and allied health sciences fell somewhere in between.
These patterns are consistent with the findings of the above-mentioned surveys.
Difficulty in responding to peer reviewer comments
In a separate question about the level of difficulty faced at various stages of publishing, 70% of survey participants reported experiencing moderate to extreme difficulty in responding to peer reviewer comments. We found that the difficulty levels did not vary remarkably by discipline and showed slight inverse proportionality with tenure.
However, when studied by region, we observed some noteworthy differences, with Japan and South Korea having substantially high proportions of respondents (84%) who reported facing difficulty and India having the lowest proportions of respondents (46%) reporting difficulty.
Other observations
The survey also had two negative statements in addition to the positive statements (not included in the infographic):
- Comments from different reviewers are often conflicting and confusing.
- Peer reviewers often ask for complex and time-consuming additional research to be conducted.
Interestingly, for these statements as well, China showed the highest agreement rates (>60%). The lowest were from India (<35%). While there were no striking discipline-specific variations, physical sciences and life sciences had slightly higher agreement rates.
Finally, we considered whether the regional patterns presented here might have been influenced by demographic composition (discipline, tenure) for each country. However, a brief review showed no distinctive patterns. China, India, and South Korea had greater proportions of respondents (>50%) with tenure of five years or less; the other countries had ≥60% respondents with tenure greater than five years. China had more respondents from the physical sciences; Brazil, the USA, and the UK had more from the humanities and social sciences; and the rest had more from medicine and allied health sciences.
What aspects of peer review did respondents want changes in?
Our survey did not systematically ask authors their opinions on different peer review processes/approaches or on motivations/incentives for reviewers. However, as noted earlier, peer review was the second most common area of change authors wanted to see in academic publishing. Below is a list of some common themes we observed, followed by representative author comments.
1. Objectivity, fairness, and accountability in peer review
The fair system which can avoid a self-protective rejection by peer reviewers needs to be established.
I suggest to establish a system which enables many researchers to read manuscripts, instead of giving an authority for some people to judge publication. Then leave the decision for readers whether a certain research is valuable or not.
I believe that the open peer review system can improve appraisals and make the system more transparent.
Peer review needs to be done in a double-blind system. It is very common for reputed researchers to have a flawed article approved by peer reviewers, or good articles to be disapproved by reviewers who consider themselves adversaries.
It should be double blind. Some people are not professionals and judge paper according to authors’ country, or race, or name, or even according to English. Reviewers should judge only results presented in the paper.
There should be some kind of post-publication follow-up review of papers (e.g. other scientists can comment on whether they could replicate the results or issues with the methods etc.).
2. Recognition, remuneration, or incentives for peer reviewers
Writing a useful review should have some benefit for the reviewer (OA publication, discount for conference, etc.).
There should be payment for reviewers and your reviews should be taken into account at tenure and promotion.
I believe that at this point peer review system needs to be a) professionalized and b) payed for. Considering that about 15% of peer-reviewers are responsible for ~80% of peer reviewing the process is becoming akin to a short-term consulting jobs and as such should be remunerated. If implemented the publisher should be able to enforce a shorter review time - both the reviewers, authors and in a longer term publishers win.
3. Evaluation of and feedback on peer reviewer comments
Quality control of peer review process must be carried out. Not all reviews are of the same quality or technically sound. This must be checked by the associate editor/editor. The editor should be more invested in the process.
I would like Editors to ensure that reviewer comments/questions/suggestions are clearly phrased - so they can be actioned or responded to by authors. In some cases comments are almost questions and you feel a need to respond to them. If the reviewers expectations are clear, then the time required for authors to respond would be reduced.
Reviews that are destructive rather than constructive should be ‘red-carded’ with mandatory training for the reviewer involved before they are asked to do any more reviews.
If authors could evaluate the qualities of journals for their editing and review, and if it could be open to the public, it may improve a quality of review and shorten the time of editing process.
4. Process changes to improve peer review quality
Authors should be prohibited to appoint a reviewer who is an acquaintance. The system which allows publishers to choose suitable reviewers is necessary.
Editors should respond to authors after appropriately judging how far additional experiments are required and whether over-requirements are made or not, instead of just swallowing the comments by peer reviewers. That is applicable to when there is contradiction between the comments. I strongly hope the editors to more responsively control the results of review in all journals.
To have no more than two reviewers on a manuscript. When there are 3 or more, the revision becomes very laborious and sometimes confusing and conflicting between reviewers.
Selection of peer reviewers is often based on who is available rather than who is qualified.
Editors most of the times do not read manuscripts, and simply follow the reviewers' suggestions. That is fine, but only if the reviewers have been carefully and properly selected so that they possess the expertise to review the manuscript.
The reviewers should be selected more carefully based on their pertinent background, expertise and ability to comment critically and constructively.
5. Training and resources for peer reviewers
Peer review guideline needs to be shown clearly. Especially the validity of statistical analysis might be ignored on critical points, if a reviewer does not have enough knowledge and experience.
Editors and peer reviewers should be trained and fully accountable for their performance to authors, not just to publishers or the owner of the journal.
Discussion points
The trends we saw in our survey data and the comments authors shared prompt a few important discussion points that need to be addressed: How can journals ensure that reviewer comments are clear, constructive, and actionable? Do studies provide sufficient evidence to determine which peer review models may be most effective, objective, and sustainable? Have incentives of any kind been found to improve review quality? Should there be universal standards for review quality and a standardized format for reviewer training programs? Can access to training encourage more authors to accept reviewing assignments, thus expanding the pool of suitable reviewers?
In the coming days, we will explore these and other related questions on peer review during a couple of interesting and interactive events we have planned for Peer Review Week: a webinar on quality in peer review and a panel discussion on challenges in peer review and the way forward. We would love to have you attend these to share your thoughts and to hear what industry professionals have to say about peer review and the issues surrounding it.
References and additional reading
- Publons (2018). 2018 Global State of Peer Review.
- Sense about Science (2009). Peer Review Survey 2009: Full Report.
- Taylor & Francis (2015). Peer review in 2015: A global view. A white paper from Taylor & Francis.
- Taylor & Francis (2016). Peer review: A global view. Motivations, training and support in peer review. Insight supplement from Taylor & Francis.
- Ware, M., and Monkman, M. (2008). Peer review in scholarly journals: Perspective of the scholarly community – an international study. Publishing Research Consortium.
How do authors feel about peer reviewers' comments.pdf
Comments
You're looking to give wings to your academic career and publication journey. We like that!
Why don't we give you complete access! Create a free account and get unlimited access to all resources & a vibrant researcher community.
