Peer Review Week 2023: Top 10 questions and answers in our live AMA
What do you really know about peer review? On the occasion of International Peer Review Week, ScienceTalks and Editage jointly invited experts to answer questions about peer review. Here are the top questions posed to the “Ask Me Anything” panel that included Editage invitees Joan Marsh, editor-in-chief of The Lancet Psychiatry, Matthew Hodgkinson, Research Integrity Manager at the UK Office for Research Integrity, Jadranka Stojanovski, Associate Professor, University of Zadar. SciNet invitees included Professor Ma Jun of Lanzhou University of Technology and researcher Yang Wei of the Institute of Geology and Earth Research, Chinese Academy of Sciences.
How do you see the future of peer review? Are there any changes to be expected?
Joan Marsh (JM): Many changes are to be expected but I believe, and hope, that journals and researchers will continue to invest time and effort in peer review because I think it substantially improves scientific literature. The publishers will improve tools, making it easier for reviewers to submit their comments, which will help.
Matthey Hodgkinson (MH): Peer review is moving towards more transparency, with open peer review. This has two parts: the authors knowing who the reviewers are (and the public knowing who they are if the article is published) and/or the reviewer reports being made public with the published article. I think that AI may help in peer review, but not chatbots/LLMs.
Jadranka Stojanovski (JS): There is no doubt that the peer review process is already evolving. Numerous changes are taking place, especially related to the transparency/openness of the peer review process, peer review of preprints, application of AI tools in the selection of reviewers, education and training of reviewers, greater representation of structured review reports and the need to respect the principles of equity, diversity and inclusion in selection reviewers. In the future, we expect that the reviewers will always have the underlying research data available and that the reviewers' reports will generally be available in open access to be used and ensure greater visibility and recognition of the reviewer's work.
What are some common biases or stereotypes you can think of during the peer review process?
JM: Common biases, according to the literature, are against people presumed to be female, from their names, and people in low/middle income countries or just with affiliations in institutions or countries not known to the reviewer.
MH: There is evidence that anonymizing the authors during peer review reduces bias favoring high-income country authors, so double anonymized review before publication and open peer review afterwards might be a combination that works well. A problem in peer review is the increased difficulty in finding reviewers, partly caused by reviewer fatigue, and also caused by editors not inviting reviewers from China, India, and Iran nearly as much as expected from their volume of articles as authors.
JS: The PR process is susceptible to biases and stereotypes that can unconsciously influence reviewers' assessments since reviewers are only humans. Reviewers may favor research that confirms their pre-existing beliefs. This bias can lead to a reluctance to accept novel or unconventional findings. Additionally, reviewers might give an advantage to the research that reports positive results over studies with neutral or negative findings. Gender bias can also influence peer review, with more criticism towards female authors, although some studies have quite opposite findings. More problematic is institutional bias favoring research from prestigious institutions or by prestigious authors. Reviewers might be influenced by the reputation, assigning more credibility to well-known senior authors, while early-career researchers might face additional scrutiny. Often, reviewers are biased against papers written in not-perfect English or some other language by authors who are not native English speakers. Cultural biases can also affect how reviewers interpret and evaluate research, leading to misunderstandings or misinterpretations based on cultural differences. Last, political or ideological biases can affect the reviewer's assessment of research on controversial topics.
What steps can be taken to prevent biases in peer review?
JM: One option is anonymizing the paper so that the reviewers do not see this information but that goes against the trend towards preprints and open science. Editors should use a wide range of reviewers, balanced for gender (at least to the same extent as the academic field in question, preferably more balanced) and from diverse institutions and countries. This requires work from editors to expand their knowledge of people in the field willing to peer review. Publishers are tackling this with tools that recommend peer reviewers, some using AI.
MH: Publishers may need to move to having reviewer panels and more staff scientific editors to ensure the appraisal of articles. We need specific tools that are validated for the assessment of scientific reporting and statistics.
JS: To prevent biases in peer review, journals should provide clear guidelines to reviewers, offer training on unbiased evaluation, and implement anonymized peer review during the PR process. At the same time, promoting diversity among reviewers, including reviewers from various cultural backgrounds and encouraging a culture of fairness, open-mindedness and transparency in the peer review process is also vital to mitigate biases and stereotypes. Additionally, encouraging reviewers to focus on the content, relevance, significance, methodological rigour, and evidence using neutral, objective and respective language in the reviewer's reports could reduce possible biases. Editors play a crucial role in monitoring and addressing potential biases in peer review.
What are some common models of peer review? and what are their pros and cons respectively? When and why do journals go with post-publication peer review?
JM: The traditional models are single/double-blind or anonymized, whereby the authors do not know who the peer reviewers are, and in the double-anonymized model, the authors' names are removed from the paper.
MH: Common models, using the terminology approved by NISO and the STM Association, are:
Single-anonymized: in which the reviewer knows who the authors are, but the reviewers are kept anonymous; Double-anonymized review: in which neither the authors nor the reviewers know who the others are; open peer review, in which the identity of the authors and reviewers are known to each other (and the public, if published) and/or the peer review reports are made public if the article is published; Post-publication peer review: in which the article is posted online and then reviewed, usually but not always by inviting reviewers.
JS: To get the best insight into the various peer review models, you can consult EASE Peer Review Toolkit and Peer Review Terminology Standard (https://www.niso.org/publications/z39106-2023-prt). Today, the most prevalent model is double anonymised peer review, where the identities of both reviewers and authors are concealed from each other throughout the peer review process.
What pros and cons of peer review models? When and why do journals go with post-publication peer review?
JM: In an era of open science and preprints, I do not think double-anonymized is sustainable, but the Institute of Physics has recently moved all its journals to this model. I believe reviewers should have the option to sign their reviews if they want to. I do not know why journals opt for post-publication peer review. Lancet journals have active Correspondence sections, if anyone wants to highlight possible errors or omissions in papers.
MH: Double-anonymized review reduces bias towards high-income country authors, as shown by a randomized trial in a Wiley ecology journal published this year, but it is difficult to implement in practice and many reviewers can guess who the authors are. I also find it limits the identification of conflicts of interest, so I would like a post-acceptance stage in which reviewers can look again knowing who the authors are. Open peer review can make it harder to find reviewers if their names will be known, but it reduces the chances reviewers will act poorly and they tend to write more. Publishing reports allows public scrutiny of journals, which is good. A combination of double-anonymization pre-publication, public reports post-publication, and the option for reviewers to be named may be a messy but acceptable compromise. Post-publication peer review is supposed to allow wider participation in peer review, but few examples I have seen treat volunteer reviews the same as those invited by the journal. Some articles languish unreviewed. In some models, authors invite their own reviewers, which I think ensures the process will be biased and poor quality. There is also usually no editor to make a decision to publish; this is outsourced to the reviewers. The argument is that this reduces gatekeeping, but I feel it takes away an important step of scrutiny and responsibility.
JS: Traditional anonymized PR has many shortcomings: susceptibility to biases concerning gender, nationality, affiliation, and language of the manuscript, potentially disregarding groundbreaking works, being unable to detect severe errors in methodology, data collection and analysis, inconsistency in the opinions and comments provided by reviewers when evaluating the same paper, prone to manipulation, fertile ground for unethical behaviors, the lack of recognition, credit, and rewards for the reviewers, and squandering knowledge and resources. Many of these shortcomings could be solved by open peer review. The benefits of OPR are greater accountability and effective error detection, building trust and confidence in the research community, empowering collaboration and exchange of knowledge and ideas among researchers, encouraging constructive criticism and helping identify flaws or areas for improvement, enhancing the visibility, recognition and reputation of reviewers and their contributions, help validate the quality of editorial work etc. However, revealing the reviewers' identities in OPR may have negative consequences, such as reducing the level of criticism, reduced honesty, privacy concerns, fear of retaliation and repercussions, or unwanted public attention.
Why do we need peer review? What are the standards of reviewing?
JM: If you compare preprints with published versions of papers in your field, you should see why we need peer review. Reviewers come in all standards, from excellent to not very good.
MH: We didn't used to have peer review. Scientists would send each other letters. However, this did not scale well. In the 17th century, journals were founded in Western Europe to allow wider circulation of these letters. They were usually reviewed by the editors, who decided if the content was of interest. After World War II in Europe and America, there was a great increase in scientific research and editors found they could not handle the volume any longer. They turned to peer review, inviting outside experts. This was also done in part because of ethical scandals in medicine and biology; peer review helped avoid outside regulation in the US. Peer review is done because authors may be incompetent, mistaken, or malicious, and they are often not able to spot their own errors. It is thought it would be better to have a few people check the work before wide distribution, to reduce the chances of serious errors or lies being spread to readers. The Committee on Publication Ethics has ethical guidelines for peer reviewers, which include keeping details confidential, declining to review if you have potential conflicts of interest or you are not qualified, and not using the content or process to your own advantage.
JS: The primary goals of peer review are to ensure the quality and credibility of research (reliability of the research methods, data analysis and interpretation, and findings), identify and correct errors, inconsistencies or flaws, and help determine whether a manuscript is suitable for publication in a scholarly journal where submitted. Reviewers evaluate the manuscript's clarity, significance and contribution to the field and suggest improving the writing and structure to make the reporting on research more accessible to readers. Reviewers can also help in addressing some ethical concerns if they exist. Their feedback allows authors to correct possible issues and improve the manuscript before publication. Peer review standards depend strongly on specific journal policies and guidelines, field and type of manuscript, usually including adherence to journal guidelines, relevance for the journal's scope and readers, confidentiality, objectivity and timeliness.
What do you think of the use of Artificial Intelligence (AI) in peer review? Are there potential ethical issues?
JM: AI cannot vouch for its opinions so how can it perform peer review? It could be used to check certain types of information are present: that is more pre-review checking than peer review.
MH: There are no legitimate uses of chatbots/LLMs in peer review. They are not validated to assess scholarly articles and submitting the content to these tools may breach confidentiality. That said, there are other AI/machine learning tools that are designed for peer review, e.g., tools that assess the quality and completeness of reporting, or tools that spot statistical errors. These are better tested and much more promising than chatbots.
JS: AI in PR can offer several benefits and enhance the efficiency and effectiveness of the peer review process. Benefits include identifying suitable reviewers, checking for plagiarism, analysing manuscript content and reviewer profiles to suggest potential reviewers with relevant expertise, assisting in language and grammar checking tools, assisting in verifying references and citations, and ensuring accuracy and adherence to citation standards. Potential ethical issues include bias of AI algorithms, training data, and data privacy (sensitive data). The use of AI in PR should adhere to ethical guidelines, including transparency in disclosing the involvement of AI tools in the PR process. AI should be seen as a complementary tool to human expertise.
On what basis do journals select reviewers?
JM: At Lancet journals, we select reviewers who we know, or think, will have expertise in the topic under review, particularly methodological expertise. We are looking for critical insight with suggestions for how to improve and clarify the paper: reviewers who write just a few lines and say Reject or Accept, are usually not invited to review again. We use a dedicated statistical review for all research articles. At Lancet Psychiatry, we invite people with lived experience of mental illness, who are not necessarily academics, to review papers that directly affect patients, such as clinical trials, and qualitative research
MH: Usually, they should have published work in the peer-reviewed literature that is closely related to the topic and/or methods of the submission. It is common that journals will have a cut-off, e.g., number of publications, years publishing, or h-index, but there is no universal standard. Some journals allow PhD students to review, often if they are mentored by their supervisor. Some reviewers may be chosen due to professional expertise demonstrated in ways other than academic publications.
JS: The practices among journals are different. Some journals maintain the reviewers' database, others ask authors to suggest reviewers. For the best practices in selecting reviewers, you can consult EASE Peer Review Toolkit and How to select reviewers infographics at https://ease.org.uk/communities/peer-review-committee/peer-review-toolkit/how-to-select-reviewers/.
To be a reviewer, what should I do? Can a person become a reviewer through self-recommendation? Do you mind sharing your own story of becoming a reviewer?
JM: There are many online training courses. EASE (ease.org.uk) has a list of these. Elsevier Research Academy has a good one for academic researchers. You can write to a journal in your field and offer to become a reviewer: preferably after you have done the training and explain the training you have done. You can also ask your supervisor whether they will train you and whether you can then help review papers they have been invited to review. At Lancet journals, reviewers can indicate whether they did the review with anyone. As a journal editor, if a senior reviewer suggests a junior member of their team as a suitable reviewer, we often invite the junior colleague. This is an important way of diversifying the reviewer pool.
JS: Yes, many journals post calls for the reviewers on their websites where researchers can express their interest. It can be helpful to reach out to journals and publishers directly, update your profiles on relevant platforms, and actively participate in academic and research activities to establish your expertise and visibility in your field. Additionally, your publication records can certainly contribute. Personally, I did not use self-recommendation, but I was always invited.
Could you share a story of being a reviewer?
MH: I wrote this 15 years ago: https://journalology.blogspot.com/2007/01/case-study-in-open-peer-review.html. I have myself been a peer reviewer several times, but I have nothing particularly memorable to share. I have always signed my reports. I have many anecdotes from being an editor.
JS: I want to share an intriguing anecdote related to a peer review experience with a scholarly journal. I had the privilege of reviewing a paper that was exceptionally well-written, meticulously structured, and highly relevant to the field. My initial review was quite positive, reflecting the paper's strong qualities. Curiosity led me to investigate whether the research data associated with the paper was accessible. Fortunately, this particular journal had embraced the practice of granting reviewers access to research data. However, what I discovered was rather alarming. The authors had taken a notably liberal and inaccurate approach to analyzing and interpreting their results. The graphical representations they employed were not grounded in the underlying research data; instead, they were based on superficial and incorrect interpretations. This story underscores the vital importance of providing reviewers with access to research data.
About the experts:
Joan Marsh, Editor in Chief, Lancet Psychiatry.
Past President of the European Association of Science Editors (EASE) and currently Chair of its Gender Policy Committee and part of the editorial team for European Science Editing.
Matthew Hodgkinson, Research Integrity Manager, UK Research Integrity Office (UKRIO).
Matthew is a council member of the Committee on Publication Ethics (COPE), Treasurer of the EASE, and a mentor for AuthorAID.
Jadranka Stojanovski, Associate Professor, University of Zadar.
Jadranka’s research interests are in peer review and research assessment, new trends in scholarly publishing, ethical issues, and research data.
Comments
You're looking to give wings to your academic career and publication journey. We like that!
Why don't we give you complete access! Create a free account and get unlimited access to all resources & a vibrant researcher community.
Subscribe to Journal Submission & Peer Review
