Podcast: What are the characteristics of good peer reviewer feedback? Interview with Sin Wang Chong

If you were asked to list the most important attributes one needs to have as a peer reviewer, what would they be? Very likely, you’ll talk about subject matter expertise, critical-thinking skills, and the ability to provide a sound, objective assessment on the quality of research presented in a manuscript. Which are all, no doubt, imperative.

But most of us rarely think of peer review as performing an important social function, as well as an intellectual one—of supporting fellow researchers in your field and building collegial relationships through peer review feedback. That’s where the concept of “feedback literacy” comes in.

In an interview conducted to celebrate Peer Review Week a couple of months ago, I spoke with Professor Sin Wang Chong, who has done extensive work exploring the value of feedback literacy in scholarly peer review. (Segments 1, 2, and 3 of this interview were published during Peer Review Week 2023.)

In this additional segment that focuses on sharing practical tips for peer reviewers, Sin Wang talks about what makes peer reviewer feedback effective (or ineffective), the factors that influence feedback quality, and how feedback quality can be improved.

About Sin Wang Chong: Sin Wang is Director of Impact and Innovation at the International Education Institute, University of St Andrews, and Head of Evidence Synthesis at the National Institute of Teaching in England. Concurrently, he is a visiting and adjunct professor at a number of universities in Asia, England, and the United States. 

He is Chair of the Scottish Association for Teaching English as a Foreign Language (SATEFL) and serves on the Council of the British Educational Research Association and the Executive Committee of the British Association for Applied Linguistics.

Sin Wang’s research interests are in evidence synthesis, educational assessment, language education, and higher education. He is Associate Editor of two SSCI-indexed journals: Innovation in Language Learning and Teaching and Higher Education Research & Development. He is co-founder and co-director (with Shannon Mason) of Scholarly Peers, a platform to support doctoral students and early career researchers to navigate journal peer review.

[Audio transcript]

Mriganka

Based on your research then, what are the characteristics of good reviewer feedback and poor reviewer feedback?

Sin Wang

A number of characteristics for good reviewer feedback.

First, I think professional peer review feedback is corrective, of course, but it’s also constructive. So you don’t want to just point out people’s errors without giving suggestions for for people to improve. And more importantly, it’s evidence-based and manuscript-focused.

And what I mean is I don’t want to put down a comment that says I feel like this is not good enough or, you know, I want it to be focusing on a particular section. Maybe I could refer to a particular paragraph of the manuscript. I’ll be able to provide evidence. Maybe there are other scholars who are arguing against this method, for example. So I don’t want to target the authors. I want to target or focus on the work itself. I think it’s a very important distinction. And secondly, I want it to be about what the best professional judgment is, but not really what my personal preference is. As I shared earlier.

Second, I think authors appreciate peer review feedback that is precise and detailed. So of course, sometimes we receive reviewers’ report that’s, you know, 10 pages long, 12 pages long. And I have mixed feelings about that kind of report as an author. I mean, on the one hand, I’m really appreciative of the time the reviewers spent and, you know, devoted on reading my paper. On the other hand, I am very, very stressed because I have 10 pages or 12 pages of feedback to incorporate and consider.

So being precise and detailed is very important, as well as providing specific and well-justified suggestions And I think suggestions sometimes can be vague. Suggestions, sometimes can be overly idealistic. For example, people may say, Can you redo this study again? And, you know, go back to the participants and collect the data again. Of course, these are suggestions, but are they actionable? And that’s why I talked about actionable feedback earlier.

Thirdly, I think while it’s unavoidable that peer reviewers bring with them their own experiences, biases and views, and it’s actually a positive thing, it’s important for reviewers to remain objective when providing feedback and to be open-minded especially when it comes to, for example, new ways of conceptualizing things or new approaches to research.

Another point, I would argue is, you know, good peer review feedback does not focus on every single problem in a manuscript, but points out the major concerns. Okay?

The next feature is I think is about language and tone. So, so far what I have talked about concerns the content and the substance of the feedback. But I think equally important is to think about the language and tone. So peer reviewers need to have a good command of written English if the language of, you know, peer review is in English, to convey their messages accurately. And also, I think it’s important to write in a respectful manner because you’re reviewing a piece of work by a fellow colleague. So we are not, you know, kind of superior than than authors.

Of course, when it comes to the organization, the peer review report, I think feedback needs to be presented logically. So, for example, following the sections of the manuscript, it’s just one suggestion you don’t have to do it, but it’s presented in a way that’s very easy to follow. One example is to refer to specific pages, paragraph numbers so that, you know, authors can always refer to their specific sections when they want to revise the manuscript.

Finally, I think it’s so obvious. Is that timely feedback is highly appreciated. You know, three, three weeks, four weeks or several months. You know, it’s not a short period of time. Of course, we understand it’s volunteer. It’s a kind of volunteering and it takes time. There are different priorities. But I always tell my students, you know, if you accept a peer review request, make sure you accept it because you have the time to do it. You don’t want to accept it, but then you delay the submission over and over again. And that’s not fair to the authors.

Going back, going back to peer review or poor peer reviewer feedback, a number of features I’d like to talk about for us to all think about and reflect on.

First, I think it’s unprofessional to what I call hijack the manuscript by kind of inserting comments that impose the reviewer’s personal views such as asking authors to cite their own work that does not have direct relevance to the manuscript.

Second, I think feedback that is too brief and ambiguous is of little use to authors when it comes to revising the manuscript. I think if it’s a rejection decision, it’s okay for feedback to be a little bit general because they may need to kind of rethink the whole project. But if it’s a major revision or minor revision decision, feedback needs to be very specific.

And then again about tone, peer review feedback, does not benefit authors when it’s demeaning, attacking the writers instead of the work done. So again, it’s work-focused. It’s not person-focused

And authors also dislike feedback that’s poorly organized. So if you have a section on the literature review somewhere in the report, and then if you have a section of findings somewhere in the report that the authors have to dig them out, that may not be the best approach.

Finally, feedback that’s very delayed of course, is something that we would try our best to avoid.

Mriganka

What factors influence these different aspects of peer review feedback, like what influences the quality? And do you have any view on what kind of approaches can help improve the quality of feedback that reviewers give?

Sin Wang

I think there are a number of factors.

Firstly, it’s the reviewers’ professional competence. So from our systematic review, we identified that the majority of the included studies pointed out that expertise in domain knowledge affects reviewers’ feedback quality the most. So that’s about the content of the feedback.

Second factor that we identified is language skills. So language skills are an essential component of reviewers’ professional competence because it dictates whether reviewers can convey their messages clearly. So it has to do with how the message is crafted, how the feedback is produced.

And there there are some internal factors that we identified from the literature that can potentially affect the kind of quality of peer review feedback. For example, whether the reviewers are confident or they have a sense of authority to make judgments. The reviewers’ kind of altruistic moral obligation to support fellow colleagues or academic peers. The reviewers’ motivation for personal development and also their personal biases as well. So these are internal factors of the reviewers.

There are also a number of external factors that will affect the quality of peer reviewer feedback.

Firstly, it’s about the policy of anonymity. So if it’s a double-blind peer review model from the literature, it usually leads to more objective and more professional peer review feedback. It also depends on the interaction between editors and reviewers. Some editors have a more kind of relaxed and kind of lenient kind of editing style where some are more kind of they’d like to have a little bit more control throughout the process. So, you know, it really depends.

And also, of course, the monitoring and the scrutiny by the public and academics. So, for example, if you have an an open peer review model where the peer review reports are published and the names of the reviewers are published, obviously you may get better quality feedback.

Finally, I think in terms of demographic factors, this is something we found from from our review that is quite interesting. There was one study that suggested that the quality of reviewers’ feedback possibly—possibly—gets poorer by age due to the increasing commitments. And the same study also found that reviewers who have higher faculty positions and more prolific publication track records are more likely to give hasty and superficial feedback.

Now I have to just add a caveat here. It’s based on one published study, one alone, you know, So it’s a there are obvious exceptions because I have experienced very detailed feedback from more senior colleagues that, you know, this is just a factor that is documented in the literature.

Responding to your second half of your question about which approaches can help improve the quality of feedback, I’m referring to my Learned Publishing article again, where at the very end of it I suggested three approaches.

The first approach is what I call a knowledge-based approach, and this is what most people are doing, what most academic publishers are doing right now. They have if you look at big international academic publishers, they have these online resources for reviewers, like, for example, they have training courses for reviewers. This this focuses on the knowledge of peer review. For example, understanding what the different steps or milestones are in the peer review process, what is the meaning of different editorial decisions. So this focuses on the knowledge.

The second approach is a skills-based approach and when I say skills, I focus more on the skills for delivering good feedback or feedback literacy, as we talked about earlier. So my suggestions in the paper are, for example, having exemplars of peer review reports and to have, you know, new reviewers kind of analyze them, try to understand the good features of the reviewers’ reports. So very recently I think there are more kind of open peer review reports released and published.

For example, think about F1000, for example. It’s where you can find tons of open peer review reports to learn from. And of course, another of my suggestion is to not only read those reports as exemplars, but also have somebody to kind of annotate those reports, as learning resources, because if you just read a published review, you have no idea whether that’s a good example or a mediocre one or a bad one. So and so, you know, there could be kind of annotation or kind of feedback on the feedback of reviewers. So that’s the skills-based approach.

Finally, it’s the community-based approach, and I think this is the most effective to build a community of researchers and scholars and academics, publishers or organizations such as yours, who care very much about peer review. They share resources, challenges, difficulties. So for example, I set up a network of scholarly peers. We have a Twitter account, we have a website where we host resources for early career researchers. We have podcasts interviewing academics about their own peer review experience. So this, this, this sense of belonging is it’s important.

I remember I just interviewed another senior academic last week and he said something that I was very impressed by. He said that he still receives a lot of rejections at his kind of career stage, and he’s like one of the leading scholars in his field. And he’s being very honest and transparent about his rejections, hoping that the junior researchers don’t feel that it’s you know, it’s peculiar to their case. So I think it’s that kind of emotional support that is so important.

So that can the whole notion of demystifying journal peer review process is so important. And I think a community-based approach can achieve that. Demystification doesn’t just mean reveal the secrets within the process, but revealing what different stakeholders truly think about review. For example, a lot of scholars may think of editors as somebody who are, you know, almost like a godly figure. But actually a lot of editors are very approachable. They’re very friendly, they’re very supportive, and they are like you and me. So, you know, that kind of layer of demystifying or humanizing the kind of peer review process, is extremely important. And I think a community-based approach is essential to achieve that.