Early career researchers and their involvement in peer review
[This post has been republished from ASAPbio's blog. It has been written by Gary McDowell, PhD is Executive Director of Future of Research, a nonprofit which wants to champion, engage and empower early career scientists with evidence-based resources to improve the scientific research endeavor.]
When it comes to peer review and the role that Early Career Researchers (ECRs) play in it, I am of course reminded of the immortal words of Steve McKnight in his President’s Message at the American Society for Biochemistry and Molecular Biology (ASBMB, emphasis mine):
the average scientist today is not of the quality of our predecessors; it’s a bit analogous to the so-called “greatest generation” of men and women of the United States who fought off fascism in World War II compared with their baby boomer children. Biomedical research is a huge enterprise now; it attracts riff-raff who never would have survived as scientists in the 1960s and 1970s. There is no doubt that highly capable scientists currently participate in the grant-review process. Likewise, unfortunately, study sections are undoubtedly contaminated by riff-raff.
The quote that launched a thousand t-shirts for our own organization, Future of Research, also initiated a tirade of criticism (I recommend this from the Drugmonkey blog). And yet, this is a refrain that I commonly hear about study sections in particular, peer review in general, and even extends to the participation of ECRs in applying for grants themselves: that they just aren’t what they used to be, back in the day.
It’s unclear how this could possibly be. The biomedical enterprise in particular has expanded continuously since the 1960s and 1970, most notably of course in the population of graduate students and postdocs (the number of tenure-track positions has not increased). The resulting hypercompetition is touted as a benefit – “the cream rises to the top” – and, of course, institutions are insistent that they are providing training for these junior researchers. So, if they are well-trained, and there’s more of them competing for the same number of positions – surely they should be better, not worse?
ECRs are not, of course, inferior to senior academics; just less experienced. But these smart people are basically restricted from the academy by those who want to maintain that they are intellectually superior. Graduate students and postdocs are barred from peer review explicitly by certain publishers (this was disclosed as Elsevier’s policy in a panel on publishing at our own 2015 Boston Future of Research Meeting) in the same way they are often excluded from applying for grants in their own right. Efforts to exclude qualified postdocs from reviewer pools prevents incorporation of new ideas and perspectives into the process.
I’d like to discuss here the role, and concerns, of ECRs (particularly focusing on graduate students and postdocs) in peer review, as part of an introduction to the ASAPbio/Wellcome/HHMI peer review meeting, and where I hope to lead or participate in a Friday unconference session addressing ECRs and their role in this scholarly pursuit.
A brief history of peer review: not much older than ECRs
Peer review is not as old as many might think; it was essentially implemented in its current form during the 1960s and ‘70s (the halcyon days that McKnight spoke of) at both journals and grant awarding agencies (see an interactive timeline here, and a longer discussion in “A multi-disciplinary perspective on emergent and future innovations in peer review”). For example, as Tennant et al. discuss (emphasis mine):
Nature, now considered a top journal, did not initiate any sort of peer review process until at least 1967, only becoming part of the formalised process in 1973…Such expansion was primarily due to the development of a modern academic prestige economy based on the perception of quality or excellence surrounding journal-based publications…Indeed, this story linking identity to peer review is taught to junior researchers as a community norm, often without the much-needed historical context
The last is the most critical part; even when McKnight is touting the good ol’ days of peer review, those are actual the very earliest days of peer review being given the emphasis it now has. Like the biomedical enterprise itself, essentially assembled post-World War II in the U.S., the systems in place were created within memory, and by those who have subsequently benefitted from the labor of ECRs. There is no great historical basis, nor momentum, supporting exclusion of ECRs from peer review.
Peer Review determines the career of the ECR; they should participate fully
Papers, and therefore peer review, are the currency for career progression for ECRs. To give full agency, and the scholarly recognition they deserve, ECRs should be able to participate in that currency, and indeed this issue is part of a larger issue of agency and full participation in the academic community that ECRs often struggle against. The exclusion of ECRs in this process is largely gatekeeping, and as we have seen, part of a system that is not even that old. As Tennant et al. point out:
A scholar’s first research article, doctoral thesis, or first book are significant life events. Membership of a community, therefore, is validated by the peers who review this newly contributed work. Community investment in the idea that these processes have “always been followed” appears very strong, but ultimately remains a fallacy.
Graduate students, as those supposed to be training in the sciences, should be participating in peer review in a manner that is conducive to their development as academics. Therefore, they should be trained in constructive peer review. Indeed some journals and societies already have such mentored programs. Postdocs are researchers in a position of mentored independence – supposed to be working on their own projects and research plans, and learning how to manage a research group from an independent principal investigators. As such, I can see no reason for postdocs not to be fully involved in the peer review process. If they are fulfilling the mission of what the postdoc position is supposed to be, and are not simply cheap hands at the bench, any postdoc who is not intellectually capable of performing peer review in their area of expertise should not be in a postdoctoral position.
Indeed graduate students and postdocs are not actually excluded from peer review; this would simply not be possible given the burden of review that currently exists in the system. Their contributions must be recognized formally to give them the credit they have earned. Denying ECRs, particularly postdocs, credit for participating in the peer review process purely because they are not in the ranks of the professoriate, is simply gatekeeping, and is intellectually dishonest. All this draws from false notions of excellence, the pervasive “riffraff,” rather than any objective assessment of scholarly ability by ECRs.
This gatekeeping is chiefly the responsibility of the journals. Again, from Tennant et al.:
Proprietary reviewer databases that enable high selectivity are seen as a good business asset. In fact, the attribution is based on the false assumption that peer review requires careful selection of specific reviewers to assure a definitive level of adequate quality, termed the “Fallacy of Misplaced Focus”
The concerns of junior and marginalized voices in peer review reform
That peer review is largely the work of the professoriate – the predominantly white, male professoriate – and in some cases explicitly excludes the graduate student and postdoc population (which is the most diverse population to draw potential reviewers from, and from which the transition to the professoriate remains stubbornly difficult, see Gibbs et al.) reinforces the lack of diversity in the reviewer population, and prevents reviewing environments from becoming more inclusive, and providing greater access to fair review from all.
There are massive structural faults with peer review at journals and funding agencies that make the processes inequitable and exclusive (see review by Lee et al.). Black researchers in the U.S. are 10 times less likely to receive major NIH awards (from the Ginther report). Journals invite too few women to referee, women are underrepresented in authorship (and of course, in one infamous case, women were even asked to put a man onto a paper’s authorship) and women are also more likely to be pressured into adding more authors. Journal editors are more likely to support peer reviewers of the same gender, and are more likely men. Women are more successful in grant review when the science is assessed, rather than the scientist.
The increased participation of a more diverse pool of reviewers is one issue that is butting against other proposed reforms for peer review, however. Junior researchers and those from marginalized communities have very real concerns about Open Peer Review (OPR), particularly signing reviews. Ross-Hellauer discusses that there is currently no clear evidence that open reviews reduce quality, produce more sycophantic/critical reviews, or that retribution is visited upon those performing open review. But it is important to recognize that both explicit and implicit biases may manifest themselves in both overt and subtle ways on members of these communities, as can be seen in other realms of academic life and has been the case historically for these communities. Terry McGlynn discusses this in “Why I prefer anonymous peer reviews” and Jeremy Yoder collected perspectives in this 2014 piece, “Why we don’t sign our peer reviews”.
My own perspective in signing reviews, which I do whenever I can, must take into account that I am a white man, and also one who takes arguing from a junior perspective as central to my identity. My own reviewing style, even with colleagues who are friends, tends also to be quite critical, so I personally like signing reviews to not only temper what I say and hold myself in check, but also to make clear that I intentionally would say such things to the person’s face, openly. The reasons to sign a peer review are so highly individual that personally I am in favor of opt-in dynamics, certainly until the appropriate checks and balances against retribution and bias can be developed with careful study. However I do also view peer review as an important and unrecognized form of scholarly work that could enhance one’s reputation as a scientist, as I and colleagues have laid out further in the Self-Journals of Science in “Novel processes and metrics for a scientific evaluation rooted in the principles of science”.
My rationale draws from the belief that scholarly identity and participation in the academic community is strongly bound up in activities like participation as an investigator on a grant, or as a peer reviewer – notably intellectual pursuits that are currently largely isolated from the junior researcher who is kept in the practical role of the bench researcher. The majority gender in these two populations in the life sciences (more women in the bench role, more men in the professorial role) is not a coincidence, but a structural inequity that associates labor with women and intellect with men; see Julie Des Jardin’s book “The Marie Curie Complex” for more discussion on this. Tennant et al. again:
By operating as a closed system, it protects the status quo and suppresses research viewed as radical, innovative, or contrary to the theoretical or established perspectives of referees
However, as they also mention, pointing out Rodríguez-Bravo et al.:
Nonetheless, careful evaluation of existing evidence and engagement with researchers, especially higher-risk or marginalized communities (e.g., Rodríguez-Bravo et al. (2017)), should be a necessary and vital step prior to implementation of any system of reviewer transparency.
Steps Forward
How can ECRs get involved? Cell’s Crosstalk blog gives some good foundational advice; but of course, without full participation in the process allowed by the journals, this can only go so far. A number of journals do allow and encourage peer review, and even provide peer review training, and it’s worth looking into your field and asking which journals are willing to facilitate this (this is also a point of focus you can direct at your society journals – if you are a member of the society, they should actively engage your participation in activities of benefit to the membership as a whole). NIH holds ECR peer review training, for example, at the Center for Scientific Review.
Journals, societies and funders can all also benefit from the inclusion of ECRs in positions of power, to hold these organizations accountable, right up to the level of the highest boards and councils. PLoS One and eLife are recent shining examples of journals that have appointed ECRs to their board of directors (Meredith Niles and Prachee Avasthi respectively). Societies such as the American Society for Microbiology and the American Fisheries Society similarly have ECRs on their board of directors. Most recently, NIH ran into trouble for its Advisory Council to the Director derailing proposals to cap funding for investigators, aimed at improving prospects for ECRs, particularly because no ECRs were able to participate in the decision-making process – this has now been addressed with the inclusion of three ECRs on their working group.
Expanding the pool of reviewers, giving credit for this work, training scholars in the pursuit and recognizing peer review as a scholarly contribution are all critical for ECRs – and for the biomedical enterprise. Currently 20% of researchers perform 75-90% of reviews, which is an unreasonable and unsustainable burden. And a proper incentive structure is critical to maintaining high quality research, as Tennant et al. discuss:
The present lack of bona fide incentives for referees is perhaps one of the main factors responsible for indifference to editorial outcomes, which ultimately leads to the increased proliferation of low quality research
Indeed, there are actually incentives for peer review, contrary to what might currently be believed. A major incentive for the subset of foreign junior researchers looking to remain in the U.S. is to build a case for immigration as an established researcher, which includes pieces of evidence like having performed peer review. Two-thirds of U.S. postdocs are estimated to be foreign-born, but it is unclear how many are looking to stay in the U.S. as few institutions track former (or accurately count current) postdocs and where they go; but I estimate there are roughly 40,000 foreign postdocs in the U.S. currently, and the number looking to remain is likely to be in the thousands. Recent advice from an immigration lawyer at a national conference stated simply that “what you need for immigration is not the same as what you need for your career.” With a massive pool of researchers eager to undertake peer review, they will find somewhere to do it, and where they do it is likely of no concern to USCIS. I would be intrigued to see if this is where the dreaded predatory open-access journals get their reviewers from (another incentive for immigration, and even in academe in many cases, is to publish often – as “well” as possible, but also often – possibly also aiding such publications). Regardless, with perverse incentives for review a possibility, the community has a duty to ensure review is carried out as well as possible, involving ECRs in a productive, constructive and well-trained process.
Published on: Sep 12, 2018
Comments
You're looking to give wings to your academic career and publication journey. We like that!
Why don't we give you complete access! Create a free account and get unlimited access to all resources & a vibrant researcher community.
