Scientists depend on the proper evaluation of research that creates the foundation for future work, and the public expects curated scientific content to be trustworthy. All forms of peer review, whether for ethical, technical and sound science criteria or for additional novelty, significance and perceived impact help ensure rigor in scientific research.
There is, however, community and public skepticism regarding the quality, trustworthiness and authenticity of the review process, from the initial stage of evaluation before reviewer assignment to the final editorial decision. Making peer review more transparent, at any stage, has the potential to revitalize the process and restore trust in the system.
Efforts to increase transparency in peer review should address challenges that include:
- Training and mentoring on how to review (see ten tips here)
- Variable publisher open science and transparency policies
- A process that is static in time and in number of people that participate
- Insights made during the review process often go uncaptured or remain hidden
Reshaping Peer Review
Change is already happening as the scientific community develops variations on themes of open and transparent, and as publishers provide more peer review offerings that range from community participation to open but anonymous, to fully open and signed reviews. While not yet functioning at scale, experiments incorporating more transparent ways to discuss and assess papers over the entire lifecycle of the research are inching their way into practice. This way, the publication of an article isn’t the single defining event in its life; it is just one chapter of its story. Many of the arguments in favor of increased transparency in peer review also hold true in the discussion of benefits of preprint submissions. According to researchers working with neuroimaging, as stated in their PLOS Biology Community Page, “preprints allow the wider community to give feedback to the authors about the manuscript and potentially improve it, which is beneficial for both the authors as well as the journal the paper will be submitted to. For example, the present paper received useful comments from three individuals in addition to the appointed peer reviewers.”
A Question of Signing
At PLOS, we’ve looked at one slice of transparency in peer review—signed reviews made available exclusively to the authors. In a research project that used a survey mechanism to collect experiences and opinions, our Publishing Operations team underwent an assessment of reviews from 2013-2016 in three of the PLOS journals (PLOS ONE, PLOS Medicine and PLOS Computational Biology).
“Our reviewer community is particularly engaged, and that’s what makes working at PLOS on this issue so exciting. Together we will be able to create solutions, both incremental and substantial, that bring constructive feedback to authors and transparency to the review process.” Helen Atkins, PLOS Director of Publishing Operations
Some of what we learned is that reviewers who were not in the habit of signing reviews simply had never been asked, or were not sure of the benefits. But it’s not as easy as just educating in these areas. These scientists also indicated that not signing allows them to be more honest and safe from retribution. We also discovered that signing can improve reviewer accountability and constructiveness, and help authors learn of a reviewer’s area of expertise. Authors who favored receiving signed reviews valued having this additional information as it provides potential for more open communication, moving research forward. The full results of this project, presented by Elizabeth Seiver, PLOS Researcher and Helen Atkins are part of the Editorial and Peer-Review Process Innovations Session on Tuesday, September 12 during the International Congress on Peer Review and Scientific Publication.
Simple but Substantive Practices
In the PLOS ONE article “Peer Review Quality and Transparency of the Peer-Review Process in Open Access and Subscription Journals,” the focus is a general aspect of transparency in peer review—how academic journals (and/or publishers) present their peer-review process to the public. According to the authors of this study, a transparent peer review system “conveys to readers and potential contributors how the peer review is implemented and how articles are selected for publication.” The researchers found that “author’s ratings of peer-review transparency predicted their assessment of the quality of peer-review at that journal.” Even small changes in practices and on publisher websites can help in this area. In addition, guidelines from the Committee on Publication Ethics, “Principles of Transparency and Best Practice in Scholarly Publishing,” stating that all journal content apart from editorial material should be subject to peer review from outside experts must be met.
Learning from Early Career Researchers
Many ECRs do not get appropriate training on how to prepare or review manuscripts. Improved transparency in peer review would not only satisfy mid-to senior-level scientists, but in providing some form of open dialogue or open accountability it enables these scientists to lead by example and provide mentorship to the next generation of reviewers.
Young and upcoming scientists have plenty of ideas when it comes to improving transparency in peer review. At PLOS, we received over 150 essays on how to revamp peer review from Early Career Researchers applying for our ECR Travel Award Program. These creative young scientists described what they consider to be characteristics of the optimal peer review process and how they might build this process either from scratch or using aspects of existing practice.
Their ideas, edited for brevity, include:
- Invite reviewers to publish reviews of the article (should they wish to reveal their identity) as an accompanying commentary, for no additional fee. If reviewers know they could gain an additional publication for their efforts, this would motivate them to review more articles and respond in a constructive and timely fashion. Victoria Leong; orcid.org/0000-0003-0666-9445
- [Provide] incentives for reviewing that encourages kind, open but fair responses; we would also be affecting a positive change in the culture of science; which will advance the science itself. Rebecca Gelding; orcid.org/0000-0003-4883-8075
- Reviews should be open, archived and after publication, reviewers should be revealed. This aims to ensure two aspects of quality control: reviewers take more seriously their job since it will be public with their name tag on it; and reviewing records can be used when considering career development. Juan Rocha; orcid.org/0000-0003-2322-5459
- Review record should also be one of the criteria judging and advancing a researcher’s professional development. Knowing that a reviewer’s identify would be revealed later and shared among peers, a reviewer would have more incentive to avoid giving low-quality comments. Xiao-Peng Song; orcid.org/0000-0002-5514-0321
- A transparent, open review process may promote accountability among reviewers. A peer reviewer whose dated comments are published as supplementary material with the article has a greater incentive to conduct a thorough and timely review of the manuscript. These same published comments could also be accessed by other researchers who are struggling to address similar issues in their own studies. Sericea Stallings-Smith; orcid.org/0000-0002-4876-9965
Slices of the Transparency Pie
Solutions that will help peer review achieve its scholarly ideal are not untenable; the challenge lies in that they must satisfy a diverse researcher and stakeholder community. While some improvements are easier to implement than others, even small slices that expedite and enrich the process of assessment in fundamentally new ways contribute to advancing science and discovery for the broader scientific community.
Publishers have an opportunity to improve both speed and efficiencies: to improve review forms that may be cumbersome or insufficient to provide thoughtful and constructive feedback to authors and to provide training for reviewers and editors that mitigate potential bias. Additional possibilities include direct or facilitated mentoring of early career researchers to improve their understanding of the principles of peer review and how it is practiced within the scientific community. Imagine the impact of a global editorial and reviewer contributor community, appropriately trained, recognized and incentivized.
For more insights on peer review listen to the following PLOScast episodes and read the following posts on The PLOS Blogs Network.
PLOScasts
- An overview of the history of peer review, how it’s changed over time, and the increased demand for transparency with Aileen Fyfe of University of St. Andrews
- A lively discussion on the future of scientific collaboration and publishing, including transparency in peer review, with former PLOS Advocacy Director Cameron Neylon, currently at Curtin University
- Learn how Knowledge Lab applies informatics and systems modeling to understand relationships between researchers, reviewers and the review process with Eamon Duende of the University of Chicago
- Ideas on how to stay up to date on all the scientific literature and the potential of preprints in paving the way for a more transparent peer review process with James Fraser of University of California, San Francisco
PLOS BLOGS
- PLOS Collections (Staff blog)
Report – PLOS iGEM Realtime Peer Review Jamboree. 2017
- PLOS Paleo (PLOS Community blog)
Peer Review from a Volunteer Editor’s Perspective. 2017
- Absolutely Maybe (Indie blog)
Good Enough? Editors, Statistics & Grant Peer Review: PRC8 Day 2. 2017
- On Science Blogs (Indie blog)
PubMed Commons and Post Publication Peer Review. 2017
- PLOS Biologue (Staff blog)
What does peer review mean when applied to computer code? 2013
Image Credit:
Etsy; FuzzyButtFarm
Views: 0
