Spotlight On: Peer Review

Jo VanEvery, Academic Career Guide · Spotlight On: Peer Review

Peer review is central to academic freedom.

It ensures that the criteria for evaluating the quality of scholarship are in the control of scholars and not subordinated to commercial, government, or other interests.

My writing on peer review over the years has primarily focused on its use in publishing. Scholarly publishing relies on peer review both for decisions about what to publish, and for editorial feedback to improve the quality of published work. This Spotlight brings together those posts. I’ve written in more depth about this in Peer Review (A Short Guide), published in 2019. Some of the posts in this Spotlight were first written for the launch of that book.

Peer review is also used in other processes in which your body of work is evaluated. Decisions that rely on evaluating the quality of scholarship generally involve some form of peer review. External reviewers are often sought for promotion applications, for example. Funding decisions are made by committees of peers, often with the guidance of more specialist reviews from reviewers. This also supports academic freedom by ensuring that research is funded, and scholars are employed and promoted, according to criteria that scholars collectively have agreed.

Publications in other peer review processes.

The questions most often asked about how your publications get used in hiring, promotion, funding, and other processes that evaluate your body of work are:

Why do some things count more than others?

and,

Why do peer reviewers use the reputation of the journal or press you published with instead of making their own evaluation of the quality of the materials?

I tried to describe some basic principles in “How scholarship is evaluated” and “Peer reviewed journals and monographs in the academic evaluation process” back in 2011. These were framed in relation to a debate about academic blogging, a specific example of scholarly conversations that often don’t “count”.

The author’s goal in publishing is to make a contribution to scholarly knowledge in a particular field, area, or discipline. (see “Communication vs Validation: why are you publishing”)

This goal is shared by the reviewers and the editor. Peer review ensures that decisions about what gets published are based primarily on academic criteria of what constitutes high quality research, and that each contribution situates itself in the ongoing conversation appropriately.

Since I wrote those things, Google Scholar has become more widely used (do clean up your profile there), but also new metrics have been developed. The problems I indicate in those 2 posts don’t seem to be resolved, and may have become more complicated. My general advice still stands:

Focus on your communicative aims and trust that your goals are similar enough to the criteria for assessment that things will be okay.

As a reviewer, try to be conscious of what the indicators are trying to assess and make sure your evaluations make sense in those terms.

As I pointed out in “What is the point of publishing peer-reviewed articles if you care about changing things out there in the world?”, non-academic audiences for your work may value peer reviewed publications as a mark of quality, which enhances the value of other kinds of research communication you engage in.

Peer review in publishing.

I write primarily to help academics make decisions to manage their workload, while prioritizing the work that is important to them in a context that is increasingly unfavourable in multiple ways.

The questions in this area include:

What do I do with these comments, especially if they conflict or I disagree with them?
How do I manage the revision and resubmission process?
Why does peer review take so long?
How many reviews should I be doing?
How do I give feedback in ways that are consistent with my values?

The clearest explanation of the value of peer reviewed publications is probably “What is the point of publishing peer-reviewed articles if you care about changing things out there in the world?” This post is less about peer review directly, and more a defence of scholarly publishing. In the light of the recent crisis, I have written about the value of peer review from the perspective of you as a reviewer in “Why peer review is worth saving”

Back in 2014, I seem to have addressed this question in a much more limited and practical way. “The role of peer review” and “How can reviewer comments improve your work” are an early attempt at teasing out the gatekeeping and editorial feedback functions of peer review.

I expanded on this considerably in Peer Review (A Short Guide) in 2019 and summarized my updated position in “Peer review supports academic writing” which I published to accompany the launch of the book.

If you are interested in the history of peer review in scholarly journal publishing, in particular, I recommend the work of Aileen Fyfe and her colleagues. In 2017, I wrote about their report “Untangling Academic Publishing” highlighting some of the key elements of the history of peer review. In 2022, they published a book A History of Scientific Journals: Publishing at the Royal Society 1665-2015 (UCL press, DOI: 10.14324/111.9781800082328).

What this means for you as an author.

It follows from all of this that revision following peer review is normal, something that I discussed with Katherine Firth in an interview which was recorded in conjunction with the launch of Peer Review (A Short Guide). Accepting post-review revision as a normal part of your writing process also means that your vision for what you are trying to achieve with this particular piece of writing is central.

Your article (or book) is not finished when you submit it. It is frustrating to wait for the reviews before you can begin the next (and hopefully final) round of revisions. In the meantime, tidy the project away to temporarily close the loop, perhaps writing a note to your future self to ease the transition back.

I’ve written some advice for when to chase up the reviews in “The Frustrations of Peer Review: Why is it taking so long”. This was written on the cusp of the transition to widespread use of online submission portals. I do not recommend checking the status in those portals frequently (or even regularly). Give the editor time to find reviewers and the reviewers time to do the reviews. Turning your attention to a different project will allow you to come back to this one with fresh eyes when you receive the decision.

When the decision is made, give yourself time to manage your emotions around the decision before opening the comments. It is normal to be upset or angry, or to experience self-doubt. If your anxiety is high, you could ask a friend to skim them for you and warn you, or sit with you (virtually or in person) while you do so.

Keeping your authorship centred as you make decisions about how to use the reviewers’ and editor’s comments to revise your manuscript is not easy. I have found the work of Liz Lerman helpful. She emphasises the importance of asking questions before offering opinions, something you might have to reverse engineer for the anonymous, asynchronous process used in academic publishing.

You will still need a process for managing the revisions, so you can ensure you have addressed everything. Ideally your process will also make it easier to communicate your decisions and changes to the editor. Several people have shared their systems and I link to these in the related posts section of Managing Manuscript Edits.

I cannot emphasize strongly enough that this is not merely a practical process of doing what the reviewers suggest.

You must make your own decisions based on your vision for this particular piece.

You might need to revise your vision, especially if it is trying to do too much. It is your vision that will enable you to reconcile competing suggestions, keep the revised version within the journal’s word limit, and address the comments of a reviewer who has either misunderstood your aims, or would prefer that you did something different.

I offer a self-paced course, with group coaching support, called Dealing With Reviewer Comments based on these principles. The class guides you through a process of reading your comments, making decisions, and creating a revision plan. The next group coaching session is in January 2023. You can register now for access to course materials and get the most from that support.

You will also be a reviewer.

You will participate in this system in both roles. One day you are receiving reviews. Another day, you are reviewing someone else’s manuscript. At some point in your career you may sit on the editorial board of a journal or even edit a journal for a few years. While publishers clearly benefit financially from all of this volunteer labour, the primary beneficiary of your peer review labour is other academics.

These posts are about your responsibilities for doing the work in a timely manner: “How you, as a peer reviewer, can contribute to a better process” and “The role of journal editors in long review times”. Brandy Schillace has also published a very important piece at The Scholarly Kitchen about how the role of journal editors has changed over the last 10 years, something that has implications for academic freedom as well as workloads.

In order to be able to do that, you need to manage your workload. You don’t have to say yes to every request you receive. I’ve written about how to decide in “Making Decisions: Peer Review”. More recently I’ve considered the recommendations I made in the light of the current crisis, and radical commitment to the community of thinkers. “Peer Review is worth saving” returns to the question of why do it, and also updates some of my thinking about how to make time for this work.

Most of my guidance on how to centre editorial feedback that contributes to the advancement of knowledge in your reviewing practice is contained in my book: Peer Review (A Short Guide). The general principles outlined in “Why questions are useful in feedback” are a good place to start, keeping in mind that your goal is to help the author.

Relevant Short Guides

In addition to not being very careful about key words when writing about peer review and related issues, I also developed a lot of my thinking on this as I wrote Peer Review (A Short Guide). If you’ve found this Spotlight helpful, the book expands on what I’ve said here and includes prompts to help you as you respond to reviewer comments or provide them as a reviewer.

The material on how peer reviewed publications relate to other forms of scholarly communication is expanded in Scholarly Publishing (A Short Guide) that revisits this discussion, with the aim of helping you make decisions about how to publish. It also contains chapters about each of your options, with prompts and questions to help you make the decisions that are right for you.

In October 2022, I combined Scholarly Publishing and Peer Review into one eBook bundle. The principles outlined in those Short Guides will guide your decisions and help you ask better questions of mentors with more knowledge of specific journals, presses, funding competitions, and institutional hiring processes.