6.7- Doing a peer review

Once you’ve been published, it won’t be long before you’re asked to do a peer review. If you’re invited, take the opportunity. Peer review is a great way to learn and to build confidence. The first time you’re asked to do a peer review, you may feel intimidated. You may feel like, I’m too inexperienced, I don’t have enough knowledge of the field, I’m just a graduate student. But if you get asked to do peer review, you should absolutely take advantage of that opportunity. Journal editors are looking for young reviewers. Young reviewers are often more up on the latest in a field, on the latest techniques. Young reviews also tend to do a more careful job than people who have been reviewing for eons. There was a research study presented at a conference a few years back where they traced the natural history of peer reviewers. And they found that the longer people had been peer reviewing, the more poorly they did the reviews. There was a deterioration effect, because the more you do, the faster you get, the less careful you are. So editors want young reviewers. And one of the greatest benefits of reviewing is that you will see the back end of the publication process. So it’s a great way to learn about the publication process. It’s also great for your confidence, because you’ll see that not everything submitted for review is of that great of quality. You’ll realize that you can do just as well or better than many of the submissions you see. You’ll also get the opportunity to see the reviews that other peer reviewers turn in. That’s also helpful for your confidence, because you’ll see, when you get the other peer reviewers comments back, that they had many of the same critiques as you. You realize that you do know something about the field. You know what you’re talking about, and also you’ll learn something from the other reviewers comments because sometimes they may see things that you missed in the paper. All right, here some things to keep in mind if you are a peer reviewer. The first thing I always like to talk about with peer review is the tone. When doing a peer review, you’re supposed to be looking for both the positives and the negatives. But because you have to make a judgement on the paper, it’s understandable that you naturally start to look for the problems more than the strengths. As a result, it’s easy to slip into a condescending and critical tone. Just because you’re being critical, doesn’t mean the tone has to be critical. There’s a way to present criticism in a positive and encouraging way. I always like to picture that on the other end of this whole review process, there’s some poor graduate student who’s the one who did all the work, who’s the first author on the paper, and their confidence is on the line. So my tone matters, I have to remind myself to be positive. For example, you could say something like, the authors should delete table 5. Not only is it completely irrelevant, but it also reveals their utter lack of statistical understanding. That’s a very harsh way of delivering a criticism. Compare that to, table 5 contains unnecessary information, for example, a Pearson’s correlation coefficient may not be appropriate here. The authors should consider revising or omitting the table. Notice these are actually the same criticisms, but they’re couched in a very different tone. The second version also gives specific examples of what the authors can do to fix the table so it’s more useful. We always want to shoot for that second way of presenting things. Notice that the second version focuses on the table, on the specific problems, rather than criticizing the authors. Remember that the point is to critique the work, not the authors. When you draft your review comments, it’s natural to say things like, the author’s got this wrong, or the author should have done this. Go back and revise that, so that you are critiquing the table, or the method, rather than the authors. That makes a big difference. Also, avoid generalizations, it doesn’t help to be overly general with your criticisms. You need to point out specific errors, otherwise it’s not helpful to the authors. Try to use positive instead of negative language. Instead of saying the paper is poorly written, you could say the writing and presentation could be improved. And then give a specific example of what you mean so it’s not too vague. The other thing is avoid lecturing to the authors. I have a tendency to do this, because I do spend a lot of time lecturing. So naturally, I want to teach the author something about statistics or something about writing, but that’s not the purpose of peer review. In fact, it comes off as condescending, so avoid that as well. Just so you are aware, there are different types of peer review that you might encounter. The most common type is the single blind peer review. This is where the authors are blinded to the reviewers so the authors will not know who reviewed their paper. But the reviewers will know who the authors are. That’s the most common type. Some journals, instead, have a double-blind review. In this case, the reviewers are also blinded to the authors. Author’s names, their institutions, things like this are blacked out. Now, it’s an imperfect system because I do review for a journal where I’m blinded to the authors, and I have to say, if I really wanted to know who the authors were, It probably wouldn’t be that hard for me to figure it out. There are only so many authors who work in certain areas. They have a publication history. You could figure it out if you wanted to, but it does make it harder for the reviewers to recognize authors, which can prevent bias. More and more journals are starting to offer open peer review. In open peer review, nobody is blinded. The reviewer knows who the authors, and the author knows who the reviewer is. And the reviewer’s name and the whole text of their review may be made publicly available. And I have to say there have been several times where I’ve reviewed a paper, and only when I went to put in my review comments into the online system, did I realize that it was an open review. And I’ll tell you, knowing that your review is going to be published online tends to make you more constructive and more positive. I’ve actually gone back and edited my comments to be more upbeat because I didn’t want to come across as overly negative and mean. So open peer review does encourage a more friendly tone. There’s also something called post-publication review. There’s a lot of this going on informally. People comment in blogs on papers and on Twitter. This is a way of vetting papers that got through peer review, but might still have problems in them. And more formal channels for post-publication review are starting to emerge such as PubMed Commons. I’ve outlined in the next couple of slides how I approach a peer review. I’m just going to review this quickly, I’ll leave it there for you to go through in more detail if you want to. First, I scan the abstract. Then I jump right into the tables and figures, I like to look at the data first because that’s the story of the paper. And I want to make my own judgement on the data before I read the author’s take on the data. Then I’ll read the paper through quickly, just to get a sense of it, and I’ll make some high level assessments. Not nit picky things but high level things. I’ll ask myself, do the authors conclusions match their data? That’s often a point where papers fall down for me. I’ll ask myself whether or not the writing is even understandable. That’s another big picture problem you might see if you have to struggle through the text. Another big picture comment I might make is that the paper is too long for the amount of novel information it contains. I may recommend that the authors shorten it greatly. Then I’ll go through and formulate specific comments on each section. For the introduction section, the main things I’m looking for are clarity and concision, and a clear statement of the hypotheses or aims of the study. For the method section, I’m often scanning the methods to answer particular questions, such as how the data were collected? Are there places biases could’ve creeped in? Did the study have a proper control? Were the correct statistics used? For the results section, again, I’m looking for clarity and concision, and I’m looking for the text to compliment rather than repeat what is in the tables and figures. The same kinds of things we talked about when we talked about writing the manuscript. Then I’ll go through each table and figure. I’m looking to make sure that each table and figure stands alone, and tells a clear and easy to understand story. I want to make sure the figures aren’t misleading in any way, and that there aren’t numerical inconsistencies in the paper. Then I’ll look at the discussion, I’m looking to make sure that that first paragraph tells me clearly and succinctly what was found in the study. I want to know if the author’s conclusions are justified or if they’re overreaching. I’m going to look carefully at that limitation section to make sure that they addressed the limitations that I think are the biggest threat to the paper’s validity. And of course, I’m looking for good, clear, concise writing. I take notes as I go along, and then I turn this into my formal review. You should always start your comments with a one-paragraph, general overview before you jump into specific points. I advise you to first state what you think is the major finding and importance of the work. Hopefully this is pretty obvious, but it allows you to start with a positive, and also lets the authors know what you took away as the major point of the paper. Just in case they haven’t conveyed that well. And then you should jump into those positive, encouraging statements about the work. Whenever you’re criticizing somebody, you always want to start with a positive. There are always positives. Even if there are problems in the methods, maybe they did a really nice job in the writing. Sometimes the research question itself is interesting or novel, you can point that out. When you start with the positive, it makes the person on the receiving end of the criticism much more receptive to your feedback, and also boost their confidence. After the positives,state what you think are the one or two major limitations of the paper, if there are any. If you’re not advising the editor to accept the paper as is, what are your major, big picture issues with the paper? Maybe the writing or data presentation is just too confusing. Maybe the authors have overstated their findings, maybe they’re missing a critical control group. Just keep in mind that you are not supposed to reveal your final recommendation, so don’t say anything about rejection or acceptance. Then you’re going to give a numbered list of specific criticisms. I usually give somewhere between 5 to 15 specific criticisms. If I’m recommending that something be accepted outright, I’m presumably going to have fewer comments. If I’m recommending that something be rejected outright, I will also have fewer comments because I’ll just focus on pointing out those big picture, fatal flaws. If I’m recommending revise and resubmit, then I’ll tend to have more specific criticisms, because I need to tell the author exactly what I think needs to be fixed. Be as specific as possible. Generalizations are not helpful, because the author can’t figure out how to address these adequately. One thing I want to mention is reviewer is not the same as copy editor. Don’t waste your time picking out every single grammar and spelling issue, and pointing all of those out to the author. I sometimes see peer reviewers who will spend an inordinate amount of time on all those little mistakes, and that’s really not the job of the peer reviewer. Journals have copy editors who can fix these errors, so focus on the big picture issues of the paper. If the grammar needs a lot of work, if there are a lot of typos, point this out in a general way. And give one or two specific examples. And then tell the authors that they need to get copy editing or English language help. In addition to the comments to the author, you may provide some comments to the editor that the authors don’t see. I don’t always provide these, but it’s an opportunity to be a little more frank. You don’t need to worry about tone as much. If there are touchy, ethical concerns with the paper, such as plagiarism, you might address these here. Some journals will also make you fill out some kind of grading or ranking sheet. I’ll show you an example in a minute. Of course, you always have to select your final recommendation as well. Here’s an example of a grading sheet that you might see for a journal. You may be asked to rank the impact of the research, or how interesting you think it will be to that journal’s audience. Or how original the results are, and sometimes, you’re asked to give an overall manuscript rank. These ranking systems can feel pretty arbitrary, but just try to be internally consistent. Papers you really like are going to get high marks from you in general, papers you think are flawed are going to get lower marks. These grading sheets do help remind you of all the aspects of the paper that you should be considering. So that’s the basic process of doing a peer review. My final comment is that the first one you do will take you a really long time. I think I spent a day on that first peer review I ever did. This is why journal editors love young peer reviewers, because you’re going to feel this enormous responsibility on your first review. And you’re going to be extremely thorough and careful. Just know that you will get faster as you do more of these. I’ve reviewed hundreds of papers in my career, and now I can do a good review, often in less than an hour, if the paper isn’t too technical. My final parting thought on this is just to review unto others as you would want to be reviewed. Be kind, be positive. When I review quickly, I tend to be curt just out of efficiency, but I try to remember to take the time at the end, and edit my review to make sure the tone is appropriate and positive.