How to review an original scientific article

[00:01:26] Dr. Tony Lang:
Good to be here.
 

[00:01:28] Dr. Alana Kirby:
I'd like to start off kind of at the beginning. As a young member, how do I know when I am an expert enough to be involved in the review of an original scientific article?
 

[00:01:39] Dr. Tony Lang:
As a young member, certainly even at a training level, hopefully, you will have the opportunity be involved in the peer review process, with your mentors guiding you through that process. The more we get exposed to the medical literature, the better. And you can't exist in this field without being intensively involved in reading the literature — and reading that literature very critically. That kind of experience allows one to, in their writing, emulate what you're reading. And in the reviewing, try to see what succeeded through the peer review. This gives you a better impression of what you should be expecting in reviewing the papers that you're seeing.
 

[00:02:24] Dr. Alana Kirby:
That’s very helpful you want to build for what good science is in order to be able to provide a good review and assess whether this is a a contribution to the scientific literature that is worthy of publishing.
 

[00:02:36] Dr. Tony Lang:
Very much so.
 

[00:02:38] Dr. Alana Kirby:
When an editor selects a reviewer, what is it that you are expecting that that reviewer will provide you that will help you to make a decision about what to do with the paper?
 

[00:02:48] Dr. Tony Lang:
So I think there are a number of qualities of good reviewers that an editor needs to be aware of. Firstly, the expertise, obviously. You need somebody that can read the paper, put it into perspective with a very solid knowledge of the literature and the background of the field, and then can be very careful and critical of the content of the paper.

You don't want somebody that reviews things in a very cursory fashion. We're all very busy. And some people are not as careful as they should be because they're so busy. And so you want somebody who can really very critically review it and carefully review it.

Also you don't want an abusive reviewer. As an editor I would occasionally come across really outstanding scientists who you could not actually share the review that they wrote with the author. So you want somebody that can be constructive in their criticism, but somebody who doesn't shy away from being very critical. You don't want somebody writing an apologetic review that will try to find all the strengths and ignore the weaknesses, or someone who will justify the weaknesses and give the author an out. If there are fatal flaws to the paper, you want a reviewer who will nail those fatal flaws right away and get on with things.

Be honest with the authors, tell them that there are significant problems that can't be rectified, and give the editor an appropriate review that allows them to, unfortunately, reject the paper, give constructive reasons for that rejection, and then move on.
 

[00:04:36] Dr. Alana Kirby:
Sometimes with original scientific articles, reviewers may feel that the paper may be strengthened by an additional analysis or an additional set of experiments. What should the threshold be for requesting that additional labor from the authors?
 

[00:04:52] Dr. Tony Lang:
That's a really good point. I think that additional analyses that are within reason that can be justified knowing, for example, the database that the authors are working with. So you're not asking them to provide data that they don't have. If it means going back to a large population to get more data, that's often unrealistic and unreasonable as well.

It's not unreasonable to request further analysis of existing data or obtaining, within reason, additional data. If you know that they can go back and get something very easily and quickly. But ... You're not doing the experiment. You're not doing the study. You're not rewriting the paper. You're dealing with what they presented you.

If you can make it better — and really good peer reviewers always make papers better. I'm always impressed by how much time and effort a good reviewer can provide that then makes the final product something really more valuable than it was to begin with.

But it's not up to the reviewer to rewrite it completely, or to redo experiments.
 

[00:05:59] Dr. Alana Kirby:
Can you comment on what you expect from a reviewer in terms of analyzing the statistics, and the rigor of the experiments described in the paper?
 

[00:06:11] Dr. Tony Lang:
Well I think you want an honest appraisal from the reviewer as to whether they feel capable of doing that. And it may not be possible for the editor to know the statistical savvy of every reviewer that they request to, to work with them.

And so I think it is reasonable to ask the reviewer what they think of the statistics. But I think that a good reviewer will admit where they lacked the knowledge base. And I always do this, I share in the above- line comments to the editor. There is a place where you make comments to the editor to a place where you make comments for the author. And in the part to the editor, I will always say, "This paper has some very complex analysis that I don't feel comfortable fully reviewing. And I really think you need a statistical reviewer with this kind of level of expertise."
 

[00:07:07] Dr. Alana Kirby:
I think that's an important point make for the listeners of our podcast. As someone who's relatively junior, I would certainly feel the pressure, that I would feel like I needed to be able to do everything. But it's actually more to the editor to be upfront about what your skills and knowledge are than to try and teach yourself something that you don’t know and not give an accurate review.
 

[00:07:29] Dr. Tony Lang:
Very much so, and you don't want to fake it and then have an editorial decision based on you faking. Second, don't forget, in most cases, unless it's an absolute rejection, the authors are allowed to come back to the editor, responding to the reviews. And if they responded and point out major flaws in your review, and some of the statements you've made in your review, the editor sees that. And I think for a junior reviewer that influences the editor's opinion of the reviewer, and could have an influence on subsequent interactions and things.
 

[00:08:08] Dr. Alana Kirby:
We've talked so far about major, big picture summaries of the impact of the paper and the quality of the experiment. How useful is it to get into the more detailed nuances of a paper? So specifically, let's talk about suggestions about the figures or the tables. Is there a role for that?
 

[00:08:31] Dr. Tony Lang:
Very much so, yes. The reviewer should be looking at the paper from the eyes of the reader as well. If you realize that tables could be improved, the text could be shortened and more easily comprehended by the reader, if the figures are sloppy or you realize that the Y axis should be re designated.... If you had a question, and it took searching out in the text, you should share that with the author to say, "okay, you can fix this up for the reader so that they understand it more readily."

I always find that figures, for example, should stand alone. A reader should be able to look at a figure and really understand that figure very easily without scrounging through the text to figure out what the authors wanted you to get out of that figure. There's a lot of wasted figures that are put into papers as well.

So you're not only reviewing the paper to make sure the science is good and make sure it hits a certain bar for quality, but you also want to enhance the readability of the paper so that the readers get something out of reading that journal. You're working for the journal; You're working on behalf of the journal, and you want to enhance the quality of what gets into that journal.
 

[00:09:47] Dr. Alana Kirby:
Another aspect of the presentation of the paper is the language that is used. My understanding is that the role of the reviewer in commenting on grammatical errors or proofreading mistakes, is it is less of a priority. What's your thought on that?
 

[00:10:08] Dr. Tony Lang:
It's less of a priority, but if the language and the grammar serves to confuse the reader, it is up to the reviewer to state that. The reviewer can say, "This is a great paper, but to enhance its readability and avoid confusion, this needs to be reviewed and rewritten." If all you've done is struggle to understand the paper, and there's all sorts of potential for confusion based on the grammatical problems , I think it's very reasonable for the reviewer to say, "This is so confusing that I just don't think it's readable at this point. And there are too many grammatical errors for me to accept it. And suggest rejection on that basis."

Where the line between those two exists is sometimes very difficult to establish, but I think a good reviewer can get through the grammar and realize that there's something really important here that we need to salvage, or the opposite.
 

[00:11:05] Dr. Alana Kirby:
For some papers that major revisions are recommended, sometimes the article will go back in its revised form to the reviewers. Do you have any tips for deciding whether the response is sufficient?
 

[00:11:21] Dr. Tony Lang:
There are two ways of that happening. I'll deal with a second way first: And that is you didn't review the paper first, and the editor has sent it to you now. I think it's really important for you to clarify your role with respect to the editor in that situation. Because then you've got a new reviewer, bringing a whole new set of eyes to the paper. And so it's like another review, and that's not fair to the authors. In this situation, the editor should have said, "We have lots of problems with this paper. I'd like you to look at it with a fresh set of eyes because of the following issues." And then you really do come in with a designated responsibility, given that role by the editor. So that's the second option.

The first option is, you were a reviewer in the first case. Now it's come back to you and you really have to review it in the light of your previous review. You don't review it brand new and say, "Oh, I didn't think of this the last time. Now I'm going to bring a whole new..."

Now sometimes, you did miss something. and it's not unreasonable. And what I do in that situation is I apologize to the author. I think it's very reasonable to give a dialogue between you as a reviewer and an author. The whole point of this is to bring science to bear and to advance our field. So it's not unreasonable as a reviewer to say, " I now realize that I missed this point the last time. And could you address it?"

But for the most part, you're reviewing it based on your previous commentary and concerns, and your job is to see how well they've addressed those concerns and comments.

It's also helpful to read the other reviews. You learn something from that. You sometimes see, "Oh my Goodness I missed that point completely, they're right!" Or you realize that they actually commented on something that you disagree with.

But your job really is to review their responses to your comments and concerns and see how well they've done in doing that. And if you still feel that they've come short, your job is to say that.
 

[00:13:31] Dr. Alana Kirby:
Is there any preferred format that you would like to receive a review in?
 

[00:13:36] Dr. Tony Lang:
There are many recommendations of how to review an article. I don't think any of them are absolutely right. Or, or necessary to the letter. An overall summary of the opinion of the reviewer. How important do they think this paper is? And often somebody will say right up front, this has major flaws that I don't think can be adequately addressed.

It's very reasonable to outline the concerns in the order in which they arise. I find if you're a very, very busy person and you're reviewing a lot of papers, it's sometimes quite difficult not to review it in the order of the concerns you've seen in reading the paper. But some sort of summary commentary at the beginning is very reasonable.

Don't overload the review with the minor comments, the minor corrections, the grammatical concerns at the beginning of your review. Note the grammatical and spelling and typo errors ... it's worthwhile putting a section right at the end of your review saying minor point. And then you can give the list of the minor points that help the writer fix things up.
 

[00:14:49] Dr. Alana Kirby:
Very good. Well, this has been very helpful. Thank you for sharing your experience with us.
 

[00:14:54] Dr. Tony Lang:
Nice talking to you.
 

[00:14:56] Dr. Alana Kirby:
Thanks, Dr. Lang.

That completes episode five of the peer review podcast. In this episode, Dr. Lang shared his recommendations for reviewing an original scientific manuscript. Next week, we will discuss how to review a review article. Thank you so much for listening.