Peer review: Past, present and future

[00:02:02] Dr. Michael Okun:
My pleasure to be here, Daniel.
 

[00:02:04] Daniel DiLuca:
Thank you very much. So today we're going to focus on the past present and future of the peer review process. And I'm also with my colleague, Dr. Prasad, who also helped conducting this interview.
 

[00:02:16] Shweta Prasad:
Thank you, Daniel, and thank you professor Okun for agreeing to be part of this with us. So we know that everything evolves. One of the fundamental concepts in biology is evolution. In the same way, we need to understand how peer review has evolved over these years.

So Dr. Okun, from the first time you received your very first set of comments for a paper you wrote to the first time you gave out a set of comments as a peer reviewer, and perhaps your roles as an editor, how do you think that peer review has evolved over these years?
 

[00:02:49] Dr. Michael Okun:
It's really informative, always, to look back at the history of things. And of course my first degree was in history and my first love, in fact. If there were good jobs in history, I might be a historian instead of being a neurologist. When I was trained in neurology, and came out of my residency and did my fellowship with the man named Mahlon DeLong at Emory university who was influential in my career, Mahlon saw something in me that I didn't see in myself in terms of thinking about research. I was thinking more about being an educator and more of a historian type. I like to write and educate and things like that. And so, learning how to write technically I think is the first step.

Right? And, and so people like me that like poetry and short story and prose and things like that, you gotta take all the adjectives out. Right. This is very technical. It's very specific. And so in a lot of ways, I won't say it's easier, but it's, it's a completely different experience.

 It's a really important, fundamental concept that we're able to submit things in writing that will live beyond our generations because I don't wanna shock both of you, but all of us are mortal. We're not gonna live forever. And one of the best things that we can do is we can leave behind some crumbs for people so they know what happened. And then allow us to evolve forward.

So in the early days, when we would submit a paper, it was done. I would write it out on a piece of paper, on what people would call a legal pad.

There used to be these things called legal pads that some people may know and still use. And then you would type it in to the computer into an early word processor or a typewriter. We did our applications on typewriters, at the beginning . And then when we start to submit these things, you would have to go to a copy machine. Some people don't know what a Xerox machine is, a copy machine. I met my wife at the copy machine, by the way. So it's a very important place. For a social area, you talk about the copy machine and the coffee machine, right, where you get to meet people. And you had to make five copies of your paper and you would send it off, and it would go to a journal. In the regular postal, snail mail, not email. Cause it's before email. And they would take your paper and they would send it out to three to five peer reviewers and you didn't get a chance to say no, right, cause you would just get it like as a peer review in the mail, and either you respond or you don't.

And so a number of people respond. And a lot of times when we got our comments back, not only would we get the comments from the reviewers, but really good editors and really thoughtful editors would in pencil, actually mark your paper. There was no track changes. And so you would get your paper marked.

And one of the most well known people in neurology that would do that was a man named Berch Griggs who's still up at the University of Rochester, former president of the American Academy of Neurology, a neuromuscular expert. And he would write in your paper. He would sometimes write Latin phrases on the top of your paper. So if you didn't know Latin, you'd have to look 'em up like pithy maxim and other little things, short quippy things that would give you a sense of what they thought. And so some of that is lost on the modern peer review system.

And then you would have to go back and forth in snail mail. So it would take a lot longer, but in some ways it was a much more human experience. Then as things evolved from the pen and paper and typewriter era into the computer era, this has allowed us to be able to be speedier. So the speed has increased. The personability of the peer review process has decreased. It's become much more impersonal, and people say no a lot more than they used to.

The number of journals went from a handful, you know, two or three journals that you might submit to, to literally hundreds of journals that are out there. Some are predatory, some are not. And so it's a different world now with the internet and with the ability to submit so many different places and to get quick peer reviews. Sometimes peer reviewers don't even respond to the email. It was different when you got a package in the mail, you almost felt compelled to help by looking at somebody's paper. And so this has changed.

It's allowed, in some ways, more opportunity for things to make it into the medical literature, but also it's changed the quality of what we see in the peer review process. And more things tend to get through and get out. Now, there's two ways to look at this: some people say, "Oh, this is completely impure. This is a bad thing." And then the other way to look at this is no, maybe it's better that these things get out, because science is all about replication. And if it's not replicable, then over time the historians and us are gonna prove it right or wrong.

 I think people misunderstand the peer review process. I think it's a good process. I think the more comments that you get, the more challenges that you get. If you want to get to higher impact journals, if you want to get to higher level science, you've gotta ask the questions and prove it. And I think a lot of people don't really appreciate how important it is to prove it and how important it is to replicate it, how important it is to ask a series of questions. You talked about my lab , about where in the brain tics come from and proving it. Each step of proving your point is important. Can you separate a tic from voluntary movement? You've gotta be able to do these fundamental things and, and build a story.

So I think things have changed over the years. Some for the better, some for the worse. I wish it was a more personalized process. And I wish the people that I've handled their papers, I wish that they understand that I've seen a lot of good science that didn't maybe get into one journal or another, but that doesn't mean that I have anything less than pure respect for them for submitting it to peer review and for thinking it through. We passed on a lot of really great papers over the years, and it's nothing personal.
 

[00:08:49] Shweta Prasad:
Well, thank you so much for that brief yet incredibly lucid history of your experience with peer review. I'd like to ask my co-chair, Dr. Kirby, to continue this discussion.
 

[00:08:58] Alana Kirby:
Thank you for talking with us about the perspective of yourself as an author and as a reviewer. I'm wondering from an editor's perspective, what are the challenges related to the peer review process in its current format?
 

[00:09:12] Dr. Michael Okun:
Yeah, I think one of the biggest challenges is that one, people are busy. And making sure that you get peer reviewers who can give you an honest opinion within a reasonable amount of time and also can represent the science in a fair way. And so I think that that's still a, a very challenging aspect of the peer review process.

I also think that some journals overweight when somebody says, "This is a completely horrible paper, and it should be done." And then they just scrub the paper. I actually think you need to look at their reviews and try to understand where they're coming from, because sometimes it's, it's actually the opposite. Maybe the people are on to something and you want to ask for a revision or a replication cohort. There was a recent paper that I was involved with where there was some uncertainty whether the finding was, was true or not. And asking for a replication cohort , they came up with a replication cohort. Sure enough, they were right. And it could have easily been rejected out of peer review. It ends up being a very important finding.

 From an editor's perspective, there's a challenge to the volume of papers that are coming through to making sure that you give people their due. And then that people understand that it's not personal. I can't tell you how many papers that I've been involved with with that got rejected where I think, gosh, that's a really good paper and that's gonna get in somewhere. And they might even appeal their rejection. And it's heartbreaking, it just didn't didn't make it for whatever reason, but I really like the paper.

As an editorial team, In general neurology we meet every Friday for several hours. It's a really fun call. And we go through all the papers and I learned so much from listening to all the different editors talk about it. Not every journal does that now. Some people, it's all digital and it's all by email, but we still do the old fashioned get together and actually talk through the papers and it's informative.

And I can't tell you how many times one paper gets saved or not, or even as a handling editor, I don't see something. And there's some beauty in that paper that somebody else sees or there's some point that didn't get emphasized or maybe the authors didn't even realize they were on to something — but the reviewers or the editors picked up on it. And so I think the challenges more of the personalized why. Certainly somebody wrote that paper for a reason. They spent a whole lot of their time. And they thought it was worth it for them to invest a whole bunch of time. And they're probably not getting a whole bunch of money or fame or anything from doing it, but they got super excited about it. And can we, in the current peer review format that is sometimes sterile, can we actually step back and say, "They thought they had something here when they got super excited to send us this paper." Can we step back and try to understand what their excitement was?

And even if we reject this paper, it's fun to watch to see what happens to papers. And sometimes watching people go back and redo the science and come back again, or you see it in a different journal. And you're like, gosh, I'm really glad that they did that. Cause I thought that was good or it just completely disappears. And then you look back and you realize it didn't replicate, which is fine in science. 95% of what we do should be negative. And we learn from the negative to push the positive and, and people misunderstand that.

So that's kind of how I look at the world, big picture, now, after a number of years, a little more wisdom and more compassion. And just to let people know, you know, feel really bad for people that take the punch of a bad review and either leave the field or it just, it takes the steam out of 'em. I really wish it was the other way around, that people would take the challenge of, let's tighten this up and keep pushing along. That's part of how science is.

We don't share that enough with people that that's part of the process and that's what's gonna make you better and what's gonna make the papers better. And some people leave the field instead of embracing it, as, you know, they pointed out a whole bunch of things I can go back and make this better and improve the field. So, I think we got some work to do in terms of keeping good scientists in science, and the peer review process is important, but a little bit misunderstood.
 

[00:13:30] Alana Kirby:
Okay, thank you for that. During this podcast series, we've heard a number of times from your colleagues about the importance ofthe editorial role in analyzing the analysis to come up with the right answer for accepting, rejecting, or asking for revisions. But I think putting it into this context of how things used to be, and hopefully how things may be, is really helpful for us. I think we haven't heard that before.
 

[00:13:56] Daniel DiLuca:
How do you see the future of the peer review process? We have a lot of journals, we have a lot of articles, not a lot of people that have enough time to do the reviews. Do you see any major changes in the next coming years?

I know it's kind of a philosophical question. So it's mostly your personal perspective and thoughts about the future of the peer review process in general.
 

[00:14:17] Dr. Michael Okun:
Yeah, well we talked a little bit about history. So if you think of history of medicine, just remember it it's only in the last a hundred years or so that people started even trusting doctors. Right. So if you're talking about medical things, medical research, doctors, healthcare professionals, submitting into peer review, I think you have to remember that if you were writing the Bible, you would still be in Genesis somewhere. The first couple pages of the, of the first chapter of what's happening in peer review. You all may think, oh, we're completely evolved in this. We're not.

I mean, my gosh, look at my own career over the last couple of decades going from paper and pencil to these systems and a handful of journals to journals you can't keep track of, to trying to figure out if the journal you're reading is a real journal or some version of entertainment, or a profit motive. These things are gonna evolve on their own and the evolution's gonna take time.

And I think that there will be more questions asked of the future generations. For example, people compete to be first. Is being first in line as important as being quality? We all hope that the evolution is gonna be that we want the quality work rather than first in line. You're already seeing this sort of foray into preprints. And most people, including myself, aren't even sure exactly what all this means. It's all about catalysis of how quickly you can get information out to people, right? So, you have something to say, How do you get it out there speedy enough, but still subjected to the peer review process. I think in the future, we're gonna have to figure that out, what does that mean? And then also figure out how to collate our information.

And then the next challenge that's evolved, and this is an internet age challenge, is, there's evolving disciplines and information versus misinformation versus truth, and what is truth. And so philosophers are gonna love this. Ethicists are gonna love this. They're gonna be busy for the next couple hundred years, sorting those issues out. What is truth and where does truth lie? And there's so much nuance in science, particularly in human science, and humans are so different, and one disease isn't necessarily one disease, even if it comes with a singular name. And so I think these are gonna be major challenges.

I think there's still questions that are unresolved about anonymizing all papers coming in. I think you will hear that from purists. And there's reasons on both sides of that equation. I'm not sure that that one settles out. I don't think that argument will probably ever come into focus where everybody accepts one or the other, but you might have a group almost like a political faction, that a large segment of journals will go to that and consider that more pure. And you might begin to see two or three different methods go at it, almost like political parties — factions of journals that believe that it should be done in one way or the other.

And so it's hard to predict where things are gonna go, but I think the hope is that more science comes out, we replicate, we ask better questions. We use the peer review process to drive the fuel forward and we catalyze their reaction, whether it's preprints or not, so that people can get information and use that information quickly, and then we have some way to deal with misinformation. And so those are kinds of the things that are on my mind when I think through that question . And that's a long-winded way of saying I don't have the answer.
 

[00:17:53] Shweta Prasad:
Well, I think no matter what really happens to the peer review process, one of the most important aspects of it would definitely be a good peer reviewer. And considering where we have this whole bunch of the young generation in front of us, who's going to be part of this future, do you have one peer review tip you would really like to share with the young trainees?
 

[00:18:13] Dr. Michael Okun:
I mentioned that our group gets together very human, and we talk about those papers. And then you see each person has a style they develop and something they add to the process. And one of the things that maybe I'm known for is the I'm the title changer and the abstract changer, the, the point the punch line. And so I think that one of the great tragedies is when you write something terrific, and you give it a bad title, and you miss the punchline.

And so the tip that I would give people is really, look at it and have people look at it because you get so caught in the details when you're writing something, and say, big picture: what is new? What am I adding here? What do I really wanna say? If I close my eyes and I'm honest with myself, and I realize that 80% of people are only gonna read the abstract and the title. If you wanna lure those 20% into reading your paper, it better be a pretty good abstract entitle that says this isn't just a me too thing. What was it that you saw, and why is this important? And so spending more time on those things I think is super important.

And then I always say a great paper has a great, what I call, "money figure." Okay. It's like Dilbert. It's easy to look at, it's like a comic strip. You look at it, it drives a point home. You don't need a lot of explanation, but it really encapsulates that study and what you're doing. And even over the years of my own writings, trying to encapsulate for the reader that you're trying to share with them. You're trying to make a point. And so having at least one really good figure, that makes a ,point that's easy to read that doesn't take like a, quote, rocket scientist or a brain scientist or whatever to interpret. That you can give it to your 12 year old, if you have a 12 year old at home and she can go, "Oh yeah, dad, I see that. I understand what that is." Those are great figures and those will help to guide the field. And so those would be the tips that I would give to young trainees along the way.

Don't get so caught in the weeds and the details that you missed the punch.
 

[00:20:18] Alana Kirby:
Well, thank you so much for those insightful comments. I'm gonna think a lot about your vision of the future of warring journals, selling alternate versions of scientific truth. It's been such a pleasure to have you on this episode of the MDS podcast. We've learned a lot talking about the past present and future of peer review, and it's really a fitting final episode to our series.

Just to recap, we've had a 10 episode series looking at tips for young members of the MDS as they embark on the important role of reviewers and take their part in the peer review process. So on behalf of Dr. DiLuca and Dr. Prasad, I'd like to thank you and all of the rest of our panelists over the course of this series and to our listeners for joining us all the way to the end.

Thank you all.
 

[00:21:15] Dr. Michael Okun:
Thank you.