Be Reasonable

I decided to take last week off from blogging and enjoy some of the holiday weekend, in part because I spent the rest of the weekend dealing with an excessive late-stage revision where one of the reviewers, as my coauthor put it, is a “dedicated foe” of our paper. Whereas the other reviewers had signed off on the paper or only had a minor comment, this reviewer wrote a five-page single-spaced review trying desperately to undermine our manuscript. It took a fair bit of self-control, but we responded patiently and respectfully to all the points raised. We all run into reviewers like this occasionally; however, we should all also work hard to not be that reviewer. Thus, this week’s blog post focuses on how to be a reasonable reviewer.

Being reasonable doesn’t mean being easy, or having low standards. You can be rigorous and hold authors to high standards while also being reasonable, recognizing where the paper’s going, and helping to make it as strong as possible rather than trying to torpedo it at every turn. I’ve reviewed papers where I recommended rejection at the initial submission and first revision stages, but the other reviewers and editor saw things differently, and the paper continued to progress. Rather than dig in my heels and try to scorch the paper, I instead acknowledged that the paper was on its way to publication, and while still noting my concerns in my private comments to the editor, also noted that reasonable people can differ, and that the editor was the decision maker. I then focused my attention in the review on the few key things I thought the authors really needed to address, given the existing positioning and framing of their study.

So why do some reviewers become unreasonable? It usually revolves around issues that are very subjective, such as whether the study makes a sufficient “contribution,” and “taste” issues with respect to topics, or the interplay of theory and context. What constitutes a “sufficient” theoretical contribution is inherently subjective. It’s like pornography: hard to define, but you know it when you see it. Some folks naturally find certain topics more interesting. But just because you find a topic boring doesn’t mean that others do. Taste issues also often revolve around whether and how the context is employed in theorizing. I personally hate “context-free” theorizing that provides no examples and is unrooted in a phenomenon because it is too arid and abstract, making the paper excruciatingly boring to read. I’d much rather see examples and/or the context woven into the theory and hypotheses development. However, there are others who equate this approach with theorizing from anecdotes, or believe integrating the context into the theorizing limits its generalizability. Since I prefer the former, my unreasonable reviewers typically fall into the latter camp.

Reviewer stridency can also occur because of different or contested methodological perspectives and approaches that take on quasi-religious meaning for them. Endogeneity, whether because of selection bias, omitted variable concerns, or some other reason, is a current favorite for many. They see endogeneity under every rock and around every corner, and are very alarmist about it, arguing it imperils interpreting any results. It’s also essentially impossible to publish a survey now where all the data were collected at the same time because of concerns about common method variance, whether real or imagined. Effect sizes also seem to garner a great deal of attention. And I’ve recently found that if you have panel data, many reviewers automatically want you to employ models that look at within-actor changes over time, even if your theoretical question focuses on between-actor differences, making such within-actor analyses inappropriate.

Given these traps, how do you avoid becoming the unreasonable reviewer? If you find yourself objecting to a manuscript on these grounds, even if others seem more satisfied, it’s important to step back, acknowledge the issue (or your zealotry), and reconsider your position. You can express your preference in your review, but at the same time recognize that opinions and tastes differ. On the methods side, it’s also important to acknowledge that just because something can be a problem, that doesn’t automatically mean it is a problem. If you have specific suggestions that will at least help partially alleviate your concerns and fit with the direction the paper is going, make them. For example, it may just mean adding some qualification statements to highlight the issue, or making sure the theory precedes the example when developing hypotheses. Make suggestions that are feasible and doable for the authors, though. If it’s methodological concern, you can suggest specific tests they can use to rule out problems. Or, at a minimum, suggest they acknowledge the issue as a potential limitation and discuss its implications for interpreting their results, either in the results or the discussion section.

You aren’t going to love every paper you review that makes it to publication, and as I said, reasonable people can disagree. However, if you are clearly in the minority with respect to the paper and can see it’s likely heading to publication, rather than trying to torpedo it, take a moment to consider your own biases, how you’d like to be treated if it were your paper, and try to offer recommendations that at least reduce your concerns, even if they can’t or won’t eliminate them.