Revising after Rejections

Like everyone, some of my papers get rejected. Indeed, in recent weeks I’ve a had a couple of papers rejected, which means I’ve been thinking about how to revise them for the next journal. After the obligatory cooling off period needed to read the reviews somewhat objectively, the task thus becomes figuring out which comments to follow up on, and which are likely idiosyncratic and safer to ignore. This is something of a guessing game, and I’ve guessed wrong plenty of times. If you make a big deal out of what turns out to be an idiosyncratic point, you run the risk of making an issue out of something the next set of reviewers would have otherwise ignored. If, on the other hand, you keep ignoring issues that need addressing, you end up burning through potential journal outlets and piling up the rejection letters. So, how do you decide what to attend to, and what to ignore? I don’t have a crystal ball, but I do have some rules of thumb I try to follow that work out more often than not when I can get myself to heed them.

First, I look for patterns. The best place to find them is often the editor’s decision letter. Assuming your editor didn’t just write a one paragraph decision letter with no meaningful comments or suggestions, they have often done much of the work for you.  Part of the editor’s job in writing rejection letters is to summarize the main issues leading to their decision. Thus, they will identify the themes in reviews that suggest a significant underlying problem. The reviewers may focus on different aspects of the issue, or use different labels to describe it, but the underlying theme and problem usually reveal themselves. However, I also make sure I read the reviews carefully myself, because I may see the issue differently than the editor does, or I may identify a theme the editor doesn’t. But, if multiple reviewers are keying in on the same issue, then it’s probably one you need to address. For example, questions about endogeneity were raised by multiple reviewers in one of my recent rejections. Although we had taken a lot of steps to address most of the sources of endogeneity, there was one source we hadn’t addressed, and they homed in on it. Although we couldn’t address it in our modeling, we were able to run a supplemental analysis that demonstrated endogeneity was unlikely to affect our results. When we sent the paper to the next journal, endogeneity issues didn’t come up.

However, not all reviewers are equally sharp; thus patterns won’t necessarily emerge within a single set of reviews. I therefore also go back to prior reviews and see if the same issue has come up more than once, even if only one reviewer in a given set of reviews raises the issue. I always get at least a couple friendly reviews from colleagues before I send a paper out for the first time, so I will compare my friendly reviews to the journal reviews I receive, and if the paper’s been rejected at more than one journal, I’ll look at prior rejection letters and reviews to see if I can identify patterns. Friendly reviewers often flag issues that I don’t want to deal with, or that I think are idiosyncratic. It’s only after the paper’s been rejected at a journal and I go back to look at the friendlies that I realize the issue was raised previously, but I chose to disregard it. Having multiple rounds of reviews to triangulate across can help surface latent problems.

At the same time, multiple reviewers sometimes identify an issue in a particular round of reviews that a different set of reviewers would ignore. That’s the random aspect of reviewing. So, you may identify a “pattern” in one set of reviews, and by responding to it create challenges for yourself with the next set of reviewers. At best, they’ll ask, “why are you talking about this?” and suggest you drop that section. However, at worst, you will make a non-issue a major issue that leads to another rejection. This trap is harder to avoid. Sometimes you just have to run an experiment and see, if I don’t address this issue, or just address it in the limitations, is that okay? It may cost you another rejection, but then you’ll know. I will say that if you identify what seems like a major theme, but it’s going to be difficult to address, or time consuming to collect additional data to address it, you are likely better served biting the bullet and dealing with it sooner rather than later. I’ve been down that road more than once, and the paper was consistently rejected until I collected the data or conducted the analysis.

There’s no foolproof way to avoid the pitfalls associated with revising your paper based on the last set of reviews for the next set of reviewers. But I hope this at least gives you some ideas for what you can probably ignore, and what you better just go ahead and do.