How to be a Good Referee

Marc Bellamare gives a thorough overview of what to do, and what not to do, as a manuscript referee in the social sciences. Some of these things should be obvious, but as someone who has received plenty of unhelpful yet strangely hostile negative reviews delivered after 11 months, I think that this advice is worth sharing and discussing. I will assign them as reading for my grad Comparative Methods course. I will also assign Chris Blattman on the same topic. One especially welcome pointer is that every review that you write should start with a summary of the manuscript, its argument, and its findings or conclusions. That not only helps the author(s) to know if his or her manuscript makes sense, it also forces you to review the manuscript that’s in front of you, not the one that you wish were in front of you.

Just to brag, my own personal record for completing a manuscript review is 2 hours. On the advice of Preston McAfee, I’m trying to move to a “clear the inbox” model for manuscript refereeing, ideally completing the review in 48 hours but never taking more than two weeks unless I’m traveling and so jetlagged that I cannot concentrate. I used to try to review articles on long flights, but I’ve found that planes are a better place to read than they are to write.

Unaddressed in Marc’s post and Chris’s post, though, is one issue that I bet flummoxes all young referees: the difference between reviewing a manuscript for a “top” or “general interest” journal versus a “subfield” journal. I’ve often found myself reviewing the same manuscript twice or even three times, as it gets dinged at the top journals and works its way down the food chain. There are three issues at play. (A) Are the standards lower for subfield journals than for top journals? (B) Does the article have to be more “important” to make it into top journals, and who decides? And (C) Are rigor and importance substitutes? The following are my positions, although I’m curious to hear from anyone who disagrees.

On point (A), the answer should be yes, although I know that others will disagree strongly with this. To me, an article that is has a glaring identification issue should not make it into a top general interest journal. Nor should a theoretical paper whose predictions are either trivial or impossible to translate into observable politics, either in a case study or a quantitative analysis. Nor should a case study that has no source of inferential leverage. But when it comes to subfield journals, I am increasingly willing to be accommodating to papers that admit various shortcomings, conditional on the authors being willing to write frankly about how these shortcomings might condition our interpretations or conclusions.

On point (B), the answer should be no, but not for reasons that you might think. That is, all manuscripts, both in the most prominent journals and in the more focused subfield journals, should demonstrate that they are important to some substantive political problem or to some debate within the discipline. I don’t hold general interest journal submissions to a higher standard. It should be up to the reviewers to make the case to the editor about why a manuscript is or is not important, and it should be up to the editor to decide if the manuscript meets the bar.

On point (C), no, rigor and importance should not be considered substitutes. I understand that they often considered to be, especially by critics of what is believed to be our discipline’s hegemonic core, but the response is not to abandon rigor. Rather, it is to do better political science, and to embrace standards for rigor that are more inclusive than “ivreg2 y x1 (x2=z)” and nothing else.

Comments 2

  1. Matt January 24, 2012

    1) Two hours is good; my own record is about half a day. The model you propose is a good one: decide if you can do the review within the week; if not turn it down.

    2) I don’t think reviewers should get a second crack at the same article for another journal. I’ll do a re-review of an R&R, but I won’t review the same paper from a 2nd journal.

    3) The summary strikes me as very useful to the editor; I do that most of the time.

    4) How does this post square with your recent critique of the obsession with identification in the social sciences? In that post, if I recall, you suggested that there is likely to be, as a property of the world, a tradeoff between things that are big and important and the ease with which they can be studied. If so, then why should one come down on the side of the most “rigorous methods” used too study things that “only a political consultant could care about” instead on the side of a less ironclad empirical strategy used to study something really important?

  2. Tom January 24, 2012

    Thanks for reading, Matt. I think that the way I square this post with my earlier post on identification (see the last hyperlink in this post) is like this. First, I reject the claim that rigor and statistical identification are by definition the same thing. Second, I recall that nothing in my post says that we should be unconcerned with identification, only that we should also be concerned about other things too: the quest for identification should not delimit the extent of what we can study.

    I recognize the tension that you observe, though. There’s no simple answer, and I can’t pretend that it easy or feasible to have it both ways.

Comments are closed.