Marc Bellamare gives a thorough overview of what to do, and what not to do, as a manuscript referee in the social sciences. Some of these things should be obvious, but as someone who has received plenty of unhelpful yet strangely hostile negative reviews delivered after 11 months, I think that this advice is worth sharing and discussing. I will assign them as reading for my grad Comparative Methods course. I will also assign Chris Blattman on the same topic. One especially welcome pointer is that every review that you write should start with a summary of the manuscript, its argument, and its findings or conclusions. That not only helps the author(s) to know if his or her manuscript makes sense, it also forces you to review the manuscript that’s in front of you, not the one that you wish were in front of you.
Just to brag, my own personal record for completing a manuscript review is 2 hours. On the advice of Preston McAfee, I’m trying to move to a “clear the inbox” model for manuscript refereeing, ideally completing the review in 48 hours but never taking more than two weeks unless I’m traveling and so jetlagged that I cannot concentrate. I used to try to review articles on long flights, but I’ve found that planes are a better place to read than they are to write.
Unaddressed in Marc’s post and Chris’s post, though, is one issue that I bet flummoxes all young referees: the difference between reviewing a manuscript for a “top” or “general interest” journal versus a “subfield” journal. I’ve often found myself reviewing the same manuscript twice or even three times, as it gets dinged at the top journals and works its way down the food chain. There are three issues at play. (A) Are the standards lower for subfield journals than for top journals? (B) Does the article have to be more “important” to make it into top journals, and who decides? And (C) Are rigor and importance substitutes? The following are my positions, although I’m curious to hear from anyone who disagrees.
On point (A), the answer should be yes, although I know that others will disagree strongly with this. To me, an article that is has a glaring identification issue should not make it into a top general interest journal. Nor should a theoretical paper whose predictions are either trivial or impossible to translate into observable politics, either in a case study or a quantitative analysis. Nor should a case study that has no source of inferential leverage. But when it comes to subfield journals, I am increasingly willing to be accommodating to papers that admit various shortcomings, conditional on the authors being willing to write frankly about how these shortcomings might condition our interpretations or conclusions.
On point (B), the answer should be no, but not for reasons that you might think. That is, all manuscripts, both in the most prominent journals and in the more focused subfield journals, should demonstrate that they are important to some substantive political problem or to some debate within the discipline. I don’t hold general interest journal submissions to a higher standard. It should be up to the reviewers to make the case to the editor about why a manuscript is or is not important, and it should be up to the editor to decide if the manuscript meets the bar.
On point (C), no, rigor and importance should not be considered substitutes. I understand that they often considered to be, especially by critics of what is believed to be our discipline’s hegemonic core, but the response is not to abandon rigor. Rather, it is to do better political science, and to embrace standards for rigor that are more inclusive than “ivreg2 y x1 (x2=z)” and nothing else.
Tom Pepinsky 