The Perilous Peer Review Process

My friend Nate Jensen has written a brutally honest and really important post about the peer review process. Every grad student and assistant professor should be required to read it. I leave it as an exercise for the reader to write down a timeline of when a paper ought to be submitted if you expect to have a good publication by a particular date.

A silly personal note: on one of Nate’s papers that endured multiple submissions, yours truly served as a reviewer at multiple journals. The manuscript certainly improved over time, but the whole process ended up being rather ridiculous. On the last review, I wrote “I have essentially run out of criticisms to make of this paper and it is time for it to be published.”

(Nate and at least one of his co-authors know that this was me—they figured it out on their own and we have talked about it since—so I am not betraying any confidences here. I knew the authors because we share work sometimes. How’s that for anonymous peer review?)

A not-so-silly personal note: I can tell similar stories. This paper was rejected four times. This paper was rejected at the journal that commissioned it, then desk-rejected at a different journal, before being accepted. This paper was desk-rejected at an Asian affairs journal, but no one informed me for a year. Which, as you can imagine, really grinds my gears. As much as my feelings have been hurt, I have never once complained to an editor, because I have never been treated unfairly (the last one was not the editor’s fault).

A final note, before returning to work on revising a paper that recently earned a fair rejection at a very good journal: I appreciate Nate’s honesty and his willingness to reveal something about how the process looks. As I tweeted yesterday,

But I do think that it raises some questions that are worth some reflection.

  1. What kind of model of science are we following when appearing in print/surviving peer review is such a capricious process? Especially if the reviews are about framing the contribution rather than the technical details of the analysis? Suggests to me that political science may aspire to be a science, but the work of political scientists is really much more like a craft. Of course, nothing special about political science here, but we ought to be clear about it.
  2. I wonder how many of Nate’s R+Rs and rejections were justified by a referee’s request for new specifications of an empirical model. We talk about “researcher degrees of freedom,” but in my experience “referee degrees of freedom” are just as problematic, especially if we go with the position that if the referee can conceive of specification in which a coefficient of interest does not cross an arbitrary line of statistical significance, then the results are not “robust.” One paper of mine—at the request of multiple rounds of reviewers—has been subjected to four times as many robustness tests than there are observations in the dataset.

Am I missing anything?

Comments 11

  1. cshendrix September 16, 2013

    Ah, you are missing the dreaded “paper is technically sound, but not of sufficient general interest to merit publication in such a widely read journal.” This is one of my personal faves because the implicit assumption is that the journal editor knows what the marketplace for ideas wants, and can tell what will have a broad impact and what won’t. Judging by the number of APSR and AJPS articles that have been cited fewer than ten times after five years, this assumption is worth questioning.

    • tompepinsky September 16, 2013

      Ugh, I’ve never experienced that one. Instead, my rejections tend to be the same old “this just isn’t good enough” type.

  2. Raul Pacheco-Vega, PhD September 16, 2013

    Tom, this is an incredibly useful post and I have to thank you for posting it.

    My horror story: I had a piece published in 2012 that I first submitted in 2004. My thinking around the issue had evolved so much I almost didn’t want it to be published.

    That’s one of the reasons why I abide by the following rules:
    – I will NOT peer review a manuscript outside of my expertise.
    – I will NOT peer review a manuscript when I know I’m strapped for time
    – I will keep my peer reviews to a maximum of one month, preferably at most 2 weeks
    – I will do thorough reviews that are actually useful to the reader, kind and constructive.

    I’ve blogged about this on my own site (“the ethics of peer reviewing”) but I can’t recall the actual link and I’m trying to finish (oh, the irony) a peer review to be submitted today!

    • tompepinsky September 16, 2013

      Thanks for reading, Raul. Those rules sound like good ones to live by, and I am trying to live by them myself! Do send along the link to your own post when you come across it, I’d be happy to share.

    • anonymous September 17, 2013

      “I will NOT peer review a manuscript outside of my expertise.”

      I think this is a bad rule, especially if asked to review for a general poli sci journal. I think these journals should always include one reviewer outside of the subject area. It also helps to prevent the “cartel” model of peer review and publishing!

  3. Pingback: Rejection | lucylynnee's Blog

  4. Rich Maass September 17, 2013

    Hi Tom, First time reader, first time commenter (I just discovered your site), but I had found Nate’s post through another source yesterday and agree that it provides an admirably candid window into the real world of peer-reviewed publishing, which we’ve all experienced but is rarely spoken of publicly. I am sympathetic to your conclusions regarding what this means for the state of our field if the results of this process are held to be among the highest standards of achievement, but I am curious about your commentary that this makes political science a “craft” rather than a “science.” In my mind, the craft/science distinction has more to do with the research design standards to which we hold ourselves (and to which our peer reviewers hold us) rather than the unpredictability of the publication process and the effort (or lack thereof) some people put into reviewing manuscripts. By deriving theories to explain the world around us, testing those theories using empirical evidence, and opening those theories and tests to the public realm for potential critiques, I would say that we are employing the scientific method. Happy to talk about this (and publication experiences) more in person! Keep up the good work!

    • tompepinsky September 17, 2013

      Thanks for reading and commenting, Rich.

      So, my comment about craft versus science is about how we form judgments about the quality of our output. I do think that we employ the scientific method in our work, but our ability to judge that process is somehow imperfect, and there is something else—aesthetics, perhaps?—explaining referee and editorial decisions. Make sense?

  5. Pingback: Links: German elections, grad student advice, IL/IR symposium, O'Bagy - IR Blog

  6. Pingback: Replicability versus Robustness | Tom Pepinsky

  7. Pingback: Approximate Blogiversary | Tom Pepinsky

Comments are closed.