Sabina Leonelli has written a very interesting new essay on reproducibility (and, tangentially, replicability) with the title “Re-Thinking Reproducibility as a Criterion for Research Quality” (PDF). The central argument is that we should not consider reproducibility to be the main criterion by which we judge research to be scientific. In her words,

While most commentators focus on the use of reproducibility as the best strategy to achieve inter-subjectively reliable outcomes, I shall argue that (1) such convergence of results can be obtained in the absence of reproduction, and (2) a different, yet crucial function of reproducibility consists in helping researchers to identify relevant variants in the first place.

The key to her argument is this: there is a continuum of research practices ranging from those that offer full control over the system under investigation (computer simulations) to those with very limited or no control over the system under investigation (participant observation). Reproducibility as a standard for distinguishing scientific from unscientific claims makes most sense for the former.

Anyone interested in reproducibility and replicability in the social sciences ought to take a look at this. One possible critical response would be that we should always prefer reliable measures and unbiased estimators to unreliable measures and biased estimators, so reproducibility has epistemic value as a description of methodological practice even if replicability does not. This perspective might follow from an understanding of science as a communal exercise in which one must be able to describe what one has done to someone else in order for that communal exercise to be possible. That “description of what one has done” is perhaps the thinnest possible way that we might conceptualize reproducibility.

It is noteworthy that there are no references to political science in this manuscript. Despite the wide interest in replication* in contemporary political science, we do not seem to be leading these debates more generally.

NOTE

* Replication for political scientists normally means what other disciplines call reproduction: can I run your code on my computer and generate the same results?