Two lessons from my methods courses early in graduate school have been fundamental to the way I do political science.

1. Lesson One. Data can never “speak for itself.” Only theory can tell you what what correlations mean, what empirical models are sensible, and in general how to interpret statistical results.
2. Lesson Two. Causal inference is one of the central endeavors in social science. Moreover, when it comes to causal inference, experiments are the gold standard, and everything else must be measured against the experimental template.

It is no exaggeration that these have shaped everything that I have done since my second year in graduate school. Both lessons seem very reasonable, if idealistic. Yet I am increasingly convinced that these two lessons are not self-evidently compatible with one another.

Here’s what I mean: causal effects in the Rubin causal model are defined as differences between outcomes in different treatment states. I emphasize here defined. In the potential outcomes framework, this is what it means for something to be a cause of something else. There is nothing deeper.

So where does theory (Lesson One) enter into this understanding of causality? Nowhere. If you have a treatment, the average treatment effect can be calculated irrespective of the theoretical implications or assumptions associated with it. And it truly is the causal effect by the definition of causality above. If you have a quasi-experiment in which you can claim that the treatment-like variable is strongly ignorable, or an instrument or forcing variable, then the some kind to treatment effect can be estimated from the data.

There is a tension, then. Of course, no one goes around randomly assigning random treatments and then calculating their effects. But if you did that…you could calculate treatment effects, and they would indeed have causal interpretations under the model of causality that underlies the experimental template for causal inference. Somehow, “proper” research means using theory (that is, following Lesson One) to guide the production of causal inferences. That is the contested and messy part of the research process, one that cannot be achieved by faithfully adhering to Lesson Two. Worse yet, Lesson One doesn’t provide clear guidelines either.