Politics is complex. For scholars of comparative politics who study domestic politics in an increasingly globalized world, understanding the interactions among local, national, transnational, regional, and global forces is essential. So how should we proceed? One view is that grasping complexity means discarding simple theories and spare models of politics that do not reflect the complexity that we know exists. The position is intuitively appealing: complex problems require complex tools.

There is, however, another view. My colleague Andrew Little and I have recently finished a new paper on formal theory in comparative politics entitled “Simple Formal Models in Comparative Politics” (PDF). It is written as part of a dialogue on the future of comparative politics, and responds (in part) to work by Philippe Schmitter (see e.g. here) in which complexity[*], multi-level politics, and the dangers of simplifying assumptions figure prominently. Part of the paper is a clarification of the state of formal theory in comparative politics. We show that formal theory still occupies a relatively small part of the work being done in comparative politics, and that there is scant evidence that this is going to change any time soon. We also comment on some common beliefs about what models and assumptions are for which, sadly, remain all too common in the discipline.

But more interesting and broadly relevant is what comes next. We argue that in a world of complex interactions, simplification—in formal theory as in other kinds of theorizing—is a virtue, not a vice. We explain why in detail in the paper, but at root is the fact that theories are always simplifications, and descriptive accuracy is but one criterion by which a theory ought to be judged. We also suggest that professional incentives lead modelers to create formal models that are more complicated than they need to be. Our suggestions for how simple models of politics (formal or otherwise) might join together with the “complexity-embracing” modes of research is a nice parallel to recent contributions by Gehlbach (PDF) and Lorentzen et al (PDF). Our perspective on theory-as-simplification also parallels Healy’s colorful reflections on “nuance” (PDF).

So yes, the politics is complex, but this does not mean our theories must be also. Instead, we need multiple just-simple-enough theories, and continuous collaboration with case experts and other empiricists to know what “just-simple-enough” means.

Note

[*] Without speaking for my coauthor, I doubt that this is a particularly revolutionary idea when the term “complex interdependence” is nearly a century old, and prominent scholars have been asking questions like “is the traditional distinction between international relations and domestic politics dead?” longer than I have been alive (see Gourevitch 1978).

The “credibility revolution” has transformed how social scientists think about the relationship between causal inference and statistical estimation. An active research agenda has developed over the past twenty years that seeks to reground or reformulate existing statistical models as treatment effects estimators, and Angrist and Pischke’s Mostly Harmless Econometrics is an early progress report.

Nevertheless, there are many more statistical models than there are derivations of treatment effects estimators associated with them. Treating a standard statistical model as a plug-and-play extension of a related model that does have treatment effects interpretations can be particularly dangerous. This post discusses an example.

My use of “plug-and-play” comes from my experience using standard statistical software. Once you realize that you have a dependent variable that requires a nonlinear model, nothing stops you from replacing (in Stata) reg Y X ... with (say) logit Y X .... Interpreting the results requires a bit more work, but the regression output looks about the same, and all too rarely do we actually interpret the substantive results (or care about the specificities of that interpretation). So for better or for worse, it is common to treat these as plug-and-play models. Just plug in the nonlinear model that fits your dependent variable and off you go.

The context of my example is the linear difference-in-differences model, where one compares the changes in a group that experienced a change between t1 and t2 with a group that did not experience such a change. The standard linear regression-based formulation of the model is

with G = group (treated or not), T = time (before and after the treated group got the treatment), and X are controls. The model relies on the assumption that the trend between periods in the treated group is the same as that in the control group. The model below (from Wikipedia) shows why such an assumption is commonly called “parallel trends.”

One convenient feature of the regression-based implementation of the diff-in-diff model is that the coefficient , the interaction term between G and T, represents the causal quantity of interest.

Now, interaction terms in general have proven troublesome for applied researchers basically forever (cavepeople were omitting constituent terms back in the Neolithic, which is why it took so long to discover that fire required fuel and heat). But in nonlinear models they are especially challenging. Most notably, in 2003 Ai and Norton (PDF) reminded applied researchers that in nonlinear models like logit and probit, the marginal effect of an interaction term is not the same as the interaction effect. They can have different magnitudes, different levels of statistical significance, even different signs. Interaction terms in the nonlinear models cannot be interpreted as they can be as in linear models.

So what if one is interested in estimating treatment effect using a nonlinear diff-in-diff model? Say, the effect of a policy change on whether individuals attend college or not. The plug-and-play extension of the above linear regression specification will require an interaction term, just as the standard diff-in-diff model does.

It would seem that the Ai and Norton conclusions ought to extend perfectly: if the same statistical model is being used, then shouldn’t the same interpretations of interaction terms follow? However, Puhani (PDF) demonstrates that this is not true.

How? Why? Puhani and also Lechner (PDF) provide extended discussions, but the core reason is that causal effects must be defined differently when bounding the dependent variable because potential outcomes too must be bounded. One consequence, as Puhani discusses, is that will necessarily have the same sign as the causal quantity of interest. But the causal quantity of interest itself is now something different than the average treatment effect as justified by a common trend assumption. Lechner’s review discusses further the challenges of invoking parallel trend assumptions in nonlinear diff-in-diff models, and suggests ways that more parametric identification assumptions may provide plausible foundations in particular applications. (The related literature is large and technical; Athey and Imbens (PDF) is one notable contribution.)

The takeaway from this discussion is that the very same statistical model must be interpreted differently when used to identify treatment effects versus summarizing partial correlations. Plug-and-play statistical models are generally dangerous in a treatment effects world. This is perhaps a further reason that many are skeptical of logit or probit as alternatives to OLS—not because the differences don’t usually matter, but because treatment effects interpretations of nonlinear models might not be obvious. The real work is in determining the conditions under which plug-and-play models are appropriate, something which new research (which uses the terminology of “plug-in estimators”) promises to do.