Tom Pepinsky

Category: Research

  • Plug-and-Play Statistical Models and Treatment Effects

    The “credibility revolution” has transformed how social scientists think about the relationship between causal inference and statistical estimation. An active research agenda has developed over the past twenty years that seeks to reground or reformulate existing statistical models as treatment effects estimators, and Angrist and Pischke’s Mostly Harmless Econometrics is an early progress report.

    Nevertheless, there are many more statistical models than there are derivations of treatment effects estimators associated with them. Treating a standard statistical model as a plug-and-play extension of a related model that does have treatment effects interpretations can be particularly dangerous. This post discusses an example.

    My use of “plug-and-play” comes from my experience using standard statistical software. Once you realize that you have a dependent variable that requires a nonlinear model, nothing stops you from replacing (in Stata) reg Y X ... with (say) logit Y X .... Interpreting the results requires a bit more work, but the regression output looks about the same, and all too rarely do we actually interpret the substantive results (or care about the specificities of that interpretation). So for better or for worse, it is common to treat these as plug-and-play models. Just plug in the nonlinear model that fits your dependent variable and off you go.

    The context of my example is the linear difference-in-differences model, where one compares the changes in a group that experienced a change between t1 and t2 with a group that did not experience such a change. The standard linear regression-based formulation of the model is
    Y = \alpha + \beta G + \gamma T + \delta G\cdot T + \theta X + \epsilon
    with G = group (treated or not), T = time (before and after the treated group got the treatment), and X are controls. The model relies on the assumption that the trend between periods in the treated group is the same as that in the control group. The model below (from Wikipedia) shows why such an assumption is commonly called “parallel trends.”

    One convenient feature of the regression-based implementation of the diff-in-diff model is that the coefficient \delta, the interaction term between G and T, represents the causal quantity of interest.

    Now, interaction terms in general have proven troublesome for applied researchers basically forever (cavepeople were omitting constituent terms back in the Neolithic, which is why it took so long to discover that fire required fuel and heat). But in nonlinear models they are especially challenging. Most notably, in 2003 Ai and Norton (PDF) reminded applied researchers that in nonlinear models like logit and probit, the marginal effect of an interaction term is not the same as the interaction effect. They can have different magnitudes, different levels of statistical significance, even different signs. Interaction terms in the nonlinear models cannot be interpreted as they can be as in linear models.

    So what if one is interested in estimating treatment effect using a nonlinear diff-in-diff model? Say, the effect of a policy change on whether individuals attend college or not. The plug-and-play extension of the above linear regression specification will require an interaction term, just as the standard diff-in-diff model does.
    Y = \Phi \{\alpha + \beta G + \gamma T + \delta G\cdot T + \theta X + \epsilon\}
    It would seem that the Ai and Norton conclusions ought to extend perfectly: if the same statistical model is being used, then shouldn’t the same interpretations of interaction terms follow? However, Puhani (PDF) demonstrates that this is not true.

    How? Why? Puhani and also Lechner (PDF) provide extended discussions, but the core reason is that causal effects must be defined differently when bounding the dependent variable because potential outcomes too must be bounded. One consequence, as Puhani discusses, is that \delta will necessarily have the same sign as the causal quantity of interest. But the causal quantity of interest itself is now something different than the average treatment effect as justified by a common trend assumption. Lechner’s review discusses further the challenges of invoking parallel trend assumptions in nonlinear diff-in-diff models, and suggests ways that more parametric identification assumptions may provide plausible foundations in particular applications. (The related literature is large and technical; Athey and Imbens (PDF) is one notable contribution.)

    The takeaway from this discussion is that the very same statistical model must be interpreted differently when used to identify treatment effects versus summarizing partial correlations. Plug-and-play statistical models are generally dangerous in a treatment effects world. This is perhaps a further reason that many are skeptical of logit or probit as alternatives to OLS—not because the differences don’t usually matter, but because treatment effects interpretations of nonlinear models might not be obvious. The real work is in determining the conditions under which plug-and-play models are appropriate, something which new research (which uses the terminology of “plug-in estimators”) promises to do.

  • How Did Southeast Asia Become A Social Fact?

    The first principle of Southeast Asian studies is the very artificiality of the concept of Southeast Asia. I have called this the “fundamental anxiety” of Southeast Asian studies, that there is no coherent argument why Southeast Asia properly includes Burma but not Bangladesh or Sri Lanka, or why it includes Indonesia but not Papua New Guinea, why Vietnam ought to fall within Southeast Asia rather than East Asia.

    And yet Southeast Asia does exist. It is now a social fact. For example, Google knows exactly what Southeast Asia is and what it is is not.

    Southeast Asian studies programs cover the same set of countries in the US, in Southeast Asia itself, in Japan, and elsewhere. The Association of Southeast Asian Nations will someday include Timor-Leste, and will never include Sri Lanka, Taiwan, or the Solomon Islands. Southeast Asianists have for decades lamented the intellectual strictures that Southeast Asian-ist thinking can place on their scholarly inquiry, and there is a rich genre of border-crossing Southeast Asian studies—on Zomia, most famously, but there are many others—that conceptualizes some subset of the peoples of the region as part of some competing configuration of peoples or states. Yet such inquiry has failed to dismantle the concept of Southeast Asia entirely, and my prediction is that it never will. Just look at how the Association for Asian Studies imagines Asia.

    In my view, the durability of Southeast Asia as a social fact is easy to explain. The much more interesting question is, how did we get here? How did Southeast Asia become a social fact, and how did it become this particular social fact? Those eleven states, no more and no less?

    The customary answer is that the concept of Southeast Asia is an external projection of Western images of the East onto the people living there. In some basic sense this is certainly true, but it leaves the details underspecified: why did the West happen to think that Vietnam is part of Southeast Asia, but not Bangladesh? Culturalist and geographic answers to such questions do not withstand close scrutiny; the Wallace line excludes the Philippines and half of Indonesia, for example, and Theravada Buddhist cultural influences are minor in much of Indonesia, the Philippines, and Vietnam. The answer upon which most scholars seem to settle is due to Ruth McVey, who looked to the idea of Southeast Asia as a lineage of the Second World War. This explains both timing and content of Southeast Asia, through

    the coincidence between Southeast Asia’s birth as a concept and the triumph of American world power. World War II was important not for the bureaucratic detail of the South East Asia Command but for the fact that the Japanese occupied the region, creating an abrupt break with the era of European domination and making it an object of American attention.

    McVey’s insightful analysis forms the core of my own understanding of why Southeast Asia is what it is. But the argument is incomplete because the South East Asia Command does not coincide with the social fact of Southeast Asia. Here are maps of Southeast Asia in the eyes of the Allied military command structure in 1942 and 1944.


    Never were the Philippines and most of Indonesia part of the South East Asia Command. The South East Asia Command, in fact, operated out of Sri Lanka. And never was it the case that Japanese occupation demarcates the boundaries of contemporary Southeast Asia, even to a first approximation.

    I am left without a good explanation for how Southeast Asia came to be this particular social fact. I wonder if readers can point me to other answers, no matter how grandiose or mundane they may be. It could be, for example, that Southeast Asia emerged as a social fact in response to the crystallization of other social facts, of South Asia, of China as including both the mainland and Taiwan, of Australia as a Western country, and of Melanesia and Micronesia as separate regions. Such an argument would raise other questions, but those questions too could help reveal why Southeast Asia is what it is.