Tom Pepinsky

Category: Research

  • What Does Randomization do to Confounders?

    Lately, I have been reading a lot of descriptions of experiments and randomization by applied researchers. One kind of phrase particularly bugs me: I frequently see language like “experimental randomization controls for potential alternative explanations…” or something similar.

    Specifically, I don’t like “controls for” as a description of the function of randomization, or more generally the metaphor of the experiment as a method of controlling for confounding variables. It comes from a regression-based model of statistical inference that was dominant in most poli sci graduate programs until somewhere in the early 2000s, where you often deal with an alternative explanation by adding a “control variable” to a regression.

    I’m no expert, but here is my one-sentence description of randomization and confounders: Randomization ensures that in expectation, each confounder is independent of the joint distribution of treatments and potential outcomes in the sample. Let me be clear: this may be a very reasonable assumption. But stating it this way helps to illustrate the differences between experiments and regression-based adjustments. Careful description of what experiments do is critical for careful thinking about what experiments do, and that is undoubtedly good social science.

    Now, the dirty little secret is that while people of my generation who are trained in multiple regression always nod when they hear the words “control for,” it’s devilishly hard to explain in words what control variables actually do. (Illustrative task: explain to a smart undergraduate student—one who can talk back—exactly what a control variable does. I bet that you will find that it’s a lot harder than you think.)

    Methodologists/experimentalists: did I get my description of randomization right? If not, how can I fix it?

  • The Shitty Politicians Theory of Indonesian Politics

    Today I had cause to think about a basic question for scholars of contemporary Indonesia. What is the mainstream explanation for policy outcomes in the local level? So, when a regency implements a good policy, or a bad policy, why does this happen? How can we explain it? It was great because it was asked by a non-specialist, who was genuinely curious.

    In trying to come up with an answer, I was kinda stumped. There are plenty of accounts of policymaking in individual regencies that describe rich interactions of political agency, material incentives, historical legacies, advocacy networks, voting and popular mobilization, etc…but they tend to be highly specific to individual contexts. There are also plenty of narratives about why Indonesia as a whole is the way that it is—it is corrupt, it is dynamic, it is Muslim, etc.—but these are not explanations that can explain variation across regencies. There are also estimates of the causal effects of things like direct local elections on policy outputs, but they are designed not to explain outcomes (to tell us “the causes of effects”), but rather to illustrate “the effects of causes,” which can exist even if these effects don’t explain a lot.

    As far as I can tell, the only framework out there that is (1) capable of explaining variation and (2) designed to explain a lot of the variation in local policymaking is something that I will call the “Shitty Politicians Theory of Indonesian Politics.” That is, good policies are implemented because there are good politicians making the decisions to implement them. Bad policies are implemented because of, well, bad politicians. And what explains bad politicians? Bad regencies.

    Now it turns out that I have made this very argument myself, in print, and not that long ago. Of course, we used the phrase “the endogenous deterioration of local governance institutions [that] undermine[s] the supposed development-enhancing promises of decentralized government,” rather than the more colorful equivalent of “shitty regencies elect shitty politicians who implement shitty policies that make the regencies even shittier.”

    Despite my belief that this is in general a useful way to think about local politics in the broadest sense, as a research agenda is deeply unsatisfying. Let me be clear, this is not only a theory of Shitty Politicians and Indonesian Politics, but also Politicians Theory of Indonesian Politics which is shitty. In this framework, basic problems of conceptualization and measurement of the quality of politicians, the quality of policy outputs, and the concrete historical/social/economic fundamentals that drive the selection of the good or bad politicians are largely swept under the rug. What’s more, there’s no room for agency or human action here: no real room for a Jokowi without torturing the theory to death.

    The scholar who can either fix the Shitty Politicians Theory, or who can propose an alternative with real explanatory power for making sense of the diverse experiences of many different locations across the archipelago, will have made a major contribution to both Indonesian studies and comparative politics.