Tom Pepinsky

Author: tompepinsky

  • A Note on Methodological Discussions in Published Articles

    This very clever (and important) paper by Peter Aronow and Allison Sovey Carnegie on recovering the Average Treatment Effect from the Local Average Treatment Effect makes an interesting observation:

    …in a review of 34 empirical articles that employ IV estimation from 2004-2009 in the American Political Science Review and the American Journal of Political Science, only two (6%) mention that the causal effect being estimated is the LATE.

    This is a specific instance of a general phenomenon which is common among methodological papers: the author(s) will note that in a review of the literature, some Very Small Number of articles will have mentioned some Very Important Thing. The author(s) make this point in order to make the case that this Very Important Thing is misunderstood or unappreciated by the discipline in general.

    This is speaking directly to people like me. I’m not one of the authors who employed IV in the sample to which Aronow and Carnegie are referring, but I could have been: on two separate occasions I have been told by reviewers to “remove the discussion of the local average treatment effect” from a manuscript under review. One reviewer did not seem to understand what the LATE is. The other wrote something along the lines of “everyone knows what the LATE is, so get on with it.”

    Despite the rhetorical power of the sort of claim of that Aronow and Carnegie are making, failure to discuss an important methodological point does not necessarily reflect the author’s failure to understand it. It could also reflect an obstreperous reviewer or editor—and these people are gatekeepers, so we have strong incentives to implement whatever they recommend. Or it could, in principle, reflect the belief that that important point is so broadly understood as not to require explicit elaboration—like, do we cite Zellner 1962 in every paper that uses seemingly unrelated regressions?

    It is probably true that in this specific example, Aronow and Carnegie are correct that most political scientists aren’t really thinking about what specific quantity the LATE represents, a claim I have made around here once or twice before. But, we ought to be careful about what a review of published studies can tell us about what their authors think, and by implication, what a discipline believes. Running manuscripts through the wringer of peer review very often produces articles that look very different from what their authors had originally intended.

  • Fractionalization and Competition Reprise

    About nine months ago I posted about a peculiar finding about Indonesian local politics: the observation that there appeared to be high levels of political fragmentation in the most ethnically homogenous districts. It makes sense to observe that ethnic heterogeneity produces a fragmented party system, but not that ethnic homogeneity would do the same. I found this puzzling.

    Sometimes, though, a puzzling result is just a mistake. I think that that is what happened here. Together with my co-authors, I have been recreating political fractionalization and ethnic fractionalization scores from the original raw data, ethnicity data from the 2000 Census and party seat shares in the district legislatures from the General Election Commission. (We had been using indices created by someone else, probably for a different purpose.) In the process of doing this, I found a load of errors in our original data, many of them quite significant.

    Here is that same scatterplot using the corrected data.

    Ethnolinguistic and Political Fractionalization Indices

    The indices of political and ethnic fractionalization are the standard Herfindahl-style indices: if pi is the proportion of ethnic group (or political party) i in a district, then a district’s total fractionalization score is

    FRACTIONALIZATION = 1–Σ(pi)2

    Very straightforward stuff. In addition to a positive correlation between ethnic and political heterogeneity, we observe in this figure as well a classic example of heteroskedasticity: there is a higher variance in political fractionalization in more ethnically homogenous districts than in more ethnically heterogenous ones (a Breusch-Pagan test strongly rejects the null of homoskedasticity; some further digging indicates that the political fractionalization index is not normally distributed). We also see that the green dashed line (the linear fit) and the red solid line (the lowess fit) are nearly identical, which suggests that there isn’t any significant non-linearity in the bivariate relationship.

    Anyway, the question about what to do about the peculiar relationship that we’ve uncovered between ethnic fractionalization and political fragmentation turns out to be an artifact of some bad data. Happily, with the better data our earlier results appear even stronger: in Indonesian local politics, more political fragmentation -> lower budget surpluses.