I am currently working on a new paper on the subnational comparative method in political science. If my earlier writings on comparative methods (PDF) are forceful advocacy in favor of comparisons across countries, this paper will offer some cautionary notes about the limits of comparisons within countries.

One prominent argument in favor of subnational research designs is that they make the unit homogeneity assumption more tenable, both by controlling for various historical or contextual factors that are specific to one country and by giving the analyst a better command of the data. But context and history also can suggest important differences across regions within a country too, and these differences may not reduce simply to linear combinations of observed variables. Here is one illustration from Indonesia: electrification rates by village across Indonesian provinces. Using the 2011 Village Potential Survey (Potensi Desa, PODES 2011), I have a calculated the proportion of each province’s villages in which no households have access to electricity. Here is a map, by province (with provinces grouped by island/island group).

To put it baldly, Papua is not like the others.

Interestingly, PODES allows us to distinguish between publicly provided and otherwise provided electricity sources. (The alternative to being electrified through a grid through Indonesia’s state electricity company [PLN] is to have generators or other privately-provided sources of electricity.) Here, there is more variation.

The results, in fact, suggest the real difference in Papua is that unlike other parts of the Indonesian periphery, it is rare to even have access to generators to make up for the failure of PLN to provide sufficient access to electricity.

Of course, precisely none of this implies that we should never rely on subnational comparisons. The point is that subnational research designs rely on the same assumptions as cross-national research designs, and they are more or less tenable depending on the comparison and the question. And of course, I am still comparing here—the distinctiveness of Papua is invisible unless we compare it to other provinces. The conclusion I have in mind is something more subtle: it appears that what sets Papua apart is not (just) that it is far from Java, or (just) that it is relatively poor, or (just) its distinct colonial history, but something else. It could be the interaction of all of those observed variables, or something else altogether. In separate results, I can show that a province-level regression of electricity availability on population, distance from Jakarta, average ruggedness of village terrain, and island-group dummies always yields a negative and statistically significant coefficient for Papua (and no other interesting results).

Recognizing this “unit-specific heterogeneity” helps us to account for the variation that we see. But if the point of our subnational comparative method is to help us to replace proper names with conceptual variables, this unit-specific heterogeneity reminds us of the limits of this endeavor even in the subnational context.