Tom Pepinsky

Author: tompepinsky

  • Measuring the Credibility Revolution in Political Science

    Measuring the Credibility Revolution in Political Science

    A new working paper entitled “The Credibility Revolution in Political Science,” by Torreblanca, Dinneen, Grossman, and Xu, is making the rounds. It is an interesting, important, and thought-provoking effort to characterize how the credibility revolution has shaped the political science discipline. The authors assemble mounds of new data on journal articles in political science, and use LLMs to identify articles that employ design-based versus model-based inference. This allows the authors to characterize phenomena like the citation premium for design-based articles and other interesting features of the data.

    I am interested in the implicit idea that one measures the influence of the credibility revolution in political science by counting the number of articles that employ the tools of the credibility revolution and comparing that to the number of articles that do not employ these tools. This is a reasonable answer to one specific kind of question: “are the tools of the credibility revolution being employed more frequently than other tools in quantitative political science.” The authors phrase this research question as follows:

    if the credibility revolution has taken hold, design-based methods should increasingly displace purely model-based approaches in quantitative explanatory research, with growth spread across multiple design-based strategies

    Again, this is a useful hypothesis and we should know the answer to it (spoiler alert: the answer is yes). But this way of thinking underestimates the impact of the credibility revolution on political science. Specifically, one of the other implications of the credibility revolution is that one should deliberately not use causal language or employ design-based inference when one’s quantitative research does not seek to estimate a causal effect, or when no design-based logic of inference is credible.

    I count myself as a partisan in the credibility revolution, but this means something different to me than it might to some of my other fellow partisans. To be a partisan does not imply that one only employs design-based inference; rather, it means always paying close attention to the distinction between causal and non-causal claims and the requirements for inferring causality from an observed association between two or more variables. Here (PDF) is an example from my own research:

    In this article I reinterpret Ng, Rangel, Vaithilingam, and Pillay’s analysis in this issue of pro-BN voting in Peninsular Malaysia in Malaysia’s 2013 general election. I show that the authors’ statistical methods are inappropriate for testing whether district ethnicity predicts district-level BN vote share, and that their modeling choices result in tests of hypotheses that do not exist and cannot be derived from standard theoretical approaches to ethnic voting in Malaysia. I then provide a range of statistical evidence that supports three main conclusions: (1) ethnicity and district area (a proxy for urbanization) both predict BN vote shares at the district level, (2) neither the effect of ethnicity nor of district area can be reduced to the other, and (3) there is no interactive effect between ethnicity and urbanization. These results are in direct contradiction with the authors’ results, and apply equally in Peninsular Malaysia and the entire country. I also discuss the broader issues that emerge when testing competing theories of BN vote share.

    This article does not purport to estimate a causal effect, and goes on at some length about the conceptual and empirical challenges of prediction from two highly correlated explanatory variables. You could not argue that this is an example of design-based inference in action, and yet this is a case where the credibility revolution caused* an author to not use causal language or design-based inference in quantitative explanatory research. I think the inferences are more compelling because of it.

    One of the upshots of the credibility revolution might be better non-causal empirical research, not just more and better causal research designs.

    NOTE

    * Counterfactually, no way I would have written this piece this way had I not been exposed to the logic of design-based inference from early in graduate school.

  • Comparative Politics Needs Area Studies, and Area Studies Needs Comparative Politics

    If you were a graduate student in political science between 1990 and 2010 or so, you probably experienced some heated debates about the future of area studies and its role in the discipline. This was a time of revolutionary entrepreneurs were advancing rational choice theory as a unified framework for the social sciences, and quantitatively-oriented scholars were developing a hegemonic language for understanding qualitative scholarship within the standard positivist template.

    At the same time, area studies in the United States and around the world was under attack, the first in many “crises of area studies” since around 1990. Political scientists were some of the main participants in the so-called “area studies wars” (see here and here and here and many other pieces). As you click on those links, you can see how connected these two developments were: rational choice theory and quantification were part of the intellectual ferment of the area studies debates. And the language was one of not just tension or disagreement, but conflict: area studies wars, scholars and perspectives on the retreat or under siege, a rational choice revolution with an intellectual vanguard, and so forth.

    Many early career researchers during this time faced pressures—either subtle or explicit—to declare their partisan loyalties either to social science or to area studies. At issue was not just what individual scholars would choose to do, but where they stood on the broader discipline’s commitment to creating knowledge of a particular form. My favorite illustration of this perspective is a point made by Gary King in 1996:

    the professional goal of all scientists should be to attempt to demonstrate that context makes no difference whatsoever.

    Read that piece in its entirety if you don’t believe me (PDF); King’s argument is sincere and thoughtfully articulated, and you cannot just dismiss it out of hand.

    If you were a professional scientist, then, King is suggesting that your goal is not just to ask whether context matters, but to prove that it does not. What is the role of an area specialist in such a world? To demonstrate that their specialize knowledge of context is irrelevant for political science.

    Time has marched on since the 1990s, and the discipline of political science has evolved. The rational choice revolution was mostly unsuccessful, with the revolutionaries putting down their weapons (just as the inclusion-moderation thesis would predict). The next intellectual revolution—the credibility revolution—has created its own revolutionary vanguard. Graduate students today do not face quite the same hostility towards area-specific knowledge. I do not know anyone who would argue in 2025 that the professional goal of the comparativist should be to demonstrate that context does not count.

    So what of the relationship between comparative politics and area studies? In a new working paper just released as an APSA preprint, Jordan Gans-Morse, Daniel Gingerich, and I take up this question, revisiting old debates and arguments in light of these and other developments in the social sciences and in area studies itself. Our argument relies on a comprehensive analysis of article data from major journals in political science from 1980-2020, and an original survey of political scientists conducted in 2022-23. Here is the abstract:

    We revisit the sharp divide that emerged in the 1990s between area studies advocates and methodologically oriented political scientists. We argue that tensions between political science and area studies are neither intrinsic nor static, but instead evolve in tandem with theoretical and methodological trends, as well as with broader political and technological developments. Drawing on an original survey of American Political Science Association members and analysis of roughly 4,500 articles in leading journals, we identify four shifts in the discipline: from a theoretical to an empirical orientation; from cross-national datasets to country- and region- specific studies; from macro- to micro-level analyses; and from descriptive to causal inference. We also document evolving patterns in language training, fieldwork, methods use, and data collection. Our findings suggest that political science and area studies are increasingly compatible and well-positioned for reconciliation, but that the state of area studies is fragile and the subfield of comparative politics must support it.

    We began this research years ago, long before the current administration set out to dismantle area studies expertise in the United States.* Our argument, though, speaks to the current moment as well. We write,

    As university models are being reevaluated around the world, the future vitality of area studies departments, programs, and infrastructure are in question; comparativists must recognize that the future of comparative politics as we currently know it depends on robust area studies institutions….

    For area studies within comparative politics to thrive, though, the discipline as a whole must reward engagement with area studies, aligning comparativists’ professional incentives as political scientists with the needs of area studies communities. One concrete measure is for political science departments to reward scholars equally for their service to area studies as to political science. Another is for APSA and other political science associations to formally recognize scholars who help promote interdisciplinary collaboration or institution building that sustains or strengthens area studies…

    Coauthoring is now the norm among quantitatively oriented scholars. But this model must be extended to reward collaboration with country experts, including local scholars from the region being studied, not as research assistants or informants or raw data collectors (as in some of the more extreme division-of-labor proposals from the 1990s) but as full and equal partners in the research enterprise. Rather than a hierarchical model of collaboration that privileges disciplinary expertise, the appropriate model is horizontal in nature, with all participants deserving of full credit for their contributions.

    We hope that our perspective can renew scholarly focus on the essential role of area studies for research and policy, for political science and for other disciplines as well.

    NOTE

    * Non-U.S. scholars might want to tut-tut about the sorry state of higher education funding for area studies in the U.S., but area studies is under threat and facing major cuts in funding in the U.S., Europe, Australia, Japan, and every other advanced industrial democracy. Find me an exception; I’ll wait here.

    Only China is expanding its area studies funding, as Gerhard Hoffstaedter argues in a recent piece. And note Hoffstaedter’s use of war terminology: “Continuing to cut area studies while China expands them amounts to intellectual disarmament.”