Author: tompepinsky

  • Language, Thought, and Reality: Does Language Shape Our Social World?

    Does language shape thought? Do the languages we speak affect how we live our lives? These are some of the oldest questions in the cognitive and social sciences, and most everybody reading these words probably has thought about them. These also speak to core questions in the philosophy of mind and language, on the role of language in intermediating between our brains and the world around us.

    An emerging literature in the social sciences has given these questions renewed prominence by arguing that language systematically affects people’s values, beliefs, and behaviors. And in a forthcoming article in Language (preprint here [PDF]), I subject this prominent new literature to conceptual and empirical scrutiny. What follows is an expanded summary of my argument, accompanied by a discussion of why it matters, based loosely on this Twitter thread.

    Linguistic Relativity and the Sapir-Whorf Hypothesis

    You may have encountered the idea that language shapes reality in some form or another before. If you learned about it in a university setting, you probably heard it described as the Sapir-Whorf hypothesis; more generally, however, linguists, anthropologists, and those working in adjacent fields use the term “linguistic relativity” to describe a set of related theses about human language. Linguistic relativity means, loosely, that—at least to some degree—the language that you speak affects how you perceive and interact with the social world. For example, Arabic-speakers and Japanese-speakers experience the world differently because their languages differ. Do note that linguistic relativity is a precise claim: it is not just that Arabic speakers and Japanese speakers differ, or that communities of Arabic speakers and communities of Japanese speakers differ. Linguistic relativity holds that in some specific way, the differences between the Arabic language and the Japanese language are responsible for some of those differences between Arabic and Japanese speaking communities.

    French versus English pronouns help to illustrate what I mean. English has one second-person singular pronoun: you.* French as two second-person singular pronouns: tu and vous. The two words, in French, differ in their familiarity, formality, or politeness. You’d address a stranger with vous, a friend with tu, a teacher with vous, a child with tu, and so forth.

    The linguistic relativity thesis holds that this is not just a quirk of French that English speakers have to master. A strong proponent of linguistic relativity would predict that French people will be more attuned to rank, status, and hierarchy than English speakers because speaking French obliges one to attend to tu/vous distinctions in order to be communicatively competent. Of course lots of other things would explain speaker-community-level differences in attention to hierarchy; language isn’t everything. But counterfactually, English would be even more status-conscious if it also had a tu/vous distinction as well.

    It is hard to think of a single research area that is so equally exciting for philosophers, experimental psychologists, ethnographers, literary theorists, and development economists. Think of the implications! Are certain language communities going to be better at certain cognitive tasks than others? Are there linguistic roots of hierarchical political systems? Is translation even possible? And it’s not just academics: growing up, I once heard a family friend announce to my parents that “the French are assholes” because of their attention to hierarchy, and used the tu/vous distinction as evidence.**

    Although people have wondered about these sorts of questions for a long time, the history of linguistic relativity as a field of scientific inquiry dates roughly to the mid-19th century, concomitant with the emergence of anthropology as a scientific discipline. One common feature of mature scientific disciplines is that they began through various procedures and claims that we today would classify as pseudo-science.*** Alchemy to chemistry, astrology to astronomy, phrenology to physical anthropology, the list goes on.

    And so too with the pseudo-scientific origins of what I consider to be a properly scientific field of inquiry on linguistic relativity. Linguistic relativity as a scientific claim developed in the context of the encounter between Europeans and the Americas and with Africa, in a time of settler colonialism and imperial race science. Although mine is not a systematic excavation of the intellectual history of linguistic relativity, it seems fairly plain that it emerged in conversation with views about “primitive peoples” versus “civilized peoples,” and capturing the notion that primitive peoples were primitive because they spoke primitive languages.

    One important intellectual figure in the development of the (pseudo-) science of linguistic relativity was Wilhelm von Humboldt, a German linguist who made important discoveries about Basque and Malayo-Polynesian languages. He postulated that language is a human universal, and it is the “organ” that makes thought possible. Reality exists outside of our thoughts, but it is only through language that we are able to translate the world “out there” into thought. And because languages vary, he argued, it must follow that languages produce different thought-systems that will, naturally, vary across linguistic communities.

    This is not an inherently racist position. The idea that thought is only possible through language is straight Wittgenstein, for example (see here [PDF] for more). But such a position is easily imported into a pseudo-scientific racist worldview as a way to justify a view of certain languages as inherently superior to others, better suited to scientific reasoning or logical exposition than others. (Guess which languages were considered the best for logic and science.) It also can be used to justify the act of creating modern or civilized people by raising them with a modern or civilized language. It is not hard to see how linguistic relativity might be used to argue that an Anishinaabe child can be made into a proper Canadian by replacing their Ojibwe thoughts with English thoughts.

    This period in the intellectual history of linguistics and anthropology is endlessly fascinating: Boas, Sapir, Bloomfield, and others built linguistics and anthropology alongside one another. So much of making anthropology in the Americas was the task of learning and classifying the languages of the Americas. They worked both with and against Saussure and the European structuralists like Lévi-Strauss. The lineages are complex and it is sometimes hard to figure out who disagrees with whom about what, exactly. The arbitrariness of the sign unites basically everyone, but from there, just about anything goes.

    In any case, by the 1950s Noam Chomsky took linguistics in a new direction by postulating a Universal Grammar inherent to all human languages, reformulating the problem as to explain the diversity of human language from the postulated common linguistic capacities that all humans possess. The limitless generativity**** of human language suggests that we must all possess some common ability to think through language.

    In this now-standard view, the null hypothesis is that languages vary but that their phonological inventories, syntactic structures, semantic distinctions, and conversational pragmatics do not fundamentally affect human cognition.

    With this history in mind, you can imagine how shocking it is for linguists to hear non-linguists arguing about the effects of language variation on thought. For example, on the TED radio hour on NPR, we have this long piece on the effect of having a grammatical future on savings rates across countries. The argument, in the words of author Keith Chen, is that

    You speak English, a futured language, and what that means is that every time you discuss the future or any kind of a future event, grammatically, you’re forced to cleave that from the present and treat it as if it’s something viscerally different. Now suppose that that visceral difference makes you suddenly disassociate the future from the present every time you speak. If that’s true, and it makes the future feel like something more distant and more different from the present, that’s going to make it harder to save.

    The research that underlies this claim appears in the American Economic Review, the flagship journal of the premier economics disciplinary association in the world.

    Most Everybody Believes Some Version of Linguistic Relativity…

    Before we dig into that kind of research, though, let’s step back to consider what we know about linguistic relativity. The Chomskian position does not rule out linguistic relativity completely. And indeed, at some basic level, some version of linguistic relativity seems almost certain to be true. For a simple example, take the word for the color blue. In English, we look at these two colors and call them both blue.

    But in Russian, these are two different colors: goluboy/голубой (light blue) and siniy/синий (dark blue). You can’t go wrong calling either blue in English, but using siniy to describe light blue would be like calling the sky green. And it turns out that Russian speakers can do certain tasks related to classifying blue colors faster than English-speakers. This article in The Proceedings of the National Academy of Science of the United States of America finds that

    ….Russian speakers were faster to discriminate two colors when they fell into different linguistic categories in Russian (one siniy and the other goluboy) than when they were from the same linguistic category (both siniy or both goluboy). 

    This means that Russian-speakers go about their world a little bit differently than English-speakers: not just different words, but different competences. This finding tells us that our language affects our reality in some small yet precise way. Yet disambiguating among shades of blue isn’t really a socially meaningful activity. Most linguists don’t think that linguistic differences like this explain too much about our everyday lives, or gross patterns in human behavior or cognition across speech communities.

    My favorite example of linguistic relativity that matters comes from speakers of Guugu Yimithirr, a language of Far North Queensland (and source of the word “kangaroo” in English). Guugu Yimithirr is probably unique among the world’s languages in that—at least in some analyses—it possesses no egocentric directions, like left/right or in front/behind. This means that to describe where something is, you have to reference cardinal directions like North/South, or “towards the mountains” or “towards the sea”. For example, at this very moment the computer that I’m typing on is to the West of me. If I turn my back to it, it is still to the West of me. In English, though, one would say that it was in front of me, and then it was behind me.

    There is evidence that Guugu Yimithirr speakers are better at dead reckoning and solving maze puzzles than are speakers of related languages who do possess egocentric directions and who live otherwise similar lives. If this evidence is sound, then this is linguistic relativity in action. But I hasten to add that the analysis of Guugu Yimithirr as having only absolute frames of reference is contested (see here [pdf]). One worries about how we interpret linguistic and anthropological data about languages that seem “exotic” to Western audiences.

    The big takeaway is that linguistic relativity is an interesting and relatively well-defined domain of research in linguistic anthropology and cognitive science, but its empirical claims are very narrow and very precise and there are disagreements even about those. There is a very good edited volume called Rethinking Linguistic Relativity that is over two decades old, but which summarizes the state of this literature for anyone who really wants to understand it. Dan Slobin’s chapter (pdf) on “thinking for speaking” is probably the most sophisticated reformulation of linguistic relativity into a psycholinguistic research program, but it does not make strong arguments about variation across human communities’ cognitive capacities.

    …But Linguists (and Others) are Inherently Skeptical of Whorfian Socioeconomics

    With that background, we now turn to the latest iteration of the linguistic relativity research program. In the past twenty years or so, a number of social scientists have embraced linguistic relativity to argue that language is a fundamental determinant of human behavior across the world. Examples? What if people who speak languages with grammatical gender in indefinite articles are more sexist than the rest of us? (Compare English the to German der/die/das.) What if people with obligatory politeness distinctions in pronouns (tu/vous) are less egalitarian than the rest of us? What if people who speak future oriented languages are more likely to care about the environment than the rest of us?

    These are big claims. And to distinguish them from linguistic relativity as a research paradigm, I term this emerging literature “Whorfian socioeconomics.” You can think of it as linguistic relativity for the big leagues. Language affects no only how we complete certain cognitive tasks, but how we go about our lives—and when aggregated up to the level of the speaker community, how whole societies are oriented.

    In my read, however, the evidentiary base for Whorfian socioeconomics is very thin. And in my forthcoming article, I review the evidence and the tools that people use to make such claims, showing it to be weak.

    The first source of evidence for Whorfian socioeconomics is aggregate evidence correlating the presence or absence of a linguistic feature in a country or region’s dominant language with some outcome like gender discrimination, authoritarianism, or savings rates. This research is plausible, but to give any such correlations as causal interpretation, we need strong assumptions about ecological validity and unconfoundedness.

    The second source of evidence is experimental: take bilingual speakers, randomly assign them to hear something or do something in two different languages, and see if their behaviors differ. Although experiments are indeed the gold standard for establishing causal relationships, such experiments are inherently limited by the multiple dimensions across which languages vary. In the language of causal inference, language is a bundled treatment, which means that if you compare two languages that differ on some characteristic D, you are also comparing languages that differ by characteristics A, B, C, … and so forth. Differences between languages might be a product of any of those characteristics.

    For example, this article compares Russian and Estonian among bilingual speakers to see if speaking a futured versus futureless language affects time preferences. But to put it mildly, Russian and Estonian differ in a lot of ways. Russian has gendered pronouns, Estonian does not. Russian has six cases (in most analyses), Estonian has fourteen. Russian has no strict stress pattern, but stress in Estonian words is always on the first syllable. Russian allows for zero-copula for predicate nominals, Estonian does not. Russian is an Indo-European language, Estonian is a Uralic language. The list goes on.

    In all, this means that a comparison of Russian and Estonian cannot isolate future marking from any other difference between these very different languages. A good experiment is one that can isolate the causal factor of interest by varying it while holding everything else equal in expectation, and that is very hard to do when comparing languages. If you want to argue that grammatical tone affects how we hear music, an experiment would need to compare a tonal language like Mandarin with a version of Mandarin that is structurally identical but which has no tones. Such comparisons are impossible because no pair of languages in the world is identical except for exactly one linguistic feature.

    Separately, the claims of linguistic relativity become very odd when applied to fluent bilinguals. Remember the premise: language affects one’s social reality. So what social reality to fluent bilinguals inhabit? By assumption they operate with full competence in both realities, or they wouldn’t be fluent. So what is it that we learn from the observations that bilinguals differ based on what language they happen to be speaking? It must be that the language that one speaks in real time affects thought in real time. That is a Slobin-compliant position (see above)—but Slobin is not the theoretical foundation for Whorfian socioeconomics!

    The only truly dispositive experiment would be the completely unethical experiment which randomly assigns babies to be raised in controlled conditions where they learn languages that are manipulated to vary by one dimension only. Not only is this unethical and practically impossible, it also raises questions about in what sense such a child would be a fully-formed social individual in our normal understanding of what makes us social beings. Tragic stories of so-called feral children show us the implications.

    Returning to the point at hand, though: the third source of evidence used in Whorfian socioeconomics is a mix between experiments and aggregate data: crossnational surveys that record lots of features of people’s lives as well as the languages that they speak, for lots of people all over the world. If you can code all the world’s languages as to whether they possess a particular linguistic feature or not, then you can generalize across languages that vary in lots of ways to isolate the relationship that you care about, net of other differences across languages that also may vary.

    The problem in this case is that statistical modeling decisions are paramount, and in my article, I show that the standard methodologies used when authors make such arguments are flawed. Specifically, they are very likely to uncover spurious statistical associations that look like evidence that linguistic features explain lots of variation across human populations. Some of you might enjoy learning just how easy it is for such an analysis to falsely discover completely outlandish linguistic effects:

    I also review some simple statistical fixes that are nearly costless to implement but which address the problems inherent in such analyses. Employing these tools, I can replicate that Ted Radio Hour/American Economic Review article and show that its results do not hold when one accounts for uncertainty properly.

    This Debate Matters

    Clearly, linguistic relativity remains an important area of research, and Whorfian socioeconomics will almost certainly continue to be of interest to social scientists interested in cataloguing the social, political, and economic correlates of human language variation. But what I did not cover in my Language article are the normative implications of Whorfian socioeconomics and linguistic relativity.

    Practically, of course, we should want to get the science right. Linguistic relativity might explain some subtle differences across speaker communities. But because these effects are probably very small, it should difficult to find them, and social scientists should accordingly start by engaging more with the voluminous post-Chomsky linguistics literature to see why linguists are so skeptical of these ideas.

    But hold that aside. Normatively, I think this it is dangerous to argue that language affects our social world.

    For reasons why, you can look back to the use of linguistic relativity to justify the practices of colonialism and imperial race science. Arguments that have been used to view certain peoples as primitive, and their languages as representing a barrier to their entrance into modernity, will continue to be used in this way—dressed up in new garb, applied to new problems, but with the same erroneous presumption that language constrains thoughts in socially meaningful ways.

    The best articulation of this position is a fantastic (and underappreciated) book by John McWhorter entitled The Language Hoax. He argue that what he calls “Neo-Whorfianism”—which I see as the general theoretical idea that undergirds Whorfian socioeconomics—is not only plainly very wrong, but dangerous. Here’s a link to a long lecture by McWhorter that summarizes his argument, and here’s in illustration from his introduction:

    Take the intransigent ultranationalist German historian Heinrich von Treitschke. Prussophile, xenophobic, and nakedly anti-Semitic, he was given in the late nineteenth century to insights such as “differences of language inevitably imply different outlooks on the world.” You can imagine the kinds of arguments and issues he couched that kind of statement in, and yet the statement itself could come straight out of Whorf, and would be celebrated as brain food by a great many today. “Surely,” after all, “the question is worth asking…”—yet somehow, we would rather von Treitschke hadn’t, and find ourselves yearning for thoughts about what we all have in common.

    And that is the point. Language really is one of the only things that makes us all equally and equivalently human. Language is, for many working in the philosophy of mind, the quintessence of what it means to be human. The languages that we speak may differentiate the people with whom we can communicate, but we are united in that we all use language to do so. It is entirely certain that a speaker of Guugu Yimithirr could convey to me the meaning of a sentence like “my mother’s uncle who is seated on my left has just seen an airplane that reminds him of a crocodile,” which is a sentence that no one in human history has ever uttered, and yet we can both fully comprehend. And so can you, no matter who you are or where you live!**** That is miraculous.

    There are no objectively good and bad languages, nor strong and weak languages. Nobody’s thoughts are crippled by the language that they speak, and nobody’s reality is fuller because their language makes it so. As humans, we should root against Whorfian socioeconomics because it promises that our languages imprison us in realities that divide us. The good news is that the science doesn’t support that.

    Notes

    * We used to have more: you versus thou (nominative) and thee (accusative).

    ** I don’t think that “the French are assholes,” but I remember this very clearly because this family friend was so adamant that pronouns were part of it.

    *** Yes, the demarcation problem is real, and it is spectacular. So too is the social nature of the scientific enterprise. Acknowledging and moving on.

    **** Attention Cornell PhD students who have noticed how I like to signal my approval of something by calling it “generative”: now you know where this turn of phrase comes from! It’s ultimately a Chomsky reference.

    ***** You may respond “but what if I’m from some community that has never encountered airplanes or crocodiles?” And I would respond “I’ve never seen a unicorn or a miracle either, but they can be explained to me.”

  • Elite Endorsement Experiments about COVID-19: Two Cheers for Null Results

    There is a lot of social science research about COVID-19, both in the United States and around the world. Much of it focuses on the role of elite messaging in affecting pandemic-related beliefs and behaviors. Most scholars believe—instinctively and intuitively—that elite messaging is a fundamentally important driver of behavior and public opinion about the pandemic. Speaking just personally, I believe this.

    But at the same time, a wealth of new research uses some of our best analytical tools to test the effects of elite messaging on public opinion about the pandemic. Specifically, this research uses elite endorsement experiments to test whether different elites giving the same message, or the same elites giving different messages, changes public opinion on COVID-19 or has any effect on COVID-19 related behaviors. And the stylized finding from all of this research is that experimentally-manipulated elite endorsements do not affect public opinion or behavior. These are null results: these papers generally find no evidence to reject the null hypothesis that elite endorsements have no effect on public opinion or behavior. Some of these papers also report evidence that this effect is, if anything, small (in other words, they reject the null hypothesis that the effect is large).

    To borrow a phrase from a colleague, elite endorsement experiments about COVID-19 are “a big nothingburger.”

    I can speak from some experience. I have been involved in three separate research projects that test the effects of elite endorsements on some COVID-19 related thing: this article with Shana Kushner Gadarian and Sara Wallace Goodman on partisan endorsements and COVID-19-related policy attitudes, this article with Nick Kuipers and Saiful Mujani on encouraging Indonesians to pray from home, and another one which is not quite ready for primetime. Three projects, three nothingburger sliders.

    And I am not alone. Here is a selection of articles and pre-prints that I pulled from Google Scholar:

    COVID-19 has compelled officials to institute social distancing policies, shuttering much of the economy. At a time of low trust in government and high political polarization, Americans may only support such disruptive policies if recommended by politicians of their own party. A related concern is that some Americans may resist advice coming from “elite” sources such as government officials, public health experts, or the news media. We test these possibilities using novel data from an April 2020 online survey of 1,912 Pennsylvania residents. We uncover partisan differences in views on several coronavirus-related policies. Yet overall, respondents report strong support for social distancing policies and high levels of trust in medical experts. Moreover, a survey experiment finds no evidence of more negative reactions to or less support for social distancing policies when they are advocated by elites, broadly defined. Instead, respondents over 65 prove more likely to adopt expert-advocated positions.

    from Bhanot and Hopkins

    Public health communication play an important role in the fight against COVID-19. We used five well-established psychological levers to improve on the efficacy of two posters used by the French authorities (one on protective behaviors and one on proper handwashing). The five levers were: simplification (streamlining the posters), sunk costs (emphasizing the costs already paid to fight the pandemic), morality (emphasizing the duty to help others), self-protection (emphasizing the personal risks), and disgust (pointing out and illustrating that not following the protective behaviors or proper handwashing had consequences that should trigger disgust). We tested on a large (N = 3000) nationally representative French sample whether versions of the posters using these levers, compared to a control condition, were clearer, better recalled, and increased people’s intention to follow the posters’ recommendations. On the whole, there were no effects of the manipulations on any of the measures. The only consistent pattern was that the control protective behavior poster was better recalled than the alternatives (except for the simplified version), possibly because it contained one fewer message. The lack of effect on behavioral intentions might be attributed to the potential saturation in terms of health communication at the time of the experiment. Our results–mostly null and potentially negative–confirm the importance of testing interventions before using them in a public health campaign, even if they are grounded in successful past interventions.

    from Hacquin et al

    Our expectation was that respondents in the treatment groups would favour, or disfavour, the incumbent and assign blame to government for the pandemic compared with the control group. We observe no such results. Several reasons may be adduced for this null finding. One reason could be that public health is not viewed as a political issue. However, people do think health is an important policy area (>85% agree) and that government has some responsibility for health (>90% agree). Another reason could be that people view public health policies through partisan lenses, which means that health is largely endogenous, and yet we find little evidence of polarisation in our data. Alternatively, it could be that the global nature of the pandemic inoculated politicians from blame and yet a majority of people do think the government is to blame for the spread of the pandemic (~50% agree).

    from Acharya et al

    Media critics frequently complain about the tendency of reporters to cover political news using partisan conflict or partisan game frames, which describe policy disagreement as sites of partisan conflict where the parties can score “wins” or “losses.” Such frames, thought to decrease trust and increase partisan polarization, may be particularly dangerous when used in the coverage of public health crises such as the COVID-19 pandemic. We report a survey experiment where 2,455 respondents were assigned to read coverage of the pandemic that was framed in non-partisan terms, in terms of partisan conflict, or as a game where one party was winning and the other losing. Contrary to expectations, we find no effect of these frames across a broad range of opinions about and actions related to the pandemic, with the exception of a small negative effect of partisan game-framed coverage on the desire to consume news about the pandemic. These results suggest that partisan framing may not have negative effects during a public health crisis or, alternately, that such effects are difficult to detect in real-time using traditional survey experiments.

    from Myers

    Does emphasizing the pandemic as a partisan issue polarize factual beliefs, attitudes, and behavioral intentions concerning the SARS-CoV-2/COVID-19 pandemic? To answer this question, we conducted a preregistered survey experiment with a “questions as treatment” design in late March 2020 with 1587 U.S. respondents recruited via Prime Panel. Respondents were randomly assigned to answer several questions about then-president Donald J. Trump and the coronavirus (including receiving an information cue by evaluating one of Trump’s tweets) either at the beginning of the survey (treated condition) or at the end of the survey (control condition). Receiving these questions at the beginning of the survey had no direct effect on COVID-19 factual beliefs, attitudes, and behavioral intentions.

    from Spälti et al

    These are just the first such experiments that I found, but uniformly, the headline result is no effect of the elite endorsement treatment.

    Now, this is not a proper or complete review of the literature, nor is this a meta-analysis. It could be that these papers are not representative of the full universe of results from COVID-19 endorsement experiments. But there are enough elite endorsement experiments with null results out there to ask, what is going on?

    Two Cheers for Null Results

    It is mostly good news that we are learning about so many of these nothingburger elite endorsement experiment results. It reflects a positive development in quantitative social science: the dissemination of null results. I do suspect that there are still a number of papers out there that will not be published because there are null results, and indeed some of these abstracts above are from pre-prints that have not yet, as far as I can tell, be accepted for publication in a peer-reviewed journal. But that so many null results are being published marks important progress in the social sciences’ ongoing battle against the so-called file-drawer problem, in which findings that do not pass an arbitrary statistical significance threshold are not accepted for publication, so that the observed distribution of published findings are not representative of what we have learned.

    But the news is not all positive. I think it is now appropriate to ask what we are doing here with all of these elite endorsement experiments, and where do we go from here.

    Rarely the Question is Asked, Is Our Discipline Learning?

    If we are serious about combating the file-drawer problem, the goal of publishing null results is not simply to establish that some experiments were run that did not produce the anticipated effects of our interventions, but rather to learn from the results of those experiments. The goal, in other words, is to reason from the results of those experiments to the broader class of substantive problems that we wish to understand through experimental manipulations.

    So let us construct a world in which we have run 10 experiments that are designed to test whether something about elite endorsements affects COVID-19 health behaviors, and each of the 10 experiments returns a null result. What do we conclude? Let us consider two starkly different possibilities.

    1. Elite endorsements don’t matter
    2. Elite endorsements matter, but elite endorsement experiments do not show how they matter

    There are other possible answers too, but let us consider these two as the two most extreme things that one might reasonably conclude from our 10 null-result elite endorsement experiments. When I look at all the null results, what should I conclude?

    You can probably anticipate where I’m going. The results of the experiments themselves cannot differentiate between (1) and (2). But more pointedly, I am starting to believe that these elite endorsement experiments cannot do this specifically because they are not strong or pointed enough to be dispositive of anything. They certainly cannot falsify any theory (my own experiments included). I continue to believe (2), that elite endorsements matter, and if I am really and truly honest my own account of how elite communication matters is almost entirely unchanged by the fact that my coauthors and I have found no evidence that there is no difference in beliefs or behaviors in response to manipulating the elites who endorse a particular pro-social health behavior.

    So what is the research program of elite endorsement experiments on COVID-19? Imre Lakatos has an answer: a degenerate research program.

    Let’s quote the Stanford Encyclopedia of philosophy on research programs:

    What is it for a research programme to be progressive? It must meet two conditions. Firstly it must be theoretically progressive. That is, each new theory in the sequence must have excess empirical content over its predecessor; it must predict novel and hitherto unexpected facts. Secondly it must be empirically progressive. Some of that novel content has to be corroborated, that is, some of the new “facts” that the theory predicts must turn out to be true. As Lakatos himself put the point, a research programme “is progressive if it is both theoretically and empirically progressive, and degenerating if it is not”. Thus a research programme is degenerating if the successive theories do not deliver novel predictions or if the novel predictions that they deliver turn out to be false.

    This is actually still rather too optimistic a take, though. I don’t think this work has delivered novel predictions, yes, but I don’t think that any theory’s predictions have even been shown to be false! The null results from elite endorsement experiments are vacuous.

    My hardest interpretation of elite endorsement experiments is that they are games. The game is, can I use this methodological tool (survey experiment) to find evidence that is consistent with an interesting conclusion? No one takes the methodology seriously enough to believe it is an honest test of any theory, but it is a methodology that produces unbiased estimates of causal effects, and we do favor those.

    Lots of Games Out There

    I reach an analogous conclusion thinking about the results of a prominent recent audit experiment, published in the high-profile Proceedings of the National Academy of Sciences of the United States of America, that found that recipients of an email inviting them to take a survey were less likely to do so if they came from a sender with a common Black name* than if they came from a sender with a common White name. This is framed as a test of the existence of what the authors term “paper cut discrimination,” a kind of widespread everyday racial discrimination that the authors say “is exhibited by all racial/ethnic subgroups—outside of Black people themselves—and is present in all geographic regions in the United States.”

    I have been critical of this article because there is more variation among the selection of “Black names”** than there is between White- and Black-names, as I show in the plot below:

    I think that readers’ conclusion would be very different had they presented this result.

    But forget that point. Is this test of everyday discrimination strong enough for me to learn from a null result? Not at all: under no circumstance would I look at this figure that I just produced, and conclude that paper cut discrimination doesn’t exist. Instead, it is neat that the authors are able to use this well-respected tool to uncover evidence of paper cut discrimination. They won the game.

    Where is Progress?

    Where to go from here? My hunch is that the discipline of political science needs a reckoning with elite endorsement experiments. Certainly we should start talking about just how many COVID-related null effects results there are out there.

    But we should also take stock of whatever research program we think these experiments are contributing to, and try to design experiments that have greater substantive and theoretical stakes. An experiment that matters is one that changes what you think when you see any result, not one that confirms what you believe when you see positive results.

    NOTES

    * I really don’t like the “Black-sounding name” phrasing. The authors are very good about explaining that what they have done is to identify names that are statistically more common among Black versus White Americans. But in the end, the internal logic of the study relies on the belief that a survey respondent infers the racial identity of the sender from the name alone.

    ** “Black name”, again, is just really an awkward phrasing. Andre Jefferson is not a Black name in any objective sense. It is a name that, in the U.S. Census, is statistically more common among Black Americans than among White Americans. The same in reverse with Nicholas Austin.