Tom Pepinsky

Category: Language

  • Uncooperative Survey Experiments

    Kieran Healy recently tweeted that

    The tweet references a new paper by Allan Dafoe, Baobao Zhang, and Devin Caughey that shows that survey respondents,

    when presented with information about one attribute, update their beliefs about others too. Labeling a country “a democracy,” for example, affects subjects’ beliefs about the country’s geographic location.

    This is a problem because if we want to estimate the effect of attribute T on some survey response by randomly presenting some subset of respondents with different values of T and holding everything else about the survey prompt constant, it better not be the case that respondents change their beliefs about other features of the survey prompt depending on what particular value of T they encounter.

    Healy’s tweet references the linguist Paul Grice‘s notion of implicature, which reminds us that what we say does not nearly capture what we mean and underlies the entire subfield of linguistics called pragmatics.

    Implicature serves a variety of goals beyond communication: maintaining good social relations, misleading without lying, style, and verbal efficiency.

    I do love a good linguistics reference applied to a political methodology problem, and it strikes me that thinking more broadly about what Grice called the “Cooperative Principle” can help us to put a finger about what can seem so artificial about survey experiments.

    The Cooperative Principle says contribute what is required by the accepted purpose of the conversation. There are various different subparts of this principle, which Grice termed the Maxims of Quality, Quantity, Relation, and Manner.

    1. The maxim of quantity, where one tries to be as informative as one possibly can, and gives as much information as is needed, and no more.
    2. The maxim of quality, where one tries to be truthful, and does not give information that is false or that is not supported by evidence.
    3. The maxim of relation, where one tries to be relevant, and says things that are pertinent to the discussion.
    4. The maxim of manner, when one tries to be as clear, as brief, and as orderly as one can in what one says, and where one avoids obscurity and ambiguity.

    When we take part in everyday conversation, we naturally try to accomplish these things, and we notice when people do not. For example, when someone overexplains something (testing the maxims of quantity, relation, and manner) it can make us suspicious. When someone responds to the question “how good is this cheeseburger?” with “it does look really tasty” (flouting the maxim of quantity) we are apt to conclude that the cheeseburger is likely not very good. It’s helpful to think about how humor works by deliberately flouting some of these maxims.

    Apply this, then, to survey experiments, such as priming or information or conjoint experiments. Most survey questions are designed to elicit truthful responses by appearing as natural as possible. Survey experiments make it difficult to do this. For a good illustration of how, I will use myself as an example. In 2012 I published a paper with Bill Liddle and Saiful Mujani in which we used a survey experiment to tease out why Islamist parties in Indonesia are more popular than non-Islamist parties. Our survey prompt was

    If there were a candidate for president from a Pancasila‐based party/Islamic party wishing to implement Islamic law, and you believed that/were unsure if that party’s economic policies would/would not develop our economy and increase the welfare of the people, would you vote for him or her?

    The structure of our question allowed us to separate party ideology from beliefs about economic platform, which was our goal. But imagine you are a survey respondent facing this question. I worried then—and I worry now—that our respondents would wonder about why we are explaining both of these things at the same time. If you assume (as a respondent) that we are being cooperative, then each of these dimensions is pertinent, and that is good for us. But it might be artificially highlighting a distinction that is not relevant if we were to have asked “do you support party X?” Worse yet, it might raise respondents’ suspicions that we are being cooperative at all, suggesting that “the accepted purpose of the conversation” (which the Cooperative Principle presupposes) is not shared by researcher and respondent. Which, to be fair, it was not.

    I think the Cooperative Principle every time I fill out a survey by political scientists in which I’m asked a question about my willingness to support, say, a female academic in Australia who supports the two-state solution and uses formal models.

    One way to think about the Dafoe et al. paper that started this discussion is to imagine that the goal of a survey experiment should be to fulfill the Cooperative Principle, and to think about ways in which the tools we use stand in the way of that goal.

  • Urbanization, Ethnic Diversity, and the Rise of Indonesian

    As part of a multi-year project on language shift in contemporary Indonesia, Abby Cohn, Maya Ravindranth, and I have been using the incredible census data provided by IPUMS to study what factors determine whether Indonesians speak Indonesian at home. The data are remarkable in that they comprise a 1% sample the 2010 Indonesian census—which means that our sample size is 2,358,774 individuals. And better yet, anyone can access these data.

    One thing that sociolinguists know is that urbanization leads to language shift in multilingual societies. In the Indonesian context, this means that speakers of ethnic languages like Javanese, Sundanese, Batak, and so forth will shift to speaking Bahasa Indonesia, the country’s national language.

    But what’s going on here? Is this a consequence of urbanization itself, and the accompanying process of “modernization” of everyday life that (1) expose you to media in the national language and (2) lead to shifting identities away from regional/ethnic to national? Or is it a consequence of the ethnic diversity found in urban areas, which lead speakers of different languages to encounter one another more regularly and thus increase the benefits of speaking a common national language? In principle these two processes are distinct: you could have urban areas without ethnic diversity, or rural areas that are highly diverse. The neat thing about Indonesia is that it is so big and heterogeneous that we have instances of urban and rural districts that are both homogenous and diverse. This allows us to distinguish the two effects from one another.

    Because we know the district (kabupaten or kota) in which every individual lives, and we know his/her ethnic group, we can calculate a district-level measure of ethnic diversity (using a so-called Ethnic Fractionalization index [PDF]). We also know whether or not each individual is classified as living in an urban residence or not, so we can use that to calculate the fraction of each district that is urban. Both of these measures range from 0 to 1. Comparing each of the 494 districts recorded in the 2010 census, here is what we find.

    The good news is just how varied Indonesian districts are. There are ethnically homogenous, wholly urban districts (Kota Blitar, on Java) as well as ethnically homogenous, entirely rural districts (Nias Barat, off the coast of Sumatra). And looking to the right side of this scatterplot, we see a range of incredibly diverse districts, all of which are on Papua, that range from highly urban to highly rural.

    From there, we fit a hierarchical/multilevel logistic regression model in which we predict whether or not an individual speaks Indonesian at home as a function of a range of individual-level characteristics (age and its square, gender, religion, education, etc.) as well as district-level urbanization, ethnic diversity, and their interaction. We then predict, based on the results of that model, the probability that an individual speaks Indonesian as a function of their district’s ethnic diversity and at the 10th, 50th, and 90th percentiles of district urbanization. Here is what we find.

    If you live in an ethnically homogenous district, the likelihood that you speak in Indonesian at home is very low, no matter how urban that district is. But as ethnic diversity increases, so does the likelihood of speaking Indonesian—and especially so in urban districts. This shows very clearly that the relationship between urbanization and language shift in a diverse country like Indonesia really does depend on whether or not urbanization comes with increasing ethnic diversity. And although the relationship between ethnic diversity and language shift is largest for urban districts, this relationship is substantively quite large in rural districts too.

    Note, though, that to reach such a conclusion, you need a really diverse country like Indonesia that allows you to separate urbanization from ethnic diversity empirically. Thanks, Indonesia.