The other day Roxani Krystalli shared a memo detailing her communication and negotiations with the US National Science Foundation on issues relating to data sharing and privacy in the context of research on violence in Colombia. The memo addresses a number of core issues that were part and parcel of the Qualitative Transparency Deliberations (QTD) several years ago. It is worth a read. It actually gives me hope about the responsiveness of our institutional funders to the actual concerns that political scientists have. But it is also worth a read for two other reasons: on the question of epistemology and transparency, and on institutionalizing rules in the production of knowledge.

On the first, as I wrote elsewhere, what I find striking about the memo and its discussion is that nothing in it is specific to an interpretivist epistemology. I would imagine that a positivist doing such research would have the same concerns and obligations. For example, on the question of anonymizing individual data:

These markers [name, title, institutional affiliation] are not the only aspects of identity that sketch the lives of my interlocutors. To illustrate, the first question I ask state officials who work within bureaucracies of victimhood is how they arrived at this position. My research to date has shown that their answers often consist of detailed accounts of the violence they and their families have observed or experienced in ways that would be identifiable even if I removed the research participant’s name, location, or professional title

Relatedly, on informed consent:

The range of disclosure requested by NSF would present three specific challenges to this process. First, for some of my interlocutors who have not had any formal education or access to the internet, a “data depository” is not a concept that would translate in their daily lives in a way that would allow them to meaningfully consent to this process. Second, for interlocutors who do understand the concept, it can bear strong connotations of state surveillance or surveillance by foreign governments. This perception would be exacerbated by the fact (which I would have to disclose to my research participants as part of the required funding disclosure in the consent process) that it is a US government grant that requires me to share data in this way.

There are other examples. These are not interpretivist problems, these are problems for anyone doing risky research in postconflict areas. They are certainly questions that I have pondered at some length, and I suspect that many others have as well.

It is on the issue of institutionalizing rules in the production of knowledge, though, that I find Krystalli’s memo even more telling. Contemporary political science is moving in a direction of greater post hoc policing of published research: replication archives, annotation for transparent inquiry, DA-RT, and so forth. There is real concern that social scientists need to create common disciplinary institutions to prevent fraudulent or plainly erroneous research from being published; or, in the case of related initiatives like pre-analysis plans, to mitigate the strategic incentives that authors face to produce certain types of findings. Much of this concern is genuine, and it responds to real problems.*

But there is a related move to ensure that political science remains a broad and inclusive discipline in which not just quantitative and experimental, but also historical, qualitative, ethnographic, and post-positivist research can be published. The logic of the replication archive is plain for a dataset; not so for field notes. The QTD initiative was an attempt to figure out if there was a way to put these together. Although I have come to believe that some proponents of initiatives like ATI wish to use these initiatives as a way to constrain the types of qualitative work that are admissible as “Real political science,” in my experience most genuinely want to find a way to ensure that other types of research are still possible.

The emerging solution seems to be something like an “opt-out” provision that allows the authors of qualitative, ethnographic, or other types of research to request that the established rule not apply to their specific research. That is, in effect, what Krystalli’s memo is.

Here is how these intersect. The establishment of a blanket standard—a rule—for analytic transparency that forces qualitative or ethnographic scholars to go through an appeal procedure to ensure that their work is not subject to that rule creates one more barrier to seeing such work published. Think of how much extra work that memo required! It also requires discretion—and good will and understanding—on the part of editors, funders, or other gatekeepers.

Well-meaning proponents of institutionalized rules who also seek to maintain a methodologically and epistemologically plural discipline should take note.

NOTE

* I am a big fan of replication archives for quantitative research, and I am also happy when I have to to create a replication archive for my own work. It enforces a discipline on analysis and coding that is annoying in the moment but welcome after the fact.