The new editorial team of the Journal of Politics–one of the premier disciplinary journals in the field of political science–has announced that going forward, all experimental research submitted to the journal must be pre-registered:

Pre-registration: authors who want to submit manuscripts containing original experimental work, including laboratory, field, and survey experiments are required to submit proof of study/design pre-registration with one of the available research registries (e.g., EGAP, RCT, Open Science). Pre-registration of other types of research design is very much encouraged.  The submission of unregistered laboratory, field, and survey experiments will not be accepted.  This policy will be phased in: For manuscripts submitted in 2021, authors need to justify in a letter to the editor why the study was not or could not be pre-registered.

There’s lots of chatter on Twitter about the implications of such a policy. In my view it runs the risk of creating a series of inequities that will make it harder for those without resources to publish experimental research, especially given how common follow-up studies are.

Setting this particular issue aside, I have two general concerns.

The first is, I am also unclear as to why this policy would be obligatory for experimental research and not for non-experimental research. I say this as someone who has 2.3 million observations from the Indonesian census sitting on my computer (through IPUMS). You want an argument that having a phone line increases your propensity to speak Indonesian, and that that effect varies by the number of family members living in your household? I can cook up those significance stars for you in 20 seconds. If we truly believe that we need to police p-hacking through obligatory pre-registration, then I do not understand the substantive argument why this would be obligatory for experimentalists but not for those working with observational data.

Now, there may be a practical argument here that pre-registering observational studies is impossible, because we can never verify that pre-registration was done before seeing the data (except for in rare circumstances such as this one). But that’s not even correct! This practical objection to pre-registering observational research only applies to the analysis of secondary data. Why not insist on pre-registration of survey data analysis? Elite interviews?

The second concern that I have is that we actually do not have a common disciplinary understanding of what constitutes a pre-registration. I will not link to examples here, but I have been shocked by what editors believe “counts” as a pre-registration, and the inequalities that emerge as a result. A vague statement that “we will collect the data, and then we will test the hypothesis that X causes Y using regression” suffices for articles that currently appear in top disciplinary and general science journals. Contrast that the “earnest/completist” version of pre-registration that many of us follow, in which we announce not only the hypothesis but also the coding rules and statistical analyses, even providing the actual computer code that we will run once the data is there.

Insistence on pre-registration for experiments pushes us back to the antecedent question of whose standards must be followed for ascertaining that a study has been pre-registered. It introduces opportunities for editorial and review discretion as a result. Is my incentive as an author to pre-register only the main analysis, and then to announce that any subgroup/heterogenous treatment effects analysis that I might cook up later is exploratory? Will I get the benefit of the doubt from the referees if I just say that? Will a PhD student get the same benefit of the doubt? Who gets to say how much exploratory research is too much?

Surely there are other reactions out there to this particular editorial policy of requiring that experimental research only be pre-registered. But these two jump out immediately to me as reasons to be careful in implementing prospective rules about how research must be conducted.

Stepping back, I generally find efforts to implement hard and fast rules to discourage p-hacking and p-fishing to be misguided. These problems are hard to solve, but I do not know a model of the scientific process that works through rigid pre-registration standards (a point I make here). I’d prefer to embrace a Bayesian approach to how we evaluate research when p-fishing is possible, a topic I first touched on here but which Andrew Little and I addressed formally here. In the JOP!

P.S. In the course of reading the Twitter chatter about the JOP‘s new editorial team, I saw some criticism of the new team for lacking an Associate Editor who covers political theory. I think it a terrible mistake that a premier disciplinary journal would not have a political theory editor. Hopefully I’m misunderstanding this situation, or that it will be rectified quickly.

The United States and Indonesia both find themselves in an unenviable spot at the moment: confronting existential threats to liberal democracy. In the Indonesian case, the hardline Islamist group Front Pembela Islam (Islamic Defenders’ Front) has recently been banned as an organization by the Jokowi administration. FPI has been responsible violence against Indonesia’s non-Muslims, against liberal and progressive groups, and others. In the US case, a riotous mob of Trump supporters invaded and occupied the Capitol yesterday, causing death, widespread destruction of property, halting the work of the Congress, and shocking the world.

These are obviously not the same problem: one is a bunch of hardline Islamists who reject the rights and liberties provided to all Indonesians under the country’s 1945 constitution, the other a bunch of MAGA/Proud Boys who honestly oppose the peaceful transfer of power via constitutional means. And yet they both force us to to grapple with a singular question: what is the proper way to respond to groups that not only violate the law, but do so in opposition to liberal democracy and the established constitutional order?

I have argued that Jokowi’s banning of FPI is mistake (here is an article in Indonesian that covers a Twitter thread). Not only will banning FPI as an organization not address the ideological support that FPI draws upon, not only will it create a narrative that the organization is being unfairly singled out, not only will it fail to address the structural preconditions that explain the emergence of groups like FPI, but it is also fundamentally illiberal. The approach that the Jokowi administration should follow—which is unlikely, for reasons that need not detain us here—is to use Indonesia’s criminal justice system to try FPI members for acts of violence and incitement, and to punish those convicted aggressively, publicly, and transparently. Although I’m unclear on the specifics on Indonesian law on organized crime, FPI as an organization could be handled as a kind of criminal syndicate.

It does not make me a “FPI defender” or “FPI denier” to make a case for using the criminal justice system to confront FPI rather than to treat it as a threat to the Constitution and national security and ban it for that reason. Indeed, it is precisely because I find the group so abhorrent that I urge Indonesians to find an effective, permanent, and public solution to the threat that FPI poses to all Indonesians as individuals, and to Indonesian democracy as a whole.

The same thoughts cross my mind as I consider how best to respond to the riotous Americans who attempted to stop the final Congressional certification of the 2020 presidential election. I do think that these events plainly call into question the American constitutional order and the peaceful transfer of power.* I want to know why those who stormed the halls of government and threatened the lives of the Vice President and Members of Congress (to say nothing of the staffers and workers there) were not met with live fire on the Capitol steps.** Everyone who downplayed the threat of Trump and Trumpism for American democracy needs to take a sober look at American politics this morning, and listen to all of us who did.

And yet I am uninterested in classifying yesterday’s events as a coup, rather than a riot, a coordinated mass action, or something else. Although my thoughts are not yet settled, I support an approach that views these acts as criminal rather than treasonous (even if I believe that they probably are treasonous). Describing the rioters as treasonous coup-makers feels satisfying in my gut, as capturing what I feel about what I saw, but it has the effect of recognizing their demands even as it rejects them.

Americans should not look at the pathetic assembly of rioters as a mass of wannabe John Browns and Patrick Henrys, and Americans should reject any framing of the events that suggests as much. There should be no outs, even suggesting that their actions had legitimate political standing according to some perspective. The response should be to identify each and every participant, charge them with the crimes for which they are responsible, and try them in a court. And the current president ought to be removed from office via constitutional means for encouraging their behavior.

Postscript

An Indonesian friend shared this on Facebook last night: a comparison of the 1998 students’ occupation of the People’s Consultative Assembly (top), and the 2021 Capitol riot (bottom).

Note

* I don’t think the outcome of a Biden presidency was ever in doubt, even in the very worst moments when guns were drawn.

** I am not exaggerating.