A much-publicized, very provocative article about the effect of gay canvassing on persuasive public opinion has been retracted. The allegations are serious: that the data was simply fraudulent. An incredible memo details the charges. Already, the public commentary is that the study was “fake.”
I hesitate to conclude once and for all at such an early stage that this study was faked or fraudulent, although things are not looking good and the retraction letter by Green is damning. But if it the allegations hold, this will be a Stapel-level scandal for political science.
As someone who is very interested in issues of transparency and replicability in social science research, I am keen to see what happens next. Already I have seen several comments on Twitter (e.g. this and this and this) suggesting that replication policies might have prevented such an outcome. I’m skeptical. Why? Because replication policies such as that of the American Journal of Political Science are designed to ensure that the numerical or graphical results in a published article can be derived from the raw data. They are not designed to do the kind of sophisticated exploratory data analysis needed to probe if the raw data itself is sound.
That conclusion might be discouraging. But it does help us to focus our attention on just what replication can and cannot do.