A much-publicized, very provocative article about the effect of gay canvassing on persuasive public opinion has been retracted. The allegations are serious: that the data was simply fraudulent. An incredible memo details the charges. Already, the public commentary is that the study was “fake.”
I hesitate to conclude once and for all at such an early stage that this study was faked or fraudulent, although things are not looking good and the retraction letter by Green is damning. But if it the allegations hold, this will be a Stapel-level scandal for political science.
As someone who is very interested in issues of transparency and replicability in social science research, I am keen to see what happens next. Already I have seen several comments on Twitter (e.g. this and this and this) suggesting that replication policies might have prevented such an outcome. I’m skeptical. Why? Because replication policies such as that of the American Journal of Political Science are designed to ensure that the numerical or graphical results in a published article can be derived from the raw data. They are not designed to do the kind of sophisticated exploratory data analysis needed to probe if the raw data itself is sound.
That conclusion might be discouraging. But it does help us to focus our attention on just what replication can and cannot do.
tompepinsky.com 
dwayne woods May 20, 2015
Not surprising. I thought the study was stupid from the outset.
dwayne woods May 20, 2015
Touché: after finishing up a R&R using confirmatory factor analysis, I wondered why we don’t see more exploratory and confirmatory analysis of data.
dwayne woods May 20, 2015
Interestingly, Stapel’s book on himself as a fraud is a great read.
ingorohlfing May 20, 2015
I do not think your conclusion is discouraging. Publicly available data, code and documentation are necessary prerequisites for detecting “unusual” findings (whatever the reason is for the former Science paper), but not sufficient. As the documentation by Broockman et al. shows, it took a lot of effort to get an idea about how the data probably came about if not via genuine data collection. It is impossible to do this for all studies before publication, so the best way to go is not to overinterpret single studies (if only because we can have significant results in a single study by chance) and reproduce and replicate as much as possible.
dwayne woods May 21, 2015
I just do not see a lot of reproduction and replication taking place. What I come across is a lot of proverbial verbiage that previous studies made these claims and this is why they were wrong or missed the point. Here is my (new) theory and I’m going to test it against the data. Hence, we have “thirty” different explanatory variables across different studies purporting to explain the same dependent variable outcome.