Author: tompepinsky

  • Could Replication Have Prevented This?

    A much-publicized, very provocative article about the effect of gay canvassing on persuasive public opinion has been retracted. The allegations are serious: that the data was simply fraudulent. An incredible memo details the charges. Already, the public commentary is that the study was “fake.”

    I hesitate to conclude once and for all at such an early stage that this study was faked or fraudulent, although things are not looking good and the retraction letter by Green is damning. But if it the allegations hold, this will be a Stapel-level scandal for political science.

    As someone who is very interested in issues of transparency and replicability in social science research, I am keen to see what happens next. Already I have seen several comments on Twitter (e.g. this and this and this) suggesting that replication policies might have prevented such an outcome. I’m skeptical. Why? Because replication policies such as that of the American Journal of Political Science are designed to ensure that the numerical or graphical results in a published article can be derived from the raw data. They are not designed to do the kind of sophisticated exploratory data analysis needed to probe if the raw data itself is sound.

    That conclusion might be discouraging. But it does help us to focus our attention on just what replication can and cannot do.

  • Paul Romer and the Principles of Academic Engagement

    Paul Romer has written a couple of really biting commentaries on the contemporary state of economic theory (this and this). The enemy is “mathiness,” a term he employed—if not coined—in a recent paper in the American Economic Review Papers and Proceedings.

    Like mathematical theory, mathiness uses a mixture of words and symbols, but instead of making tight links, it leaves ample room for slippage between statements in natural versus formal language and between statements with theoretical as opposed to empirical content.

    There’s a lot more to this debate, and it will no doubt continue to unfold in the coming weeks. (And in the academic economics blogosphere, no less!) The debate among economic theorists has very little relevance for how to do contemporary political science research, but it will still be interesting to follow for anyone interested in the history of economic theory, the politics of macroeconomic policymaking, and so forth. I do agree that reading Romer’s interpretation of the point of Milton Friedman’s “The Methodology of Positive Economics” helps me to put it in a new light.

    What catches my eye, though, is how Romer describes the foundations of academic collaboration in a science of economics.

    a) We trust that what each person says is an honest account of what he or she thinks is true.
    b) We all recognize that reasonable people can differ and that no one has privileged access to the truth.
    c) We take seriously the claims of people who disagree with us.
    d) We are ready to admit that others might be right that each of us might be wrong.
    e) In our discussions, claims that are recognized by a clear plurality of members of the community by as being better supported by logic and evidence are the ones that are provisionally accepted as being true.
    f) In judging what constitutes a “clear plurality,” we put more weight on the views of people who have more status in the community and are recognized as having more expertise on the topic.
    g) We update the status of a members of our community on the basis of his or her contribution to progress a clearer understanding of what is true, not on the basis of “unwavering conviction” or “loyalty to the team.”
    h) We shun, or exclude from the community, someone who reveals the he or she is not committed to these working principles.

    I’m not sure I agree about the place of experts with status in the community: it puts a heavy burden on the experts to be self-critical in a way that I just don’t think is realistic. But in general this seems like a good set of principles. If we hold the commentary on experts and status aside, in fact, Romer’s principles can be simplified into just four.

    Modified Principles for Academic Engagement
    1. Mostly everyone is really smart
    2. Mostly everyone is really trying
    3. Mostly everyone is usually wrong
    4. All of these apply to you too, especially 3