Tom Pepinsky

Author: tompepinsky

  • What is Implementation Bias?

    Via Marc Bellemare, an interesting paper (PDF) by Bold et al. on “scaling up” randomized controlled trials, with an application to educational reform in Kenya. The general problem here is that RCTs are a great way to identify the effects of particular interventions on particular outcomes, but armed with a finding that, say, “exposure to a contract teacher in government schools in Western Kenya raises test scores by 0.21 standard deviations relative to being taught by civil service teachers,” we often have a difficult time “scaling up” this outcome to the national level. The problems are both practical (how to get the entire country to do something?) and conceptual (should we expect that the Western Kenya findings are applicable across Kenya? To say nothing about Uganda, India, Bolivia, etc.). We call latter a question about external validity.

    Marc describes another problem, which he labels implementation bias:”with only one implementing partner, it is impossible to tell whether the success or failure of any intervention is due to the perception people in the treatment group have of the implementing partner.” I have no disagreement that that’s an issue, and Bold et al.’s paper is a great way to explore it, but I never thought about calling that implementation bias. To me, the term implementation bias conjures up two other problems.

    1. An intervention may only be possible in the very places where it’s most (or perhaps least) likely to be successful. If you want to generalize about the effects of an public accountability intervention in Yogyakarta, Indonesia, you ought to wonder if that intervention could have been implemented in the first place in Solo. This is related to the “OMFG exogenous variation” syndrome: we seize on findings that we can find using the techniques we prefer, and ignore ones that we cannot. The danger is that there is some feature of Yogya that allows public accountability to be manipulated and which is related to the outcome we want to measure.
    2. We may only implement things that are relatively uncontroversial to implement. Take the classic paper (PDF) by Miguel and Kremer on deworming. The authors find (once again in Kenya) that where “school-based mass treatment with deworming drugs was randomly phased into schools” there are large externalities to deworming: the benefits spill over to some children who did not experience the treatment. This is a great finding and a great paper, and we can think of its findings as evidence that public health interventions are investments in human capital, but I wonder how many minds it changed. Take something a lot more expensive and a lot more controversial—something like forced sterilization of people with low IQs, which I of course do not support—could we ever know if that would be a cost-effective investment in human capital? Probably not, because we can’t implement it. Between the two extremes of deworming and forced sterilization there are lots of public health interventions that are more or less controversial but which may or may not be good investments. We might only implement the ones that are probably the most likely to yield good outcomes.

    Are all of these things implementation bias? Or are there other proper names for them? I don’t know, and I suppose that it doesn’t matter substantively, but as an interested observer I’d like to know.

  • Three in One, Almost Done

    This is my job!

    It’s early May, and that means that this semester’s teaching is finally done. Phew. This has been the heaviest teaching semester that I’ve ever had: one 65+ person lecture plus two 15-person seminars (both of which are “new preps,” which means that I had never taught them before). I know that for most faculty, three courses in a semester is the norm: among the invisible professoriate three in one with two new preps is normal, even light. JMP once taught FIVE courses in one semester. But among regular faculty in research-oriented departments in research-intensive universities, it is not the norm.

    But I have a confession. Right now I may have the bleary-eyed look of Donald Sutherland in Animal House (and I actually do have that jacket) but unlike his Jennings, teaching isn’t “just a way to pay the bills until I finish my novel.” I like teaching, and I don’t care who knows it. Of course I don’t like everything about teaching, but I do like the basics: I like to talk about Southeast Asian politics to people who are eager to learn about it, and I like to debate basic questions about, say, why Asia’s material prosperity has grown over the past half century, or how we ought to go about learning how politics works around the world. On the whole, I’ve had a good time this semester. I’ll be glad to turn my attention to research again, but not because I haven’t enjoyed what I’ve been doing.

    The costs of teaching this much are that I have had scarcely any time to do any research or writing since January. I have, though, done a lot of productive thinking. I have a couple of ideas saved up on multimethod research, for example, that I’ll share here in coming weeks. Some of these ideas are going into a new paper on context and method in Southeast Asian politics, which I’ll be presenting in three short weeks at Uni Freiburg (this conference program can only be described as bad-ass…MacIntyre, Malesky, von Luebke, Shair-Rosenfield, Kuhonta, Tajima, and the great Emmerson too).

    However, before that happens, we have to have Slope Day here at Cornell. I have a couple of other Animal House images that I could post in honor of Slope Day, but I leave that to your imaginations.