Yesterday on Twitter, David Siegel stated what he considered an unpopular opinion: that “puzzles” are overused as a way to motivate or frame political science research.
Unpopular opinion: There is entirely too much focus on puzzles as motivating social scientific research questions. Things are puzzling if they surprise us, but surprise is both endogenous and context-dependent. We should be trying to understand the world, period.
— David A Siegel (@DaveASiegel) February 6, 2019
As it turns out, this opinion is plenty popular, as a quick look at the ensuing discussion will confirm. The problem is that most self-described puzzles are not puzzling, at least in the eyes of those consuming or evaluating the research. A puzzle is only puzzling relative to what the audience knows about it, or relative to the tools that the audience has to solve it. If I tell you that it’s puzzling that Singapore has such a high GDP per capita for a postcolonial country, this will only prove satisfying if you don’t have a good explanation for economic development in the postcolonial world or a good understanding of Singapore. If you already do have a good explanation for that, or a good understanding of Singapore, then the puzzle is not a puzzle.
And if your argument for why the audience should care about your research relies on it being puzzling, there’s no backup plan if it’s not.
For all these reasons, the puzzle can be risky as a motivation or framing device.* The best puzzles in social science are those research questions about which we know almost nothing, and where we plainly have no theoretical tools to make sense of them. Those are bound to be rare. If you describe some phenomenon as puzzling simply because one theoretical paradigm doesn’t account for it, well, then you are bound to learn that an audience familiar alternative theoretical paradigms is annoyed. “Puzzling for a realist” or “puzzling for an Americanist” are not good looks.**
But the pile-on against puzzles that has followed David’s tweet has missed, I think, the point of the puzzle as a framing device. Puzzles are not features of research projects, they are rhetorical devices that we use to motivate research or to draw in an audience. Importantly, puzzles are rhetorical devices that are particularly useful for drawing an audience to exploratory, qualitative, and country- or region-specific research. Push too hard against puzzles, and you risk pushing such research to the margins.
Consider this argument about puzzles by comparing the other ways that we motivate social science research.
- We don’t know how common a phenomenon is around the world (or around the country).
- We have not looked at this relationship yet.
- We don’t know what the structure of this thing is.
- We don’t know if this theory makes sense.
- Theory A and Theory B make different predictions about An Important Thing, and we ought to see which one works better.
- We don’t have a credible research design that tells us the answer to this causal question.
Each of these has value, but each also recommends a fairly normal science approach to formulating research questions. Collect data, test hypotheses, formulate concept or measurement, specify theory, evaluate predictions, estimate causal relationship. I take it as a given that these are good things that should lie at the core of social science. But they presume that we know what there is to know and the only issue is to do the social science.
I hold the view that we also can learn from the exceptions, oddities, and unremarked-upon patterns that characterize the social world but lie outside of what we already know to look for. The puzzle framing is a rhetorical device that legitimizes exploratory research by placing in the context of what we do know and promising that we can know more if we look elsewhere. It has affinities for qualitative research—which I defined here, abusing terminology just a touch, as research that focuses on uncovering causal processes rather than characterizing statistical patterns—because it implies that the historical or contextual details will provide clues that have generalizable implications.
Framing the results of such research in terms of a puzzle is a way to relate it to the existing body of social science research.*** It also clarifies—helpfully—that we have not learned much from exploratory, qualitative, country- or region-specific research if it just confirms what we already know.**** The puzzle framing, so conceived, is a bridge between this kind of research and social science research in the normal science mode.
A brief illustration may help to make the point clear. Benedict Kerkvliet‘s landmark book The Huk Rebellion tells the story of rural peasant revolt in the Philippines. One way to justify why one ought to read this book is to lead with the findings that the revolt emerged from the breakdown of traditional social relations that followed land reform. Another way to justify it is to remark that our existing theories of peasant rebellion do not accurately describe how this rebellion emerged; relative to what theoretical tools were available to understand peasant revolts, the details of the Huk Rebellion were puzzling. Although Kerkvliet does not use the word puzzle, he follows the latter strategy, and that is why the book is not just relevant to scholars of the Philippines or of land reform.
There are two takeaway messages from this discussion. First, puzzles are one way of motivating research, not the only way. I suspect that many if not most social science research problems are best framed otherwise. I have listed six options above, each of which is probably less likely to raise hackles.
Second, when motivating a research problem as a puzzle, the task must be to ensure that we really do not have the tools to make sense of it yet. If it turns out that intensive study of the puzzle shows that it has an easy solution, then it’s time to find another framing. In turn, when consuming research framed as a puzzle, think through the objective of the framing as much as whether you are clever enough to solve it before the author does.
* PhD students are welcome to confirm how frequently I say “that’s not puzzling.”
** Saying this amounts to saying “I have read incompletely on this research question, and I’m about to use a jerry-rigged theoretical apparatus to tell you what you already know. Either that, or I’m just going to steal your ideas.”
*** There might be other ways that researchers invoke puzzles, as a shorthand for “unknown thing” or “theoretical contradiction,” but I want to be fair to the more provocative use of the term.
**** This point has implications that I suspect that not all critics of normal-science or quantitative social science will want to endorse. Qualitative, exploratory, case-based research does not contribute much to social science unless it tells us something that we don’t already know. Now think through the implications for the qualitative researcher: should we should only observe qualitative research that produces novel findings that solve puzzles? How to square this with results-free peer review (PDF)?