Talking Points Memo’s Dylan Scott broke a story about a political science field experiment gone somewhat awry:
[O]ther political scientists consulted by TPM described the study as “malpractice” and “improper and unethical” because, by introducing the ideological position of non-partisan candidates, the flyers could — intentionally or not — influence the results of the elections.“It’s basically political science malpractice. That’s what I’d call it,” Jennifer Lawless, professor of government at American University in Washington, D.C., told TPM. “When you’re going to engage in an experiment as a political scientist, I think you have a responsibility not to affect election outcomes, let alone break the law.”“When you can conduct an experiment in a real-world setting or amidst a campaign, it does add generalizability and real-world components to your results,” she continued. “That said, there is a difference between trying to have generalizable results and playing electoral god.”“This strikes me as a lapse in judgment. If the election’s actually happening, they’re intervening in it,” Theda Skocpol, a political scientist at Harvard University, told TPM, though she cautioned that state authorities “ought to take a deep breath” before pursuing legal action.Jeffrey Tulis, associate professor of government at the University of Texas-Austin, told TPM in an email after being alerted to the study: “My initial reaction is that this quasi-experiment is improper and unethical.”
Now what’s interesting about those judgments is that it appears that Lawless, Skocpol and Tulis do not have any background whatsoever in conducting experiments. Skocpol’s statement in particular seems rather odd — how exactly would one conduct a field experiment on an election that is not happening?
When one looks to see what political scientists who conduct experiments think about the Montana case, the responses are a bit different. This Math of Politics post gets at the basic question:
The most interesting criticism of the study rests on concerns that the study itself might have affected the election outcome. The presumption in this criticism is that affecting the election outcome is bad. I don’t accept that premise, but I don’t reject it either. A key question in my mind is whether the intent of the research was to influence the election outcome and, if so, to what end. I think it is fair to assume that the researchers didn’t have some ulterior motive in this case. Period.That said, along these lines, Chris Blattman makes a related point about whether it is permissible to want to affect the election outcome. I’ll take the argument a step farther and say that the field is supposed to generate work that might guide the choice of public policies, the design of institutions, and ultimately individual behavior itself. Otherwise, why the heck are we in this business? (emphasis added)
For more experimentalist takes, see Thomas Leeper, Melissa Michelson and Benjamin Lauderdale. As some of them observed, the irony of this complaint is that the political scientists were doing something that any advocacy group would do to try to influence voters. As Blattman notes, “my hunch is that it’s the fact that the academics are the interveners that makes ‘other political scientists’ question the appropriateness.”
The deeper irony here is that political scientists appear to be damned if they do and damned if they don’t conduct experiments. In the absence of experimental methods, the standard criticism of political science is that it’s not really a science because of [INSERT YOUR PREJUDICE OF CHOICE AGAINST THE SOCIAL SCIENCES HERE]. The presence of experimental methods, however, threatens to send critics into a new and altogether more manic forms of “POLITICAL SCIENTISTS ARE PLAYING GOD!!” panic.
Unfortunately, because there is the cut-and-dried issue of misusing the state seal, this is not an ideal case to use to defend the nature of experimental methods. Even defenders of this kind of experimental method will have to acknowledge that Mistakes were Made. But the deeper debate about political scientists running experiments will not go away. This is the new fad in the field. Provided the appropriate ethical guidelines are adhered to, this is a kind of political science that should be encouraged and not pilloried — and I say this as a non-experimentalist.
Am I missing anything?