The Washington PostDemocracy Dies in Darkness

When political scientists practice experimental political science …

Montana officials said Thursday, Oct. 23, 2014, that they are investigating whether mailers like the one shown violated laws by appearing to come from the state. The mailers are part of a Stanford University and Dartmouth College political science research project, and they rate how liberal or conservative the four nonpartisan candidates for Montana’s state Supreme Court are. (Matt Volz/AP)
Placeholder while article actions load

Talking Points Memo’s Dylan Scott broke a story about a political science field experiment gone somewhat awry:

Political scientists from two of the nation’s most highly respected universities, usually impartial observers of political firestorms, now find themselves at the center of an electoral drama with tens of thousands of dollars and the election of two state supreme court justices at stake.
Their research experiment, which involved sending official-looking flyers to 100,000 Montana voters just weeks before Election Day, is now the subject of an official state inquiry that could lead to substantial fines against them or their schools. Their peers in the field have ripped their social science experiment as a “misjudgment” or — stronger still — “malpractice.”

Read the whole thing to see what exactly went wrong. The most obvious problem is the “official-looking” part of the flyer, which uses the state seal of Montana and therefore does apparently violate Montana election law. Beyond that, however, let’s see what these political science “peers” quoted in Scott’s story have to say about this experiment:

[O]ther political scientists consulted by TPM described the study as “malpractice” and “improper and unethical” because, by introducing the ideological position of non-partisan candidates, the flyers could — intentionally or not — influence the results of the elections.
“It’s basically political science malpractice. That’s what I’d call it,” Jennifer Lawless, professor of government at American University in Washington, D.C., told TPM. “When you’re going to engage in an experiment as a political scientist, I think you have a responsibility not to affect election outcomes, let alone break the law.”
“When you can conduct an experiment in a real-world setting or amidst a campaign, it does add generalizability and real-world components to your results,” she continued. “That said, there is a difference between trying to have generalizable results and playing electoral god.”
“This strikes me as a lapse in judgment. If the election’s actually happening, they’re intervening in it,” Theda Skocpol, a political scientist at Harvard University, told TPM, though she cautioned that state authorities “ought to take a deep breath” before pursuing legal action.
Jeffrey Tulis, associate professor of government at the University of Texas-Austin, told TPM in an email after being alerted to the study: “My initial reaction is that this quasi-experiment is improper and unethical.”

Now what’s interesting about those judgments is that it appears that Lawless, Skocpol and Tulis do not have any background whatsoever in conducting experiments. Skocpol’s statement in particular seems rather odd — how exactly would one conduct a field experiment on an election that is not happening?

When one looks to see what political scientists who conduct experiments think about the Montana case, the responses are a bit different. This Math of Politics post gets at the basic question:

The most interesting criticism of the study rests on concerns that the study itself might have affected the election outcome.  The presumption in this criticism is that affecting the election outcome is bad.  I don’t accept that premise, but I don’t reject it either.  A key question in my mind is whether the intent of the research was to influence the election outcome and, if so, to what end.  I think it is fair to assume that the researchers didn’t have some ulterior motive in this case.  Period.
That said, along these lines, Chris Blattman makes a related point about whether it is permissible to want to affect the election outcome. I’ll take the argument a step farther and say that the field is supposed to generate work that might guide the choice of public policies, the design of institutions, and ultimately individual behavior itself.  Otherwise, why the heck are we in this business? (emphasis added)

For more experimentalist takes, see Thomas Leeper, Melissa Michelson and Benjamin Lauderdale.  As some of them observed, the irony of this complaint is that the political scientists were doing something that any advocacy group would do to try to influence voters. As Blattman notes, “my hunch is that it’s the fact that the academics are the interveners that makes ‘other political scientists’ question the appropriateness.”

The deeper irony here is that political scientists appear to be damned if they do and damned if they don’t conduct experiments. In the absence of experimental methods, the standard criticism of political science is that it’s not really a science because of [INSERT YOUR PREJUDICE OF CHOICE AGAINST THE SOCIAL SCIENCES HERE]. The presence of experimental methods, however, threatens to send critics into a new and altogether more manic forms of “POLITICAL SCIENTISTS ARE PLAYING GOD!!” panic.

Unfortunately, because there is the cut-and-dried issue of misusing the state seal, this is not an ideal case to use to defend the nature of experimental methods. Even defenders of this kind of experimental method will have to acknowledge that Mistakes were Made. But the deeper debate about political scientists running experiments will not go away. This is the new fad in the field. Provided the appropriate ethical guidelines are adhered to, this is a kind of political science that should be encouraged and not pilloried — and I say this as a non-experimentalist.

Am I missing anything?