Since controversy erupted over a field experiment in Montana, the secretary of state of Montana has filed a complaint, the presidents of Stanford and Dartmouth have apologized, and the study has been widely critiqued by academics and journalists (see, for example here, here, here, here, here and here.)
Ironically, although this experiment has proved controversial, it is typical of dozens of similar experiments that have been run all over the world during elections. These studies are all considered field experiments: interventions conducted “in the real world,” often with citizens who are unaware that they are part of a study. These types of studies hold great promise due to their realism, as they can show how things work outside an artificial laboratory environment. But they also have raised questions about subjects’ rights and interference in democratic processes.
Two questions in particular are central to the current debate but also apply more broadly to all interventions in elections: First, is it ethical to run an experiment that may affect an election outcome, or some other feature of “the real world”? Second, is it ethical to conduct experiments where individuals do not know that they are research subjects and that their environment is being manipulated?
Let’s consider first the question of conducting a field experiment during an actual election. In our edited volume, Brigitte Zimmerman points out that successful interventions in elections always cause some harm, precisely because elections are zero-sum competitions. Any change in election outcome caused by a study means a candidate and his or her supporters are harmed. And for those of us in comparative politics, the idea that U.S. research money might change an election outcome overseas seems dangerous. At the extreme, this implies a Prime Directive principal of observational studies that don’t risk any direct impact.
On the other hand, there are strong arguments that we should not worry about impact. The goal of empirical science is, in fact, to do just that: impact the real world. Further, it is easy to argue that treatments that mobilize underrepresented groups or provide more information to voters are beneficial to democracy. As discussed in the introduction to our volume, the treatments in these experiments — political mailers or advertisements — are being used by many other political actors with the precise intent of having an impact, and their motives are often less altruistic than those of scholars. (John Patty and Chris Blattman have explored these issues further in recent blog posts.)
But the question of affecting elections has a practical side as well as an ethical one. Many citizens and politicians don’t like the idea of scientists experimenting during real elections. As a discipline of largely public servants supported by public money, it seems ill-advised to make our principals angry.
The second concern with field experiments is that subjects and bystanders often do not even know that they are in a study. This violates the principle of informed consent, which means that all subjects must be told about the nature, aims and risks of the study, and only after being informed, they must give voluntary consent. Historically, most of the great scandals of human research involve a lack of informed consent, including Milgram’s Obedience to Authority and the Tuskegee Syphilis study. For this reason, informed consent is required in almost every research study with human subjects. Scholarly experiments can skip informed consent only when the risks to subjects are minimal and the research could not be carried out otherwise (45CFR46.116.d) .
How are these two questions related in field experiments? Whether impact is an issue depends on whether subjects are informed and consenting. Suppose, hypothetically, that everyone in Montana had been invited to participate in a study in which they would receive some mailers, and suppose that they received a flier only if they had agreed to participate. I doubt that anyone would be upset if all the subjects knew they were in a study and all had consented to receive a treatment.
This suggests that the critical issue to resolve is not whether we are impacting the real world, but whether our subjects are aware that they are subjects. If we can resolve the consent question, we can worry less about the impact question. Unfortunately, fully informed consent might render some hypotheses difficult to test.
One possible solution is exploring other types of acceptable consent, as discussed in a nice paper from the conference by Macartan Humphreys. Humphreys proposes a series of alternative forms of consent, for example, superset consent: listing many possible treatments and having the subject consent to the entire set, not knowing which one of them will be used in the experiment. Another option is proposed by bioethicist Barbara A. Koenig, who has argued that in some cases, consent could be granted by a representative citizens’ body. A third possibility is part of federal human subjects’ regulations but is rarely used by political scientists: instead of seeking consent, simply inform all subjects after the study. Presumably scholars would anticipate subjects’ reactions and avoid controversy in experimental design.
I would like to suggest that there are three important lessons from the Montana incident that should be noted going forward. First, as a discipline we should engage these issues directly and work toward shared norms. The past several days have shown that political scientists have widely differing views on these issues. Yet many scholars do not want to even talk about ethics. This is unfortunate. The future of field experiments as a viable way of testing hypotheses about politics and political behavior depends on engaging directly with these issues. This is especially important because there are many other ethical issues emerging across experimental political science, some of which are examined in our volume. We need more dialogue and research on these issues to avoid harm to subjects and to avoid damaging the reputation and future of the field.
Second, as Vanderbilt University political scientist Liz Zechmeister argues in our edited volume, we can’t outsource ethical judgments to institutional review boards. IRB approval is neither a blank check nor absolution from sins past, present and future. IRBs may do their best (or worst), but scholars are the experts who know their cases and must take responsibility for keeping their research legal and ethical.
Lastly, we all need to acknowledge that when it comes to ethics and research, each and every one of us has a conflict of interest. There is always a tradeoff between protecting human subjects and executing our research agendas. Protecting subjects will almost always make research harder. There are strong career incentives to overlook rules and risks and push for bigger, more aggressive treatments that have a better chance of being published. It’s easy to convince ourselves that the costs to subjects are low and the benefits to society are high — even when they aren’t. Be careful out there.