The Washington PostDemocracy Dies in Darkness

Do presidential debates usually matter? Political scientists say no.

This was first posted last night, but we're bumping it in advance of tonight's debate. - Dylan

The first of three debates between President Obama and Mitt Romney airs tomorrow. History is littered with examples of debate performances that allegedly decided elections. There was John F. Kennedy beating Richard Nixon due to the latter's not-ready-for-prime-time scruff. There was Gerald Ford losing after asserting that the Soviet Union didn't dominate Eastern Europe. There was Ronald Reagan's "there you go again" comeback to Jimmy Carter, and Lloyd Bensten's admonition that Dan Quayle is "no Jack Kennedy."

But did any of those actually matter? The best political science says no.

The polls don't shift

Perhaps the most compelling evidence for the irrelevance of debates is that polling in past races hasn't changed much at all following them. John Sides, a political scientist at GWU and official friend-of-the-blog, summarized the research on this in the Washington Monthly. He notes that a study by UNC's James Stimson found few noticeable changes in polling after debates, and cites the work of Columbia's Robert Erikson and Temple's Christopher Wlezien, who focus on election forecasting using national polling.

Erikson and Wlezien find that there's a "a fairly strong degree of continuity from before to after the debates." Indeed, if you plot polls from before and after debates against each other, they line up very tightly:

There one exception: the 1976 race. But after the debates, Jimmy Carter did not grow his margin off of Gerald Ford's Eastern Europe gaffe. In fact, Carter's lead actually shrunk. Sides quotes Erikson and Wlezien: “Carter’s downward slide during the fall campaign seems to belie that this debate gaffe did much lasting harm.”

Princeton's Sam Wang — who's an amateur forecaster rather than a political scientist — disagrees with this conclusion. He notes that his forecasting model saw notable shifts in what outcome it predicted in 2004 after debates:

He argues that this shows that Kerry picked up 30 electoral votes following his first debate. But it's hard to disaggregate this from other things happening around the same time. For example, Iraq war casualties spiked sharply up in September 2004, which some models, notably Douglas Hibbs's "Bread and Peace" model, suggest affected presidential outcomes. Moreover, national polls didn't move at all as the scatterplot above shows.

Nate Silver claims you can show slight national poll gains for challengers following debates:

However, the effect is small, with an average shift of 2.3 percentage points, and it's hard to infer causality with such a small sample. In any case, only two elections — 1980 and 2000 — saw the candidates trade places in the polls following the debate, and in every case the poll leader after the first debate won the electoral college. So Obama can rest easy assuming he still leads after tonight.

Harvard's Sunshine Hillygus and Stanford's Simon Jackson found minor negative effects of the 2000 debates on Al Gore, but mostly among undecided or weakly partisan voters.

The evidence for debate effects on election outcomes is thus weak at best, and at worst nonexistent.

Media spin could matter more

Even if debates do matter, there's even less evidence suggesting that the actual candidates' performances do. The media seems to be the far more important player.   Arizona's Kim Fridkin and her colleagues conducted an experiment to test this proposition at the 2004 debate in Tempe, Ariz. They asked 74 voters to watch the debate and say who they thought won. 25 watched the debate without seeing commentary afterwards, 25 watched and saw commentary from NBC News which suggested that George W. Bush won, and 24 watched and saw commentary from CNN which suggested that John Kerry won. It turns out that the effects of cable news spin were enormous*:

Without spin, voters thought Kerry won, and CNN's coverage didn't have much effect, given that it was pro-Kerry. But people who watched NBC were convinced that Bush won. This suggests that voters' judgments on these matters are very malleable. UNC's Jennifer Brubraker and Kent State's Gary Hanson did a similar experiment comparing CNN and Fox News, and found that voters who were shown CNN were more supportive of Kerry, whereas watching Fox lead to an enormous 18.3 point swing toward Bush.

The way that the media talks about the debates also matters. Ohio State's Ray Pingree, Lousiana State's Rosanne Scholl and Ohio State's Andrea Quenette showed 700 students a five-minute clip from a 2004 debate.** A third read no news coverage, a third read "horse-race" coverage that framed the debate as a competition between the candidates, and a third read substantive coverage focused on the candidates' policy differences.

Pingree and his coauthors then asked the students to write out a description of the debate. They found that the descriptions by students who'd read the substantive coverage contained the greatest number of opinions provided with supporting evidence, whereas those who'd read the horse-race coverage wrote descriptions with much less substance. This suggests that the media can trivialize the debate, making viewers less focused on policy, or they can focus on policy themselves and recenter voters' minds on it.

The media even affects how "negative" voters think a debate is.  Erik Voeten of Georgetown and the late Lee Siegelman (then at GWU) studied polls asking whether voters in 2000 found the Bush campaign or the Gore campaign more "negative and/or nasty". They found that the first debate had a permanent effect, causing voters to view Gore as more negative:

But this was uncorrelated with actual negative statements made by either candidate. This suggests that the media coverage of the debate, rather than Gore's actual statements, lead to this impression.

Appearance matters, and it gets worse

John Wihbey at the Kennedy School has compiled a list of studies on debate effects, and many study factors that one wouldn't think would have any impact at all, like what television setting a voter is using. But these things do matter, at least a little bit.

Several studies suggest that a candidate's appearance during the debates could have a big impact. MIT's Gabriel Lenz and Chappell Lawson have found that attractive candidates disproportionately benefit from debates, with new support coming especially from less informed voters. The College of Wooster's Angela Bos, Bas van Doorn and Abbey Smanik found that HDTV hurt John McCain in 2008, with viewers reacting negatively to his appearance on higher-resolution screens.

Williams' Steven Fein, Richmond's George Goethals and Princeton's Matthew Kugler found that being exposed to other viewers' reactions to the debates greatly influenced voters' perceptions, suggesting that social networks could be important. U.C. Davis' Jaeho Cho and Yerheen Ha found that conversations with friends after the debate amplified their impact, reinforcing the theory that social networks influence how debates are perceived. Cho has also found that viewers who watch debates using a split screen are less likely to have partisan reactions. Yes, picture-in-picture is the enemy of bias.

Summing up

In short, the effects on debates on eventual votes are likely mild, and, in most cases, effectively nil. Moreover, what effects do exist are often caused by factors wholly beyond the candidates' control, like media coverage, attractiveness, and whether voters are watching a Nats game in the other panel of their TV.

* Thanks to Kevin Collins for the study pointer.

** Hat tip John Sides.