If it’s accurate, the finding has disturbing implications about our citizenry — and about democracy in general. As linguist Steven Pinker wrote in his blurb for “Democracy for Realists,” “however cynical you are about the democratic process, it’s worse than you think.”
But Anthony Fowler and Andy Hall have just published a new paper that makes a convincing case that Achen and Bartels’s analysis was statistically flawed. Yes, Achen and Bartels found a statistically significant result, which in the absence of any real effect should occur with probability less than 5 percent — but this claim of statistical significance had two problems.
First, correlation: Election results from different counties are not independent events; hence the effective sample size in their analysis was lower than one would think based on a simple calculation.
Second, forking paths: There were many possible ways that Achen and Bartels could have analyzed their data, giving them many different ways to obtain statistical significance. But these two criticisms together and the New Jersey 1916 data are seen to be consistent with a null effect.
As Fowler and Hall put it, in more technical terms:
The standard errors are biased downward because nearby coastal counties are subject to correlated political shocks. While Achen and Bartels report a p-value of about .01, a more appropriate estimate of their p-value is about .32, the frequency with which we would expect to see an estimate as large as theirs if shark attacks have no effect.
So what? So, a lot of things. This is not just a statistical quarrel; it has larger political implications. As Fowler and Hall write,
Our analyses suggest that recent concerns about voter competence, at least as they relate to voters’ abilities in ignoring irrelevant events, are overblown. The evidence on shark attacks and presidential voting is not strong enough to warrant dismay toward voters or the democratic process.
Fowler’s been on this before. Last year he wrote a paper with B. Pablo Montagnes rejecting an earlier claim that presidential elections can be determined by the outcomes of college football games. In reporting that work, I gave this take-home message: “Voters aren’t as ‘irrational and emotional’ as is sometimes claimed.”
On the other hand, follow-up research has found some evidence of short-term attitudes following football wins and losses, so the full story there is not so clear.
From the other side, Bartels has made other statements suggesting that voters are irrational. For instance, he has said that subliminal exposure to cartoon smiley faces have “produced substantial changes in policy attitudes.” As with the shark attack study, I didn’t see any convincing statistical evidence there.
It would be easy to laugh at all this. Smiley faces, football games, shark attacks: It is indeed a bit ridiculous to suppose that these can determine political attitudes election outcomes.
After all, it’s hard enough for political actors to change voters’ opinions even when they try. Considering the zillions of possible subconscious stimuli (college football, pro football, basketball, TV sitcoms, bad weather, good weather, random celebrity news, indigestion), it would indeed be surprising if these particular inputs have the large and consistent effects being claimed.
Fowler and Hall make this point in the concluding section of their paper. With so many potential influences, it’s no surprise that researchers will be able to find patterns in the noise — and, with correlated data, simple statistical methods can easily find apparent statistical significance just by chance. In Fowler and Hall’s words:
[T]hough published studies have presented a variety of irrelevant factors that purportedly influence behavior, we have no way of knowing how many other irrelevant factors were examined and found to have no effect. Furthermore, when a researcher has decided to test for a purportedly irrational behavior, all bets are off regarding the nature of that effect. . . . The sheer number of hypotheses and specifications that could be presented as consistent with irrational behavior should make readers worried about the validity of the few statistically significant effects reported.
It’s easy for me to say this all now, not so easy a few years ago. For example, here’s me writing in 2012:
I took a look at the study [of college football games and election outcomes] . . . and it seemed reasonable to me. . . . this sort of pattern among voters is disturbing, similar to Chris Achen and Larry Bartels’s famous study of the electoral effects of shark attacks.
Since then my own views, and those of many others, have changed. We’ve gradually become more and more aware of the problems with selection bias, and have learned no longer to trust these sorts of isolated claims that are backed up by a combination of statistical significance and a flexible sort of theory that can fit just about any data pattern.
So I don’t see Achen and Bartels as particularly foolish here. They did their shark attack study back in 2002 when we were willing to believe all sorts of claims based on weak evidence, and then they got caught in the riptide of the replication crisis. Once an idea gets out there, it can stay in the collective discussion for a long time.
Just to be clear, I’m not claiming to be sure that shark attacks can’t have big effects or that they didn’t affect the 1916 election. I’m just saying that the claimed statistical evidence isn’t there. Such things are possible but the evidence is also consistent with null effects, or effects in the opposite direction, or effects that are positive in some settings and negative in others.
Flaws in any particular study should not blind us to the real issues raised by Achen and Bartels. In “Democracy for Realists,” they write:
… voters — even those who are well-informed and politically engaged — mostly choose parties and candidates on the basis of social identities and partisan loyalties, not political issues. They also show that voters adjust their policy views and even their perceptions of basic matters of fact to match those loyalties. When parties are roughly evenly matched, elections often turn on irrelevant or misleading considerations such as economic spurts or downturns beyond the incumbents’ control.
There’s a lot of evidence for these claims. This view of politics does not require the validity of claims about football games, smiley faces, or shark attacks. It should be possible to disengage some of the more extreme claims of voter irrationality from the larger story of how complex voters’ decision-making can be.