On Monday, The Post's op-ed page ran a piece on "How Twitter can help predict an election" suggesting campaigns no longer need pollsters. It lit up the polling world. (We also took a look at the study here on The Fix.) Rob Santos, the president of The American Association for Public Opinion Research and a senior statistician at the Urban Institute weighs in below with a rebuttal.
The polling world was caught up in the buzz this week about research showing how to predict congressional elections using Twitter. Researchers from Indiana University investigated how to leverage ‘tweetnados’ of candidate names before elections in order to predict winners. A sampling of headlines from Web sites and blogs about this research proclaim:
- How Twitter could predict elections — in one eye-catching study
- Tweets Can Foretell Votes
- Big Buzz On Twitter Means Better Chances On Election Day
The hype is about a study that shows how ‘tweet shares’ of congressional candidates can be used to correctly predict 403 of 406 elections in 2010. One of the authors declares their methods “show that Twitter discussions are an unusually good predictor of U.S. House elections." Their conclusion: “…anybody with a laptop computer can come up with the forecast of an election that may be on par or better as a traditional poll.”
Does this portend the demise of traditional polling? Will Nate Silver need to totally revamp his approach to predicting elections? And will election night coverage devolve into flashing a chart of real-time tweet flows? Of course not. As the adage goes: if it looks too good to be true, it is too good to be true. In the present case of predictive Tweetnados, the devil is in the details.
Congressional elections are generally easier to call than one might think. The reason is that incumbents usually win. As Stu Rothenberg noted in his blog, Congressional incumbents have won reelection over 90 percent of the time in 19 of 23 past elections. He writes “In most cases, all you need to know is incumbency (or the district’s political bent) and the candidates’ parties to predict who will win.” Not bad, huh? So the real measure of tweetnado success is how much better one can predict elections beyond the standard use of readily available data. The answer may well be “not much”.
I reviewed the research paper that started this craze, entitled “More Tweets, More Votes: Social Media as a Quantitative Indicator of Political Behavior.” Tweets contribute relatively little to the outcome (Republican vote margin) when you include the “big boys” of election prediction – incumbency status and district partisanship. For instance, partisanship contributes a share of 240,000 votes to the Republican margin in their statistical model, whereas tweet-mentions assigns only a share of 15,500 votes.
being an incumbent predicts almost a 50,000 vote contribution to the Republican margin in their statistical model, whereas receiving 100 percent (all!) of tweet-mentions gets you only 155 votes.
Moreover, the paper’s "full" prediction model may be almost as accurate when you exclude the tweet share and just rely on traditional variables. Unfortunately, the authors failed to include that insightful model in their paper. Regardless, it appears that tweets can at best help at the margin relative to other more profound predictors that are easily available from existing public election data. In fact, there could be much better predictors than the ones used (e.g., measures of past voting patterns from each district instead of %McCain vote in 2008). Who knows, with more effective modeling, the contribution of tweets to the vote margin could be rendered negligible even if its inclusion attains the holy grail of “statistical significance.” Keep in mind that “statistical significance” does not always equal “practical relevance”. Is a model that -- on the basis of tweet shares to a candidate -- contributes 1/15th to the vote margin relative to that of partisanship an “unusually good predictor of U.S. House elections”? You be the judge.
And then there is the ‘face validity’ issue. Data are only data. Data do not know that they should behave one way for a congressional elections and another way for a presidential primary or even a New York election. If in fact tweet data are an “unusually” good predictor of elections, then Anthony Weiner should be optimistic about his mayoral aspirations given his ‘explosion’ of tweet attention and despite his free fall in the polls. In fact, this study’s finding that the tweet content is inconsequential prompted Salon.com to post the following headline: “Good news for Weiner: All Twitter publicity is good publicity.”
But there is actually much more going on here from a public opinion research perspective. The attempt to use tweets for election prediction illustrates a more general movement to leverage social media data to measure public sentiment. These methods often employ sophisticated statistical modeling to arrive at an estimate or conclusion.
In the public opinion research industry we refer to these approaches as “nonprobability methods” because of their divergence from typical rigorous probability sampling that underlies the traditional polling industry. Traditional methods have a proven track record and well established statistical theory to back them up.
The new methods – like the approach to incorporate tweets in election predictions – make some big assumptions in order to yield useable results. That does not mean all such approaches are inherently bad. They are an inevitable part of our future. But new uses of social media represent yet another item in the toolkit of public opinion research rather than a replacement of all others. This is especially true of the “workhorse” probability sample poll whose value goes far beyond just seeing which candidate is ahead in a given day. And so our Tweetnado fades back into the clouds… but watch out! The storm clouds are still rumbling.
Robert Santos is President of the American Association for Public Opinion Research and a senior statistician at the Urban Institute, Washington, DC.
NOTE -- The original blog posted was edited on Aug. 19, 2013 at 3:20 pm. The authors of the tweet paper indicated that comparing incumbency and tweet share was not an apples-to-apples comparison due to a scaling factor in their regression equation. The post has been edited to indicate that partisanship contributes a portion of 240,000 votes to the Republican margin in their statistical model, whereas tweet-mentions assigns only a portion of 15,500 votes. Tweet share has 1/15th the impact of partisanship on the prediction.