This post has been updated to reflect the fact that the Gravis poll was a so-called "informed ballot."
Political junkies have been yukking it up since last Tuesday over a Gravis Marketing poll that got a Maryland congressional race wrong by 96 — yes, 96 — percentage points. The Gravis survey, conducted in January, showed primary challenger Michael Smigiel leading Rep. Andy Harris by 29 percentage points in their Republican primary contest; Smigiel actually lost by 67 percentage points, 78 percent to 11 percent.
We all know polls aren't perfect (remember the Democratic presidential primary in Michigan?), but the margin of error in this case was truly exceptional. It was conducted three months before primary day, yes, but there's just no way it was even close to accurate at the time. Smigiel is a former Maryland state delegate, but he was never more than an underfunded long-shot who was going to get trounced by an incumbent; the $1,900 he paid Gravis to conduct the survey for him was one-tenth of his entire campaign spending. Harris spent about 15 times more than Smigiel.
As Daily Kos Elections pointed out way back when, this Gravis poll was not a true ballot test but rather an "informed ballot" survey; it planted negative information about Harris in respondents' minds before gauging their support. As such, it shouldn't necessarily be directly compared to the eventual result — except that Gravis's client presented it as a normal poll, so it was covered that way.
But it wasn't entirely stunning that Gravis would be far off the mark, given its track record.
In Ohio, Gravis had primary challenger Matt Lynch up by eight percentage points on Rep. David Joyce a month before the March vote in their GOP primary; Joyce cruised to a 20-point win. (This one was not an informed ballot.)
Shortly before the Democratic presidential primary in New York last month, Gravis put Sen. Bernie Sanders within six percentage points of Hillary Clinton -- the closest of any pollster; he lost by 16.
During the 2014 primary season, a Gravis poll showed Sen. Mitch McConnell (R-Ky.), then the Senate's minority leader, up by 14 points over Matt Bevin (now the state's governor) on the eve of their primary. Once again, their polling showed the race more competitive than any other survey. McConnell, now the majority leader, trounced Bevin by 25 percentage points.
Gravis also vastly underestimated the strength of Sen. John Cornyn (R-Tex.) in his primary battle against Rep. Steve Stockman that year. The pollster gave Cornyn a 15-point edge, while another had him up 46 points. He won by 40.
In an article for Slate two years ago, now-Washington Post reporter David Weigel noted the Kentucky and Texas primary botch jobs under an emphatic headline: "The Worst Poll in America."
That's probably a little hyperbolic. The data addicts at FiveThirtyEight give Gravis a "C" grade among more than 300 polling outfits — not good but not as bad as, say, Strategic Vision or the Pharos Research Group, who have "F" grades and are excluded from the site's calculations altogether.
Other Gravis surveys have been accurate. In Republican presidential primaries in Connecticut, Rhode Island and Delaware last week, the Florida-based firm came close to actual vote shares of winner Donald Trump and runner-up Ohio Gov. John Kasich.
Gravis managing partner Doug Kaplan said congressional primaries, in general, are "very difficult to poll." He noted the firm was working for Smigiel and Lynch when its surveys showed them leading races they ultimately lost badly but emphasized that "every poll is in good faith."
I had asked about the integrity of Gravis polling because there seems to be a pattern to the double-digit misses. You likely noticed it, too, among the examples I cited. The polls that were way off the mark tended to make blowout contests appear more compelling -- to make incumbents appear more vulnerable in primaries than they actually were. Call it a pro-drama bias — a problem to which the political press is susceptible, too.
This shared vice might help explain why news outlets like the New Yorker, the Guardian and, yes, even The Washington Post — occasionally and with caveats — continue to cite Gravis surveys. Another reason is that Gravis polling is sometimes the only game in town, as it was in the Delaware primary.
Kaplan emphasized the good-faith nature of the polls, but Gravis's survey methods lend themselves to unreliable results. The Fix's Philip Bump explained the pitfalls in some detail last summer, as Trump was touting a Gravis poll that supposedly proved his popularity among Hispanics. The issues basically boil down to relying heavily on landlines, which many voters don't use, to administer automated phone questionnaires, which have very low response rates.
At minimum, the media ought to note Gravis's methodology and inconsistency as a predictor in instances where they feel they have to use the firm's polls. And they should take care not to cite Gravis simply because one of its surveys fits an interesting narrative. Interesting has proven to be inaccurate too many times.
(h/t Daily Kos Elections)