“President Obama and Mayor Bloomberg are pushing gun control. America’s police say they are wrong. Seventy-one percent of police say that Obama’s gun ban will have zero effect on violent crime. Eighty percent of police say that more background checks will have no effect. Ninety-one percent say the right answer is swift prosecution and mandatory sentencing. Tell your senator to listen to police, instead of listening to Obama and Bloomberg.”
— voiceover of a new ad by the National Rifle Association, citing a new “national survey” by PoliceOne.com
The NRA’s new ad caught our attention, given that it was featured prominently on the Washington Post Web site Wednesday.
But there are polls — and there are polls.
Regular readers know that we have urged caution against relying on opt-in Internet surveys that appear to make broad claims about estimating population values. Indeed, the American Association for Public Opinion Research's 2010 task force, in its top recommendation, warned researchers to avoid “nonprobability online panels” when trying to accurately estimate population values.
For that reason, we gave Two Pinocchios to President Obama in 2012 for claiming that a majority of millionaires support the Buffett rule. He was relying on an opt-in Internet poll.
Let’s find out more about this survey, and whether it justifies the language used by the NRA.
PoliceOne.com is a Web site that caters to law enforcement and is part of San Francisco-based Praetorian Group, a family of Web sites for first responders. The headline on the news release announcing the poll last week said: “PoliceOne.com Releases Survey of 15,000 Law Enforcement Professionals about U.S. Gun Control Policies.” PoliceOne’s Web site describes the survey-takers as “more than 15,000 verified law enforcement professionals.”
The number — 15,000 — sounds impressive, but it turns out that once again, this was an opt-in survey, promoted on the Web site and through e-mails to 260,000 newsletter subscribers.
Jon Hughes, Praetorian’s vice president for content, said that a qualifying question at the beginning, asking whether a survey-taker was a current or former law enforcement member, was intended to weed out people who were not connected to law enforcement.
“While that falls short of a 100 percent guarantee that no non-law enforcement members took the survey, we found the responses reflective of the general tone of discussion on our site (e.g. member comments) and the rank distribution and department size distribution closely matches that of our registered member base as well as the law enforcement community as a whole,” Hughes said. “There was no cross-checking after the survey was finalized.”
Still, let’s do the math.
The reported membership of PoliceOne is 400,000, so the response rate among members is just 3.75 percent. Moreover, there are some 765,000 sworn local and state law enforcement officers — not to mention federal officers.
This makes it all but impossible to claim that this survey is representative of law enforcement opinions on gun-control measures, and Hughes conceded that “the survey was not scientific by definition.” He added: “Nowhere in our release did we claim to be speaking on behalf of U.S. law enforcement in its entirety.”
Moreover, people who opposed gun control might have been more motivated to take the survey because of the way the Web site promoted it. Doug Wyllie, the site’s editor in chief, wrote a column in which he knocked the “made-for-TV imagery” used by “certain politicians” who had rows of uniformed officials behind them. “In reality, those cops are there under orders, silently standing on stage like so much furniture,” Wyllie wrote.
Hughes noted that the column got only 2,500 page views. “We were not trying to sway responses in any way,” he said. “Did the self-selection by respondents have a skewing effect on the poll? It’s possible, as it is with any such survey. Do we believe the results were inaccurate as a result? No.”
We won’t go into detail about the wording of the questions, but those also raise some concerns. One question, for instance refers to the “White House’s currently proposed legislation.” In these partisan times, that could skew the results, as Gallup discovered when it inserted Obama’s name in polling on gun control.
The survey methodology also discloses that a question on criminal background checks was removed from the survey “due to flaws with the question details, highlighted by a handful of users.” That’s rather unusual, especially given that background checks are at the center of the gun-control debate.
“I agree that the removal of the background-check question was unfortunate, though one that should be chalked up to an error in execution rather than any intent to suppress,” Hughes said.
(Perhaps it is just a coincidence, but the NRA also has not surveyed its members about background checks, which most surveys show has broad support.)
Given that the main question on background checks was removed, how could the NRA cite a question on background checks? That’s because the organization relied on the results for this question: “Do you think that a federal law prohibiting private, non-dealer transfers of firearms between individuals would reduce violent crime?”
This question doesn’t really say background checks, but it does reflect what the NRA claims would be a consequence of the background-check proposals. Generally, however, pollsters try to ask broader questions about policies, not consequences. This question is a bit like trying to discern if someone likes hard candy by asking if they like to have cavities pulled.
CNN, for instance, asked a series of questions about various restrictions in the background-check proposals, ending with “if the buyer is purchasing a gun from a family member or receiving it as a gift.” That got 54 percent approval, compared to the standard 80 to 90 percent for “universal background checks.”
NRA spokesman Andrew Arulanandam countered that such policy-worded questions do not get at the core of legislation and thus are intrinsically misleading. “We are providing information about what could happen,” he said. Surveys that show support for background checks “do not explain the implications.”
We will leave it to readers to decide whether more neutral language yields more accurate survey results. (For the record, the recent Washington Post poll on Maryland proposals described at length the requirements of the law — and still found overwhelming strong support.)
“How the NRA decided to use our survey data, we had no influence over nor involvement with,” Hughes said. “Obviously, the data ended up supportive of their perspective, but that was certainly not by design.” He stressed that “our survey release was in no way ‘political advertising,’ nor do we have any stake in the game.”
Indeed, PoliceOne, in its news release, chose to emphasize a different result on background checks: “Respondents were more split on background checks, with 31 percent agreeing that mental health background checks in all gun sales would help reduce mass shootings, while 45 percent disagreed.” The news release did not mention the result that the NRA describes as a background-check question.
It is difficult to compare the results of this survey to other surveys of police on this issue because many suffer from the same problems identified above. We found one 2006 survey that relied on a random sample of police chiefs, but the questions aren’t very relevant to this year’s debate.
The Pinocchio Test
It is quite possible that a truly randomized survey of police officers would turn up results that mirror this poll, but in the meantime, the NRA can’t claim that it reflects the views of “America’s police.” It only reflects the views of the self-selected people who took the survey — nothing more.
The NRA is also pushing the envelope with how it characterized one of the questions on the survey. But we will keep the rating at Two Pinocchios — similar to other situations when we have assessed statements based on online opt-in polls.
UPDATE: Upon reflection, we are increasing this rating to Three Pinocchios. As readers noted, a number of police organizations, including the National Fraternal Order of Police with 325,000 members, have endorsed expanded background checks. Moreover, in our desire to maintain consistency on how we treat such online polls, we failed to place enough weight on how the NRA portrayed the question that it says refers to background checks.
Check out our candidate Pinocchio Tracker