“Despite the Administration’s delay of the employer mandate by a year, small businesses expect the requirement to negatively impact their employees. 27% say they will cut hours to reduce full time employees, 24% will reduce hiring, and 23% plan to replace full time employees (30 hours per week or more) with part-time workers to avoid triggering the mandate.”
— United States Chamber of Commerce Small Business Outlook Study, conducted by Harris Interactive, released July 16, 2013
We have long warned readers about the perils of relying on data from opt-in Internet polls, especially those that make broad claims about estimating population values. We have given Pinocchios both to President Obama, for relying on an opt-in poll when he claimed that a majority of millionaires support the Buffett rule, and the National Rifle Association, for asserting that an opt-in poll reflected the views of the nation’s police.
This is a yet another case, but with a wrinkle. Here, the polling company, Harris Interactive, and the sponsor, the U.S. Chamber of Commerce, presented the data in a highly misleading way — and then made false claims about the type of poll that had been conducted.
The Chamber has been a fierce opponent of the health-care law, a.k.a. Obamacare, and we frequently warn readers they should always be skeptical of polls peddled by partisan organizations. Perhaps it should be no surprise that this poll was released just as the GOP-led House of Representatives scheduled a vote to repeal the law.
Looking at the language in the report, highlighted above, is it any wonder that House Speaker John A. Boehner would tweet: “Study: ‘74% of small businesses will fire workers, cut hours under #Obamacare.’”
Or that Sen. Marco Rubio (R-Fla.) would write: “75% of small businesses now say they are going to be forced to either fire workers or cut their hours.”
Of course, that was the intended message. If there was any doubt, Chamber senior vice president Rob Engstrom tweeted a thank-you to Fox News host Bill O’Reilly for mentioning the survey on the air, adding: “74% will fire or reduce hours bc of employer mandate.”
But that’s not what the survey says — not by a long shot. Let’s dig into the data.
In an apparent effort to give the poll statistical authority, Harris included this line: “Margin of sampling error: +/- 2.5 percentage points.” That is supposed to mean that 95 out of 100 times the poll was conducted, the numbers in the poll should fall within in that range.
But we thought this notation was curious, because the poll was created through on-line surveys of two groups — 499 Chamber of Commerce members and 805 non-Chamber members (“small business executives” in firms with less than $25 million in revenues), which Harris claimed was weighted to be representative of small businesses.
But this is not a random sample, and such margins of error apply only to random samples from a population. (An example of that would be a poll based on random digital dialing, in which survey participants are randomly selected for interviews.)
Both Harris and the Chamber claim that this was an inadvertent mistake. We can understand a one-time screw-up. But it turns out that every Harris “small business” survey released every quarter by the Chamber since 2011 has had this bogus notation about a margin of error.
Why is this important? By reporting a margin of error, Harris indicated to readers that this is a probability sample where all businesses had some chance of inclusion, which is not correct. This notation boasts a higher level of statistical rigor than a sample cobbled from Chamber members and convenient lists. The nature of the actual poll greatly limits the ability to generalize the results to all small businesses.
The American Association for Public Opinion Research’s 2010 task force, in its top recommendation, warned researchers to avoid “nonprobability online panels” when trying to accurately estimate population values. In other words, despite suggestions that this poll represents the views of small businesses, it really only expresses the views of the business executives who happened to have answered an online survey, including members of an organization dedicated to repealing the health-care law.
As for the factoid eagerly repeated by Republican foes of Obamacare, there is a hint in the Chamber news release (but not in the more detailed Harris poll document) that a crucial piece of information is missing: “Among small businesses that will be impacted by the employer mandate, one-half of small businesses say that they will either cut hours to reduce full time employees OR replace full time employees with part-timers to avoid the mandate. 24% say they will reduce hiring to stay under 50 employees.”
“Among the small businesses that will be impacted by the employer mandate” suggests that these figures come from a subsample of the poll.
A Chamber spokeswoman, who declined to be identified, acknowledged that only 17 percent of the businesses surveyed said they would be affected by the employer mandate. Put another way, the poll found that 83 percent of small businesses surveyed said they would not be affected by an employer mandate that the Chamber of Commerce has said is a burden on small businesses.
Instead, Harris and the Chamber highlighted the answers of only those affected by the employer mandate. Moreover, they did not disclose that respondents could select as many options as they wanted, meaning the numbers could not be added up, because some executives may have selected more than one answer.
To find out the real values, a two-step process is necessary. First, the numbers have to be significantly reduced to account for the fact that only 17 percent of the sample (222 businesses) answered the question. Then you have to account for the possibility of multiple answers.
We will spare you the calculations, but here’s the bottom line: just 4.5 to 8.5 percent of small business executives surveyed said they will reduce hours or full-time staff in response to the employer mandate. That certainly has a different ring than “74 percent.”
Given that the law requires companies with more than 50 full-time employees to provide health care or pay a fine, it is quite possible some businesses will take steps to avoid the employer mandate.
Scott Clement survey research analyst for The Washington Post’s Capital Insight, noted that in April, Gallup conducted a random-sample poll in which it asked small business owners (drawn from a Dun & Bradstreet list of active companies with less than $20 million in revenues) if they had taken any specific actions in response to the health care law. Among the answers: 41 percent said they held off plans to hire new workers and 38 percent said they had pulled back plans to grow the business.
But Gallup also noted an important caveat: small-business owners have a “tendency to be more Republican than Democratic, higher income, more against big government, more conservative, and less optimistic than Americans overall.”
Chamber officials would not speak for the record.
“We are trying to tell a story and not every number fits in,” explained Sarah Simmons, a Harris vice president and senior consultant. “We were not attempting to be misleading.”
Simmons disputed the idea that survey report had created confusion about the results, saying she had received only one other media inquiry — from FactCheck.org, the nonpartisan, nonprofit project of the Annenberg Public Policy Center at the University of Pennsylvania. “FactCheck.org is a partisan organization,” asserted Simmons, who in 2008 had been director of strategy for Sen. John McCain’s presidential campaign.
“Harris’ good name and strong history in the research world would indicate that we, as a practice, do our level best to represent the data and let the data tell the story — regardless of our clients’ interests,” Simmons said.
The Pinocchio Test
Maybe the Chamber of Commerce got exactly what it paid for in this poll.
But in their zeal to bolster Republican talking points, the Chamber and Harris Interactive offered a misleading picture of what the poll said — and the nature of the poll itself. They should offer apologies to the Republicans who so quickly took the bait — and conduct a thorough review of how such polls are presented in the future.
Check out our candidate Pinocchio Tracker