The September jobs report is already spurring a number of conspiracy theories, alleging that the numbers have been tampered with for political gain. Ezra has dispatched with those quite nicely. But it raises a broader question. How does the Bureau of Labor Statistics derive these numbers? How accurate is that process? And how susceptible is it to tampering?
The jobs report actually compiles the results of two different surveys: the Current Population Survey (CPS), which is commonly dubbed the "household survey," and Current Economic Statistics (CES), or the "employer survey." The household survey produces the unemployment rate, and the employer survey produces the "nonfarm payrolls" number, which is the most common measure of jobs gained.
The CPS is a monthly survey of households that asks which people in the household have worked (or were temporarily absent) in the past week; which have actively looked for work in the past month but did not work; and which neither looked for work nor worked. It's jointly run by BLS and the Census Bureau and has been conducted since 1940.
The number of people who either worked or looked for work is defined as the "labor force". The CPS calculates both the "labor force participation rate" — which is the fraction of people over 16 and not in prison who are in the labor force — and the "unemployment rate", which is the percentage of the labor force not working.
It also calculates a number of "alternative" unemployment measures by asking if members of the household have concluded that no work is available (the "U-4" measure), have given up looking for work but would still like a job (the "U-5" measure), or are working part-time but would like to be working full-time (the "U-6" measure). These tend to be higher than the regular unemployment rate. This month, U-6 was unchanged because a lot of the job gains came in part-time work, but all the others went down.
BLS also asks if members of the household lost their jobs in the month in question, and if they've been unemployed more than 15 weeks. Both of these measures went down last month, which is another sign that the labor market is recovering.
The CPS sample includes about 66,000 households. Every month, 25 percent of the sample is changed, so that the sample is not just studying the same group of people. The response rate is very high - usually more than 90 percent. By contrast, opinion polls have a response rate of around 9 percent, and contain samples a fraction of that size.
But even still, there is a lot of room for error. The BLS estimates that the standard error for the unemployment rate is 0.1 points. That means that the margin of error, like that you'd use for polls, is about 0.196 points:
It's higher for changes in the unemployment rate - the standard error is 0.12, so the margin of error is around 0.2352. But the drop in unemployment last month was 0.3 points - so outside the margin of error, and statistically significant. As former Labor Department economist Betsy Stevenson explains, that's a clear improvement.
What about the CES? That surveys 141,000 employers, both private and public, at 486,000 worksites and asks for the number of employees, hours worked, and earnings. "Employees" includes both part-time and full-time labor, and is a snapshot taken at the twelfth of the month. "Employment is the total number of persons on establishment payrolls employed full-or part-time who received pay for any part of the pay period that includes the 12th day of the month," BLS explains. It excludes the self-employed, volunteers, farm workers, and domestic workers. But it includes workers who are on strike, and those working for two different businesses are double-counted.
The standard error for CES is enormous - 55,254 for monthly change. That means the margin of error (at a 95 percent confidence level) is 108,293 - nearly the total number of jobs gained last month. So this month's job gain was statistically different from zero, but many past ones were not:
What's this about the seasons?
Both the CPS and the CES use a process called "seasonal adjustment." They figure that employment usually follows certain seasonal patterns. Construction workers don't work in the winter, high school and college graduates get hired in the spring, teachers get laid off for the summer, etc. They then adjust the number of employed and unemployed people by what you'd expect to change due to seasonal factors.
As Brad has noted, there's some evidence that the adjustment formula has gone awry, as there is a wide divergence in the summer and winter jobs numbers, with the latter usually stronger, as you can see in the above figure.
How secure is it?
As my colleague Eli Saslow notes, the BLS process is highly confidential. Economists are put on an eight-day security lockdown in advance of the report, signing confidentiality agreements every morning. The computers they use feature heavy encryption, and data is placed in a safe even for bathroom breaks. The Wednesday the week before the release, the CPS data comes in, followed by the CES data a few days after.
On the day before the release, three copies of the report and a CD-ROM are placed in a safe and taken to downtown Washington from the secure location where they were prepared, and presented to the few White House officials who have permission for a sneak peek at the numbers. Journalists are given access to the information 30 minutes before release but have to connect to a secure network that prevents them from sending out the data ahead of its official release.
As nonpartisan as government gets
The BLS is a highly nonpartisan operation, existing since 1884 and headed by Jack Galvin, a career employee who ran the employment and unemployment statistics division from 1998 to 2011, and has held a variety of positions there since 1978. Prior to him, Keith Hall headed the agency from 2008 to 2012, following career positions at the Council of Economic Advisors and the International Trade Commission. It's normal for BLS commissioners to span administrations and parties. Janet Norwood headed the agency from 1979 to 1991, spanning Carter, Reagan, and Bush I, while her predecessor Julius Shiskin headed it under Nixon, Ford and Carter.
Hall has told the Wall Street Journal that it is "impossible" to alter the numbers for political gain. But that hasn't stopped some from harboring conspiracy theories about their political manipulation. Former GE chief Jack Welch today tweeted that "these Chicago guys will do anything…can't debate so change numbers," implying the Obama administration artificially inflated the figures.
But presidents themselves have worried about the agency in the past. Richard Nixon infamously asked his aide Fred Malek to count the number of Jews working in BLS, based on his delusion that Jewish liberals were trying to sabotage him through bad jobs numbers. Suffice it to say, none of these conspiracy theories — be they anti-semitic or anti-Obama — have any truth to them.
The original version of the post confused benchmark revisions to the CES with ones to the CPS. Apologies.