In 2006, a senior U.S. official said, the NSA was collecting “closer to 100” percent of Americans’ phone records from a number of U.S. companies under a then-classified program, but as of last summer that share had plummeted to less than 30 percent.
The government is taking steps to restore the collection — which does not include the content of conversations — closer to previous levels. The NSA is preparing to seek court orders to compel wireless companies that currently do not hand over records to the government to do so, said the current and former officials, who spoke on the condition of anonymity to discuss internal deliberations.
That effort comes in the wake of President Obama’s decision last month to find a way to move the data out of the government’s hands to assuage concerns about intrusions on privacy. Obama has given the Justice Department and the intelligence community until March 28 to come up with a plan.
The actual percentage of records gathered is somewhere between 20 and 30 percent and reflects Americans’ increasing turn away from the use of land lines to cellphones. Officials also have faced technical challenges in preparing the NSA database to handle large amounts of new records without taking in data such as cell tower locations that are not authorized for collection.
That low percentage still probably represents tens of billions of records going back five years, a cause of great concern to privacy and civil liberties advocates. “For innocent Americans, 20 or 30 percent is still a significant number and will chill legitimate lawful activities,’’ said Christopher Soghoian, chief technologist for the American Civil Liberties Union.
In defending the program, administration officials have emphasized the need to gather all records. “If you’re looking for the needle in the haystack, you have to have the entire haystack to look through,” Deputy Attorney General James Cole told Congress in July.
Edward Felten, a Princeton University computer scientist who has studied the program from a technological perspective, said the revelation “calls into question whether the rationale offered for the program is consistent with the way the program has been operating.”
But collection of even a quarter of the records is valuable, officials say.
“It’s better than zero,” NSA Deputy Director Rick Ledgett said Thursday in an interview, without describing the program’s exact scope. “If it’s zero, there’s no chance.”
One former senior official acknowledged that 100 percent was the goal but asserted that as long as the collection “is fairly spread across the different vendors in the geographic area that you’re covering,” the collection provides value.
The NSA, for instance, is still able to obtain the call records of some customers whose phone companies are not covered by the program. When the customers of a non-covered carrier call customers of a covered carrier, the latter’s records should reflect both ends of the call.
Some industry officials said that the 20 to 30 percent figure can only be explained if the NSA is also missing records from companies that provide Internet-based calls.
According to industry and government figures, the number of land lines in use fell from 141 million in 2008 to 96 million in 2012, a 32 percent drop. By contrast, the number of cellphones in use in the United States jumped from 255 million in 2007 to 326 million in 2012, a 28 percent rise. And Internet-based subscribers, according to the Federal Communications Commission, doubled from 21 million in 2008 to 42 million in 2012.
The NSA collection program began without court or congressional approval after the Sept. 11, 2001, attacks but was placed under court supervision in 2006 when American phone companies balked at providing the data solely at the request of the executive branch.
Under the program, the NSA receives daily transfers of call “metadata” from several of the nation’s largest phone companies. Those records include numbers called and the calls’ time and duration but not the content of conversations, subscriber names or cell tower location data.
The bulk collection began largely as a land-line program, focusing on carriers such as AT&T and Verizon Business Network Services. At least two large wireless companies are not covered — Verizon Wireless and T-Mobile U.S., which was first reported by the Wall Street Journal.
Industry officials have speculated that partial foreign ownership has made the NSA reluctant to issue orders to those carriers. But U.S. officials said that was not a reason.
“They’re doing business in the United States; they’re required to comply with U.S. law,” said one senior U.S. official. “A court order is a court order.”
Rather, the official said, the drop in collection stems from several factors.
Apart from the decline in land-line use, the agency has struggled to prepare its database to handle vast amounts of cellphone data, current and former officials say. For instance, cellphone records may contain geolocation data, which the NSA is not permitted to receive.
“It’s not simply the ability to go to the court and order some vendor to give you more records, but you have to make sure that the [agency’s collection system] is prepared and ready to take the data and meet all the requirements of the court,” the former official said. “You don’t want to turn it on and get hundreds of millions of records, only to find out that you’ve got the moral equivalent of raw sewage spilling into the Chesapeake Bay.”
The process of preparing the system can take months, said the senior U.S. official, adding that mobile calls have different data elements than land-line calls. “That’s a really detailed set of activities where we get sample data in, and we march it through our systems,” the official said. “We do that again and again and again. We put in auditing procedures to make sure it works. So before we turn on that mobility data, we make sure it works. . . . It’s very complex.”
Compounding the challenge, the agency in 2009 struggled with compliance issues, including what a surveillance court found were “daily violations of the minimization procedures set forth in [court] orders” designed to protect Americans’ call records that “could not otherwise have been legally captured in bulk.”
As a result, the NSA’s director, Gen. Keith Alexander, ordered an “end-to-end” review of the program, during which additional compliance incidents were discovered and reported to the court. The process of uncovering problems and fixing them took months, and the same people working to address the compliance problems were the ones who would have to prepare the database to handle more records.
The NSA fell behind, the former official said.
In June, the program was revealed through a leak of a court order to Verizon by former NSA contractor Edward Snowden, setting off an intense national debate over the wisdom and efficacy of bulk collection.
The same NSA personnel were also tasked to answer inquiries from congressional overseers and others about how the program and its controls worked. “At a time when you’re behind, it’s hard to catch up,” the former official said.
Storage and implementing new features to comply with court requirements also cost money, and that has been difficult in an era of budget cutbacks, the former official said.
The agency did not go to the court to seek new orders, because it was not prepared, officials said.
“Until you are sure that you have an architecture that has the right features and the right capacity, you wouldn’t go to the time and trouble of getting the court to authorize the collection and retention of the data,” the former official said. “Because the court would want to know that you’ve followed through on that and you had a material intent to get it and use it.”