Although researchers said the pilot had demonstrated the concept of information sharing, they also cited deficiencies in the way it was implemented. The test program, which began last May, relied on NSA “signatures” or fingerprints of malicious computer code that in initial stages were “stale when deployed” and in many cases did not prevent intrusions that the companies could not have blocked themselves, according to the report, which was not publicly released by the Pentagon but was shared with Congress this week.
The unclassified study, which was obtained by The Washington Post, underscores the operational, legal and policy challenges in building a robust defense of critical U.S. computer networks as foreign rivals and other adversaries seek to penetrate systems, steal data and perhaps lay the groundwork for a destructive attack.
“Unfortunately, the report highlights one of my continuing points: that there is no silver bullet in cybersecurity,” said Rep. James R. Langevin (D-R.I.), co-founder of the Congressional Cybersecurity Caucus. “Signature-based defenses alone will never be enough to secure our critical infrastructure. We need a comprehensive approach that incorporates innovative information sharing with industry, while holding them accountable for stronger security.”
The Defense Industrial Base cyber pilot includes 17 defense companies, among them Bethesda-based Lockheed Martin, which several years ago had terabytes of data related to the Pentagon’s Joint Strike Fighter project stolen from its networks.
In the pilot, the Internet carriers AT&T, Verizon and CenturyLink filter firms’ incoming e-mail for malicious software using classified NSA signatures. The pilot tested two concepts: Incoming malicious e-mails were quarantined and outbound traffic headed for suspicious Web sites was redirected.
The e-mail measure was considered effective, yielding few false positives. But companies reported large numbers of false positives in the redirecting of outbound traffic headed for bad Web sites, the report said. Still, these two measures should “play a critical role” in a broader effort to secure critical networks, the report said.
The pilot allows companies to share data with the government, and some companies have opted to send information to the Department of Defense Cyber Crime Center, the report stated.
Some of the program’s flaws resulted from unrealized expectations.
For instance, the report said, many Defense Industrial Base companies thought that the pilot aimed to prove that NSA signatures would provide an optimum level of protection not available through any other source. But, the report said,“when this result did not fully materialize,” the goal was scaled back to showing “a baseline level of protection.”
In fact, the study said, of 52 incidents of malicious activity detected during the test program, only two were “unique” or resulted from NSA threat data that the companies did not already have themselves.
Said one unnamed company official quoted in the report: “Public pronouncements of success were unencumbered by facts.”
Another factor was that many of the firms, which include other defense giants such as Northrop Grumman and Raytheon, already have sophisticated monitoring capabilities. Indeed, “the added value of the classified signatures relative to already available signatures was not conclusively demonstrated during the pilot,” the report stated.
To evaluate efficacy more fully, the pilot would need to include a broader sample of companies and more comprehensive set of signatures, the report said.
Nonetheless, the researchers said the approach could be valuable as part of a broader effort to secure critical networks. They recommended that the pilot be expanded to a more diverse set of defense companies. It also recommended that Department of Homeland Security participation be increased, as initially planned.
Based on the study’s results, the Obama administration in November decided to continue the program and put DHS in charge of the relationship with the Internet carriers, officials said.
“This program is a step in the right direction,” said Sen. Joseph I. Lieberman (I-Conn.), Homeland Security Committee chairman. “I’m pleased that the administration has directed the Department of Homeland Security to continue the development and potential expansion of this effort.”
One irony is that the classified data used in the pilot remain secure because they are transmitted in low-tech “sneaker net” fashion. A courier hand-delivers to carriers 80 to 90 threat signatures every two days or so, on paper. Those signatures then need to be hand-typed into the system, an industry official said.
“We had successes in proving this concept, but there were areas in which we encountered unanticipated challenges,” Pentagon spokeswoman April Cunningham said. “[Carnegie Mellon’s] analysis will further help us build upon the lessons learned.”
The pilot “has prompted interest in other sectors,” said an administration official who was not authorized to speak for the record, “and there is certainly underlying contemplation of future expansion.”