The UCLA Health system, for example, had communications records — including doctors’s names, e-mail addresses and phone numbers — visible online. The Pentagon’s Defense Information Systems Agency, which maintains secure military networks, exposed a contracting database appearing to show $164 million in purchases. Two Texas state agencies failed to protect the personal details of people receiving government services.
This was not the work of sophisticated Russian hackers or Chinese cyber-warriors, who typically get blamed for problems in computer networks. Instead, researchers are pointing to humble system administrators for making routine errors that left the data unsecured.
The scale of the mix-up has highlighted how in an era of soaring national investment in cyber-security, the weakest link often involves the inherent fallibility of humans. Experts say even the most skilled system administrators struggle to keep every computer at large institutions running smoothly, with the proper software updates, security patches and configurations.
“There’s an old joke,” said Columbia University professor Steven M. Bellovin, “that computers need a ‘Do-What-I-Mean’ function.”
Bellovin, who teaches computer science, added, “Some systems are just impossible to configure correctly… The code is complex.”
Security research Dana Taylor, founder of NI@root and a self-described “ethical hacker,” said she discovered flaws in Oracle’s Reports software in 2011 and reported them to the company. Oracle began warning customers and issuing patches to solve the problem in 2012, but uptake has been uneven.
“Oracle notified all of our customers directly that they should apply the patch,” said Oracle spokeswoman Deborah Hellinger. “This process is commonplace in the industry."
Yet even two years later, anyone with an Internet connection and knowledge of certain Oracle software commands can download sensitive information from some of the servers running affected — but unpatched — software.
Administrative credentials for the systems also have been left exposed in many cases, giving hackers the opportunity to probe deeply into the compromised networks. Though it’s not clear how much of the data — if any -- was collected by hackers, some sensitive information could be found through a simple Google query.
“These are some really horrific exposures,” Taylor said. “Basically, it’s a case where the database administrators are not familiar with the actual Oracle commands that are on their server.”
The issue broke into public view last month when CNN reported that the Texas State Department of Family and Protective Services had left the names, birth dates and other information about children exposed to the Internet, along with other instances. Then Brian Krebs, a journalist who specializes in computer security issues, followed with a report on how MBIA, the nation’s largest insurer of municipal bonds, had suffered what Krebs called a “huge data leak” involving information stored on Oracle software.
The Wall Street Journal last week reported on a similar lapse at the Port of Baltimore, where logs of visitors were left vulnerable. They included information about foreign diplomats, State Department employees and officials from the Defense Intelligence Agency.
Port Administration spokesman Richard Scher confirmed the problem but said investigators found no evidence that the leaked data was collected by unauthorized people. “We are doing everything in our power to make sure this type of exposure does not occur again,” Scher said.
Security researchers say the sweep of the problem still remains poorly understood, even as they work with law enforcement officials and the affected institutions to help lock down the systems. Some have patched the problem; many haven’t, the researchers say. And even some institutions that have successfully protected some data have left data exposed on other servers.
“I would suspect that each one of these organizations have dozens or more routers and switches that manage which systems are connected to the Internet and which ports are exposed,” said Ben Caudill of Rhino Security Labs. “A lot of time organizations don’t really know what’s publicly accessible, and that becomes a real big problem.”
At Purdue University Calumet, the data of anyone who applied to or attended the school in recent years was exposed, said Bryan Seely, chief executive of Seely Security, who also has worked on the Oracle software issue. Social Security numbers, dates of birth, home addresses and contact information that could be used to pursue identity theft was included in the database available online until recent weeks, when online access was closed off after the researchers alerted the institution to the problem.
The university says it has reviewed its IT system maintenance and administrative processes and has now applied all patches available for its servers. “While we regret that information of our students was exposed, we are doing everything possible to protect the data of our students,” university spokesman Wes Lukoshus said in a statement.
The Kansas state court system was also leaking data, Seely said, with traffic conviction records that included drivers license numbers, dates of birth and, in some cases, Social Security numbers left unprotected. Lisa Taylor, spokeswoman for the Kansas Office of Judicial Administration, acknowledged the incident, which she said is now resolved.
UCLA Health also acknowledged that it is seeking to correct “potential” issues with records on its paging system. A spokeswoman for the Defense Information Systems Agency said it has “multiple network protections in place” and that the agency’s exposed database “was not on a classified network.”
Some experts lay at least some of the blame on Oracle for issuing software that was both complicated to use properly and had default settings that left security weak. If a software patch must be issued to correct a problem, there will always be some computers that are left vulnerable — especially at a time when many information technology departments are understaffed.
Security issues typically get attention and resources, experts say, only when something goes wrong.
“People think that there’s all this wonderful technology and it’s great. But at the same time a lot of these institutions may have one full-time technical person who they staff to do this stuff,” said Joseph Lorenzo Hall, chief technologist for the Center for Democracy & Technology. “To think that a local government IT administrator in a small town is going to be able to adequately protect from all threats is woefully misguided.”