Their company, Richmond-based Tridium, would succeed — but with far-reaching implications for the security of the online universe known as cyberspace.
Tridium’s driving technology, 4 million lines of software code called the Niagara Framework, is a marvel of innovation. With the click of a mouse, Niagara enables plant managers to view video streams, high-rise superintendents to operate air conditioners and elevators, security officials to track personnel inside U.S. military facilities, and nurses to monitor medical devices in hospitals.
At least 11 million devices and machines in 52 countries, including security and surveillance systems in homes, have been linked to the Internet through Niagara, most of them in the past two years. But behind that success is a looming threat: an unknown number of Niagara-run networks are vulnerable to attacks from hackers, an examination by The Washington Post has found.
Last week, after more than a month of conversations with The Post, the company in a confidential security bulletin warned customers about the vulnerabilities and described ways to mitigate them.
“We’re not going to say Niagara is secure,” Sublett said in an interview. “We try to soften it and say we’re trying to make it as secure as possible.”
Tridium’s story illustrates the unintended consequences of the world’s rush to connect machines and devices in cyberspace. It also demonstrates how even small missteps in writing software or configuring systems can have huge implications. In cyberspace, determined hackers routinely transform obscure gaps into major security holes.
Over the past two years, hackers and cyberwarriors who once focused primarily on traditional computers and networks have put control systems in their crosshairs,
damaging machinery, stealing information from networks and spying on facilities. Warnings from the Department of Homeland Security about the threats have become a drumbeat, while officials at the Pentagon and the White House consider them a national security priority.
After discussing Tridium with a Post reporter, a pair of security researchers decided on their own to zero in on Niagara and discovered gaps that would enable hackers to download and decrypt user names and passwords. The researchers, Billy Rios and Terry McCorkle, shared their findings with The Post and reported them to cybersecurity officials at the Department of Homeland Security, who recommended several measures to Tridium, including better security training for customers.
“There are hundreds of thousands of installations on networks, including [Defense Department] installations and Fortune 500 firms,” said Rios, a 34-year-old security researcher and a co-author of “Hacking: The Next Generation,” a handbook for security experts. “These customers have no idea they are exposed.”
In interviews, Sublett defended Niagara’s security, saying it follows industry “best practices” and “is basically secure.” He said Tridium has long recommended, to customers who asked, that users protect against intrusions by placing Niagara behind more-secure “virtual private networks.”
Sublett said executives learned about the vulnerabilities almost a year ago, when a Niagara customer that uses the software to manage Pentagon facilities turned up issues in an audit. He said Tridium is working on fixes. The firm also is doing more to train customers about security than it has in the past, he said.
“We’re committed to making our framework more secure,” Sublett said. “And we know it’s our responsibility to educate our community.”
No longer off the radar
For more than a decade, few people gave much thought to the security of commercial control systems on the Internet.
Tridium executives said attacks seemed unlikely, because hackers had not traditionally targeted such systems. In interviews, the executives said they and their customers generally assumed that control systems were buffered somewhat by their obscurity.
Rios and other “white hat” hackers — those who seek to improve security by exposing flaws — noticed Niagara systems popping up online in recent years, as office buildings, apartment complexes and other facilities automated heating systems, security and other operations. One researcher documented thousands of “portals” online. Others shared details about Niagara locations and speculated about their security.
But the interest did not take hold beyond a small cadre of security specialists, because few people grasped the implications of Tridium’s business and its expanding reach.
Commercial control systems rely on computer software, microprocessors and networks. They do not have to be as quick or as finely calibrated as the industrial control systems that run power generators, manufacturing equipment and other heavy machinery. But they are far more numerous.
In 1996, Sublett and five colleagues wanted to give customers options to connect a wide variety of devices into a single network. At the time, devices had to be controlled separately by each manufacturer’s software.
The team formed Tridium and designed Niagara to leverage the technology that is at the core of the Web. The Web uses the universal Hypertext Transfer Protocol, or HTTP, to facilitate computer-to-computer communication over the Internet.
In 1999, Tridium introduced the software framework to the market. Niagara permitted other companies to tailor the software to incorporate any device. At its core, Niagara serves as a kind of middleman that transforms the electronic babble of every network-connected device into a single manageable language.
Like the Web, Niagara was easy to use, and word spread in the world of control systems. Over the next few years, so many smaller companies began licensing Niagara to use in building automation projects that the big makers of commercial control systems began to follow suit.
In 2005, as the notion of connecting machines to cyberspace was gaining momentum, Honeywell International bought the company for an undisclosed amount and allowed Tridium to operate independently. Sales of the system soared, and Tridium’s promotional materials include examples of Niagara’s growing presence around the world.
In Chicago, the Niagara software manages heating, lighting, security and more for two federal government buildings that house the FBI, the Drug Enforcement Administration, the U.S. Attorney’s Office, the Internal Revenue Service and other agencies.
In Dubai, in the United Arab Emirates, managers of the 53-story 21st Century Tower apartment complex use the software to control fire detection, security sensors, air conditioning and myriad other operations.
At Singapore’s Changi Airport, Niagara helps manage more than 110,000 devices and sensors. The James Cook University Hospital in Middlesbrough, England, relies on Niagara to manage everything from “critical medical systems” to elevators, security, lighting and kitchen refrigeration.
Some Defense Department facilities in the United States also depend on Niagara. That includes the giant Tobyhanna Army Depot in Pennsylvania, which uses Niagara to control boilers.Some military installations use Niagara to provide surveillance and access control at “high security” facilities, said Marc Petock, Tridium’s vice president for global marketing and communications.
Growing numbers of home-automation companies are using Niagara to enable homeowners to control lighting and security systems.
One of the most widespread uses of Niagara involves about 575 Wawa convenience stores nationwide, where the software connects oven doors, gasoline pumps, exterior lights, freezers and security cameras, all controlled or monitored from a Wawa command center.
Wawa embodies Tridium’s lofty ambitions, as Petock describes them: “Any device, any system, any network, any protocol from anywhere at any time.”
Rios first noticed Niagara more than a year ago, while working on a security project. He became intrigued about the framework while attending a security conference in January, as he drank beers and smoked a cigar on a veranda at a conference center in Miami.
Rios is a former Marine captain who served in Iraq with a signals intelligence unit and later as an information assurance analyst at the Defense Department, helping to protect networks. Since then, he has held senior security positions at Microsoft and Google.
On their own time, Rios and his research partner, McCorkle, also a corporate security specialist, have made a specialty of finding vulnerabilities in industrial and commercial control systems. The two have been credited with reporting at least 25 serious vulnerabilities to cybersecurity officials at the Department of Homeland Security and vendors.
Rios was troubled by what he was hearing about Niagara in Miami. Another researcher described how he had mapped thousands of Niagara-driven networks. He said they were linked directly to the Internet even though many apparently required only user names and passwords for access. To a hacker such as Rios, that was virtually no security at all.
Following the conference, after a Post reporter called and shared new details about Tridium — including the use of Niagara by the Pentagon and other federal agencies — Rios decided to take a closer look.
On Jan. 26, sitting in his family’s San Jose living room and working on a laptop, Rios began to work. He created an account on a Tridium Web site devoted to helping users, lurking quietly as he read complaints and recommendations. The comments from users helped him to construct a “mental map” of how the software worked.
“I’m learning about its weaknesses, and I’m learning about the common configurations that I’ll likely see in the wild,” he said later.
A key moment came when someone in the online forum referred to technical manuals and a Web address where they were located. Rios and McCorkle downloaded the manuals and pored over them for clues. Rios saw a reference to a Niagara demonstration site online and turned his attention there.
As he toyed with the demo and hunted for an exploit, his insights gleaned from the manuals crystallized. “Within five minutes, I’ve found what I’m looking for,” Rios later wrote in an e-mail to The Post. “I find a flaw that gives remote attackers the ability to download all the user names and passwords for all the users on the Niagara server. I test it against the demo server . . . it works. I test it against a couple of other places . . . it works. The attack is trivial and very reliable.”
The exploit, well known in the hacker world, is called a directory traversal attack. It enabled Rios to turn the Web’s core function — the communication protocol that is intended to make everything easier — to his advantage. With some deft alterations to the Niagara Framework’s Web address, he was able to order the framework to perform certain tasks. One of them was to electronically hand over a “configuration file,” which happens to contain user names, passwords and other sensitive material.
By the time he was finished, it was 3 a.m. on Saturday, Jan. 28. Rios crafted a technical note about his exploit to cybersecurity officials at DHS and, after encrypting the message for security, sent it off. “All total, it took me 2 days to go from zero knowledge to remote password theft,” Rios wrote.
The passwords Rios had grabbed were scrambled for security by a mathematical formula called a “hash.” But that offered limited protection. Automated computer tools can crack the hashed passwords with relative ease. (In a recent attack on LinkedIn, a social-networking site, hackers made off with 6.5 million hashed passwords and immediately began cracking them.) “Once the passwords are decrypted, you can simply log in to the Niagara Framework as any user you desire,” Rios said.
A senior cybersecurity official at DHS acknowledged that the department had received Rios’s information and had talked to Tridium.
Tridium officials attributed the demo vulnerability to an employee who they said set it up incorrectly. They said some Niagara Framework users have also misconfigured their systems. Other users have never bothered to take secure measures, such as using a hard-to-hack virtual private network.
Sublett said the company intends to change the location of the configuration file to make it harder for hackers to find. Tridium also is trying to figure out the best way to change the framework’s default security settings “so it’s not as easy to make a mistake.” And it is going to improve the hash that scrambles passwords because “it’s not as strong as it should be,” Sublett said.
The company, still in talks with DHS officials, has begun a push to better communicate to Niagara users about the security risks in cyberspace. The security bulletin last week was part of that process.
“We’re not out there claiming we are bulletproof secure,” Sublett said. “Nobody is 100 percent secure.”