On Thursday, the Department of the Treasury announced new sanctions against a number of Russians and Russian entities believed to be linked to hacking efforts both during the 2016 election and since. Included in that announcement was a broader warning to the public: Since at least March 2016 — the same month during which the email account of Hillary Clinton’s campaign chairman is believed to have been first compromised — Russian hackers have “targeted U.S. government entities and multiple U.S. critical infrastructure sectors, including the energy, nuclear, commercial facilities, water, aviation, and critical manufacturing sectors.” As part of the government’s efforts to counteract that activity, Homeland Security released an extensive alert documenting the tools used by the hackers in their efforts — and detailing that incident in which the hackers accessed the control mechanism for that turbine.
The natural question that emerges is: How serious is this hacking? The idea of Russian hackers having access to the control switches of America’s power infrastructure is particularly unnerving, raising the idea of waking up one morning to learn that the United States has simply been switched off.
Several experts who spoke with The Washington Post, though, explained that this is not only oversimplistic but also that it is almost certainly impossible. The effects of infiltration of America’s power grid would be much more geographically limited thanks to the distributed, redundant nature of the system.
In fact, it’s more than a little like another alert issued by the government about Russian infiltration efforts: the one on Oct. 7, 2016, warning about Russian efforts to tamper with state voting systems.
That announcement came from the director of national intelligence and the Department of Homeland Security and indicated that Russian hackers were “scanning and probing” election-related systems. The message, one of the few public responses to Russian interference from the Obama administration, didn’t get as much play as it might have, given that it came out the same day as The Post published the “Access Hollywood” tape.
Meaning that the caveats in the announcement — that our voting mechanisms were protected by being disconnected from the Internet during the election, by being distributed throughout thousands of counties and by having after-the-fact statistical checks on accuracy — weren’t really absorbed. The idea that Russian hackers might have significantly altered election results in 2016 persists out of a misunderstanding of how remarkably hard it would be to subvert the process, particularly without being noticed.
In several ways, those looking to disrupt the electrical grid would have an advantage over a hacker trying to throw a national election. But generally speaking and in important ways, it’s not much less difficult.
What “the grid” is
What we think of as “the grid” — a nationwide network of transmission lines crisscrossing the country, interconnected. Something like this.
It is that, and it isn’t.
Sergio Caltagirone is director of threat intelligence at the firm Dragos, which creates tools to protect industrial control systems. In a phone conversation with The Post, he explained how the grid is composed.
“It is not a single entity,” Caltagirone said. “We’re talking about almost a thousand independent utilities operating across the United States in a partially connected environment.”
The “grid” is three separate grids, he said. The Eastern Interconnection covers most of the United States east of the Rocky Mountains. The Western Interconnection covers the other side of that range. And then there’s the Electric Reliability Council of Texas, because Texas does its own thing.
Taking down all three of those grids at once is probably beyond the technical capability of any hackers, Caltagirone said, and would require so much preparation that it would almost certainly not escape notice.
Within each of the interconnections are those independent utilities, running power generation systems (coal-burning power plants, wind turbines, etc.), transmission lines (carrying high-voltage electricity long distances) and distribution systems (the substations that convert the high-voltage electricity to something your house can use and then the lines that carry it to you).
At the micro level, the system is designed for redundancy.
“Electric grids are fairly resilient,” Brad Bauch, lead on cybersecurity and privacy at PwC, said in a phone conversation. “When there’s physical damage, such as ice storms, hurricanes, that sort of thing — that tends to wreak a good deal of havoc, but overall they’re fairly resilient and well-engineered, so there’s a lot of redundancy. So if a certain transmission line goes down, then there is, for the most part, some redundancy for the power to be rerouted in another direction.”
“The thing that saves the grid, honestly, is that the grid goes down all the time,” Caltagirone said. “The grid is highly reliable because it has to be, because the environment that it’s in is unreliable.”
What’s more, many key systems aren’t directly connected to the Internet — or to one another.
“A lot of these systems are disconnected, but as we’ve built the smart grid and we’ve started to put in more advanced metering infrastructure, they are starting to become more and more interconnected — but interconnected in an individual utility,” Bauch said. “For the most part, they are disconnected or should be disconnected from other corporate systems or the Internet.”
If a Russian hacker (or any hacker) gains access to Con Edison in New Jersey, they might be able to create some significant headaches. But the experts with whom we spoke indicated that it was all but impossible that, from there, the hackers would be able to put the entire nation at risk. The Con Ed system isn’t going to let them take down Georgia Power or AEP. Those are separate systems running in separate ways and running independently of one another.
We go back to our elections-systems analogy. A hacker who manages to get access to the voter file in Autauga County, Ala., isn’t going to be able to change the results of the presidential contest in Montana.
What the Russians allegedly did
An official with the Department of Homeland Security indicated there was no evidence that, having gained access to the control system for that turbine, the hackers did anything to manipulate the system in the real world. So it raises the question: What were they doing?
“What we’re seeing here is what we would call the precursors or the preparation of the environment,” Caltagirone said. “Meaning that in order to achieve any successful disruption, whether they want to do it now or they’re preparing for a potential conflict way out in the future, what’s necessary are some critical pieces of intelligence on how the grid or any of these operators actually operate the system because none of these operators are cookie-cutter.”
Many grid operators use similar components to make up their systems, the same relays from the same manufacturer and that sort of thing. But the way they’re deployed and connected to one another varies widely. It’s like the plumbing in your house: If a stranger gains access to your bathroom, he can pretty easily figure out how to turn the water on and off. But from the outside, it’s going to be a lot harder to figure out where the pipes come in and how they route to your sinks and showers. Each house is different.
“In order to actually achieve effects here, you have to understand how each of these operators are operating: What technologies they’re using, how they’re configured and so forth,” Caltagirone said. That means breaking into each system and getting to know how it works.
The hackers would need to become almost as adept at using the system as the operators employed by the power company, he said, to know how and where to have an effect.
“Operational networks are very complicated. Just because you push a bunch of buttons — if you sat at a control panel and you just pushed and slammed a bunch of buttons, it doesn’t mean that anything is going to happen,” Caltagirone said.
A report released by the Department of Energy in January 2017 outlines how that access is gained. (It mirrors the specific example used in Thursday’s alert from DHS.) After a target is identified, the hackers try to gain access to public-facing Internet sites controlled by the company. Often this is done through “spear-phishing,” sending an employee (or a number of employees) a disguised link meant to trick them into entering their system credentials on a hacker-controlled server.
With those credentials in-hand, the hackers establish a footprint in the network, often creating new accounts or collecting other credentials in order to assure continued access to the system. The hackers also then download malware — malevolent software — that allows them to explore and manipulate the system. In the example cited in this week’s alert, the hackers installed a client that allowed them to connect to the company’s network through a virtual private network. They also installed an additional hacking tools downloaded from public and private sites. And: They poke around. Figure out how it works.
The natural question that arises is how a hacker gets from the public-facing website of a company to the control system that operates things like power-generation systems. Obviously the power companies are going to avoid making critical control systems accessible to anyone who knows the URL.
This is where we go back to Bauch’s point about the increasing connectedness of smart grids and metering infrastructure. As Caltagirone pointed out, at some point the data on, say, how much electricity you use needs to flow from the distribution system into the public-facing website so you can view your bill. To oversimplify things, the hackers use that pathway in reverse.
Figuring out how to do that is “very, very hard work,” he said. Last summer, the government warned about attempted intrusions at nuclear facilities. In those cases, the hackers appear to have not been able to get from the public-facing systems to the control systems behind the scenes.
There are other ways to gain access to a network or leverage over a grid operator. The sanctions announced on Thursday included a reference to the NotPetya attack last year.
“That is really a supply-chain attack vector,” Bauch said. “The companies weren’t attacked individually or specifically: A software company was attacked, and malware was then injected into the software that a lot of companies use. It was then sophisticated enough to spread itself around.” The hackers were able to get into multiple systems, in other words, without having to hack each system separately.
Here, again, there are echoes with the election intrusions. Shortly before the 2016 election, Russian hackers are believed to have compromised the network of a voting machine manufacturer, according to information stolen from the National Security Agency and provided to The Intercept.
What they might be able to do
Now our hackers are in place and have studied the control systems for one or more of the power companies dotting the grid. What’s next?
To answer that question, each of the experts I spoke with pointed to a 2015 attack in Ukraine.
As explained in a detailed account published by Wired, on the afternoon of Dec. 23 hackers began to systematically shut down power systems in Ukraine and to lock out the systems operators who watched as cursors slid across their screens, controlled by an unseen hand. Eventually, 230,000 people in the country lost power.
That attack began as a phishing attack, explained Doug Westlund, senior vice president of the consulting firm AESI. He noted that the Ukraine incident has become a go-to case study for how to prepare for such an attack.
In this case, users were sent a document that asked them to enable macros; if they did so, Wired reports, malware was installed on their computers. Unable to break through the firewall protecting the control systems network from the public-facing system, the hackers explored the system until they found a way in. They also corrupted the software on devices that allowed remote control of the physical substation systems, making it that much harder to Ukraine to get back online.
Ukraine had one advantage for getting back online, Caltagirone said: After repeated attacks from Russia, the country was already operating on a manual basis that allowed them, in certain places, to switch back on quickly.
In the United States, that might not be the case, since there’s “such a mix of ways” in which power is generated and distributed.
If hackers were to similarly shut down elements of the grid here, it could take days or weeks to restore operation. It depends on what sort of damage was done. Shutting off power is one thing. Deploying software that causes physical damage could extend recovery time significantly.
Dragos, Caltagirone’s company, identified one such piece of malware last year. Called TRISIS, it affects safety systems that are triggered in the event of other failures. If there’s a crisis in a facility and the safety system is deactivated, the situation could become life-threatening. An experiment in 2007 demonstrated another way in which code could do physical damage, by causing a generator to self-destruct.
There are statistics, Bauch said, showing that “the amount of malware being produced in the world has now surpassed the amount of valid software produced.”
Again, though, this damage would be localized. Perhaps, Bauch said, a hacker who’d gained access to a regional power provider or generation system might be able to knock out power to tens or hundreds of thousands of people at a time. That would be significant, but it’s not taking out the whole grid. Attacking multiple providers across the country at specific times of vulnerability — like on a hot day in the west when power supplies are strained — could multiply those effects. But it would require a significant amount of planning, coordination and access to have an impact on a massive scale.
Consider the tree that cut power to millions.
In 2003, 50 million people in the Eastern Interconnection lost power when a system failure cascaded through a number of providers. The initial culprit was a tree coming into contact with a power line that began to sag because of the high current it was carrying. That failure pushed high current into several other lines, which sagged, hit trees, and shut down. The effects continued from there.
“What you learn” from reading the report of what went wrong in 2003, Caltagirone said, “is that the grid is so darn complicated that we didn’t know how this happened.” The idea that hackers could find a vulnerability as effective as the one that tree stumbled upon seems unlikely.
But, as was the case in Ukraine, shutting off power for everyone might not be the goal. In addition to shutting off power, the hackers in that case also flooded the power company’s complaint lines with calls in an effort to make it harder and more frustrating for customers who were suddenly in the dark. Not only did they not have power, they couldn’t even phone the power company to complain about it. It was an added level of frustration apparently meant, experts interviewed by Wired suggested, to stoke frustrations with the Ukrainian government.
Which is to say that much of this is psychological. Taking out power for hundreds of thousands of people for 24 to 48 hours is a big deal, certainly, and would cause huge economic damage. But demonstrating that vulnerability would have intangible effects, too, prompting all sorts of concerns about the vulnerability of American infrastructure. In our conversation, Westlund repeated this point consistently: The grid is more robust than people seem to think and the perception that the grid is at risk itself contributes to the problem.
Thanks to the distributed nature of our elections and the barriers to changing votes, hacking our elections is trickier than most people realize. Thanks to the distributed and often disconnected nature of our electrical system and the barriers to accessing it, the same can be said of hacking the grid.
That screenshot of the control system for that turbine shows that systems can be compromised. That’s why the Department of Homeland Security released the image and the details of past intrusions: To inform industrial control system operators about what to look for.
It’s a serious situation, warranting the sort of dramatic response we saw from the government on Thursday. But do not expect to wake up some day soon and learn that Russian President Vladimir Putin now controls the flow of electricity to your house. Real life, as always, is less dramatic than the movies.