The Washington PostDemocracy Dies in Darkness

Federal investigators to examine equipment from 2016 N.C. election amid renewed fears of Russian hacking

Students of North Carolina Central University wait in a long line during a problem in the elections computer system in Durham County, N.C., on Nov. 8, 2016. (Sara D. Davis/Getty Images)

It was a single phrase, offered without elaboration, in special counsel Robert S. Mueller III’s report: In August 2016, hackers working for Russian military intelligence “installed malware on the company network” of an unnamed voter registration technology vendor in the United States.

The claim amounts to one of the first indications that Russians successfully executed a cyberattack against a private company supporting American election infrastructure. And it has set off a scramble for answers in North Carolina, where officials have long been concerned about the security of a voting technology company called VR Systems — so much so that the state tried to halt the use of its electronic poll books, equipment used to check in voters.

Problems checking in voters on Election Day 2016 in Durham County, one of the largest counties in the state, made national headlines. Later, leaked documents revealed that Russians had tried to hack VR Systems shortly before the election. After the Mueller report, state officials wanted to know: Was VR Systems the company referenced? What effect, if any, did the malware have? And how could they prevent it from happening in 2020?

An examination of North Carolina’s struggle to answer those questions, detailed in court records and a dozen interviews, hints at the difficulties state officials face in shoring up security ahead of next year’s elections — a lack of technical expertise, poor communication between state and federal officials, and the apparent unwillingness, in this instance, of the federal agency involved to share information. Russian hackers targeted elections infrastructure in at least 21 states, and likely more, in 2016, federal officials have said.

“It would be great to know what vulnerabilities exist that we need to be preventing,” Josh Lawson, who was general counsel for the North Carolina State Board of Elections until last week, said in an interview. “It’d be nice to know how [poll books] can be gotten to, and if they’re gotten to, what that can look like.”

Hackers could not directly alter vote totals by breaching poll books. But they could create substantial delays or even prevent voters from casting ballots.

On Tuesday, the Department of Homeland Security told The Washington Post it will conduct a forensic analysis of the laptops used in Durham County elections in 2016. Lawson said North Carolina first asked the department to conduct such a review more than 18 months ago, though he added that DHS has generally been a “good partner” on election security.

“We appreciate the Department of Homeland Security’s willingness to make this a priority so the lingering questions from 2016 can be addressed in advance of 2020,” said Karen Brinson Bell, the newly appointed executive director of the State Board of Elections.

In a recent statement to The Post, VR Systems said that “we think we are the company referenced” in the Mueller report — a document produced by a team of prosecutors and FBI investigators — but denied that its network was ever breached and said its technology did not fail in Durham County in 2016. The company’s election equipment is used in at least a half-dozen states, according to its website.

Spokesmen for the Justice Department and the FBI declined to comment on the company’s claims or to elaborate on the malware reference.

The FBI came under fire from Florida lawmakers last month for being slow to share information on another Mueller report contention — that Russian hackers had accessed “the network of at least one Florida county government in 2016.” 

The FBI now says two Florida counties’ voter registration databases were breached. The bureau has not publicly named the counties, and it did not disclose them to state officials until the Mueller report touched off a demand for answers. (Citing U.S. officials, The Post reported last month that one of the counties was Washington County, which has a population of about 25,000 people and is in Florida’s Panhandle.) 

“It is untenable to continue to hold this information classified and not to let the public know,” Rep. Stephanie Murphy (D-Fla.) told The Post after the FBI briefed lawmakers. She has since announced plans to seek legislation that will force federal law enforcement to more quickly report election-related breaches.

Interrupted voting

On Election Day 2016, poll workers across Durham County reported problems with the VR Systems poll book, known as “EViD.” The software was showing that some voters had already cast ballots, while the voters themselves said they had not. It was also prompting poll workers to ask for some voters’ picture IDs, even though a North Carolina law that required those IDs had been struck down.

Alarmed, local officials decided to stop using EViD for the rest of the day. That slowed things down further, as workers scrambled to get paper poll books. In one precinct, voting stopped for two hours, local media reported. The county asked state officials to extend voting by 90 minutes in all its precincts; ultimately, extensions of 20 to 60 minutes were granted. 

No one knew the root cause at the time. But Susan Greenhalgh, an election security advocate who was fielding calls from voters and poll workers for the nonpartisan Election Protection Coalition Hotline that day, thought of a disturbing connection: CNN had recently reported an alleged Russian cyberattack against a “contractor for Florida’s election system.” She’d come to suspect, through her own digging, that the vendor was VR Systems. 

“This is the same vendor that was hacked,” she remembers thinking. 

After the election, Durham County hired a firm called Protus3 to dig into what happened. The security consultant said it appeared the problems were caused by user error but ended its 12-page report with a list of recommendations that included examining computers in a lab setting and interviewing more election workers.

Durham County elections director Derek Bowens said he is comfortable with the report’s conclusions. Even so, in 2017, the county switched to electronic poll books created by the state. Bowens said in an interview that the state’s software would save money and is, in his view, better.

But for North Carolina officials, concerns resurfaced in June 2017 when the website Intercept posted a leaked National Security Agency report referencing “cyber espionage operations against a . . . U.S. company in August 2016.” The NSA report said that “it was likely that at least one account was compromised.”

VR Systems soon acknowledged that hackers had targeted the company but insisted that its network had not been breached.

North Carolina officials weren’t so sure.

“This was the first leak that indicated anything like a nation-state actor targeting a voting systems vendor,” Lawson said.

The state elections board soon launched its own investigation, seizing 40 laptops from Durham in July. And it suspended the certification that allowed more than 20 North Carolina counties to use VR Systems’ poll books during elections, an action that would later land in court. “Over the past few months there has been a considerable change in the election security landscape and the level of scrutiny we receive,” the board wrote in a letter explaining its decision to VR Systems.

No one working for the board had the technical expertise to do a forensic examination of the machines for signs of intrusion. Staffers asked DHS for technical help but did not get a substantive answer for a year and a half, Lawson said. The board went so far as to file a Freedom of Information Act request with DHS in the fall of that year.

It never got a response, Lawson said.

DHS spokeswoman Sara Sendek declined to comment on Lawson’s account. “DHS has been in regular contact with the State Board of Elections since 2017 offering and providing a variety of services,” she said.

The agency said it routinely scans North Carolina election systems for vulnerabilities. It said it has no information about “any previous or ongoing issues regarding election systems” in North Carolina.

Suspension appealed

In September 2017, three months after the Intercept report, VR Systems for the first time heard from DHS about the hacking attempt, the company said. The communication “came in the form of a voicemail left around 2:00 a.m.,” the company wrote in court filings. “The voicemail stated something to the effect that VR Systems may have been the subject of a phishing attempt and VR Systems should give Homeland Security a call.”

The call struck VR Systems chief operating officer Ben Martin as being so unusual that he called his contacts in law enforcement “to ask if this was a hoax or if it was actually DHS,” he said in an email to The Post. He was assured the message was real.

“We had already been in close contact with the FBI and DHS and continued to work with them after that,” he said.

That same month, the elections board said in a news release that it had received reassuring information from DHS: North Carolina was not thought to be among the 21 states whose election systems were targeted in 2016.

“We are greatly relieved to hear that North Carolina’s systems were not directly targeted by Russian hackers,” Kim Westbrook Strach, then the board’s executive director, said in the Sept. 22, 2017, release.

But in an affidavit filed later in court, Strach noted a caveat. “The Department could not state any conclusion regarding the success of hacking activities against third-party election vendors operating in North Carolina, including VR Systems,” she wrote.

Later that fall, less than a month before local elections, VR Systems appealed its suspension, arguing that it had not received proper notice or a hearing. An administrative law judge agreed and issued a temporary order that forced the state to allow counties to continue using VR Systems.

North Carolina appealed the order in state court on Nov. 6, the day before the elections. It argued that the “malfunctions of EViD last year in Durham combined with a Russian cyber espionage campaign against VR Systems combine to pose a significant potential harm to the State’s elections.”

The court dismissed the case that same day, finding that it lacked jurisdiction and letting the administrative law judge’s ruling stand. An appellate court later agreed. Neither court addressed the merits of the state’s election security concerns.

In early 2018, North Carolina Attorney General Josh Stein (D) sent a letter to DHS pleading for more information about VR Systems and the problems in Durham County. “North Carolina’s situation illustrates the need for information-sharing among all levels of government,” he wrote. “. . . Without this information, state officials may be unable to take appropriate and necessary actions to ensure that hacking and other forms of disruption by malicious actors do not occur in upcoming elections.”

The attorney general did not get a “substantive response,” spokeswoman Laura Brewer said in an email. “We continue to be very concerned about this issue.”

Last July, the FBI indicted 12 Russian government agents for allegedly interfering in U.S. elections. The documents once again referred to a company that matched the description of VR Systems, stating that “in or around August 2016” the accused “hacked into the computers of a U.S. vendor that supplied software used to verify voter registration information for the 2016 U.S. elections.” 

'A right to know'

Shortly after the release of Mueller’s final report, Sen. Ron Wyden (D-Ore.), a member of the Senate Select Committee on Intelligence and one of the most outspoken U.S. lawmakers on issues of election security, sent VR Systems a letter asking for assurances of the security of its network.

“Given the voting problems caused by the failure of electronic poll books manufactured by your company in November 2016, the American people have a right to know if there was any connection to the Russian cyber attack against your company three months earlier,” Wyden wrote.

In response, VR Systems said a cybersecurity firm it hired to review its computer network in 2017 found no evidence of a hack. A subsequent review by DHS also found no issues, the company said. VR Systems declined to give Wyden documentation of those reviews, citing the need to protect proprietary information.

Wyden in a statement to The Post accused VR Systems of “stonewalling congressional oversight.”

A senior U.S. official confirmed DHS’s review of VR Systems’s network to The Post and noted that by the time agency investigators arrived, a commercial vendor had already “swept” the networks. “I can’t tell you what happened before the commercial vendor came in there,” the official said, speaking on the condition of anonymity to discuss a sensitive matter. The official said DHS has no information about the malware mentioned in the Mueller report. 

Today, VR Systems continues to operate in many North Carolina counties. To avoid future court battles, state lawmakers explicitly granted the Board of Elections the power to“halt the use of electronic poll books.” The law also says that companies must agree to cover the cost of a new election if their products fail. Voting machine vendor Election Systems & Software has agreed to pay $17 million in the event of failure, while VR Systems and the state are still negotiating over a number, board spokesman Patrick Gannon said.

North Carolina officials say they’ve made progress in securing the state’s elections since 2016. After receiving $10 million in federal grant money last year, the state is in the process of hiring its first chief information security officer along with analysts dedicated to election integrity.

Two staff members, Lawson and Strach, obtained security clearances so they could receive classified briefings on election-related issues. But last month, the State Board of Elections voted to replace Strach; the board is now controlled by Democrats, and Strach had been appointed by Republicans. As a result of the decision, a week ago, Lawson, too, left the agency.

Ellen Nakashima contributed to this report.