The virus struck in an e-mail 81 days ago, flagged by a federal team that monitors cyberthreats. The target was a small job-development bureau in the Commerce Department. The infiltration was so vicious it put Commerce’s entire computer network at risk.
To avert a crisis, the Economic Development Administration (EDA) unplugged its operating system — and plunged its staff into the bureaucratic Dark Ages.
E-mail? Gone. Attachments, scans, Google searches? Until further notice, no such thing.
Employees became reacquainted with their neighborhood post office and the beep-squeak-hiss of the fax spitting out paper. The must-have office supply became toner for the machine.
Twelve weeks offline and the longest intrusion into a federal network in recent history is still wreaking havoc.
“We don’t yet have any deeper understanding of what happened,” Commerce Secretary John Bryson said in an interview. “But we have the best resources in the federal government looking into this.”
The hackers so far have outrun those investigators; the malware’s origin remains unknown.
The EDA gives grants to distressed communities out of six regional offices, with a small Washington presence. It has 215 employees, a tiny corner of the federal landscape.
But its crippled system is evidence that every government network is vulnerable to cyberattacks that could disrupt business and spread. The number of intrusions into federal systems reported to the Department of Homeland Security’s U.S. Computer Emergency Readiness Team exploded to 44,000 in fiscal year 2011 from 5,500 in fiscal 2007. They ranged in severity from malicious software to unauthorized computer use.
Most of the attacks did not knock out entire networks. They were erased or swatted away with anti-virus tools, password changes and other security steps.
Other attacks were serious. In recent years, hackers have penetrated e-mail and other systems at the Defense and State departments and NASA and disabled another Commerce bureau that handles sensitive information.
Cyber-experts have repeatedly pointed to a lack of system security at the Commerce Department. The agency’s IT systems “are constantly exposed to an increasing number of cyber attacks, which are becoming more sophisticated and more difficult to detect,” Inspector General Todd J. Zinser wrote last year.
As an outside security team tries to isolate the current culprit, the EDA has spent weeks building from scratch a new operating network that requires servers and equipment and a complex security firewall to prevent another virus from working its way into the new system.
Business has limped along as employees slowly are brought back online on the new network. The hackers’ motives, whether economic espionage or something else, are unknown.
The bottom line for now: Make do.
The already long vetting process for grants slowed. How fast, after all, could it move when paperwork had to be sent by snail mail?
In the field, the first sign of trouble in January was bouncing e-mails.
In Rochelle, Ill., economic development director Jason Anderson was waiting for word on funding for a railroad spur between a freight line and a new rail car plant under construction in his city. Finally, he dialed the EDA’s Chicago office.
“I said, ‘Is there a problem on your end?’ ” he recalled. “They said, ‘Yeah, there’s a problem. We’ve just had a major computer meltdown here in the Region 5 office.’ ”
Rochelle, 75 miles west of Chicago, is struggling to create jobs. Anderson was hearing little from Chicago. He was getting impatient.
Then the official reviewing his application did something very unbureaucratic. He gave Anderson his private cellphone number.
“He would make sure if I needed an answer to a question I got it,” Anderson recalled.
The announcement for a $2.4 million award arrived in February — by fax.
People are rediscovering what it is like to scribble down a “When you were out” slip. They pick up the phone, calling congressional staff members, for example, to announce a grant in their districts. They meet potential clients face to face.
With their data frozen on infected PCs and no place in the field to scan federal forms, staff members have retyped hundreds of pages into word processors, key by key.
“If someone told me I wouldn’t have e-mail for this long, I would have said it’s not possible,” said Jane Reimer, a planner in the Denver office who manually processed hundreds of grant applications. “I thought it was my lifeline.”
Employees refer to the outage as “the disruption.” At Commerce Department headquarters on Constitution Avenue NW, managers panicked at first. How would business get done?
“There were things like, ‘How are we going [to] do our payroll?’ ” external-affairs chief Angela Martinez recalled.
Work hours were submitted from local libraries, home computers or mobile phones. The payroll went out on time.
Employees were instructed to call their clients and ask how they wanted to communicate without the Internet.
“It may just be better instead of trying to fax someone 100 pages to decide it’s going to be in the mail,” said Philip Paradice Jr., director of the Atlanta office.
“It’s not necessarily something you look forward to in life,” he said. “But there’s a certain invigoration that’s come up. We’ve come up with work-arounds.”
Some managers at headquarters were able to log on to servers at other Commerce Department sites, but in limited doses. The risk of spreading the infection was too great.
For some veterans, the technological blackout had a familiar feel.
The EDA is a slow-turnover place. Plenty of people there remember when they looked up numbers in the Yellow Pages or addresses on a map.
“In those days, we had secretaries and clerical people,” said Shirley Marshall, a project engineer in Denver who started in the agency’s West Virginia office in 1966. “We wrote reports and gave them to the secretary to type up. I’m sure living without the Internet has been easier for me than some younger employees.”
The agency is starting over, issuing employees new e-mail addresses, Blackberrys and laptops on loan from the Census Bureau. A skeletal Web site was restored last week. Fax machines have been ordered for staff members who work from home. Scanning and attachments are off-limits for now, however, and files and e-mail from the infected computers have not been recovered. Bugs are slowly being worked out. With security concerns so high, logging in to the new system is cumbersome, employees said.
Commerce officials declined to say how much the crisis is costing.
There has, however, been an upside: human contact.
“You pick up your phone and you get back to some human interaction,” said Chris Massengill of the Delta Regional Authority in Clarksdale, Miss., which works with the federal government to jump-start development in the Delta, “which in my opinion is never a bad thing, especially for government.”