Crash of Va. computer network has implications for tech world, state politics
Thursday, September 2, 2010
RICHMOND -- The data storage unit that failed in a warehouse outside of Richmond last week, wreaking havoc in the computer networks of a number of Virginia agencies for more than a week, is a ubiquitous bit of technology used by virtually every major company and government in the country.
The crash -- still baffling to state officials -- exposes the vulnerability of modern, massively complex interconnected computer networks, and is being closely watched by information technology professionals across the country.
"People in the industry are watching in horrified fascination as this unfolds," said Robin Harris, an Arizona technology analyst who writes a blog on computer storage systems. "There's a lot of 'there but for the grace of God go I' kind of thinking."
It is also a political headache for Gov. Robert F. McDonnell (R), who promised on taking office that he would straighten out the state's troubled $2.4 billion computing contract with Northrop Grumman and is now grappling with what appears to be the worst network failure since the company took over the state's technology overhaul in 2003.
More than a week after the initial hardware crash, state officials said operations would be fully restored at Virginia's 74 Department of Motor Vehicles offices Thursday. They estimate that as many as 45,000 people have been unable to renew their driver's licenses while computers have been down, and the agency will extend its hours in coming days and weekends to process the backlog. Officials reported Wednesday that the state's Department of Taxation was able to access taxpayer accounts and issue refunds and liens for the first time in seven days.
Employees throughout state government worked long hours for days to restore computer functions. At the Department of Social Services, local and state staffers had to work through the weekend to ensure that food stamps and welfare checks due to 380,000 residents were not delayed. The Department of Juvenile Justice was unable to release inmates. The Department of Veterans Services, which manages two long-term care centers and two cemeteries, couldn't pay its bills. In all, computers at 26 of the state's 89 agencies were affected.
Chief Information Officer Sam Nixon said Thursday that the problem began Aug. 25 with the crash of a pair of three-year-old memory cards -- one was supposed to back up another. That led to 485 of the state's 4,800 data servers being knocked off-line.
"The thing that is never supposed to happen, happened," he said.
Built by Massachusetts-based EMC, the storage units are fundamental building blocks used to hold the mass databases of information necessary to run complex organizations in a digital age. Industry analysts say EMC's drives are used throughout the world of finance and government and have generally been considered highly reliable. If the units were found to have flaws that caused the failure, the impact could be felt far beyond Virginia.
"This is surprising -- it's a selling point for them when they talk to a major organization, that this stuff never goes down," said Bill Kreher, a senior technology analyst who monitors EMC at the investment firm Edward Jones.
Experts say that even more troubling than the failure of the initial storage drive -- which was repaired by Friday -- is how long it has taken state agencies, Northrop Grumman and the Virginia Information Technologies Agency to restore computer applications.
Nixon acknowledged that it took much longer than it should have for the system to be restored and said he plans to study ways to speed up the process.
In a statement this week, Northrop Grumman, which will face a state fine of more than $100,000, vowed to learn from the problems and correct them. EMC officials did not return calls for comment. State officials have said the company reported that the Virginia outage was unprecedented, an occurrence never before seen in 1 billion hours of system use.
The outage has compounded a public relations disaster for VITA and Northrop Grumman and could have political implications for McDonnell as well.
McDonnell, who last year criticized his Democratic predecessor, Timothy M. Kaine, for failing to properly manage the Northrop Grumman contract, pledged to run such programs as businesses. His solution was to rework the massive contract with Northrop Grumman, extending the 10-year agreement by three years and agreeing to pay the company $100 million more than originally envisioned, but adding new penalties for poor service.
Democrats are using the incident to call into question McDonnell's management style and his propensity for privatization.
"We're paying Northrop to run modern, quality service -- and they have failed to do it every step of the way," Sen. Janet D. Howell (D-Fairfax) said. "At the time, I was angry that the governor extended their contract for three years when they still weren't performing. The anger is only growing."
McDonnell and his administration have commented only briefly about the issues over the past week, choosing to let VITA take the lead on speaking to the media. The governor has called for an independent, third-party investigation into the network crash but has yet to issue details. Jim Duffey, McDonnell's secretary of technology, declined to comment.
Then-governor Mark R. Warner (D) proposed consolidating the state's computer operations into one agency and overhauling the system after a legislative review in 2002 showed that the state's computer system was out of date and increasingly expensive to maintain. The General Assembly, then controlled by Republicans, approved.
The state accepted bids and selected Northrop Grumman, the giant Los Angeles-based defense company, for the contract, the largest of any kind in Virginia's history. But the company's contract has been plagued by problems from the start, including missed deadlines and poor service. A blistering legislative audit released in October found that the computer system had caused problems at almost every state agency that uses computers.
"Big, newly installed systems often have quirks -- bugs," said Del. Joe T. May (R-Loudoun), who chairs the Joint Commission on Technology and Science. "This turned out to be much more difficult than we thought."
Researcher Meg Smith contributed to this report.