Verizon details errors in derecho, calls response to 911 outages ‘insufficient’
Verizon officials did not know that 911 emergency service was out in Fairfax County during June’s derecho storm until the county called to tell them, according to a Verizon executive and a company report to be released this week to local governments.
The report, to be presented Wednesday to the Metropolitan Washington Council of Governments, for the first time acknowledges widespread failures in Verizon’s 911 emergency backup systems during the storm. It also admits significant problems with Arlington County’s 911 service that Verizon had previously denied.
In addition, the report says drained batteries, faulty generators and the failure of a technician to thoroughly explore the cause of troubles extended the delays in restoring full 911 service to nearly 2.3 million residents in Fairfax, Arlington and Prince William counties and Manassas and Manassas Park.
Maureen Davis, vice president of Verizon’s network operations for the Mid-Atlantic region, said in an interview that the company also made a mistake in treating the problem as a service complaint rather than the large-scale outage that it was. That meant that the company did not implement its own emergency command center procedures, even as the problems escalated at two offices — Verizon’s central office in Arlington and another in Fairfax — that route calls for multiple 911 centers, Davis said.
That left Verizon working “with a gang of people” but “not the army” it needed for a “full-blown emergency,” she said. The company’s response “was insufficient for what we were dealing with,” she said.
The failures left some emergency dispatchers unable to see incoming numbers and caller addresses from the beginning of the storm on June 29 until July 3, the report said. The report also includes remedies that Verizon said it has started to pursue.
The COG and Virginia and federal regulators are investigating what brought down one of the most essential emergency lines in the area.
Steve Souder, director of Fairfax County’s emergency services operations center, said the report was “a good first step on a journey that is a ways to go.”
But it also shows “they had a lot of weak links in their networks,” Souder said. “I can’t say I’m happy, but I’d rather find it out now, in a bad storm, than in a heck-of-a-lot-worse environment.”
In its report, Verizon says its backup power systems “should have withstood the Derecho without the resulting 911 problems.” But they went out, the report says, because of internal problems at Verizon.
After the storm knocked out power at Verizon’s Arlington and Fairfax facilities, batteries carried the system for about six hours at each site. Backup generators should then have shouldered the load. But the generators failed, Verizon said in the report, despite having been tested three days earlier.
At the Arlington site, the routine and limited testing had not checked whether a generator could carry a full power load in an emergency, Davis said.
The testing also did not identify that air had entered the fuel system in Arlington, a flaw found in a post-storm review, the report said.
At Verizon’s Fairfax site, the company’s tests did not check whether equipment that automatically signals a generator to work during a blackout was functioning properly, Davis said. The equipment failed during the storm.
The fuel lines in Arlington and the controller equipment in Fairfax have been replaced, Davis said, and over the next 60 days Verizon will be auditing systems throughout the greater Washington area to check on similar issues.
In the case of the service technician, a functioning generator was powering the lights and air conditioning in the part of the Verizon building where he was working, and four hours passed before another technician who came to the scene realized that a second generator was not working properly.
The failures at the Arlington site have implications beyond Northern Virginia. The location serves as a relay center to alert Verizon if network equipment serving other parts of Northern Virginia, the District and parts of Maryland are not working. The arrangement is similar to the system that alerts a home security company when something trips an alarm at a customer’s house.
So the lapse in Arlington left Verizon blind to the fact that Fairfax also had lost 911 service, Davis said.
The report also faulted Verizon’s communication with the local jurisdictions. Verizon relied on routine e-mails to emergency workers to say 911 was out and did not include more expansive alerts to more senior-level officials, the report said.