Updates to the Weather Service’s public-facing website, Weather.gov, ceased publishing.
In an email to staff on Tuesday, David Michaud, the director of the Weather Service’s Office of Central Processing, said a power outage had triggered the outage and characterized the impacts as “significant”. The cause of the outage was under review, a Weather Service spokesperson said.
“[I] want to ensure you that everyone involved is working hard to avoid these outages in the future and find ways to better communicate to employees across the agency in real time when outages occur,” Michaud’s email said.
Ryan Hickman, chief technology officer for AllisonHouse, a weather data provider who relies on the Weather Service feeds that were compromised, called the situation “catastrophic.”
Hickman said two core routers for transmitting information from the Weather Service offices out to satellites, which beam the information back to public service providers, stopped working. “There is a primary and a backup, and both have failed,” Hickman said.
At 1:33 p.m. Monday, the Weather Service sent out a message confirming that the core routers had gone down. “Engineers are looking for a work around,” the message said. “Most data for the U.S. is affected including RADARS.”
The Weather Service’s Michaud said when both routers failed, its Network Control Facility (NCF) “was not able to immediately remotely access the Backup NCF located in Fairmont, W. Va., and had difficulty successfully switching operations to the backup site.”
Ryan Hanrahan, a broadcast meteorologist in Connecticut, called the data outage “an embarrassing infrastructure failure” on Twitter.
While access to Weather Service updates on the Internet was cut off, some service providers who obtain Weather Service information through fiber optic landlines still received some data. In addition, National Oceanic and Atmospheric Administration Weather Radio, which broadcasts forecasts and warnings, continued operating.
Weather Service offices were also able to manually share information through social media.
During the outage, the Weather Service posted two messages on Twitter:
1) NWS is aware of the data outage impacting customers across the nation. We will pass along more detailed information as soon as possible.
2) In the meantime, please follow your local forecast office’s social media accounts for updates.
NOAA and the Weather Service have a history of problems with information dissemination. As Hurricane Matthew charged toward Florida in October, a “partial service disruption” made its website unavailable for many users. It also experienced a “major network issue” in July that impeded the issuance of forecasts and warnings. In 2013, it endured a host of systems failures.
During a media roundtable in late September, Weather Service Director Louis Uccellini said that dissemination upgrades were underway and that the agency had made “a lot of progress.” It completed its first phase of reconstructing and bulking up its dissemination system and, in its second phase, is working toward 99.9 percent reliability and having full backup capacity. “We’re about a year away,” Uccellini said at the time.