If you tried to check the National Weather Service forecast on Monday and were left waiting, and waiting, then you weren’t alone. The National Weather Service’s website, weather.gov, was made inaccessible for most of the afternoon and evening by a flood of data requests from a non-Weather Service Android app.
The website was slow to return data for most of Monday, and the point forecasts were completely unavailable. At 4:11 p.m. ET, the NWS Telecommunications Operation Center posted this message, indicating the culprit:
TO – ALL CUSTOMERS SUBJECT – POINT FORECAST ISSUES . WE ARE PROVIDING NOTICE TO ALL THAT NIDS HAS IDENTIFIED AN ABUSING ANDROID APP THAT IS IMPACTING FORECAST.WEATHER.GOV. WE HAVE FORCED ALL SITES TO ZONES WHILE WE WORK WITH THE DEVELOPER. AKAMAI IS BEING ENGAGED TO BLOCK THE APPLICATION. WE CONTINUE TO WORK ON THIS ISSUE AND APPRECIATE YOUR PATIENCE AS WE WORK TO RESOLVE THIS ISSUE. . NIDS – KM
This is why we can’t have nice things->A Single Android App Is Crippling the Nat’l Weather Service’s Website http://t.co/xHxAo6oAE0
— Brad Panovich (@wxbrad) August 25, 2014
In an email, Chris Vaccaro, the NWS Director of Public Affairs, described the problem:
Yesterday’s issue involved a phone app that scrapes forecast information directly from forecast.weather.gov. When a user launches the app, the phone queries forecast.weather.gov to get the forecast for defined points and keeps querying to keep the forecast current. But due to a programming error, the app was querying us thousands of times a second. We worked with the app developer to address the issue on their end and the NWS also installed a filter that prevents the faulty version of the app from hitting us.
Fortunately, it appears that watches, warnings, and advisories were still being disseminated to media and other applications, though if you rely on weather.gov to get your weather alerts, you were mostly out of luck (although the home page was loading, so it was possible to get a “big picture” view of the advisories across the country).
This was not the case in May when a firewall upgrade took down the NWS warning dissemination system during a severe weather event. That outage prevented a tornado warning near Albany, N.Y. from going out, and here in the D.C. area, severe storms were firing without warnings. That outage also included radar data, making it even more difficult for media to gather information on the approaching severe weather.
While Monday’s outage did not impact the more critical advisory arm of the Weather Service operations, the forecast suite outage is yet another concerning example of a system that is seemingly fragile and in need of improvement.
Maybe what is most surprising in Monday’s case was the amount of time that the website seems to have been impacted before a message was sent out that indicated the Weather Service was aware of the problem, and working on it. Access to the website was impaired for users beginning in the early afternoon (at least), and continued that way until the first message from the NWS operations center was sent around 4 p.m.
Monday’s outage was not the result of a malicious attack, but it’s not beyond imagination that malicious software could cripple the National Weather Service in the same way during a dangerous weather situation, like a land-falling hurricane or a severe weather outbreak. A safeguard algorithm to block the source of too-frequent requests could prevent outages like Monday’s in the future.
The problem was resolved by Monday night, when the application was blocked from requesting data from the National Weather Service website. At 10:43 ET, the telecommunications operating center posted this message:
AKAMAI HAS INSTALLED FILTERS WHICH BLOCK THE OFFENDING TRAFFIC. NIDS HAS VERIFIED THAT THE TRAFFIC IS BEING BLOCKED. ALL SYSTEM ARE NORMALIZED. WE APPRECIATE YOUR PATIENCE. PLEASE NOTIFY THE TOC AT TOC.NWSTG-AT-NOAA.GOV IF ANY FURTHER ISSUES ARE IDENTIFIED. THANKS FOR YOUR PATIENCE.
Update at 1:20 p.m. ET: In a separate email, Director of Communications Chris Vaccaro noted that NWS Director Louis Uccellini will be addressing the Weather Service’s technical issues in an upcoming episode of WxGeeks on The Weather Channel. “Clearly, the string of IT/dissemination issues is unacceptable and has the attention and action of leadership,” wrote Vaccaro.