“Facebook, Instagram, WhatsApp and Messenger are coming back online now,” chief executive Mark Zuckerberg posted late Monday. “Sorry for the disruption today — I know how much you rely on our services to stay connected with the people you care about.”
The hours-long outage again shed light on the company’s huge swaths of power, something regulators and lawmakers are scrutinizing in the wake of new revelations from a whistleblower about the company that she alleges proves it has been negligent in eliminating violence and misinformation from its platform.
The problems weren’t limited to external users. Facebook’s internal communication platform, Workplace, went down altogether for most of the work day, said a person familiar with the matter who spoke on the condition of anonymity because they weren’t authorized to speak publicly. And as employees turned to third-party tools such as Slack, many found themselves locked out of even those, because Facebook’s mechanism for signing on to them was not working, said another person familiar with the matter who spoke under the same conditions.
Reports on Downdetector suggest users were affected globally. The problems began about 11:40 a.m. Eastern time.
Facebook confirmed late Monday night that the issue was caused by configuration changes that interrupted network traffic between its data centers.
“We want to make clear that there was no malicious activity behind this outage — its root cause was a faulty configuration change on our end. We also have no evidence that user data was compromised as a result of this downtime,” Facebook vice president of infrastructure Santosh Janardhan wrote in a blog post.
In a separate post Tuesday, Facebook provided more technical details of the outage, saying the error happened during one of its “routine maintenance jobs,” which inadvertently took down Facebook’s network connections, and disconnected it from data centers that allow it to communicate online.
Earlier Monday, outside experts analyzed Facebook’s outage as a problem caused by an update internally.
“Something happened internally at Facebook that messed with their network settings on how Facebook talks to the rest of the world and accesses the Internet,” said Courtney Nash, senior research analyst at security company Verica.
The issue seems to be with Facebook’s border gateway protocol routes, or paths that allow routers to exchange information, said Doug Madory, director of Internet analysis for Kentik, a network monitoring company. Madory called them the “underpinnings of how the Internet operates.”
Facebook’s routes were withdrawn Monday morning, he said, and its apps couldn’t be found online because those routes contained the addresses of Facebook’s domain name system (DNS) servers, which translate familiar Web addresses, such as Facebook.com, into a string of numbers that computers can read. When the servers have trouble communicating, it can make websites unreachable.
It’s nearly unheard of to have such a large company go down for so long, Madory said.
“This is massive,” Madory said during the outage. “It’s completely dead.”
It’s also possible the outage was affecting other Internet services, Nash said. When the services went down, so many users tried to load the sites that it caused a run on traffic on the Internet’s DNS infrastructure.
“The reason these failures are so crazy is because there’s so much interconnectiveness of the Internet we rely on,” Nash said.
While the social networks were down, some users flocked to Twitter instead to complain. The hashtags #facebookdown and #instagramdown took off.
After Twitter tweeted “hello literally everyone,” it appeared the surge in usage prompted some problems.
“Sometimes more people than usual use Twitter,” it tweeted later. “We prepare for these moments, but today things didn’t go exactly as planned. Some of you may have had an issue seeing replies and DMs as a result. This has been fixed. Sorry about that!”
Some of Facebook’s leaders also turned to Twitter to share their thoughts.
“*Sincere* apologies to everyone impacted by outages of Facebook powered services right now. We are experiencing networking issues and teams are working as fast as possible to debug and restore as fast as possible,” Facebook’s chief technical officer Mike Schroepfer posted.
Instagram head Adam Mosseri tweeted that it “does feel like a snow day.”
The WhatsApp outage was particularly hard for a huge swath of the world that relies on it heavily for messaging, especially in the around two dozen nations where the app is the messaging market leader.
According to the Global Web Index’s 2020 Social Media User Trends Report, in seven countries — including Kenya, Malaysia and Colombia — more than 90 percent of those ages 16 to 64 are monthly WhatsApp users.
In the Middle East, where the public and governments rely heavily on Facebook and WhatsApp, the outage meant a near-complete communications blackout.
Phone calls and text messages are expensive in countries such as Lebanon and Jordan, causing residents to turn to WhatsApp in particular. The app also offers encrypted voice calls, an important feature in a region rife with government surveillance.
In some countries, including Lebanon, political and public announcements are made almost exclusively via Facebook.
Several international newspapers, from South Asia to South America, were running news of the shutdown as the top story. El Tiempo, a news outlet in Colombia, quickly published a list of alternatives to WhatsApp, including Telegram. In the United Kingdom, digital news outlet the Independent was running a live update file on the shutdown.
India has about 400 million WhatsApp users, and the service plays a heavy role during elections.
The World Health Organization capitalized on the moment to push forward more pandemic public health messaging.
Sammy Westfall, Sarah Dadouch, Will Oremus, Elizabeth Dwoskin and Heather Kelly contributed to this report.