The Washington Post

Here’s why your Internet might have been slow on Tuesday

Some users were frustrated to find some of their favorite Web sites were unresponsive or otherwise inaccessible Tuesday. But it wasn't a data center outage or a squirrel chewing through a cable line causing the disruption. Instead, structural problems with one of the core technologies that keeps the Internet working were to blame, researchers say.

You may not have heard of Border Gate Protocol (or BGP) routing, but you likely rely on it every day if you're online. While we often think of the Internet as one big network, it's really a collection of smaller, interconnected networks -- and BGP is how traffic finds its way from one part of the globe to another.

"It's kind of like the telephone directory for every Internet Service Provider, showing how it sends traffic to every other part of the Internet," explains Doug Madory, a senior analyst at Internet monitoring firm Renesys. "We have a vast, global Internet here, and it can be described in about 500,000 lines, which is kind of amazing."

"The beauty of it," he says, "is if one change occurs in one part of the world, within seconds it is propagated throughout the rest of the world."  Much of the network depends on trust: Servers publicly announce their routes, which are then rebroadcast to help networks know the best path for their data. Because of that variation, the total number of routes changes depending on the time they're accessed and other factors.

But some older hardware has trouble handling things when the number of routes reaches above 512,000 because of restrictions in the hardware's specific type of memory. And on Tuesday, researchers say, it appears some servers vulnerable to that issue crossed that threshold.

According to a blog post from Andree Toonk at BGPMon, starting at  around 7:48 UTC roughly 15,000 new prefixes were introduced into the global BGP routing table. The prefixes "almost all" came from some Verizon autonomous systems and appear to be more specific announcements for paths in their larger aggregate blocks, he wrote. Essentially, it appears that Verizon leaked out internal routes that divide up existing pathways into much smaller pieces to the routing tables that direct traffic on the global Internet.

The spike in routes advertising the leaked Verizon prefixes yesterday. (Renesys)

According to Madory, BGP messages carrying the leaked Verizon prefixes peaked at over 5 million per minute. Verizon declined to comment by press time.

But the issue shouldn't have been a huge shock. Cisco published a blog post reminding administrators about the 512,000 threshold and suggesting ways to work around the problem when the BGP routing tables started to pass the 500,000 market earlier this year.

"As an industry, we’ve known for some time that the Internet routing table growth could cause Ternary Content Addressable Memory (TCAM) resource exhaustion for some networking products," the company wrote. "Most platforms have more than enough space to support larger routing tables, but the default configurations might require adjustment."

While the hiccups seem to be mostly resolved now, the issues experienced by many users on Tuesday serve as a testament to the somewhat precarious nature of the systems the Internet is built upon -- even if the issue didn't amount to a full-blown Internet emergency.

BGP is tightly integrated into the way the Web works, and it would be quite time-consuming to transition the infrastructure associated with it to another system. So it's unlikely the Internet will move on to a new framework any time soon.  And BGP is a somewhat elegant solution to the problem of how to find the best routes for Internet traffic sent around the world -- it's fluid enough that, for the most part, things get to where they are needed.

But BGP also contains known vulnerabilities. In the past, researchers have been concerned about potential man-in-the-middle attacks leveraging the system, where an adversary could potentially abuse the trust in the network to snoop or even modify traffic. And just last week, Dell cybersecurity researchers said that an "unknown entity" repeatedly used BGP hijacking to direct networks belonging to Amazon, Digital Ocean, OVH and other large hosting companies to mine bitcoins between February and May 2014, walking away with an estimated $83,000 in just over four months.

Andrea Peterson covers technology policy for The Washington Post, with an emphasis on cybersecurity, consumer privacy, transparency, surveillance and open government.
Show Comments

To keep reading, please enter your email address.

You’ll also receive from The Washington Post:
  • A free 6-week digital subscription
  • Our daily newsletter in your inbox

Please enter a valid email address

I have read and agree to the Terms of Service and Privacy Policy.

Please indicate agreement.

Thank you.

Check your inbox. We’ve sent an email explaining how to set up an account and activate your free digital subscription.