Some users were frustrated to find some of their favorite Web sites were unresponsive or otherwise inaccessible Tuesday. But it wasn't a data center outage or a squirrel chewing through a cable line causing the disruption. Instead, structural problems with one of the core technologies that keeps the Internet working were to blame, researchers say.
You may not have heard of Border Gate Protocol (or BGP) routing, but you likely rely on it every day if you're online. While we often think of the Internet as one big network, it's really a collection of smaller, interconnected networks -- and BGP is how traffic finds its way from one part of the globe to another.
"It's kind of like the telephone directory for every Internet Service Provider, showing how it sends traffic to every other part of the Internet," explains Doug Madory, a senior analyst at Internet monitoring firm Renesys. "We have a vast, global Internet here, and it can be described in about 500,000 lines, which is kind of amazing."
"The beauty of it," he says, "is if one change occurs in one part of the world, within seconds it is propagated throughout the rest of the world." Much of the network depends on trust: Servers publicly announce their routes, which are then rebroadcast to help networks know the best path for their data. Because of that variation, the total number of routes changes depending on the time they're accessed and other factors.
But some older hardware has trouble handling things when the number of routes reaches above 512,000 because of restrictions in the hardware's specific type of memory. And on Tuesday, researchers say, it appears some servers vulnerable to that issue crossed that threshold.
According to a blog post from Andree Toonk at BGPMon, starting at around 7:48 UTC roughly 15,000 new prefixes were introduced into the global BGP routing table. The prefixes "almost all" came from some Verizon autonomous systems and appear to be more specific announcements for paths in their larger aggregate blocks, he wrote. Essentially, it appears that Verizon leaked out internal routes that divide up existing pathways into much smaller pieces to the routing tables that direct traffic on the global Internet.
According to Madory, BGP messages carrying the leaked Verizon prefixes peaked at over 5 million per minute. Verizon declined to comment by press time.
But the issue shouldn't have been a huge shock. Cisco published a blog post reminding administrators about the 512,000 threshold and suggesting ways to work around the problem when the BGP routing tables started to pass the 500,000 market earlier this year.
"As an industry, we’ve known for some time that the Internet routing table growth could cause Ternary Content Addressable Memory (TCAM) resource exhaustion for some networking products," the company wrote. "Most platforms have more than enough space to support larger routing tables, but the default configurations might require adjustment."
While the hiccups seem to be mostly resolved now, the issues experienced by many users on Tuesday serve as a testament to the somewhat precarious nature of the systems the Internet is built upon -- even if the issue didn't amount to a full-blown Internet emergency.
BGP is tightly integrated into the way the Web works, and it would be quite time-consuming to transition the infrastructure associated with it to another system. So it's unlikely the Internet will move on to a new framework any time soon. And BGP is a somewhat elegant solution to the problem of how to find the best routes for Internet traffic sent around the world -- it's fluid enough that, for the most part, things get to where they are needed.
But BGP also contains known vulnerabilities. In the past, researchers have been concerned about potential man-in-the-middle attacks leveraging the system, where an adversary could potentially abuse the trust in the network to snoop or even modify traffic. And just last week, Dell cybersecurity researchers said that an "unknown entity" repeatedly used BGP hijacking to direct networks belonging to Amazon, Digital Ocean, OVH and other large hosting companies to mine bitcoins between February and May 2014, walking away with an estimated $83,000 in just over four months.