A major flaw revealed this week in widely used encryption software has highlighted one of the enduring — and terrifying — realities of the Internet: It is inherently chaotic, built by multitudes and continuously tweaked, with nobody in charge of it all.
The Heartbleed bug, which security experts first publicly revealed on Monday, was a product of the online world’s makeshift nature. While users see the logos of big, multibillion-dollar companies when they shop, bank and communicate over the Internet, nearly all of those companies rely on free software — often built and maintained by volunteers — to help make those services secure.
Heartbleed, security experts say, was lodged in a section of code that had been approved two years ago by a developer that helps maintain OpenSSL, a piece of free software created in the mid-1990s and still used by companies and government agencies almost everywhere.
While the extent of the damage caused by the bug may never be known, the possibilities for data theft are enormous. At the very least, many companies and government agencies will have to replace their encryption keys, and millions of users will have to create new passwords on sites where they are accustomed to seeing the small lock icon that symbolizes online encryption.
“This was old code. Everyone depends on it. And I think that just everyone assumed that somebody else was dealing with it,” said Christopher Soghoian, principal technologist for the American Civil Liberties Union.
The group that was actually dealing with it consisted of fewer than a dozen encryption enthusiasts sprawled across four continents. Many have never met each other in person. Their headquarters — to the extent one exists at all — is a sprawling home office outside Frederick, Md., on the shoulders of Sugarloaf Mountain, where a single employee lives and works amid racks of servers and an industrial-grade Internet connection.
The total donations to the group last year, in support of work that keeps billions of dollars of commerce and countless personal secrets flowing safely across the Internet: less than $2,000. The group also makes money from consulting work.
“When you consider how complicated and significant a piece of software it is, and how critical a piece of infrastructure it is, it is kind of mind-boggling,” said Steve Marquess, president of the OpenSSL Software Foundation and a former federal technology contractor who works out of his Frederick-area house. “It’s such a thin thread.”
The Internet grew from research by the Defense Department in the late 1960s, but there has never been a master plan. One group built the Web browser, another search technology, another payment networks. Still others made the encryption technology that is increasingly demanded — and scrutinized — in the aftermath of revelations by former National Security Agency contractor Edward Snowden about the power and pervasiveness of Internet surveillance.
The flaw could allow hackers to access encrypted data online, including user names, passwords, credit card numbers and Social Security numbers. Some researchers believe that hackers might even have been able to access encryption keys that can unlock Internet traffic on a mass scale, even when the data have been stored for years.
Companies and government agencies have been scrambling for days to correct the flaw by updating software. Dozens of popular Web sites, including Yahoo, have proved vulnerable to data theft this week. Because the exploitation of the flaw leaves no traces, it is extremely difficult — and perhaps impossible — to know what sites were infiltrated between the introduction of the flaw in March 2012 and its discovery two years later.
Such problems were supposed to be less likely with “open source” software, produced by groups that publish the entirety of the computer code online, for all to see and scrutinize for flaws and potential improvements.
Open-source advocates often claim that their work, as opposed to software produced by private companies such as Microsoft, has fewer problems, because of the inherent transparency of the process. The belief is captured in a saying popular among the community: “Given enough eyeballs, all bugs are shallow” — meaning flaws are not terribly serious and are quickly fixed.
But security experts have warned for years that open-source software can harbor serious problems because the volunteers and nonprofit groups that often create them lack the time and expertise to continually update their work, especially as hackers become more prevalent and sophisticated. While some open-source projects, such as the Ubuntu operating system or the Firefox browser, have foundations supporting them, many others do not. Some private companies also produce open-source software.
In 2009, Columbia University computer scientist Steve Bellovin wrote in a blog post focusing on problems in Firefox that “if the open source movement is to fulfill its promise, it needs to solve its buggy code problem.”
On Wednesday, he said by e-mail, “There has been some effort to improve the process, but it’s not been enough.”
The OpenSSL Software Foundation is the product of one such effort. Marquess was contracting for federal agencies that relied on the software when he realized how poorly supported it was. He created the foundation in 2009, at first as a hobby, to provide financial support for the even-more-loosely organized OpenSSL Project, which was the official steward of the code.
The foundation arranged consulting deals in which developers got paid to do work on OpenSSL projects sought by individual companies or government agencies. That often involved adding new functions that benefited all users of OpenSSL, Marquess said.The developers who worked as consultants for the foundation also moonlighted as volunteers maintaining and improving the open-source code — a process that also includes reviewing, approving and occasionally rejecting improvements proposed by others.
“These are guys who are working very hard for very little money,” said Matthew Green, a Johns Hopkins University cryptography expert who has attempted to help the foundation. “Yahoo and all these companies are getting all this value out of this. If they just gave a small fraction of that [to the foundation], everyone would be better off.”
The foundation, a for-profit company, made less than $1 million last year, almost entirely in consulting contracts. Its $2,000 in outright donations, received in small increments mainly from overseas supporters of encryption, was not nearly enough to initiate a deeper revamping of the underlying code. There also has not been enough skilled people or money to undergo a security audit of OpenSSL, a meticulous process that involves testing the software for vulnerabilities.
“This is the sort of thing that nobody is going to come along and offer a wad of money to do,” Marquess said. “No one person is going to get a benefit out of that. It’s kind of a ‘tragedy of the commons’ kind of thing,” referring to a 1968 essay by economist Garrett Hardin, who argued that individual self-interest often leads to neglect of publicly important resources.