Since August 1995, a British Internet company called Netcraft has tried. It basically uses a series of bots to ping every Web site, evaluate whether they’re active, and pull the data into a monthly report. (In August, for instance, Netcraft estimated that 992 million Web sites were online; of those, domains ending in .xyz saw the greatest growth.) Internet Live Stats then runs formulas against those monthly reports that, together with historical data and current projections, informs the site’s “live” counter.
According to that counter, the Internet passed a billion pages on Sept. 17.
But even Peter Collins, Internet Live Stats’s overseer, admits that’s an inexact science — more an educated estimate than an actual “statistic,” with all the open-and-shut authority the word implies. Netcraft’s surveys are not 100-percent conclusive: sending too many requests from bots could overload a server, so Netcraft sometimes just takes a sample and extrapolates from there. Internet Live Stats, meanwhile, has long recorded unpredictable fluctuations in the number of “inactive” sites, which means the Internet could easily be down below 1 billion pages next month. Collins also said the site is missing some data that he said could make its formulas more accurate: He wishes Google would provide better, more timely search data, for instance.
Still, the estimate and this particular milestone serve an important role, giving shape to an otherwise boundless, ill-defined Internet. Thanks to estimates from Netcraft, we know that 80 percent of all registered domains are just “parked,” or undeveloped: There’s no actual, unique content on that site. We know that the greatest jump in Web site volume occurred between 2011 and 2012. And over time, Collins thinks, we could know exactly how and when the modern Web developed, and how that’s come to shape our offline world.
“We strongly believe in the value of keeping track of the history of the Internet,” Collins said. “This is something very important because it gives a sense of the development and the usage through its brief history … and ultimately allows us to [understand] the resulting changes in our culture and society.”