We’re now creating more data than we know what to do with. We started by measuring data creation in kilobytes and megabytes and gigabytes and we are now at exabytes, zettabytes and yottabytes. We are running out of words to describe how much data we have. By one estimate, the amount of information created every two days on the Internet is equivalent that created between the dawn of civilization and 2003. MIT’s Andrew McAfee and others have actually proposed that we settle on “hellabyte” – as in, “helluva lot of data,” to describe the next stage of the data deluge.
All of this, of course, has implications the brave new world of big data – the belief held by technologists that all of the data we are producing on our digital devices will lead to breakthrough innovations in fields ranging from medicine to astronomy to manufacturing. Or, as some have quipped, “Big data is the belief that any sufficiently large pile of s— contains a pony.” It’s no wonder that big data has emerged as one of the most talked about buzzwords of the tech world. Companies want to hire big data specialists. Universities are setting up big data degree programs. Consulting firms such as McKinsey are pitching big data as the next big innovation trend. Top executives from Silicon Valley constantly tell us how much data we are producing each day and each hour.
This inevitably sets us up for disappointment. Amidst all the claims that big data will help us find the magic needle in an ever-larger haystack, there’s the risk that we’ll eventually start to become frustrated when all of this big data, well, doesn’t seem to produce much of anything — except calls for more spending on higher and higher storage capacity and faster and faster computing ability to make sense of it all. When big data doesn’t pan out the way the marketers and bloggers tell us it will, we’ll then start to talk about the “big data myths.” At which time, someone will start talking about “big data lies,” which will then set off a spirited search for new trendy ideas in Silicon Valley.
What we have to keep in mind is that we’ll never have enough data. We can supersize our data. We can order our data in tall, venti and grande and it just won’t matter.
After all, the same type of thinking about big data has happened before. During the Enlightenment, when people such as Isaac Newton were beginning to develop highly accurate laws for how the physical universe operates. For the first time in history, mankind felt it was within its grasp to understand everything. Suddenly, fields such as biology and physics and chemistry seemed capable of unlocking all of life’s secrets. A new theory – “scientific determinism” – even became trendy, suggesting that all of the mysterious activities of the universe could be finally understood once we knew all the precise mechanical laws that controlled the universe. All we needed was enough computing power, and we could predict everything from the weather to the future.
While the Enlightenment was one of the greatest periods in innovative thinking, we haven’t found a way to unlock the mysteries of the universe. Isaac Newton’s precise mechanical laws of classical physics? They’ve been supplanted by Einstein’s theories of quantum physics. At each level, there’s a new abstraction, a new complexity that now makes technology an integral part of humanity’s eternal quest for meaning.
But that shouldn’t stop us from trying. If anything, we should embrace the exponential proliferation of data in our lives, however we decide to measure it. We should realize that it’s not the hard, quantitative data — the traditional data of 1s and os — that will result in the next innovation breakthrough, it’s all the contextual and qualitative data about our lives — if we could just somehow capture it — that could lead to truly disruptive new ideas about the complex way the world really works.