Get ready for Hadoop as you’ve never seen it before
The new year should bring good news and bad news if you’re sick of hearing about Hadoop. The bad news is that you’re going to hear more about it. The good news is that you’re probably going to hear about it in a whole new light.
All over Silicon Valley, startups are working on ways to turn Hadoop into something more useful than a batch-processing engine that runs on tens or hundreds of physical servers. Sometimes, it’s as simple as offering Hadoop as a cloud service. Amazon Web Services, Mortar Data, Infochimps and others already do this, and newer offerings such as VertiCloud — headed by former Yahoo CTO Raymie Stata — and Microsoft Windows Azure HDInsight are also coming down the pike.
Probably more important, though, are the rash of companies trying to make Hadoop more useful by turning it into a platform for something other than running MapReduce jobs. Applications are the key to the adoption of any technology platform, and there just aren’t too many compelling applications for Hadoop in its current state. We’ve covered a handful of the companies doing this work already — Continuuity, Platfora, Drawn to Scale — but they’re just the tip of the iceberg.
I have it on good authority, for example, that we’ll see more real-time/streaming Hadoop frameworks in the coming months. These probably won’t be the only MapReduce alternative we see pop up as projects such as Drill and Impala mature, and Apache Hadoop’s YARN feature catches on. And it’s hard to imagine Hadoop won’t be used more and more as the core of enterprise software applications targeting companies in major fields such as manufacturing, financial services, or oil and gas.
Yes, there are other technologies floating around that do certain things better, but Hadoop is still the biggest thing in big data. It’s not going anywhere.
The Google-Ray-Kurzweil singularity
Ray Kurzweil’s decision to come on board as Google’s director of engineering is fascinating for so many reasons. Not the least of which is while Google-bashing has become high art in some circles, the company is nonetheless one of the biggest drivers of innovation — and useful innovation — on the planet. Self-driving cars could improve road safety dramatically, and having just returned from a week and a half in China with next to no knowledge of Mandarin, I’m even more excited about the possibilities of Google’s’ Project Glass.