GRASPING AT TEA LEAVES

So You Can't Pick the Hits. Neither Can Anyone Else.

By Duncan Watts
Sunday, January 4, 2009

Humans love to make predictions -- whether about the movements of the stars, the gyrations of the stock market or the upcoming season's hot color -- and the beginning of a new year brings out this tendency even more than usual. Unfortunately, the predictions most of us make will be wrong. When I was a kid, the future was supposed to be filled with flying cars, orbiting space cities and endless free time; instead, we work more hours than ever, drive internal combustion cars on crumbling, congested freeways and endure endless cuts in airline service. Meanwhile, Web searching, handheld GPS systems and online shopping -- the technologies that have, in fact, changed our lives -- came more or less out of nowhere. Nor is it just the megatrends that are hard to predict -- elections, recessions and risk assessments are equally slippery. And publishers, producers and marketers routinely fail to predict which books, movies and products will become the next big hits.

Why is predicting so difficult?

Well, for lots of reasons, but two fundamental ones stand out. First, individuals are much harder to predict than they seem, not because people are infinitely complex, but because how we are apt to behave depends on subtle details of the situation. In recent decades, psychologists have conducted innumerable experiments showing that subtle changes in how a situation is framed, or even such seemingly irrelevant factors as background music or a writing font, can all have an impact on individual decision-making. Second, social phenomena are never just the product of individual people making decisions, but emerge out of many people making decisions in conjunction with each other.

Thanks to the Web, it's actually possible to see this phenomenon at work. My collaborators, Matthew Salganik and Peter Dodds, and I have conducted a series of experiments to explore how certain songs become hits while so many others never crack the Top 100. We recruited tens of thousands of people to a Web site where they could make choices about what music they liked. Some people saw only the names of the songs and the bands that performed them, but others also saw how many times each song had been downloaded previously. In addition, we split this second group into a series of parallel universes in which history could unfold many times, revealing how much of a song's success depends on its intrinsic qualities and how much on peer influence. When participants knew what others liked, the popular songs became more popular and the unpopular songs less popular than when people made their choices independently. More surprisingly, however, we found that which particular songs become the most popular also became more unpredictable -- in some cases social influence caused luck and randomness to overtake intrinsic appeal as the main factors driving success.

The implications of these experiments stretch beyond speculation about the merits of Madonna or Justin Timberlake. The point really is that whenever people make decisions based in part on what other people are doing, predicting the outcome is going to be susceptible to large errors, no matter how careful you are. In many cases, it's probably impossible.

So why do we still think that we can forecast future events?

My suspicion is that we are so deeply attached to the idea of prediction that even when we make mistakes, we feel as if we could have predicted the future correctly if only we'd paid attention to the right information. The usual trick we use to maintain this illusion is to find someone who did foretell the outcome correctly and assume that if only everyone had known what this person knew, we'd all have been okay.

It's a great trick, and it almost always works, because at any given time there are so many people out there making predictions about so many things that no matter what happens, it's almost certain that somebody somewhere will have said it would.

That only works in hindsight, however, and what we ideally want from predictions is knowledge about the future that we can use before it arrives. If we were serious about knowing which experts could predict what, we would insist that they write down all their forecasts in advance and track their total performance as the events themselves unfolded.

Believe it or not, someone has done exactly this kind of assessment. In an ingenious 20-year study, the psychologist Philip Tetlock asked 284 analysts to make about 100 predictions each regarding future political and economic events. The results weren't pretty. Overall, Tetlock found that his experts were only slightly more accurate than simple rules such as "the home team wins 60 percent of the time, so always bet on the home team." More disturbingly, they were actually better at making predictions outside their areas of expertise than in them.

In part because of disappointing findings such as this , an increasingly popular substitute for expert opinions are so-called "prediction markets," in which individuals buy and sell contracts on various outcomes, such as football game point spreads or presidential elections. The market prices for these contracts then effectively aggregate the knowledge and judgment of the many into a single prediction, which often turns out to be more accurate than all but the best individual guesses.

But even if these markets do perform better than experts, they don't necessarily do a good enough job to rely on. Recently, my colleagues have started tracking the performance of one popular prediction market, at forecasting the outcome of weekly NFL games . So far, what they're finding is that the market predictions are better than the simple rule of always betting on the home team, but only slightly so -- which, oddly, is very similar to what Tetlock found regarding his experts. Some outcomes, in other words, and possibly the outcomes we care about the most, simply aren't "predictable" in the way we would like.

Many people find this conclusion just a little depressing, because they associate a lack of predictability with an absence of meaning. But just because outcomes aren't predictable doesn't mean that there's nothing to be done. Consider venture capital firms. They expend a great deal of effort trying to identify the next "black swan" among the many pitches they receive. But they also recognize that no matter how hard they try, most of their swans will be white, and that a large percentage of their investments will fail to make money. These sound like lousy statistics, but many of those firms are doing quite nicely, thanks very much, and most of them are entirely comfortable with their apparent inability to predict winners. Unpredictability is built into their business model.

Naturally, you can't apply this model to everything: The United States cannot invest in six different foreign policies and wait to see which one pans out. Nevertheless, there are many circumstances that allow for far more experimentation than is currently the norm. And simply by acknowledging the limits of predictability, we can at least guard against the risk of overconfidence.

So as we embark on what many prognosticators are calling a dark year, hold on to your hats and try to enjoy the ride -- because no one knows where we'll be 12 months from now.

Duncan Watts, a principal research scientist at Yahoo! Research and a professor of sociology at Columbia University, is the author of "Six Degrees: The Science of a Connected Age."


© 2009 The Washington Post Company