An attempt to identify flu outbreaks by tracking people’s Google searches about the illness hasn’t lived up to its initial promise, a new paper argues.
Google Flu Trends, an attempt to track flu outbreaks based on search terms, dramatically overestimated the number of flu cases in the 2012-2013 season, and the latest data do not look promising, says David Lazer, a computer scientist and political scientist at Northeastern University in an article published last week in the journal Science.
“There’s a huge amount of potential there, but there’s also a lot of potential to make mistakes,” Lazer said.
In February 2013, researchers reported in Nature that the program was estimating about twice the number of flu cases as recorded by the Centers for Disease Control and Prevention (CDC), which tracks actual reported cases.
“When it went off the rails, it really went off the rails,” Lazer said.
Google Flu Trends also struggled in 2009, missing a nonseasonal flu outbreak of H1NI entirely. The mistakes have led the Google team to retool its algorithm, but an early look at the latest flu season’s data suggests these changes have not fixed the problem, according to a preliminary analysis by Lazer and colleagues.
The problem is not unique to Google Flu Trends, Lazer said. All social science “big data,” or the analysis of huge swaths of the population from mobile or social media technology, face the same challenges.
Figuring out what went wrong with Google Flu Trends is not easy, because the company does not disclose what search terms it uses to track flu.
“They get an F on replication,” Lazer said, meaning that scientists don’t have enough information about the methods to test and reproduce the findings.
But Lazer and his colleagues have a sense of what went wrong. A major problem, he said, is that Google is a business interested in promoting searches, not a scientific team collecting data. The Google algorithm, then, prompts related searches: If someone searches “flu symptoms,” he will likely be prompted to try a search for “flu vaccines,” for example. Thus, the number of flu-related searches can snowball even if flu cases don’t.
Another problem, Lazer said, is that the Google flu team had to differentiate between flu-related searches and searches that are correlated with the flu season but not related to it. To do so, they took more than 50 million search terms and matched them up with about 1,100 data points on flu prevalence from the CDC.
Playing the correlation game with so many terms is bound to return a few weird, nonsensical results, Lazer said, “just like monkeys can type Shakespeare eventually.” Google picked out obviously spurious correlations and removed them, but exactly what terms they removed and the logic of doing so is unclear. Some terms, such as “coughs” and “fever,” might look flu-related but actually signal other seasonal diseases, Lazer said.
“It was part flu detector, and part winter detector,” he said.
The Google team altered its algorithm after both the 2009 and 2013 misses, but made the most recent changes on the assumption that a spike in media coverage of the 2012-2013 flu season caused the problems, Lazer and his colleagues wrote in their paper. That assumption discounts the major media coverage of the 2009 H1N1 pandemic and fails to explain errors in the 2011-2012 flu season, the researchers argue.
A Google spokeswoman pointed to a blog post on the Google Flu Trends updates that calls the efforts to improve “an iterative process.”
Lazer was quick to point out that he wasn’t picking on Google, calling the flu tracker “a great idea.”
Social scientists deal with problems of unstable data all the time, and Google’s flu data are fixable, Lazer said.
“My sense, looking at the data and how it went off, is this is something you could rectify without Google tweaking their own business model,” he said.