Data mining software searched for red flag words and phrases like “sick,” “vomit,” “diarrhea” and “food poisoning." Researchers also checked for reports of multiple illnesses and an incubation period of at least 10 hours, which would indicate a food-borne illness.
They ended up with nearly 900 cases that were flagged for further review by epidemiologists.
From there, the cases continued to be whittled down. Three hundred and ninety-four reviews were disqualified because they may have contained key words but didn't indicate that someone had actually gotten sick, but more than half indicate that at least one person had symptoms that were consistent with a food-borne illness.
"With food borne illnesses, its very important to speak with the people involved in the illness," said Sharon Balter, a medial epidemiologist at the New York City Department of Health, and a co-author of the study. "Most people think they got sick from the last thing they ate," but there is actually a 10-hour period of incubation for most food borne illnesses.
City public health departments rely on reports from the public and treated cases of illnesses to find and investigate food safety issues. But because most symptoms are usually not life threatening — the Centers for Disease Control report only 3,000 deaths out of 48 million food-related illnesses each year — many cases go unreported.
In this case, only 3 percent of the 468 reviewers that indicated recent illnesses had actually been reported to the city using the “311” hotline.
Yelp reviews may not always be the most timely – users are typically reviewing their experiences days or even weeks after their visit— but the system makes it easy to follow up, which proved to be a key component of this experiment.
Using Yelp’s internal messaging system, epidemiologists reached out to about 129 suspected cases of scombroid poisoning, which typically results from eating bad seafood, or severe neurologic illness, which come with symptoms like numbness, tingling or hallucination.
Most of those people never responded to requests for phone interviews, but after analyzing the ones who did, officials found evidence of three separate outbreaks that has never been reported.
Sixteen people had gotten sick from contamination in either house salad, shrimp and lobster cannelloni, or macaroni and cheese spring rolls at city restaurants.
That may not sound like a lot, but on average, health officials receive 3,000 calls a year to the 311 tip line, reporting potential outbreaks. They follow up with all of those people, and identify on average about 30 outbreaks a year, Balter said.
With this pilot program, Yelp becomes the latest company to use its massive stores of user provided data to aid public health officials in tracking down unreported outbreaks of illnesses and disease.
Google’s flu tracking service offers the promise of using its search data to estimate flu activity across the country — though that program is far from perfect. And researchers at Johns Hopkins University and George Washington University found that Twitter offers the same promise of helping public health officials estimate the concentration of a flu outbreak, as its happening.
The technology is already improving. Each time the software successfully identifies a potential outbreak, that information is used to make the search more accurate.
Yelp is now providing data to New York health officials on a daily basis, which can speed up the investigation process and increase the likelihood that reviewers will participate in the process. And they are also exploring using other sites like Twitter to collect data on potential outbreaks.
[This post has been updated.]