Discovering foodborne illness in online restaurant reviews.

Objective: We developed a system for the discovery of foodborne illness mentioned in online Yelp restaurant reviews using text classification. The system is used by the New York City Department of Health and Mental Hygiene (DOHMH) to monitor Yelp for foodborne illness complaints. Materials and Methods: We built classifiers for 2 tasks: (1) determining if a review indicated a person experiencing foodborne illness and (2) determining if a review indicated multiple people experiencing foodborne illness. We first developed a prototype classifier in 2012 for both tasks using a small labeled dataset. Over years of system deployment, DOHMH epidemiologists labeled 13 526 reviews selected by this classifier. We used these biased data and a sample of complementary reviews in a principled bias-adjusted training scheme to develop significantly improved classifiers. Finally, we performed an error analysis of the best resulting classifiers. Results: We found that logistic regression trained with bias-adjusted augmented data performed best for both classification tasks, with F1-scores of 87% and 66% for tasks 1 and 2, respectively. Discussion: Our error analysis revealed that the inability of our models to account for long phrases caused the most errors. Our bias-adjusted training scheme illustrates how to improve a classification system iteratively by exploiting available biased labeled data. Conclusions: Our system has been instrumental in the identification of 10 outbreaks and 8523 complaints of foodborne illness associated with New York City restaurants since July 2012. Our evaluation has identified strong classifiers for both tasks, whose deployment will allow DOHMH epidemiologists to more effectively monitor Yelp for foodborne illness investigations.

Using online reviews by restaurant patrons to identify unreported cases of foodborne illness - New York City, 2012-2013.

While investigating an outbreak of gastrointestinal disease associated with a restaurant, the New York City Department of Health and Mental Hygiene (DOHMH) noted that patrons had reported illnesses on the business review website Yelp (http://www.yelp.com) that had not been reported to DOHMH. To explore the potential of using Yelp to identify unreported outbreaks, DOHMH worked with Columbia University and Yelp on a pilot project to prospectively identify restaurant reviews on Yelp that referred to foodborne illness. During July 1, 2012-March 31, 2013, approximately 294,000 Yelp restaurant reviews were analyzed by a software program developed for the project. The program identified 893 reviews that required further evaluation by a foodborne disease epidemiologist. Of the 893 reviews, 499 (56%) described an event consistent with foodborne illness (e.g., patrons reported diarrhea or vomiting after their meal), and 468 of those described an illness within 4 weeks of the review or did not provide a period. Only 3% of the illnesses referred to in the 468 reviews had also been reported directly to DOHMH via telephone and online systems during the same period. Closer examination determined that 129 of the 468 reviews required further investigation, resulting in telephone interviews with 27 reviewers. From those 27 interviews, three previously unreported restaurant-related outbreaks linked to 16 illnesses met DOHMH outbreak investigation criteria; environmental investigation of the three restaurants identified multiple food-handling violations. The results suggest that online restaurant reviews might help to identify unreported outbreaks of foodborne illness and restaurants with deficiencies in food handling. However, investigating reports of illness in this manner might require considerable time and resources.