How good are large language models at product risk assessment?

Product safety professionals must assess the risks to consumers associated with the foreseeable uses and misuses of products. In this study, we investigate the utility of generative artificial intelligence (AI), specifically large language models (LLMs) such as ChatGPT, across a number of tasks involved in the product risk assessment process. For a set of six consumer products, prompts were developed related to failure mode identification, the construction and population of a failure mode and effects analysis (FMEA) table, risk mitigation identification, and guidance to product designers, users, and regulators. These prompts were input into ChatGPT and the outputs were recorded. A survey was administered to product safety professionals to ascertain the quality of the outputs. We found that ChatGPT generally performed better at divergent thinking tasks such as brainstorming potential failure modes and risk mitigations. However, there were errors and inconsistencies in some of the results, and the guidance provided was perceived as overly generic, occasionally outlandish, and not reflective of the depth of knowledge held by a subject matter expert. When tested against a sample of other LLMs, similar patterns in strengths and weaknesses were demonstrated. Despite these challenges, a role for LLMs may still exist in product risk assessment to assist in ideation, while experts may shift their focus to critical review of AI-generated content.

A semiautomated risk assessment method for consumer products.

In this study, we develop a model that assesses product risk using online reviews from Amazon.com. We first identify unique words and phrases capable of identifying hazards. Second, we estimate risk severity using hazard type weights and risk likelihood using total reviews as a proxy for sales volume. In addition, we obtain expert assessments of product hazard risk (risk likelihood and severity) from a sample of high- and low-risk consumer products identified by a computerized risk assessment model we have developed. Third, we assess the validity of our computerized product risk assessment scoring model by utilizing the experts' survey responses. We find that our model is especially consistent with expert judgments of hazard likelihood but not as consistent with expert judgments of hazard severity. This model helps organizations to determine the risk severity, risk likelihood, and overall risk level of a specific product. The model produced by this study is helpful for product safety practitioners in product risk identification, characterization, and mitigation.

Text Mining Approaches for Postmarket Food Safety Surveillance Using Online Media.

Food contamination and food poisoning pose enormous risks to consumers across the world. As discussions of consumer experiences have spread through online media, we propose the use of text mining to rapidly screen online media for mentions of food safety hazards. We compile a large data set of labeled consumer posts spanning two major websites. Utilizing text mining and supervised machine learning, we identify unique words and phrases in online posts that identify consumers' interactions with hazardous food products. We compare our methods to traditional sentiment-based text mining. We assess performance in a high-volume setting, utilizing a data set of over 4 million online reviews. Our methods were 77-90% accurate in top-ranking reviews, while sentiment analysis was just 11-26% accurate. Moreover, we aggregate review-level results to make product-level risk assessments. A panel of 21 food safety experts assessed our model's hazard-flagged products to exhibit substantially higher risk than baseline products. We suggest the use of these tools to profile food items and assess risk, building a postmarket decision support system to identify hazardous food products. Our research contributes to the literature and practice by providing practical and inexpensive means for rapidly monitoring food safety in real time.