How good are large language models at product risk assessment?

Product safety professionals must assess the risks to consumers associated with the foreseeable uses and misuses of products. In this study, we investigate the utility of generative artificial intelligence (AI), specifically large language models (LLMs) such as ChatGPT, across a number of tasks involved in the product risk assessment process. For a set of six consumer products, prompts were developed related to failure mode identification, the construction and population of a failure mode and effects analysis (FMEA) table, risk mitigation identification, and guidance to product designers, users, and regulators. These prompts were input into ChatGPT and the outputs were recorded. A survey was administered to product safety professionals to ascertain the quality of the outputs. We found that ChatGPT generally performed better at divergent thinking tasks such as brainstorming potential failure modes and risk mitigations. However, there were errors and inconsistencies in some of the results, and the guidance provided was perceived as overly generic, occasionally outlandish, and not reflective of the depth of knowledge held by a subject matter expert. When tested against a sample of other LLMs, similar patterns in strengths and weaknesses were demonstrated. Despite these challenges, a role for LLMs may still exist in product risk assessment to assist in ideation, while experts may shift their focus to critical review of AI-generated content.

Multicriteria Decision Framework for Cybersecurity Risk Assessment and Management.

Risk assessors and managers face many difficult challenges related to novel cyber systems. Among these challenges are the constantly changing nature of cyber systems caused by technical advances, their distribution across the physical, information, and sociocognitive domains, and the complex network structures often including thousands of nodes. Here, we review probabilistic and risk-based decision-making techniques applied to cyber systems and conclude that existing approaches typically do not address all components of the risk assessment triplet (threat, vulnerability, consequence) and lack the ability to integrate across multiple domains of cyber systems to provide guidance for enhancing cybersecurity. We present a decision-analysis-based approach that quantifies threat, vulnerability, and consequences through a set of criteria designed to assess the overall utility of cybersecurity management alternatives. The proposed framework bridges the gap between risk assessment and risk management, allowing an analyst to ensure a structured and transparent process of selecting risk management alternatives. The use of this technique is illustrated for a hypothetical, but realistic, case study exemplifying the process of evaluating and ranking five cybersecurity enhancement strategies. The approach presented does not necessarily eliminate biases and subjectivity necessary for selecting countermeasures, but provides justifiable methods for selecting risk management actions consistent with stakeholder and decisionmaker values and technical data.

Selection of invasive wild pig countermeasures using multicriteria decision analysis.

Wild pigs are a widespread invasive species that pose significant environmental and social risks. A number of wild pig eradication and control measures exist, but many eradication campaigns are ultimately unsuccessful. Decision making regarding how to design and execute an eradication plan is difficult because of multiple costs and benefits spanning various decision criteria that are associated with different eradication and control countermeasures. Moreover, multiple stakeholders are often involved with differing and sometimes competing objectives, and wild pigs are adaptive adversaries, meaning that the ideal countermeasure may change over time. In this paper, we propose the use of formal decision analytic tools which can structure decision problems into a set of relevant criteria, countermeasures, and stakeholder preferences to facilitate the evaluation of tradeoffs. We operationalize this method in a simple Excel-based decision tool and conclude with a path forward regarding how to successfully implement such tools for effective wild pig control.

A weight of evidence assessment approach for adverse outcome pathways.

The adverse outcome pathway (AOP) is a framework to mechanistically link molecular initiating events to adverse biological outcomes. From a regulatory perspective, it is of crucial importance to determine the confidence for the AOP in question as well as the quality of data available in supporting this evaluation. A weight of evidence approach has been proposed for this task, but many of the existing frameworks for weight of evidence evaluation are qualitative and there is not clear guidance regarding how weight of evidence should be calculated for an AOP. In this paper we advocate the use of a subject matter expertise driven approach for weight of evidence evaluation based on criteria and metrics related to data quality and the strength of causal linkages between key events. As a demonstration, we notionally determine weight of evidence scores for two AOPs: Non-competitive ionotropic GABA receptor antagonism leading to epileptic seizures, and Antagonist-binding and stabilization of a co-repressor to the peroxisome proliferator-activated receptor α (PPARα) signaling complex ultimately causing starvation-like weight loss.

Limitations of toxicity characterization in life cycle assessment: Can adverse outcome pathways provide a new foundation?

Life cycle assessment (LCA) has considerable merit for holistic evaluation of product planning, development, production, and disposal, with the inherent benefit of providing a forecast of potential health and environmental impacts. However, a technical review of current life cycle impact assessment (LCIA) methods revealed limitations within the biological effects assessment protocols, including: simplistic assessment approaches and models; an inability to integrate emerging types of toxicity data; a reliance on linear impact assessment models; a lack of methods to mitigate uncertainty; and no explicit consideration of effects in species of concern. The purpose of the current study is to demonstrate that a new concept in toxicological and regulatory assessment, the adverse outcome pathway (AOP), has many useful attributes of potential use to ameliorate many of these problems, to expand data utility and model robustness, and to enable more accurate and defensible biological effects assessments within LCIA. Background, context, and examples have been provided to demonstrate these potential benefits. We additionally propose that these benefits can be most effectively realized through development of quantitative AOPs (qAOPs) crafted to meet the needs of the LCIA framework. As a means to stimulate qAOP research and development in support of LCIA, we propose 3 conceptual classes of qAOP, each with unique inherent attributes for supporting LCIA: 1) mechanistic, including computational toxicology models; 2) probabilistic, including Bayesian networks and supervised machine learning models; and 3) weight of evidence, including models built using decision-analytic methods. Overall, we have highlighted a number of potential applications of qAOPs that can refine and add value to LCIA. As the AOP concept and support framework matures, we see the potential for qAOPs to serve a foundational role for next-generation effects characterization within LCIA. Integr Environ Assess Manag 2016;12:580-590. Published 2015. This article is a US Government work and is in the public domain in the USA.

Uncertainty in multi-media fate and transport models: a case study for TNT life cycle assessment.

Life cycle assessment (LCA) is an evaluation method used by decision-makers to help assess the relative environmental impacts of various industrial processes. Despite that many LCA methods remain sensitive to uncertain input data, which can reduce the utility of their results, uncertainty arising from constituent LCA models remains poorly understood. Here, we begin to address this problem by evaluating the extent to which parameter-value uncertainty affects the SimpleBox 2.0 fate and transport model, which serves as a backbone for many LCA ecotoxicological impact categories. Two Monte Carlo type sampling methods were used to evaluate dispersion in steady-state concentration values for three chemicals involved in grenade production: toluene, 2,4-dinitrotoluene (2,4-DNT), and 2,4,6-trinitrotoluene (TNT). Parameters were first sampled stochastically one-at-a-time, then by randomly exploring a local patch of the parameter space. We confirmed that global temperatures contribute primarily to the overall variance of model results, which at most spanned approximately 8 decades in magnitude. These results are consistent with previous results obtained for the whole of the LCA method. LCA methods carry out calculations iteratively; a reduction in the error of a single component, such as the fate and transport model, may therefore improve its performance and utility as a decision-making aid.

A decision analytic approach to exposure-based chemical prioritization.

The manufacture of novel synthetic chemicals has increased in volume and variety, but often the environmental and health risks are not fully understood in terms of toxicity and, in particular, exposure. While efforts to assess risks have generally been effective when sufficient data are available, the hazard and exposure data necessary to assess risks adequately are unavailable for the vast majority of chemicals in commerce. The US Environmental Protection Agency has initiated the ExpoCast Program to develop tools for rapid chemical evaluation based on potential for exposure. In this context, a model is presented in which chemicals are evaluated based on inherent chemical properties and behaviorally-based usage characteristics over the chemical's life cycle. These criteria are assessed and integrated within a decision analytic framework, facilitating rapid assessment and prioritization for future targeted testing and systems modeling. A case study outlines the prioritization process using 51 chemicals. The results show a preliminary relative ranking of chemicals based on exposure potential. The strength of this approach is the ability to integrate relevant statistical and mechanistic data with expert judgment, allowing for an initial tier assessment that can further inform targeted testing and risk management strategies.