Use of artificial intelligence in critical care: opportunities and obstacles.

BACKGROUND: Perhaps nowhere else in the healthcare system than in the intensive care unit environment are the challenges to create useful models with direct time-critical clinical applications more relevant and the obstacles to achieving those goals more massive. Machine learning-based artificial intelligence (AI) techniques to define states and predict future events are commonplace activities of modern life. However, their penetration into acute care medicine has been slow, stuttering and uneven. Major obstacles to widespread effective application of AI approaches to the real-time care of the critically ill patient exist and need to be addressed. MAIN BODY: Clinical decision support systems (CDSSs) in acute and critical care environments support clinicians, not replace them at the bedside. As will be discussed in this review, the reasons are many and include the immaturity of AI-based systems to have situational awareness, the fundamental bias in many large databases that do not reflect the target population of patient being treated making fairness an important issue to address and technical barriers to the timely access to valid data and its display in a fashion useful for clinical workflow. The inherent "black-box" nature of many predictive algorithms and CDSS makes trustworthiness and acceptance by the medical community difficult. Logistically, collating and curating in real-time multidimensional data streams of various sources needed to inform the algorithms and ultimately display relevant clinical decisions support format that adapt to individual patient responses and signatures represent the efferent limb of these systems and is often ignored during initial validation efforts. Similarly, legal and commercial barriers to the access to many existing clinical databases limit studies to address fairness and generalizability of predictive models and management tools. CONCLUSIONS: AI-based CDSS are evolving and are here to stay. It is our obligation to be good shepherds of their use and further development.

Empowering nurses to champion Health equity & BE FAIR: Bias elimination for fair and responsible AI in healthcare.

BACKGROUND: The concept of health equity by design encompasses a multifaceted approach that integrates actions aimed at eliminating biased, unjust, and correctable differences among groups of people as a fundamental element in the design of algorithms. As algorithmic tools are increasingly integrated into clinical practice at multiple levels, nurses are uniquely positioned to address challenges posed by the historical marginalization of minority groups and its intersections with the use of "big data" in healthcare settings; however, a coherent framework is needed to ensure that nurses receive appropriate training in these domains and are equipped to act effectively. PURPOSE: We introduce the Bias Elimination for Fair AI in Healthcare (BE FAIR) framework, a comprehensive strategic approach that incorporates principles of health equity by design, for nurses to employ when seeking to mitigate bias and prevent discriminatory practices arising from the use of clinical algorithms in healthcare. By using examples from a "real-world" AI governance framework, we aim to initiate a wider discourse on equipping nurses with the skills needed to champion the BE FAIR initiative. METHODS: Drawing on principles recently articulated by the Office of the National Coordinator for Health Information Technology, we conducted a critical examination of the concept of health equity by design. We also reviewed recent literature describing the risks of artificial intelligence (AI) technologies in healthcare as well as their potential for advancing health equity. Building on this context, we describe the BE FAIR framework, which has the potential to enable nurses to take a leadership role within health systems by implementing a governance structure to oversee the fairness and quality of clinical algorithms. We then examine leading frameworks for promoting health equity to inform the operationalization of BE FAIR within a local AI governance framework. RESULTS: The application of the BE FAIR framework within the context of a working governance system for clinical AI technologies demonstrates how nurses can leverage their expertise to support the development and deployment of clinical algorithms, mitigating risks such as bias and promoting ethical, high-quality care powered by big data and AI technologies. CONCLUSION AND RELEVANCE: As health systems learn how well-intentioned clinical algorithms can potentially perpetuate health disparities, we have an opportunity and an obligation to do better. New efforts empowering nurses to advocate for BE FAIR, involving them in AI governance, data collection methods, and the evaluation of tools intended to reduce bias, mark important steps in achieving equitable healthcare for all.

A Federated Registration System for Artificial Intelligence in Health.

This Viewpoint discusses a suggested framework of local registries to record and track all health artificial intelligence technologies used in clinical care, with the goal of providing transparency on these technologies and helping speed adoption while also protecting patient well-being.

Translating ethical and quality principles for the effective, safe and fair development, deployment and use of artificial intelligence technologies in healthcare.

OBJECTIVE: The complexity and rapid pace of development of algorithmic technologies pose challenges for their regulation and oversight in healthcare settings. We sought to improve our institution's approach to evaluation and governance of algorithmic technologies used in clinical care and operations by creating an Implementation Guide that standardizes evaluation criteria so that local oversight is performed in an objective fashion. MATERIALS AND METHODS: Building on a framework that applies key ethical and quality principles (clinical value and safety, fairness and equity, usability and adoption, transparency and accountability, and regulatory compliance), we created concrete guidelines for evaluating algorithmic technologies at our institution. RESULTS: An Implementation Guide articulates evaluation criteria used during review of algorithmic technologies and details what evidence supports the implementation of ethical and quality principles for trustworthy health AI. Application of the processes described in the Implementation Guide can lead to algorithms that are safer as well as more effective, fair, and equitable upon implementation, as illustrated through 4 examples of technologies at different phases of the algorithmic lifecycle that underwent evaluation at our academic medical center. DISCUSSION: By providing clear descriptions/definitions of evaluation criteria and embedding them within standardized processes, we streamlined oversight processes and educated communities using and developing algorithmic technologies within our institution. CONCLUSIONS: We developed a scalable, adaptable framework for translating principles into evaluation criteria and specific requirements that support trustworthy implementation of algorithmic technologies in patient care and healthcare operations.

Predictive Modeling to Identify Children With Complex Health Needs At Risk for Hospitalization.

BACKGROUND: Identifying children at high risk with complex health needs (CCHN) who have intersecting medical and social needs is challenging. This study's objectives were to (1) develop and evaluate an electronic health record (EHR)-based clinical predictive model ("model") for identifying high-risk CCHN and (2) compare the model's performance as a clinical decision support (CDS) to other CDS tools available for identifying high-risk CCHN. METHODS: This retrospective cohort study included children aged 0 to 20 years with established care within a single health system. The model development/validation cohort included 33 months (January 1, 2016-September 30, 2018) and the testing cohort included 18 months (October 1, 2018-March 31, 2020) of EHR data. Machine learning methods generated a model that predicted probability (0%-100%) for hospitalization within 6 months. Model performance measures included sensitivity, positive predictive value, area under receiver-operator curve, and area under precision-recall curve. Three CDS rules for identifying high-risk CCHN were compared: (1) hospitalization probability ≥10% (model-predicted); (2) complex chronic disease classification (using Pediatric Medical Complexity Algorithm [PMCA]); and (3) previous high hospital utilization. RESULTS: Model development and testing cohorts included 116 799 and 27 087 patients, respectively. The model demonstrated area under receiver-operator curve = 0.79 and area under precision-recall curve = 0.13. PMCA had the highest sensitivity (52.4%) and classified the most children as high risk (17.3%). Positive predictive value of the model-based CDS rule (19%) was higher than CDS based on the PMCA (1.9%) and previous hospital utilization (15%). CONCLUSIONS: A novel EHR-based predictive model was developed and validated as a population-level CDS tool for identifying CCHN at high risk for future hospitalization.

Mitigating Racial And Ethnic Bias And Advancing Health Equity In Clinical Algorithms: A Scoping Review.

In August 2022 the Department of Health and Human Services (HHS) issued a notice of proposed rulemaking prohibiting covered entities, which include health care providers and health plans, from discriminating against individuals when using clinical algorithms in decision making. However, HHS did not provide specific guidelines on how covered entities should prevent discrimination. We conducted a scoping review of literature published during the period 2011-22 to identify health care applications, frameworks, reviews and perspectives, and assessment tools that identify and mitigate bias in clinical algorithms, with a specific focus on racial and ethnic bias. Our scoping review encompassed 109 articles comprising 45 empirical health care applications that included tools tested in health care settings, 16 frameworks, and 48 reviews and perspectives. We identified a wide range of technical, operational, and systemwide bias mitigation strategies for clinical algorithms, but there was no consensus in the literature on a single best practice that covered entities could employ to meet the HHS requirements. Future research should identify optimal bias mitigation methods for various scenarios, depending on factors such as patient population, clinical setting, algorithm design, and types of bias to be addressed.

A framework for the oversight and local deployment of safe and high-quality prediction models.

Artificial intelligence/machine learning models are being rapidly developed and used in clinical practice. However, many models are deployed without a clear understanding of clinical or operational impact and frequently lack monitoring plans that can detect potential safety signals. There is a lack of consensus in establishing governance to deploy, pilot, and monitor algorithms within operational healthcare delivery workflows. Here, we describe a governance framework that combines current regulatory best practices and lifecycle management of predictive models being used for clinical care. Since January 2021, we have successfully added models to our governance portfolio and are currently managing 52 models.

Artificial Intelligence Can Be Regulated Using Current Patient Safety Procedures and Infrastructure in Hospitals.

This Viewpoint describes the potential benefits and harms of using artificial intelligence (AI) in health care decision-making processes.