Achieving large-scale clinician adoption of AI-enabled decision support.

Computerised decision support (CDS) tools enabled by artificial intelligence (AI) seek to enhance accuracy and efficiency of clinician decision-making at the point of care. Statistical models developed using machine learning (ML) underpin most current tools. However, despite thousands of models and hundreds of regulator-approved tools internationally, large-scale uptake into routine clinical practice has proved elusive. While underdeveloped system readiness and investment in AI/ML within Australia and perhaps other countries are impediments, clinician ambivalence towards adopting these tools at scale could be a major inhibitor. We propose a set of principles and several strategic enablers for obtaining broad clinician acceptance of AI/ML-enabled CDS tools.

How should artificial intelligence be used in Australian health care? Recommendations from a citizens' jury.

OBJECTIVE: To support a diverse sample of Australians to make recommendations about the use of artificial intelligence (AI) technology in health care. STUDY DESIGN: Citizens' jury, deliberating the question: "Under which circumstances, if any, should artificial intelligence be used in Australian health systems to detect or diagnose disease?" SETTING, PARTICIPANTS: Thirty Australian adults recruited by Sortition Foundation using random invitation and stratified selection to reflect population proportions by gender, age, ancestry, highest level of education, and residential location (state/territory; urban, regional, rural). The jury process took 18 days (16 March - 2 April 2023): fifteen days online and three days face-to-face in Sydney, where the jurors, both in small groups and together, were informed about and discussed the question, and developed recommendations with reasons. Jurors received extensive information: a printed handbook, online documents, and recorded presentations by four expert speakers. Jurors asked questions and received answers from the experts during the online period of the process, and during the first day of the face-to-face meeting. MAIN OUTCOME MEASURES: Jury recommendations, with reasons. RESULTS: The jurors recommended an overarching, independently governed charter and framework for health care AI. The other nine recommendation categories concerned balancing benefits and harms; fairness and bias; patients' rights and choices; clinical governance and training; technical governance and standards; data governance and use; open source software; AI evaluation and assessment; and education and communication. CONCLUSIONS: The deliberative process supported a nationally representative sample of citizens to construct recommendations about how AI in health care should be developed, used, and governed. Recommendations derived using such methods could guide clinicians, policy makers, AI researchers and developers, and health service users to develop approaches that ensure trustworthy and responsible use of this technology.

Using Clinical Simulation to Evaluate AI-Enabled Decision Support.

Clinical simulation is a useful method for evaluating AI-enabled clinical decision support (CDS). Simulation studies permit patient- and risk-free evaluation and far greater experimental control than is possible with clinical studies. The effect of CDS assisted and unassisted patient scenarios on meaningful downstream decisions and actions within the information value chain can be evaluated as outcome measures. This paper discusses the use of clinical simulation in CDS evaluation and presents a case study to demonstrate feasibility of its application.

Assessing the Safety of a New Clinical Decision Support System for a National Helpline.

We assessed the safety of a new clinical decision support system (CDSS) for nurses on Australia's national consumer helpline. Accuracy and safety of triage advice was assessed by testing the CDSS using 78 standardised patient vignettes (48 published and 30 proprietary). Testing was undertaken in two cycles using the CDSS vendor's online evaluation tool (Cycle 1: 47 vignettes; Cycle 2: 41 vignettes). Safety equivalence was examined by testing the existing CDSS with the 47 vignettes from Cycle 1. The new CDSS triaged 66% of vignettes correctly compared to 57% by the existing CDSS. 15% of vignettes were overtriaged by the new CDSS compared to 28% by the existing CDSS. 19% of vignettes were undertriaged by the new CDSS compared to 15% by the existing CDSS. Overall performance of the new CDSS appears consistent and comparable with current studies. The new CDSS is at least as safe as the old CDSS.

Automating the Identification of Safety Events Involving Machine Learning-Enabled Medical Devices.

With growing use of machine learning (ML)-enabled medical devices by clinicians and consumers safety events involving these systems are emerging. Current analysis of safety events heavily relies on retrospective review by experts, which is time consuming and cost ineffective. This study develops automated text classifiers and evaluates their potential to identify rare ML safety events from the US FDA's MAUDE. Four stratified classifiers were evaluated using a real-world data distribution with different feature sets: report text; text and device brand name; text and generic device type; and all information combined. We found that stratified classifiers using the generic type of devices were the most effective technique when tested on both stratified (F1-score=85%) and external datasets (precision=100%). All true positives on the external dataset were consistently identified by the three stratified classifiers, indicating the ensemble results from them can be used directly to monitor ML events reported to MAUDE.

How Well Do AI-Enabled Decision Support Systems Perform in Clinical Settings?

Real-world performance of machine learning (ML) models is crucial for safely and effectively embedding them into clinical decision support (CDS) systems. We examined evidence about the performance of contemporary ML-based CDS in clinical settings. A systematic search of four bibliographic databases identified 32 studies over a 5-year period. The CDS task, ML type, ML method and real-world performance was extracted and analysed. Most ML-based CDS supported image recognition and interpretation (n=12; 38%) and risk assessment (n=9; 28%). The majority used supervised learning (n=28; 88%) to train random forests (n=7; 22%) and convolutional neural networks (n=7; 22%). Only 12 studies reported real-world performance using heterogenous metrics; and performance degraded in clinical settings compared to model validation. The reporting of model performance is fundamental to ensuring safe and effective use of ML-based CDS in clinical settings. There remain opportunities to improve reporting.

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings.

AIMS AND OBJECTIVES: To examine the nature and use of automation in contemporary clinical information systems by reviewing studies reporting the implementation and evaluation of artificial intelligence (AI) technologies in healthcare settings. METHOD: PubMed/MEDLINE, Web of Science, EMBASE, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies reporting evaluation of AI in clinical settings from January 2021 to December 2022. We documented the clinical application areas and tasks supported, and the level of system autonomy. Reported effects on user experience, decision-making, care delivery and outcomes were summarised. RESULTS: AI technologies are being applied in a wide variety of clinical areas. Most contemporary systems utilise deep learning, use routinely collected data, support diagnosis and triage, are assistive (requiring users to confirm or approve AI provided information or decisions), and are used by doctors in acute care settings in high-income nations. AI systems are integrated and used within existing clinical information systems including electronic medical records. There is limited support for One Health goals. Evaluation is largely based on quantitative methods measuring effects on decision-making. CONCLUSION: AI systems are being implemented and evaluated in many clinical areas. There remain many opportunities to understand patterns of routine use and evaluate effects on decision-making, care delivery and patient outcomes using mixed-methods. Support for One Health including integrating data about environmental factors and social determinants needs further exploration.

Clinical decision support versus a paper-based protocol for massive transfusion: Impact on decision outcomes in a simulation study.

BACKGROUND: Management of major hemorrhage frequently requires massive transfusion (MT) support, which should be delivered effectively and efficiently. We have previously developed a clinical decision support system (CDS) for MT using a multicenter multidisciplinary user-centered design study. Here we examine its impact when administering a MT. STUDY DESIGN AND METHODS: We conducted a randomized simulation trial to compare a CDS for MT with a paper-based MT protocol for the management of simulated hemorrhage. A total of 44 specialist physicians, trainees (residents), and nurses were recruited across critical care to participate in two 20-min simulated bleeding scenarios. The primary outcome was the decision velocity (correct decisions per hour) and overall task completion. Secondary outcomes included cognitive workload and System Usability Scale (SUS). RESULTS: There was a statistically significant increase in decision velocity for CDS-based management (mean 8.5 decisions per hour) compared to paper based (mean 6.9 decisions per hour; p .003, 95% CI 0.6-2.6). There was no significant difference in the overall task completion using CDS-based management (mean 13.3) compared to paper-based (mean 13.2; p .92, 95% CI -1.2-1.3). Cognitive workload was statistically significantly lower using the CDS compared to the paper protocol (mean 57.1 vs. mean 64.5, p .005, 95% CI 2.4-12.5). CDS usability was assessed as a SUS score of 82.5 (IQR 75-87.5). DISCUSSION: Compared to paper-based management, CDS-based MT supports more time-efficient decision-making by users with limited CDS training and achieves similar overall task completion while reducing cognitive load. Clinical implementation will determine whether the benefits demonstrated translate to improved patient outcomes.

Artificial intelligence in medicine: has the time come to hang up the stethoscope?

The question of whether the time has come to hang up the stethoscope is bound up in the promises of artificial intelligence (AI), promises that have so far proven difficult to deliver, perhaps because of the mismatch between the technical capability of AI and its use in real-world clinical settings. This perspective argues that it is time to move away from discussing the generalised promise of disembodied AI and focus on specifics. We need to focus on how the computational method underlying AI, i.e. machine learning (ML), is embedded into tools, how those tools contribute to clinical tasks and decisions and to what extent they can be relied on. Accordingly, we pose four questions that must be asked to make the discussion real and to understand how ML tools contribute to health care: (1) What does the ML algorithm do? (2) How is output of the ML algorithm used in clinical tools? (3) What does the ML tool contribute to clinical tasks or decisions? (4) Can clinicians act or rely on the ML tool? Two exemplar ML tools are examined to show how these questions can be used to better understand the role of ML in supporting clinical tasks and decisions. Ultimately, ML is just a fancy method of automation. We show that it is useful in automating specific and narrowly defined clinical tasks but likely incapable of automating the full gamut of decisions and tasks performed by clinicians.

Effects of machine learning-based clinical decision support systems on decision-making, care delivery, and patient outcomes: a scoping review.

OBJECTIVE: This study aims to summarize the research literature evaluating machine learning (ML)-based clinical decision support (CDS) systems in healthcare settings. MATERIALS AND METHODS: We conducted a review in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta Analyses extension for Scoping Review). Four databases, including PubMed, Medline, Embase, and Scopus were searched for studies published from January 2016 to April 2021 evaluating the use of ML-based CDS in clinical settings. We extracted the study design, care setting, clinical task, CDS task, and ML method. The level of CDS autonomy was examined using a previously published 3-level classification based on the division of clinical tasks between the clinician and CDS; effects on decision-making, care delivery, and patient outcomes were summarized. RESULTS: Thirty-two studies evaluating the use of ML-based CDS in clinical settings were identified. All were undertaken in developed countries and largely in secondary and tertiary care settings. The most common clinical tasks supported by ML-based CDS were image recognition and interpretation (n = 12) and risk assessment (n = 9). The majority of studies examined assistive CDS (n = 23) which required clinicians to confirm or approve CDS recommendations for risk assessment in sepsis and for interpreting cancerous lesions in colonoscopy. Effects on decision-making, care delivery, and patient outcomes were mixed. CONCLUSION: ML-based CDS are being evaluated in many clinical areas. There remain many opportunities to apply and evaluate effects of ML-based CDS on decision-making, care delivery, and patient outcomes, particularly in resource-constrained settings.

The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision.

Despite the renewed interest in Artificial Intelligence-based clinical decision support systems (AI-CDS), there is still a lack of empirical evidence supporting their effectiveness. This underscores the need for rigorous and continuous evaluation and monitoring of processes and outcomes associated with the introduction of health information technology. We illustrate how the emergence of AI-CDS has helped to bring to the fore the critical importance of evaluation principles and action regarding all health information technology applications, as these hitherto have received limited attention. Key aspects include assessment of design, implementation and adoption contexts; ensuring systems support and optimise human performance (which in turn requires understanding clinical and system logics); and ensuring that design of systems prioritises ethics, equity, effectiveness, and outcomes. Going forward, information technology strategy, implementation and assessment need to actively incorporate these dimensions. International policy makers, regulators and strategic decision makers in implementing organisations therefore need to be cognisant of these aspects and incorporate them in decision-making and in prioritising investment. In particular, the emphasis needs to be on stronger and more evidence-based evaluation surrounding system limitations and risks as well as optimisation of outcomes, whilst ensuring learning and contextual review. Otherwise, there is a risk that applications will be sub-optimally embodied in health systems with unintended consequences and without yielding intended benefits.

Evaluation of machine learning-based models for prediction of clinical deterioration: A systematic literature review.

BACKGROUND AND OBJECTIVE: Early identification of patients at risk of deterioration can prevent life-threatening adverse events and shorten length of stay. Although there are numerous models applied to predict patient clinical deterioration, most are based on vital signs and have methodological shortcomings that are not able to provide accurate estimates of deterioration risk. The aim of this systematic review is to examine the effectiveness, challenges, and limitations of using machine learning (ML) techniques to predict patient clinical deterioration in hospital settings. METHODS: A systematic review was performed in accordance with the Preferred Reporting Items for Systematic Reviews and meta-Analyses (PRISMA) guidelines using EMBASE, MEDLINE Complete, CINAHL Complete, and IEEExplore databases. Citation searching was carried out for studies that met inclusion criteria. Two reviewers used the inclusion/exclusion criteria to independently screen studies and extract data. To address any discrepancies in the screening process, the two reviewers discussed their findings and a third reviewer was consulted as needed to reach a consensus. Studies focusing on use of ML in predicting patient clinical deterioration that were published from inception to July 2022 were included. RESULTS: A total of 29 primary studies that evaluated ML models to predict patient clinical deterioration were identified. After reviewing these studies, we found that 15 types of ML techniques have been employed to predict patient clinical deterioration. While six studies used a single technique exclusively, several others utilised a combination of classical techniques, unsupervised and supervised learning, as well as other novel techniques. Depending on which ML model was applied and the type of input features, ML models predicted outcomes with an area under the curve from 0.55 to 0.99. CONCLUSIONS: Numerous ML methods have been employed to automate the identification of patient deterioration. Despite these advancements, there is still a need for further investigation to examine the application and effectiveness of these methods in real-world situations.

More than algorithms: an analysis of safety events involving ML-enabled medical devices reported to the FDA.

OBJECTIVE: To examine the real-world safety problems involving machine learning (ML)-enabled medical devices. MATERIALS AND METHODS: We analyzed 266 safety events involving approved ML medical devices reported to the US FDA's MAUDE program between 2015 and October 2021. Events were reviewed against an existing framework for safety problems with Health IT to identify whether a reported problem was due to the ML device (device problem) or its use, and key contributors to the problem. Consequences of events were also classified. RESULTS: Events described hazards with potential to harm (66%), actual harm (16%), consequences for healthcare delivery (9%), near misses that would have led to harm if not for intervention (4%), no harm or consequences (3%), and complaints (2%). While most events involved device problems (93%), use problems (7%) were 4 times more likely to harm (relative risk 4.2; 95% CI 2.5-7). Problems with data input to ML devices were the top contributor to events (82%). DISCUSSION: Much of what is known about ML safety comes from case studies and the theoretical limitations of ML. We contribute a systematic analysis of ML safety problems captured as part of the FDA's routine post-market surveillance. Most problems involved devices and concerned the acquisition of data for processing by algorithms. However, problems with the use of devices were more likely to harm. CONCLUSIONS: Safety problems with ML devices involve more than algorithms, highlighting the need for a whole-of-system approach to safe implementation with a special focus on how users interact with devices.

Multicenter, multidisciplinary user-centered design of a clinical decision-support and simulation system for massive transfusion.

BACKGROUND: Managing critical bleeding with massive transfusion (MT) requires a multidisciplinary team, often physically separated, to perform several simultaneous tasks at short notice. This places a significant cognitive load on team members, who must maintain situational awareness in rapidly changing scenarios. Similar resuscitation scenarios have benefited from the use of clinical decision support (CDS) tools. STUDY DESIGN AND METHODS: A multicenter, multidisciplinary, user-centered design (UCD) study was conducted to design a computerized CDS for MT. This study included analysis of the problem context with a cognitive walkthrough, development of a user requirement statement, and co-design with users of prototypes for testing. The final prototype was evaluated using qualitative assessment and the System Usability Scale (SUS). RESULTS: Eighteen participants were recruited across four institutions. The first UCD cycle resulted in the development of four prototype interfaces that addressed the user requirements and context of implementation. Of these, the preferred interface was further developed in the second UCD cycle to create a high-fidelity web-based CDS for MT. This prototype was evaluated by 15 participants using a simulated bleeding scenario and demonstrated an average SUS of 69.3 (above average, SD 16) and a clear interface with easy-to-follow blood product tracking. DISCUSSION: We used a UCD process to explore a highly complex clinical scenario and develop a prototype CDS for MT that incorporates distributive situational awareness, supports multiple user roles, and allows simulated MT training. Evaluation of the impact of this prototype on the efficacy and efficiency of managing MT is currently underway.

Using automated methods to detect safety problems with health information technology: a scoping review.

OBJECTIVE: To summarize the research literature evaluating automated methods for early detection of safety problems with health information technology (HIT). MATERIALS AND METHODS: We searched bibliographic databases including MEDLINE, ACM Digital, Embase, CINAHL Complete, PsycINFO, and Web of Science from January 2010 to June 2021 for studies evaluating the performance of automated methods to detect HIT problems. HIT problems were reviewed using an existing classification for safety concerns. Automated methods were categorized into rule-based, statistical, and machine learning methods, and their performance in detecting HIT problems was assessed. The review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta Analyses extension for Scoping Reviews statement. RESULTS: Of the 45 studies identified, the majority (n = 27, 60%) focused on detecting use errors involving electronic health records and order entry systems. Machine learning (n = 22) and statistical modeling (n = 17) were the most common methods. Unsupervised learning was used to detect use errors in laboratory test results, prescriptions, and patient records while supervised learning was used to detect technical errors arising from hardware or software issues. Statistical modeling was used to detect use errors, unauthorized access, and clinical decision support system malfunctions while rule-based methods primarily focused on use errors. CONCLUSIONS: A wide variety of rule-based, statistical, and machine learning methods have been applied to automate the detection of safety problems with HIT. Many opportunities remain to systematically study their application and effectiveness in real-world settings.

Using artificial intelligence for personal protective equipment guidance for healthcare workers in the COVID-19 pandemic and beyond.

Background: Current procedures for effective personal protective equipment (PPE) usage rely on the availability of trained observers or 'buddies' who, during the COVID-19 pandemic, are not always available. The application of artificial intelligence (AI) has the potential to overcome this limitation by assisting in complex task analysis. To date, AI use for PPE protocols has not been studied. In this paper we validate the performance of an AI PPE system in a hospital setting. Methods: A clinical cohort study of 74 healthcare workers (HCW) at a 144-bed University teaching hospital. Participants were recruited to use the AI system for PPE donning and doffing. Performance was validated by the current gold standard double-buddy system across seven donning and ten doffing steps based on local infection control guidelines. Results: The AI-PPE platform was 98.9% sensitive on doffing and 85.3% sensitive on donning, when compared to remediated double buddy. On average, buddy correction of PPE was required 3.8 ± 1.5% of the time. The average time taken to don was 240 ± 51.5 seconds and doff was 241 ± 35.3 seconds. Conclusion: This study demonstrates the ability of an AI model to analyse PPE donning and doffing with real-time feedback for remediation. The AI platform can identify complex multi-task PPE donning and doffing in a single validated system. This AI system can be employed to train, audit, and thereby improve compliance whilst reducing reliance on limited HCW resources. Further studies may permit the development of this educational tool into a medical device with other industry uses for safety.

Digital health for climate change mitigation and response: a scoping review.

OBJECTIVE: Climate change poses a major threat to the operation of global health systems, triggering large scale health events, and disrupting normal system operation. Digital health may have a role in the management of such challenges and in greenhouse gas emission reduction. This scoping review explores recent work on digital health responses and mitigation approaches to climate change. MATERIALS AND METHODS: We searched Medline up to February 11, 2022, using terms for digital health and climate change. Included articles were categorized into 3 application domains (mitigation, infectious disease, or environmental health risk management), and 6 technical tasks (data sensing, monitoring, electronic data capture, modeling, decision support, and communication). The review was PRISMA-ScR compliant. RESULTS: The 142 included publications reported a wide variety of research designs. Publication numbers have grown substantially in recent years, but few come from low- and middle-income countries. Digital health has the potential to reduce health system greenhouse gas emissions, for example by shifting to virtual services. It can assist in managing changing patterns of infectious diseases as well as environmental health events by timely detection, reducing exposure to risk factors, and facilitating the delivery of care to under-resourced areas. DISCUSSION: While digital health has real potential to help in managing climate change, research remains preliminary with little real-world evaluation. CONCLUSION: Significant acceleration in the quality and quantity of digital health climate change research is urgently needed, given the enormity of the global challenge.

The Role of Formative Evaluation in Promoting Digitally-based Health Equity and Reducing Bias for Resilient Health Systems: The Case of Patient Portals.

OBJECTIVES: Patient portals are increasingly implemented to improve patient involvement and engagement. We here seek to provide an overview of ways to mitigate existing concerns that these technologies increase inequity and bias and do not reach those who could benefit most from them. METHODS: Based on the current literature, we review the limitations of existing evaluations of patient portals in relation to addressing health equity, literacy and bias; outline challenges evaluators face when conducting such evaluations; and suggest methodological approaches that may address existing shortcomings. RESULTS: Various stakeholder needs should be addressed before deploying patient portals, involving vulnerable groups in user-centred design, and studying unanticipated consequences and impacts of information systems in use over time. CONCLUSIONS: Formative approaches to evaluation can help to address existing shortcomings and facilitate the development and implementation of patient portals in an equitable way thereby promoting the creation of resilient health systems.

Ethical Guidance for Hard Decisions: A Critical Review of Early International COVID-19 ICU Triage Guidelines.

This article provides a critical comparative analysis of the substantive and procedural values and ethical concepts articulated in guidelines for allocating scarce resources in the COVID-19 pandemic. We identified 21 local and national guidelines written in English, Spanish, German and French; applicable to specific and identifiable jurisdictions; and providing guidance to clinicians for decision making when allocating critical care resources during the COVID-19 pandemic. US guidelines were not included, as these had recently been reviewed elsewhere. Information was extracted from each guideline on: 1) the development process; 2) the presence and nature of ethical, medical and social criteria for allocating critical care resources; and 3) the membership of and decision-making procedure of any triage committees. Results of our analysis show the majority appealed primarily to consequentialist reasoning in making allocation decisions, tempered by a largely pluralistic approach to other substantive and procedural values and ethical concepts. Medical and social criteria included medical need, co-morbidities, prognosis, age, disability and other factors, with a focus on seemingly objective medical criteria. There was little or no guidance on how to reconcile competing criteria, and little attention to internal contradictions within individual guidelines. Our analysis reveals the challenges in developing sound ethical guidance for allocating scarce medical resources, highlighting problems in operationalising ethical concepts and principles, divergence between guidelines, unresolved contradictions within the same guideline, and use of naïve objectivism in employing widely used medical criteria for allocating ICU resources.