Prognostic clinical decision support for pneumonia in the emergency department: A randomized trial.

BACKGROUND: Hospitalization rates for childhood pneumonia vary widely. Risk-based clinical decision support (CDS) interventions may reduce unwarranted variation. METHODS: We conducted a pragmatic randomized trial in two US pediatric emergency departments (EDs) comparing electronic health record (EHR)-integrated prognostic CDS versus usual care for promoting appropriate ED disposition in children (<18 years) with pneumonia. Encounters were randomized 1:1 to usual care versus custom CDS featuring a validated pneumonia severity score predicting risk for severe in-hospital outcomes. Clinicians retained full decision-making authority. The primary outcome was inappropriate ED disposition, defined as early transition to lower- or higher-level care. Safety and implementation outcomes were also evaluated. RESULTS: The study enrolled 536 encounters (269 usual care and 267 CDS). Baseline characteristics were similar across arms. Inappropriate disposition occurred in 3% of usual care encounters and 2% of CDS encounters (adjusted odds ratio: 0.99, 95% confidence interval: [0.32, 2.95]) Length of stay was also similar and adverse safety outcomes were uncommon in both arms. The tool's custom user interface and content were viewed as strengths by surveyed clinicians (>70% satisfied). Implementation barriers include intrinsic (e.g., reaching the right person at the right time) and extrinsic factors (i.e., global pandemic). CONCLUSIONS: EHR-based prognostic CDS did not improve ED disposition decisions for children with pneumonia. Although the intervention's content was favorably received, low subject accrual and workflow integration problems likely limited effectiveness. Clinical Trials Registration: NCT06033079.

User-Centered Design and Implementation of an Interoperable FHIR Application for Pediatric Pneumonia Prognostication in a Randomized Trial.

OBJECTIVE: To support a pragmatic, electronic health record (EHR)-based randomized controlled trial, we applied user-centered design (UCD) principles, evidence-based risk communication strategies, and interoperable software architecture to design, test, and deploy a prognostic tool for children in emergency departments (EDs) with pneumonia. METHODS: Risk for severe in-hospital outcomes was estimated using a validated ordinal logistic regression model to classify pneumonia severity. To render the results usable for ED clinicians, we created an integrated SMART on FHIR web application built for interoperable use in two pediatric EDs using different EHR vendors: Epic and Cerner. We followed a UCD framework, including problem analysis and user research, conceptual design and early prototyping, user interface development, formative evaluation, and post-deployment summative evaluation. RESULTS: Problem analysis and user research from 39 clinicians and nurses revealed user preferences for risk aversion, accessibility, and timing of risk communication. Early prototyping and iterative design incorporated evidence-based design principles, including numeracy, risk framing, and best-practice visualization techniques. After rigorous unit and end-to-end testing, the application was successfully deployed in both EDs, which facilitatd enrollment, randomization, model visualization, data capture, and reporting for trial purposes. CONCLUSIONS: The successful implementation of a custom application for pneumonia prognosis and clinical trial support in two health systems on different EHRs demonstrates the importance of UCD, adherence to modern clinical data standards, and rigorous testing. Key lessons included the need for understanding users' real-world needs, regular knowledge management, application maintenance, and the recognition that FHIR applications require careful configuration for interoperability.

Randomized Controlled Comparative Effectiveness Trial of Risk Model-Guided Clinical Decision Support for Suicide Screening.

Suicide prevention requires risk identification, appropriate intervention, and follow-up. Traditional risk identification relies on patient self-reporting, support network reporting, or face-to-face screening with validated instruments or history and physical exam. In the last decade, statistical risk models have been studied and more recently deployed to augment clinical judgment. Models have generally been found to be low precision or problematic at scale due to low incidence. Few have been tested in clinical practice, and none have been tested in clinical trials to our knowledge. Methods: We report the results of a pragmatic randomized controlled trial (RCT) in three outpatient adult Neurology clinic settings. This two-arm trial compared the effectiveness of Interruptive and Non-Interruptive Clinical Decision Support (CDS) to prompt further screening of suicidal ideation for those predicted to be high risk using a real-time, validated statistical risk model of suicide attempt risk, with the decision to screen as the primary end point. Secondary outcomes included rates of suicidal ideation and attempts in both arms. Manual chart review of every trial encounter was used to determine if suicide risk assessment was subsequently documented. Results: From August 16, 2022, through February 16, 2023, our study randomized 596 patient encounters across 561 patients for providers to receive either Interruptive or Non-Interruptive CDS in a 1:1 ratio. Adjusting for provider cluster effects, Interruptive CDS led to significantly higher numbers of decisions to screen (42%=121/289 encounters) compared to Non-Interruptive CDS (4%=12/307) (odds ratio=17.7, p-value <0.001). Secondarily, no documented episodes of suicidal ideation or attempts occurred in either arm. While the proportion of documented assessments among those noting the decision to screen was higher for providers in the Non-Interruptive arm (92%=11/12) than in the Interruptive arm (52%=63/121), the interruptive CDS was associated with more frequent documentation of suicide risk assessment (63/289 encounters compared to 11/307, p-value<0.001). Conclusions: In this pragmatic RCT of real-time predictive CDS to guide suicide risk assessment, Interruptive CDS led to higher numbers of decisions to screen and documented suicide risk assessments. Well-powered large-scale trials randomizing this type of CDS compared to standard of care are indicated to measure effectiveness in reducing suicidal self-harm. ClinicalTrials.gov Identifier: NCT05312437.

Adapting Cognitive Task Analysis Methods for Use in a Large Sample Simulation Study of High-Risk Healthcare Events.

Cognitive task analysis (CTA) methods are traditionally used to conduct small-sample, in-depth studies. In this case study, CTA methods were adapted for a large multi-site study in which 102 anesthesiologists worked through four different high-fidelity simulated high-consequence incidents. Cognitive interviews were used to elicit decision processes following each simulated incident. In this paper, we highlight three practical challenges that arose: (1) standardizing the interview techniques for use across a large, distributed team of diverse backgrounds; (2) developing effective training; and (3) developing a strategy to analyze the resulting large amount of qualitative data. We reflect on how we addressed these challenges by increasing standardization, developing focused training, overcoming social norms that hindered interview effectiveness, and conducting a staged analysis. We share findings from a preliminary analysis that provides early validation of the strategy employed. Analysis of a subset of 64 interview transcripts using a decompositional analysis approach suggests that interviewers successfully elicited descriptions of decision processes that varied due to the different challenges presented by the four simulated incidents. A holistic analysis of the same 64 transcripts revealed individual differences in how anesthesiologists interpreted and managed the same case.

External Validation and Updating of a Statistical Civilian-Based Suicide Risk Model in US Naval Primary Care.

Importance: Suicide remains an ongoing concern in the US military. Statistical models have not been broadly disseminated for US Navy service members. Objective: To externally validate and update a statistical suicide risk model initially developed in a civilian setting with an emphasis on primary care. Design, Setting, and Participants: This retrospective cohort study used data collected from 2007 through 2017 among active-duty US Navy service members. The external civilian model was applied to every visit at Naval Medical Center Portsmouth (NMCP), its NMCP Naval Branch Health Clinics (NBHCs), and TRICARE Prime Clinics (TPCs) that fall within the NMCP area. The model was retrained and recalibrated using visits to NBHCs and TPCs and updated using Department of Defense (DoD)-specific billing codes and demographic characteristics, including expanded race and ethnicity categories. Domain and temporal analyses were performed with bootstrap validation. Data analysis was performed from September 2020 to December 2022. Exposure: Visit to US NMCP. Main Outcomes and Measures: Recorded suicidal behavior on the day of or within 30 days of a visit. Performance was assessed using area under the receiver operating curve (AUROC), area under the precision recall curve (AUPRC), Brier score, and Spiegelhalter z-test statistic. Results: Of the 260 583 service members, 6529 (2.5%) had a recorded suicidal behavior, 206 412 (79.2%) were male; 104 835 (40.2%) were aged 20 to 24 years; and 9458 (3.6%) were Asian, 56 715 (21.8%) were Black or African American, and 158 277 (60.7%) were White. Applying the civilian-trained model resulted in an AUROC of 0.77 (95% CI, 0.74-0.79) and an AUPRC of 0.004 (95% CI, 0.003-0.005) at NBHCs with poor calibration (Spiegelhalter P < .001). Retraining the algorithm improved AUROC to 0.92 (95% CI, 0.91-0.93) and AUPRC to 0.66 (95% CI, 0.63-0.68). Number needed to screen in the top risk tiers was 366 for the external model and 200 for the retrained model; the lower number indicates better performance. Domain validation showed AUROC of 0.90 (95% CI, 0.90-0.91) and AUPRC of 0.01 (95% CI, 0.01-0.01), and temporal validation showed AUROC of 0.75 (95% CI, 0.72-0.78) and AUPRC of 0.003 (95% CI, 0.003-0.005). Conclusions and Relevance: In this cohort study of active-duty Navy service members, a civilian suicide attempt risk model was externally validated. Retraining and updating with DoD-specific variables improved performance. Domain and temporal validation results were similar to external validation, suggesting that implementing an external model in US Navy primary care clinics may bypass the need for costly internal development and expedite the automation of suicide prevention in these clinics.

Inpatient nurses' preferences and decisions with risk information visualization.

OBJECTIVE: We examined the influence of 4 different risk information formats on inpatient nurses' preferences and decisions with an acute clinical deterioration decision-support system. MATERIALS AND METHODS: We conducted a comparative usability evaluation in which participants provided responses to multiple user interface options in a simulated setting. We collected qualitative data using think aloud methods. We collected quantitative data by asking participants which action they would perform after each time point in 3 different patient scenarios. RESULTS: More participants (n = 6) preferred the probability format over relative risk ratios (n = 2), absolute differences (n = 2), and number of persons out of 100 (n = 0). Participants liked average lines, having a trend graph to supplement the risk estimate, and consistent colors between trend graphs and possible actions. Participants did not like too much text information or the presence of confidence intervals. From a decision-making perspective, use of the probability format was associated with greater concordance in actions taken by participants compared to the other 3 risk information formats. DISCUSSION: By focusing on nurses' preferences and decisions with several risk information display formats and collecting both qualitative and quantitative data, we have provided meaningful insights for the design of clinical decision-support systems containing complex quantitative information. CONCLUSION: This study adds to our knowledge of presenting risk information to nurses within clinical decision-support systems. We encourage those developing risk-based systems for inpatient nurses to consider expressing risk in a probability format and include a graph (with average line) to display the patient's recent trends.

Decision-making in anesthesiology: will artificial intelligence make intraoperative care safer?

PURPOSE OF REVIEW: This article explores the impact of recent applications of artificial intelligence on clinical anesthesiologists' decision-making. RECENT FINDINGS: Naturalistic decision-making, a rich research field that aims to understand how cognitive work is accomplished in complex environments, provides insight into anesthesiologists' decision processes. Due to the complexity of clinical work and limits of human decision-making (e.g. fatigue, distraction, and cognitive biases), attention on the role of artificial intelligence to support anesthesiologists' decision-making has grown. Artificial intelligence, a computer's ability to perform human-like cognitive functions, is increasingly used in anesthesiology. Examples include aiding in the prediction of intraoperative hypotension and postoperative complications, as well as enhancing structure localization for regional and neuraxial anesthesia through artificial intelligence integration with ultrasound. SUMMARY: To fully realize the benefits of artificial intelligence in anesthesiology, several important considerations must be addressed, including its usability and workflow integration, appropriate level of trust placed on artificial intelligence, its impact on decision-making, the potential de-skilling of practitioners, and issues of accountability. Further research is needed to enhance anesthesiologists' clinical decision-making in collaboration with artificial intelligence.

Decision-Making During High-Risk Events: A Systematic Literature Review.

Effective decision-making in crisis events is challenging due to time pressure, uncertainty, and dynamic decisional environments. We conducted a systematic literature review in PubMed and PsycINFO, identifying 32 empiric research papers that examine how trained professionals make naturalistic decisions under pressure. We used structured qualitative analysis methods to extract key themes. The studies explored different aspects of decision-making across multiple domains. The majority (19) focused on healthcare; military, fire and rescue, oil installation, and aviation domains were also represented. We found appreciable variability in research focus, methodology, and decision-making descriptions. We identified five main themes: (1) decision-making strategy, (2) time pressure, (3) stress, (4) uncertainty, and (5) errors. Recognition-primed decision-making (RPD) strategies were reported in all studies that analyzed this aspect. Analytical strategies were also prominent, appearing more frequently in contexts with less time pressure and explicit training to generate multiple explanations. Practitioner experience, time pressure, stress, and uncertainty were major influencing factors. Professionals must adapt to the time available, types of uncertainty, and individual skills when making decisions in high-risk situations. Improved understanding of these decisional factors can inform evidence-based enhancements to training, technology, and process design.

Medication Safety Amid Technological Change: Usability Evaluation to Inform Inpatient Nurses' Electronic Health Record System Transition.

BACKGROUND: Electronic health record (EHR) system transitions are challenging for healthcare organizations. High-volume, safety-critical tasks like barcode medication administration (BCMA) should be evaluated, yet standards for ensuring safety during transition have not been established. OBJECTIVE: Identify risks in common and problem-prone medication tasks to inform safe transition between BCMA systems and establish benchmarks for future system changes. DESIGN: Staff nurses completed simulation-based usability testing in the legacy system (R1) and new system pre- (R2) and post-go-live (R3). Tasks included (1) Hold/Administer, (2) IV Fluids, (3) PRN Pain, (4) Insulin, (5) Downtime/PRN, and (6) Messaging. Audiovisual recordings of task performance were systematically analyzed for time, navigation, and errors. The System Usability Scale measured perceived usability and satisfaction. Post-simulation interviews captured nurses' qualitative comments and perceptions of the systems. PARTICIPANTS: Fifteen staff nurses completed 2-3-h simulation sessions. Eleven completed both R1 and R2, and seven completed all three rounds. Clinical experience ranged from novice (< 1 year) to experienced (> 10 years). Practice settings included adult and pediatric patient populations in ICU, stepdown, and acute care departments. MAIN MEASURES: Task completion rates/times, safety and non-safety-related use errors (interaction difficulties), and user satisfaction. KEY RESULTS: Overall success rates remained relatively stable in all tasks except two: IV Fluids task success increased substantially (R1: 17%, R2: 54%, R3: 100%) and Downtime/PRN task success decreased (R1: 92%, R2: 64%, R3: 22%). Among the seven nurses who completed all rounds, overall safety-related errors decreased 53% from R1 to R3 and 50% from R2 to R3, and average task times for successfully completed tasks decreased 22% from R1 to R3 and 38% from R2 to R3. CONCLUSIONS: Usability testing is a reasonable approach to compare different BCMA tasks to anticipate transition problems and establish benchmarks with which to monitor and evaluate system changes going forward.

Empowering patients to address diabetes care gaps: formative usability testing of a novel patient portal intervention.

Objective: The aim of this study was to design and assess the formative usability of a novel patient portal intervention designed to empower patients with diabetes to initiate orders for diabetes-related monitoring and preventive services. Materials and Methods: We used a user-centered Design Sprint methodology to create our intervention prototype and assess its usability with 3 rounds of iterative testing. Participants (5/round) were presented with the prototype and asked to perform common, standardized tasks using think-aloud procedures. A facilitator rated task performance using a scale: (1) completed with ease, (2) completed with difficulty, and (3) failed. Participants completed the System Usability Scale (SUS) scored 0-worst to 100-best. All testing occurred remotely via Zoom. Results: We identified 3 main categories of usability issues: distrust about the automated system, content concerns, and layout difficulties. Changes included improving clarity about the ordering process and simplifying language; however, design constraints inherent to the electronic health record system limited our ability to respond to all usability issues (eg, could not modify fixed elements in layout). Percent of tasks completed with ease across each round were 67%, 60%, and 80%, respectively. Average SUS scores were 87, 74, and 93, respectively. Across rounds, participants found the intervention valuable and appreciated the concept of patient-initiated ordering. Conclusions: Through iterative user-centered design and testing, we improved the usability of the patient portal intervention. A tool that empowers patients to initiate orders for disease-specific services as part of their existing patient portal account has potential to enhance the completion of recommended health services and improve clinical outcomes.

Antibiotic clinical decision support for pneumonia in the ED: A randomized trial.

BACKGROUND: Electronic health record-based clinical decision support (CDS) is a promising antibiotic stewardship strategy. Few studies have evaluated the effectiveness of antibiotic CDS in the pediatric emergency department (ED). OBJECTIVE: To compare the effectiveness of antibiotic CDS vs. usual care for promoting guideline-concordant antibiotic prescribing for pneumonia in the pediatric ED. DESIGN: Pragmatic randomized clinical trial. SETTING AND PARTICIPANTS: Encounters for children (6 months-18 years) with pneumonia presenting to two tertiary care children s hospital EDs in the United States. INTERVENTION: CDS or usual care was randomly assigned during 4-week periods within each site. The CDS intervention provided antibiotic recommendations tailored to each encounter and in accordance with national guidelines. MAIN OUTCOME AND MEASURES: The primary outcome was exclusive guideline-concordant antibiotic prescribing within the first 24 h of care. Safety outcomes included time to first antibiotic order, encounter length of stay, delayed intensive care, and 3- and 7-day revisits. RESULTS: 1027 encounters were included, encompassing 478 randomized to usual care and 549 to CDS. Exclusive guideline-concordant prescribing did not differ at 24 h (CDS, 51.7% vs. usual care, 53.3%; odds ratio [OR] 0.94 [95% confidence interval [CI]: 0.73, 1.20]). In pre-specified stratified analyses, CDS was associated with guideline-concordant prescribing among encounters discharged from the ED (74.9% vs. 66.0%; OR 1.53 [95% CI: 1.01, 2.33]), but not among hospitalized encounters. Mean time to first antibiotic was shorter in the CDS group (3.0 vs 3.4 h; p = .024). There were no differences in safety outcomes. CONCLUSIONS: Effectiveness of ED-based antibiotic CDS was greatest among those discharged from the ED. Longitudinal interventions designed to target both ED and inpatient clinicians and to address common implementation challenges may enhance the effectiveness of CDS as a stewardship tool.

Trust in AI: why we should be designing for APPROPRIATE reliance.

Use of artificial intelligence in healthcare, such as machine learning-based predictive algorithms, holds promise for advancing outcomes, but few systems are used in routine clinical practice. Trust has been cited as an important challenge to meaningful use of artificial intelligence in clinical practice. Artificial intelligence systems often involve automating cognitively challenging tasks. Therefore, previous literature on trust in automation may hold important lessons for artificial intelligence applications in healthcare. In this perspective, we argue that informatics should take lessons from literature on trust in automation such that the goal should be to foster appropriate trust in artificial intelligence based on the purpose of the tool, its process for making recommendations, and its performance in the given context. We adapt a conceptual model to support this argument and present recommendations for future work.

User-Centered Design of a Machine Learning Intervention for Suicide Risk Prediction in a Military Setting.

Primary care represents a major opportunity for suicide prevention in the military. Significant advances have been made in using electronic health record data to predict suicide attempts in patient populations. With a user-centered design approach, we are developing an intervention that uses predictive analytics to inform care teams about their patients' risk of suicide attempt. We present our experience working with clinicians and staff in a military primary care setting to create preliminary designs and a context-specific usability testing plan for the deployment of the suicide risk indicator.

Descriptive Usability Study of CirrODS: Clinical Decision and Workflow Support Tool for Management of Patients With Cirrhosis.

BACKGROUND: There are gaps in delivering evidence-based care for patients with chronic liver disease and cirrhosis. OBJECTIVE: Our objective was to use interactive user-centered design methods to develop the Cirrhosis Order Set and Clinical Decision Support (CirrODS) tool in order to improve clinical decision-making and workflow. METHODS: Two work groups were convened with clinicians, user experience designers, human factors and health services researchers, and information technologists to create user interface designs. CirrODS prototypes underwent several rounds of formative design. Physicians (n=20) at three hospitals were provided with clinical scenarios of patients with cirrhosis, and the admission orders made with and without the CirrODS tool were compared. The physicians rated their experience using CirrODS and provided comments, which we coded into categories and themes. We assessed the safety, usability, and quality of CirrODS using qualitative and quantitative methods. RESULTS: We created an interactive CirrODS prototype that displays an alert when existing electronic data indicate a patient is at risk for cirrhosis. The tool consists of two primary frames, presenting relevant patient data and allowing recommended evidence-based tests and treatments to be ordered and categorized. Physicians viewed the tool positively and suggested that it would be most useful at the time of admission. When using the tool, the clinicians placed fewer orders than they placed when not using the tool, but more of the orders placed were considered to be high priority when the tool was used than when it was not used. The physicians' ratings of CirrODS indicated above average usability. CONCLUSIONS: We developed a novel Web-based combined clinical decision-making and workflow support tool to alert and assist clinicians caring for patients with cirrhosis. Further studies are underway to assess the impact on quality of care for patients with cirrhosis in actual practice.

Facilitated Nurse Medication-Related Event Reporting to Improve Medication Management Quality and Safety in Intensive Care Units.

BACKGROUND: Medication safety presents an ongoing challenge for nurses working in complex, fast-paced, intensive care unit (ICU) environments. Studying ICU nurse's medication management-especially medication-related events (MREs)-provides an approach to analyze and improve medication safety and quality. OBJECTIVES: The goal of this study was to explore the utility of facilitated MRE reporting in identifying system deficiencies and the relationship between MREs and nurses' work in the ICUs. METHODS: We conducted 124 structured 4-hour observations of nurses in three different ICUs. Each observation included measurement of nurse's moment-to-moment activity and self-reports of workload and negative mood. The observer then obtained MRE reports from the nurse using a structured tool. The MREs were analyzed by three experts. RESULTS: MREs were reported in 35% of observations. The 60 total MREs included four medication errors and seven adverse drug events. Of the 49 remaining MREs, 65% were associated with negative patient impact. Task/process deficiencies were the most common contributory factor for MREs. MRE occurrence was correlated with increased total task volume. MREs also correlated with increased workload, especially during night shifts. DISCUSSION: Most of these MREs would not be captured by traditional event reporting systems. Facilitated MRE reporting provides a robust information source about potential breakdowns in medication management safety and opportunities for system improvement.