Long-term Effect of Machine Learning-Triggered Behavioral Nudges on Serious Illness Conversations and End-of-Life Outcomes Among Patients With Cancer: A Randomized Clinical Trial.

Importance: Serious illness conversations (SICs) between oncology clinicians and patients are associated with improved quality of life and may reduce aggressive end-of-life care. However, most patients with cancer die without a documented SIC. Objective: To test the impact of behavioral nudges to clinicians to prompt SICs on the SIC rate and end-of-life outcomes among patients at high risk of death within 180 days (high-risk patients) as identified by a machine learning algorithm. Design, Setting, and Participants: This prespecified 40-week analysis of a stepped-wedge randomized clinical trial conducted between June 17, 2019, and April 20, 2020 (including 16 weeks of intervention rollout and 24 weeks of follow-up), included 20 506 patients with cancer representing 41 021 encounters at 9 tertiary or community-based medical oncology clinics in a large academic health system. The current analyses were conducted from June 1, 2021, to May 31, 2022. Intervention: High-risk patients were identified using a validated electronic health record machine learning algorithm to predict 6-month mortality. The intervention consisted of (1) weekly emails to clinicians comparing their SIC rates for all patients against peers' rates, (2) weekly lists of high-risk patients, and (3) opt-out text messages to prompt SICs before encounters with high-risk patients. Main Outcomes and Measures: The primary outcome was SIC rates for all and high-risk patient encounters; secondary end-of-life outcomes among decedents included inpatient death, hospice enrollment and length of stay, and intensive care unit admission and systemic therapy close to death. Intention-to-treat analyses were adjusted for clinic and wedge fixed effects and clustered at the oncologist level. Results: The study included 20 506 patients (mean [SD] age, 60.0 [14.0] years) and 41 021 patient encounters: 22 259 (54%) encounters with female patients, 28 907 (70.5%) with non-Hispanic White patients, and 5520 (13.5%) with high-risk patients; 1417 patients (6.9%) died by the end of follow-up. There were no meaningful differences in demographic characteristics in the control and intervention periods. Among high-risk patient encounters, the unadjusted SIC rates were 3.4% (59 of 1754 encounters) in the control period and 13.5% (510 of 3765 encounters) in the intervention period. In adjusted analyses, the intervention was associated with increased SICs for all patients (adjusted odds ratio, 2.09 [95% CI, 1.53-2.87]; P < .001) and decreased end-of-life systemic therapy (7.5% [72 of 957 patients] vs 10.4% [24 of 231 patients]; adjusted odds ratio, 0.25 [95% CI, 0.11-0.57]; P = .001) relative to controls, but there was no effect on hospice enrollment or length of stay, inpatient death, or end-of-life ICU use. Conclusions and Relevance: In this randomized clinical trial, a machine learning-based behavioral intervention and behavioral nudges to clinicans led to an increase in SICs and reduction in end-of-life systemic therapy but no changes in other end-of-life outcomes among outpatients with cancer. These results suggest that machine learning and behavioral nudges can lead to long-lasting improvements in cancer care delivery. Trial Registration: ClinicalTrials.gov Identifier: NCT03984773.

Desired Characteristics of a Clinical Decision Support System for Early Sepsis Recognition: Interview Study Among Hospital-Based Clinicians.

BACKGROUND: Sepsis is a major burden for health care systems in the United States, with over 750,000 cases annually and a total cost of approximately US $20 billion. The hallmark of sepsis treatment is early and appropriate initiation of antibiotic therapy. Although sepsis clinical decision support (CDS) systems can provide clinicians with early predictions of suspected sepsis or imminent clinical decline, such systems have not reliably demonstrated improvements in clinical outcomes or care processes. Growing evidence suggests that the challenges of integrating sepsis CDS systems into clinical workflows, gaining the trust of clinicians, and making sepsis CDS systems clinically relevant at the bedside are all obstacles to successful deployment. However, there are significant knowledge gaps regarding the achievement of these implementation and deployment goals. OBJECTIVE: We aimed to identify perceptions of predictive information in sepsis CDS systems based on clinicians' past experiences, explore clinicians' perceptions of a hypothetical sepsis CDS system, and identify the characteristics of a CDS system that would be helpful in promoting timely recognition and management of suspected sepsis in a multidisciplinary, team-based clinical setting. METHODS: We conducted semistructured interviews with practicing bedside nurses, advanced practice providers, and physicians at a large academic medical center between September 2020 and March 2021. We used modified human factor methods (contextual interview and cognitive walkthrough performed over video calls because of the COVID-19 pandemic) and conducted a thematic analysis using an abductive approach for coding to identify important patterns and concepts in the interview transcripts. RESULTS: We interviewed 6 bedside nurses and 9 clinicians responsible for ordering antibiotics (advanced practice providers or physicians) who had a median of 4 (IQR 4-6.5) years of experience working in an inpatient setting. We then synthesized critical content from the thematic analysis of the data into four domains: clinician perceptions of prediction models and alerts; previous experiences of clinician encounters with predictive information and risk scores; desired characteristics of a CDS system build, including predictions, supporting information, and delivery methods for a potential alert; and the clinical relevance and potential utility of a CDS system. These 4 domains were strongly linked to clinicians' perceptions of the likelihood of adoption and the impact on clinical workflows when diagnosing and managing patients with suspected sepsis. Ultimately, clinicians desired a trusted and actionable CDS system to improve sepsis care. CONCLUSIONS: Building a trusted and actionable sepsis CDS alert is paramount to achieving acceptability and use among clinicians. These findings can inform the development, implementation, and deployment strategies for CDS systems that support the early detection and treatment of sepsis. This study also highlights several key opportunities when eliciting clinician input before the development and deployment of prediction models.

Clinician perspectives on machine learning prognostic algorithms in the routine care of patients with cancer: a qualitative study.

PURPOSE: Oncologists may overestimate prognosis for patients with cancer, leading to delayed or missed conversations about patients' goals and subsequent low-quality end-of-life care. Machine learning algorithms may accurately predict mortality risk in cancer, but it is unclear how oncology clinicians would use such algorithms in practice. METHODS: The purpose of this qualitative study was to assess oncology clinicians' perceptions on the utility and barriers of machine learning prognostic algorithms to prompt advance care planning. Participants included medical oncology physicians and advanced practice providers (APPs) practicing in tertiary and community practices within a large academic healthcare system. Transcripts were coded and analyzed inductively using NVivo software. RESULTS: The study included 29 oncology clinicians (19 physicians, 10 APPs) across 6 practice sites (1 tertiary, 5 community) in the USA. Fourteen participants had previously had exposure to an automated machine learning-based prognostic algorithm as part of a pragmatic randomized trial. Clinicians believed that there was utility for algorithms in validating their own intuition about prognosis and prompting conversations about patient goals and preferences. However, this enthusiasm was tempered by concerns about algorithm accuracy, over-reliance on algorithm predictions, and the ethical implications around disclosure of an algorithm prediction. There was significant variation in tolerance for false positive vs. false negative predictions. CONCLUSION: While oncologists believe there are applications for advanced prognostic algorithms in routine care of patients with cancer, they are concerned about algorithm accuracy, confirmation and automation biases, and ethical issues of prognostic disclosure.

Effect of Integrating Machine Learning Mortality Estimates With Behavioral Nudges to Clinicians on Serious Illness Conversations Among Patients With Cancer: A Stepped-Wedge Cluster Randomized Clinical Trial.

IMPORTANCE: Serious illness conversations (SICs) are structured conversations between clinicians and patients about prognosis, treatment goals, and end-of-life preferences. Interventions that increase the rate of SICs between oncology clinicians and patients may improve goal-concordant care and patient outcomes. OBJECTIVE: To determine the effect of a clinician-directed intervention integrating machine learning mortality predictions with behavioral nudges on motivating clinician-patient SICs. DESIGN, SETTING, AND PARTICIPANTS: This stepped-wedge cluster randomized clinical trial was conducted across 20 weeks (from June 17 to November 1, 2019) at 9 medical oncology clinics (8 subspecialty oncology and 1 general oncology clinics) within a large academic health system in Pennsylvania. Clinicians at the 2 smallest subspecialty clinics were grouped together, resulting in 8 clinic groups randomly assigned to the 4 intervention wedge periods. Included participants in the intention-to-treat analyses were 78 oncology clinicians who received SIC training and their patients (N = 14 607) who had an outpatient oncology encounter during the study period. INTERVENTIONS: (1) Weekly emails to oncology clinicians with SIC performance feedback and peer comparisons; (2) a list of up to 6 high-risk patients (≥10% predicted risk of 180-day mortality) scheduled for the next week, estimated using a validated machine learning algorithm; and (3) opt-out text message prompts to clinicians on the patient's appointment day to consider an SIC. Clinicians in the control group received usual care consisting of weekly emails with cumulative SIC performance. MAIN OUTCOMES AND MEASURES: Percentage of patient encounters with an SIC in the intervention group vs the usual care (control) group. RESULTS: The sample consisted of 78 clinicians and 14 607 patients. The mean (SD) age of patients was 61.9 (14.2) years, 53.7% were female, and 70.4% were White. For all encounters, SICs were conducted among 1.3% in the control group and 4.6% in the intervention group, a significant difference (adjusted difference in percentage points, 3.3; 95% CI, 2.3-4.5; P < .001). Among 4124 high-risk patient encounters, SICs were conducted among 3.6% in the control group and 15.2% in the intervention group, a significant difference (adjusted difference in percentage points, 11.6; 95% CI, 8.2-12.5; P < .001). CONCLUSIONS AND RELEVANCE: In this stepped-wedge cluster randomized clinical trial, an intervention that delivered machine learning mortality predictions with behavioral nudges to oncology clinicians significantly increased the rate of SICs among all patients and among patients with high mortality risk who were targeted by the intervention. Behavioral nudges combined with machine learning mortality predictions can positively influence clinician behavior and may be applied more broadly to improve care near the end of life. TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT03984773.

Integrating machine-generated mortality estimates and behavioral nudges to promote serious illness conversations for cancer patients: Design and methods for a stepped-wedge cluster randomized controlled trial.

INTRODUCTION: Patients with cancer often receive care that is not aligned with their personal values and goals. Serious illness conversations (SICs) between clinicians and patients can help increase a patient's understanding of their prognosis, goals and values. METHODS AND ANALYSIS: In this study, we describe the design of a stepped-wedge cluster randomized trial to evaluate the impact of an intervention that employs machine learning-based prognostic algorithms and behavioral nudges to prompt oncologists to have SICs with patients at high risk of short-term mortality. Data are collected on documented SICs, documented advance care planning discussions, and end-of-life care utilization (emergency room and inpatient admissions, chemotherapy and hospice utilization) for patients of all enrolled clinicians. CONCLUSION: This trial represents a novel application of machine-generated mortality predictions combined with behavioral nudges in the routine care of outpatients with cancer. Findings from the trial may inform strategies to encourage early serious illness conversations and the application of mortality risk predictions in clinical settings. TRIAL REGISTRATION: Clinicaltrials.gov Identifier: NCT03984773.

Electronic Health Record Mortality Prediction Model for Targeted Palliative Care Among Hospitalized Medical Patients: a Pilot Quasi-experimental Study.

BACKGROUND: Development of electronic health record (EHR) prediction models to improve palliative care delivery is on the rise, yet the clinical impact of such models has not been evaluated. OBJECTIVE: To assess the clinical impact of triggering palliative care using an EHR prediction model. DESIGN: Pilot prospective before-after study on the general medical wards at an urban academic medical center. PARTICIPANTS: Adults with a predicted probability of 6-month mortality of ≥ 0.3. INTERVENTION: Triggered (with opt-out) palliative care consult on hospital day 2. MAIN MEASURES: Frequencies of consults, advance care planning (ACP) documentation, home palliative care and hospice referrals, code status changes, and pre-consult length of stay (LOS). KEY RESULTS: The control and intervention periods included 8 weeks each and 138 admissions and 134 admissions, respectively. Characteristics between the groups were similar, with a mean (standard deviation) risk of 6-month mortality of 0.5 (0.2). Seventy-seven (57%) triggered consults were accepted by the primary team and 8 consults were requested per usual care during the intervention period. Compared to historical controls, consultation increased by 74% (22 [16%] vs 85 [63%], P < .001), median (interquartile range) pre-consult LOS decreased by 1.4 days (2.6 [1.1, 6.2] vs 1.2 [0.8, 2.7], P = .02), ACP documentation increased by 38% (23 [17%] vs 37 [28%], P = .03), and home palliative care referrals increased by 61% (9 [7%] vs 23 [17%], P = .01). There were no differences between the control and intervention groups in hospice referrals (14 [10] vs 22 [16], P = .13), code status changes (42 [30] vs 39 [29]; P = .81), or consult requests for lower risk (< 0.3) patients (48/1004 [5] vs 33/798 [4]; P = .48). CONCLUSIONS: Targeting hospital-based palliative care using an EHR mortality prediction model is a clinically promising approach to improve the quality of care among seriously ill medical patients. More evidence is needed to determine the generalizability of this approach and its impact on patient- and caregiver-reported outcomes.