Detection of Medication Taking Using a Wrist-Worn Commercially Available Wearable Device.

PURPOSE: Medication nonadherence is a persistent and costly problem across health care. Measures of medication adherence are ineffective. Methods such as self-report, prescription claims data, or smart pill bottles have been used to monitor medication adherence, but these are subject to recall bias, lack real-time feedback, and are often expensive. METHODS: We proposed a method for monitoring medication adherence using a commercially available wearable device. Passively collected motion data were analyzed on the basis of the Movelet algorithm, a dictionary learning framework that builds person-specific chapters of movements from short frames of elemental activities within the movements. We adapted and extended the Movelet method to construct a within-patient prediction model that identifies medication-taking behaviors. RESULTS: Using 15 activity features recorded from wrist-worn wearable devices of 10 patients with breast cancer on endocrine therapy, we demonstrated that medication-taking behavior can be predicted in a controlled clinical environment with a median accuracy of 85%. CONCLUSION: These results in a patient-specific population are exemplar of the potential to measure real-time medication adherence using a wrist-worn commercially available wearable device.

Economics of a health system's direct-to-consumer telemedicine for its employees.

OBJECTIVES: To compare the mean per-episode unit cost for a direct-to-consumer (DTC) telemedicine service for medical center employees (OnDemand) with that of in-person care and to estimate whether the offered service increased the use of care. STUDY DESIGN: Propensity score-matched retrospective cohort study of adult employees and dependents of a large academic health system between July 7, 2017, and December 31, 2019. METHODS: To estimate differences in per-episode unit costs within 7 days, we compared costs between OnDemand encounters and conventional in-person encounters (primary care, urgent care, and emergency department) for any similar condition using a generalized linear model. We used interrupted time series analyses limited to the top 10 clinical conditions managed by OnDemand to estimate the effect of OnDemand's availability on the trends for overall employee per-month encounters. RESULTS: A total of 10,826 encounters among 7793 beneficiaries were included (mean [SD] age, 38.5 [10.9] years; 81.6% were women). The mean (SE) 7-day per-episode cost among employees and beneficiaries was lower for OnDemand encounters at $379.76 ($19.83) relative to non-OnDemand encounters at $493.49 ($25.53), a mean per-episode savings of $113.73 (95% CI, $50.36-$177.10; P < .001). After the introduction of OnDemand, among employees with encounters for the top 10 clinical conditions managed by OnDemand, the trend for encounter rates per 100 employees per month increased marginally (0.03; 95% CI, 0.00-0.05; P = .03). CONCLUSIONS: These results suggest that DTC telemedicine staffed by an academic health system and offered directly to employees reduced the per-episode unit costs and only marginally increased utilization, suggesting lower cost overall.

Long-term Effect of Machine Learning-Triggered Behavioral Nudges on Serious Illness Conversations and End-of-Life Outcomes Among Patients With Cancer: A Randomized Clinical Trial.

Importance: Serious illness conversations (SICs) between oncology clinicians and patients are associated with improved quality of life and may reduce aggressive end-of-life care. However, most patients with cancer die without a documented SIC. Objective: To test the impact of behavioral nudges to clinicians to prompt SICs on the SIC rate and end-of-life outcomes among patients at high risk of death within 180 days (high-risk patients) as identified by a machine learning algorithm. Design, Setting, and Participants: This prespecified 40-week analysis of a stepped-wedge randomized clinical trial conducted between June 17, 2019, and April 20, 2020 (including 16 weeks of intervention rollout and 24 weeks of follow-up), included 20 506 patients with cancer representing 41 021 encounters at 9 tertiary or community-based medical oncology clinics in a large academic health system. The current analyses were conducted from June 1, 2021, to May 31, 2022. Intervention: High-risk patients were identified using a validated electronic health record machine learning algorithm to predict 6-month mortality. The intervention consisted of (1) weekly emails to clinicians comparing their SIC rates for all patients against peers' rates, (2) weekly lists of high-risk patients, and (3) opt-out text messages to prompt SICs before encounters with high-risk patients. Main Outcomes and Measures: The primary outcome was SIC rates for all and high-risk patient encounters; secondary end-of-life outcomes among decedents included inpatient death, hospice enrollment and length of stay, and intensive care unit admission and systemic therapy close to death. Intention-to-treat analyses were adjusted for clinic and wedge fixed effects and clustered at the oncologist level. Results: The study included 20 506 patients (mean [SD] age, 60.0 [14.0] years) and 41 021 patient encounters: 22 259 (54%) encounters with female patients, 28 907 (70.5%) with non-Hispanic White patients, and 5520 (13.5%) with high-risk patients; 1417 patients (6.9%) died by the end of follow-up. There were no meaningful differences in demographic characteristics in the control and intervention periods. Among high-risk patient encounters, the unadjusted SIC rates were 3.4% (59 of 1754 encounters) in the control period and 13.5% (510 of 3765 encounters) in the intervention period. In adjusted analyses, the intervention was associated with increased SICs for all patients (adjusted odds ratio, 2.09 [95% CI, 1.53-2.87]; P < .001) and decreased end-of-life systemic therapy (7.5% [72 of 957 patients] vs 10.4% [24 of 231 patients]; adjusted odds ratio, 0.25 [95% CI, 0.11-0.57]; P = .001) relative to controls, but there was no effect on hospice enrollment or length of stay, inpatient death, or end-of-life ICU use. Conclusions and Relevance: In this randomized clinical trial, a machine learning-based behavioral intervention and behavioral nudges to clinicans led to an increase in SICs and reduction in end-of-life systemic therapy but no changes in other end-of-life outcomes among outpatients with cancer. These results suggest that machine learning and behavioral nudges can lead to long-lasting improvements in cancer care delivery. Trial Registration: ClinicalTrials.gov Identifier: NCT03984773.

Training digital natives to transform healthcare: a 5-tiered approach for integrating clinical informatics into undergraduate medical education.

Expansive growth in the use of health information technology (HIT) has dramatically altered medicine without translating to fully realized improvements in healthcare delivery. Bridging this divide will require healthcare professionals with all levels of expertise in clinical informatics. However, due to scarce opportunities for exposure and training in informatics, medical students remain an underdeveloped source of potential informaticists. To address this gap, our institution developed and implemented a 5-tiered clinical informatics curriculum at the undergraduate medical education level: (1) a practical orientation to HIT for rising clerkship students; (2) an elective for junior students; (3) an elective for senior students; (4) a longitudinal area of concentration; and (5) a yearlong predoctoral fellowship in operational informatics at the health system level. Most students found these offerings valuable for their training and professional development. We share lessons and recommendations for medical schools and health systems looking to implement similar opportunities.

Operation analysis of the tele-critical care service demonstrates value delivery, service adaptation over time, and distress among tele-providers.

Background: Our study addresses the gaps in knowledge of the characterizations of operations by remote tele-critical care medicine (tele-CCM) service providers interacting with the bedside team. The duration of engagements, the evolution of the tele-CCM service over time, and the distress during interactions with the bedside team have not been characterized systematically. These characteristics are critical for planning the deployment of teleICU services and preventing burnout among remote teleICU providers. Methods: REDCap self-reported activity logs collected engagement duration, triggers (emergency button, tele-CCM software platform, autonomous algorithm, asymmetrical communication platform, phone), expediency, nature (proactive rounding, predetermined task, response to medical needs), communication modes, and acceptance. Seven hospitals with 16 ICUs were overseen between 9/2020 and 9/2021 by teams consisting of telemedicine medical doctors (eMD), telemedicine registered nurses (eRN), and telemedicine respiratory therapists (eRT). Results: 39,915 total engagements were registered. eMDs had a significantly higher percentage of emergent and urgent engagements (31.9%) vs. eRN (9.8%) or eRT (1.7%). The average tele-CCM intervention took 16.1 ± 10.39 min for eMD, 18.1 ± 16.23 for eRN, and 8.2 ± 4.98 min for eRT, significantly varied between engagement, and expediency, hospitals, and ICUs types. During the observation period, there was a shift in intervention triggers with an increase in autonomous algorithmic ARDS detection concomitant with predominant utilization of asynchronous communication, phone engagements, and the tele-CCM module of electronic medical records at the expense of the share of proactive rounding. eRT communicated more frequently with bedside staff (% MD = 37.8%; % RN = 36.8, % RT = 49.0%) but mostly with other eRTs. In contrast, the eMD communicated with all ICU stakeholders while the eRN communicated chiefly with other RN and house staff at the patient's bedside. The rate of distress reported by tele-CCM staff was 2% among all interactions, with the entity hospital being the dominant factor. Conclusions: Delivery of tele-CCM services has to be tailored to the specific beneficiary of tele-CCM services to optimize care delivery and minimize distress. In addition, the duration of the average intervention must be considered while creating an efficient workflow.

Pilot of rapid implementation of the advanced practice provider in the workflow of an existing tele-critical care program.

Incorporating the advanced practice provider (APP) in the delivery of tele critical care medicine (teleCCM) addresses the critical care provider shortage. However, the current literature lacks details of potential workflows, deployment difficulties and implementation outcomes while suggesting that expanding teleCCM service may be difficult. Here, we demonstrate the implementation of a telemedicine APP (eAPP) pilot service within an existing teleCCM program with the objective of determining the feasibility and ease of deployment. The goal is to augment an existing tele-ICU system with a balanced APP service to assess the feasibility and potential impact on the ICU performance in several hospitals affiliated within a large academic center. A REDCap survey was used to assess eAPP workflows, expediency of interventions, duration of tasks, and types of assignments within different service locations. Between 02/01/2021 and 08/31/2021, 204 interventions (across 133 12-h shift) were recorded by eAPP (nroutine = 109 (53.4%); nurgent = 82 (40.2%); nemergent = 13 (6.4%). The average task duration was 10.9 ± 6.22 min, but there was a significant difference based on the expediency of the task (F [2; 202] = 3.89; p < 0.022) and type of tasks (F [7; 220] = 6.69; p < 0.001). Furthermore, the eAPP task type and expediency varied depending upon the unit engaged and timeframe since implementation. The eAPP interventions were effectively communicated with bedside staff with only 0.5% of suggestions rejected. Only in 2% cases did the eAPP report distress. In summary, the eAPP can be rapidly deployed in existing teleCCM settings, providing adaptable and valuable care that addresses the specific needs of different ICUs while simultaneously enhancing the delivery of ICU care. Further studies are needed to quantify the input more robustly.

Guiding Efficient, Effective, and Patient-Oriented Electrolyte Replacement in Critical Care: An Artificial Intelligence Reinforcement Learning Approach.

Both provider- and protocol-driven electrolyte replacement have been linked to the over-prescription of ubiquitous electrolytes. Here, we describe the development and retrospective validation of a data-driven clinical decision support tool that uses reinforcement learning (RL) algorithms to recommend patient-tailored electrolyte replacement policies for ICU patients. We used electronic health records (EHR) data that originated from two institutions (UPHS; MIMIC-IV). The tool uses a set of patient characteristics, such as their physiological and pharmacological state, a pre-defined set of possible repletion actions, and a set of clinical goals to present clinicians with a recommendation for the route and dose of an electrolyte. RL-driven electrolyte repletion substantially reduces the frequency of magnesium and potassium replacements (up to 60%), adjusts the timing of interventions in all three electrolytes considered (potassium, magnesium, and phosphate), and shifts them towards orally administered repletion over intravenous replacement. This shift in recommended treatment limits risk of the potentially harmful effects of over-repletion and implies monetary savings. Overall, the RL-driven electrolyte repletion recommendations reduce excess electrolyte replacements and improve the safety, precision, efficacy, and cost of each electrolyte repletion event, while showing robust performance across patient cohorts and hospital systems.

The Rapid Implementation of Ad Hoc Tele-Critical Care Respiratory Therapy (eRT) Service in the Wake of the COVID-19 Surge.

A 24/7 telemedicine respiratory therapist (eRT) service was set up as part of the established University of Pennsylvania teleICU (PENN E-LERT®) service during the COVID-19 pandemic, serving five hospitals and 320 critical care beds to deliver effective remote care in lieu of a unit-based RT. The eRT interventions were components of an evidence-based care bundle and included ventilator liberation protocols, low tidal volume protocols, tube patency, and an extubation checklist. In addition, the proactive rounding of patients, including ventilator checks, was included. A standardized data collection sheet was used to facilitate the review of medical records, direct audio-visual inspection, or direct interactions with staff. In May 2020, a total of 1548 interventions took place, 93.86% of which were coded as "routine" based on established workflows, 4.71% as "urgent", 0.26% "emergent", and 1.17% were missing descriptors. Based on the number of coded interventions, we tracked the number of COVID-19 patients in the system. The average intervention took 6.1 ± 3.79 min. In 16% of all the interactions, no communication with the bedside team took place. The eRT connected with the in-house respiratory therapist (RT) in 66.6% of all the interventions, followed by house staff (9.8%), advanced practice providers (APP; 2.8%), and RN (2.6%). Most of the interaction took place over the telephone (88%), secure text message (16%), or audio-video telemedicine ICU platform (1.7%). A total of 5115 minutes were spent on tasks that a bedside clinician would have otherwise executed, reducing their exposure to COVID-19. The eRT service was instrumental in several emergent and urgent critical interventions. This study shows that an eRT service can support the bedside RT providers, effectively monitor best practice bundles, and carry out patient-ventilator assessments. It was effective in certain emergent situations and reduced the exposure of RTs to COVID-19. We plan to continue the service as part of an integrated RT service and hope to provide a framework for developing similar services in other facilities.

Why Is the Electronic Health Record So Challenging for Research and Clinical Care?

BACKGROUND: The electronic health record (EHR) has become increasingly ubiquitous. At the same time, health professionals have been turning to this resource for access to data that is needed for the delivery of health care and for clinical research. There is little doubt that the EHR has made both of these functions easier than earlier days when we relied on paper-based clinical records. Coupled with modern database and data warehouse systems, high-speed networks, and the ability to share clinical data with others are large number of challenges that arguably limit the optimal use of the EHR OBJECTIVES: Our goal was to provide an exhaustive reference for those who use the EHR in clinical and research contexts, but also for health information systems professionals as they design, implement, and maintain EHR systems. METHODS: This study includes a panel of 24 biomedical informatics researchers, information technology professionals, and clinicians, all of whom have extensive experience in design, implementation, and maintenance of EHR systems, or in using the EHR as clinicians or researchers. All members of the panel are affiliated with Penn Medicine at the University of Pennsylvania and have experience with a variety of different EHR platforms and systems and how they have evolved over time. RESULTS: Each of the authors has shared their knowledge and experience in using the EHR in a suite of 20 short essays, each representing a specific challenge and classified according to a functional hierarchy of interlocking facets such as usability and usefulness, data quality, standards, governance, data integration, clinical care, and clinical research. CONCLUSION: We provide here a set of perspectives on the challenges posed by the EHR to clinical and research users.

Insourcing and scaling a telemedicine solution in under 2 weeks: Lessons for the digital transformation of health care.

The Covid-19 pandemic required rapid scale of telemedicine as well as other digital workflows to maintain access to care while reducing infection risk. Both patients and clinicians who hadn't used telemedicine before were suddenly faced with a multi-step setup process to log into a virtual meeting. Unlike in-person examination rooms, locking a virtual meeting room was more error-prone and posed a risk of multiple patients joining the same online session. There was administrative burden on the practice staff who were generating and manually sending links to patients, and educating patients on device set up was time-consuming and unsustainable. A solution had to be deployed rapidly system-wide, without the usual roll out across months. Our answer was to design and implement a novel EHR-integrated web application called the Switchboard, in just two weeks. The Switchboard leverages a commercial, cloud-based video meeting platform and facilitates an end-to-end virtual care encounter workflow, from pre-visit reminders to post-visit SMS text message-based measurement of patient experience, with tools to extend contact-less workflows to in-person appointments. Over the first 11 months of the pandemic, the in-house platform has been adopted across 6 hospitals and >200 practices, scaled to 8,800 clinicians who at their peak conducted an average of 30,000 telemedicine appointments/week, and enabled over 10,000-20,000 text messages/day to be exchanged through the platform. Furthermore, it enabled our organization to convert from an average of 75% of telehealth visits being conducted via telephone to 75% conducted via video within weeks.

Remote Monitoring of Critically-Ill Post-Surgical Patients: Lessons from a Biosensor Implementation Trial.

Biosensors represent one of the numerous promising technologies envisioned to extend healthcare delivery. In perioperative care, the healthcare delivery system can use biosensors to remotely supervise patients who would otherwise be admitted to a hospital. This novel technology has gained a foothold in healthcare with significant acceleration due to the COVID-19 pandemic. However, few studies have attempted to narrate, or systematically analyze, the process of their implementation. We performed an observational study of biosensor implementation. The data accuracy provided by the commercially available biosensors was compared to those offered by standard clinical monitoring on patients admitted to the intensive care unit/perioperative unit. Surveys were also conducted to examine the acceptance of technology by patients and medical staff. We demonstrated a significant difference in vital signs between sensors and standard monitoring which was very dependent on the measured variables. Sensors seemed to integrate into the workflow relatively quickly, with almost no reported problems. The acceptance of the biosensors was high by patients and slightly less by nurses directly involved in the patients' care. The staff forecast a broad implementation of biosensors in approximately three to five years, yet are eager to learn more about them. Reliability considerations proved particularly troublesome in our implementation trial. Careful evaluation of sensor readiness is most likely necessary prior to system-wide implementation by each hospital to assess for data accuracy and acceptance by the staff.

Using design and innovation principles to reduce avoidable emergency department visits among employees of a large academic medical center.

1: Most large employers self-insure their employee health benefits, creating a motivation for employers to improve health care's value. 2: Employers who are also health care providers can aim for value through the direct provision of clinical services, not just through wellness programs or the design of insurance products. 3: Innovation and design methods can be systematically applied to health care problems to guide decisions about solutions which should or should not be scaled. 4: A virtual, on-demand urgent care service provided by a health care provider organization to its employees has the potential to reduce unnecessary emergency department visits and decrease the total cost of care.

Characteristics, Outcomes, and Trends of Patients With COVID-19-Related Critical Illness at a Learning Health System in the United States.

BACKGROUND: The coronavirus disease 2019 (COVID-19) pandemic continues to surge in the United States and globally. OBJECTIVE: To describe the epidemiology of COVID-19-related critical illness, including trends in outcomes and care delivery. DESIGN: Single-health system, multihospital retrospective cohort study. SETTING: 5 hospitals within the University of Pennsylvania Health System. PATIENTS: Adults with COVID-19-related critical illness who were admitted to an intensive care unit (ICU) with acute respiratory failure or shock during the initial surge of the pandemic. MEASUREMENTS: The primary exposure for outcomes and care delivery trend analyses was longitudinal time during the pandemic. The primary outcome was all-cause 28-day in-hospital mortality. Secondary outcomes were all-cause death at any time, receipt of mechanical ventilation (MV), and readmissions. RESULTS: Among 468 patients with COVID-19-related critical illness, 319 (68.2%) were treated with MV and 121 (25.9%) with vasopressors. Outcomes were notable for an all-cause 28-day in-hospital mortality rate of 29.9%, a median ICU stay of 8 days (interquartile range [IQR], 3 to 17 days), a median hospital stay of 13 days (IQR, 7 to 25 days), and an all-cause 30-day readmission rate (among nonhospice survivors) of 10.8%. Mortality decreased over time, from 43.5% (95% CI, 31.3% to 53.8%) to 19.2% (CI, 11.6% to 26.7%) between the first and last 15-day periods in the core adjusted model, whereas patient acuity and other factors did not change. LIMITATIONS: Single-health system study; use of, or highly dynamic trends in, other clinical interventions were not evaluated, nor were complications. CONCLUSION: Among patients with COVID-19-related critical illness admitted to ICUs of a learning health system in the United States, mortality seemed to decrease over time despite stable patient characteristics. Further studies are necessary to confirm this result and to investigate causal mechanisms. PRIMARY FUNDING SOURCE: Agency for Healthcare Research and Quality.

Developing the eMedical Student (eMS)-A Pilot Project Integrating Medical Students into the Tele-ICU during the COVID-19 Pandemic and beyond.

The COVID-19 pandemic has accelerated the demand for virtual healthcare delivery and highlighted the scarcity of telehealth medical student curricula, particularly tele-critical care. In partnership with the Penn E-lert program and the Department of Anesthesiology and Critical Care, the Perelman School of Medicine (PSOM) established a tele-ICU rotation to support the care of patients diagnosed with COVID-19 in the Intensive Care Unit (ICU). The four-week course had seven elements: (1) 60 h of clinical engagement; (2) multiple-choice pretest; (3) faculty-supervised, student-led case and topic presentations; (4) faculty-led debriefing sessions; (5) evidence-based-medicine discussion forum; (6) multiple-choice post-test; and (7) final reflection. Five third- and fourth-year medical students completed 300 h of supervised clinical engagement, following 16 patients over three weeks and documenting 70 clinical interventions. Knowledge of critical care and telehealth was demonstrated through improvement between pre-test and post-test scores. Professional development was demonstrated through post-course preceptor and learner feedback. This tele-ICU rotation allowed students to gain telemedicine exposure and participate in the care of COVID patients in a safe environment.

Patient Characteristics Associated With Telemedicine Access for Primary and Specialty Ambulatory Care During the COVID-19 Pandemic.

Importance: The coronavirus disease 2019 (COVID-19) pandemic has required a shift in health care delivery platforms, necessitating a new reliance on telemedicine. Objective: To evaluate whether inequities are present in telemedicine use and video visit use for telemedicine visits during the COVID-19 pandemic. Design, Setting, and Participants: In this cohort study, a retrospective medical record review was conducted from March 16 to May 11, 2020, of all patients scheduled for telemedicine visits in primary care and specialty ambulatory clinics at a large academic health system. Age, race/ethnicity, sex, language, median household income, and insurance type were all identified from the electronic medical record. Main Outcomes and Measures: A successfully completed telemedicine visit and video (vs telephone) visit for a telemedicine encounter. Multivariable models were used to assess the association between sociodemographic factors, including sex, race/ethnicity, socioeconomic status, and language, and the use of telemedicine visits, as well as video use specifically. Results: A total of 148 402 unique patients (86 055 women [58.0%]; mean [SD] age, 56.5 [17.7] years) had scheduled telemedicine visits during the study period; 80 780 patients (54.4%) completed visits. Of 78 539 patients with completed visits in which visit modality was specified, 35 824 (45.6%) were conducted via video, whereas 24 025 (56.9%) had a telephone visit. In multivariable models, older age (adjusted odds ratio [aOR], 0.85 [95% CI, 0.83-0.88] for those aged 55-64 years; aOR, 0.75 [95% CI, 0.72-0.78] for those aged 65-74 years; aOR, 0.67 [95% CI, 0.64-0.70] for those aged ≥75 years), Asian race (aOR, 0.69 [95% CI, 0.66-0.73]), non-English language as the patient's preferred language (aOR, 0.84 [95% CI, 0.78-0.90]), and Medicaid insurance (aOR, 0.93 [95% CI, 0.89-0.97]) were independently associated with fewer completed telemedicine visits. Older age (aOR, 0.79 [95% CI, 0.76-0.82] for those aged 55-64 years; aOR, 0.78 [95% CI, 0.74-0.83] for those aged 65-74 years; aOR, 0.49 [95% CI, 0.46-0.53] for those aged ≥75 years), female sex (aOR, 0.92 [95% CI, 0.90-0.95]), Black race (aOR, 0.65 [95% CI, 0.62-0.68]), Latinx ethnicity (aOR, 0.90 [95% CI, 0.83-0.97]), and lower household income (aOR, 0.57 [95% CI, 0.54-0.60] for income <$50 000; aOR, 0.89 [95% CI, 0.85-0.92], for $50 000-$100 000) were associated with less video use for telemedicine visits. These results were similar across medical specialties. Conclusions and Relevance: In this cohort study of patients scheduled for primary care and medical specialty ambulatory telemedicine visits at a large academic health system during the early phase of the COVID-19 pandemic, older patients, Asian patients, and non-English-speaking patients had lower rates of telemedicine use, while older patients, female patients, Black, Latinx, and poorer patients had less video use. Inequities in accessing telemedicine care are present, which warrant further attention.

Effect of Integrating Machine Learning Mortality Estimates With Behavioral Nudges to Clinicians on Serious Illness Conversations Among Patients With Cancer: A Stepped-Wedge Cluster Randomized Clinical Trial.

IMPORTANCE: Serious illness conversations (SICs) are structured conversations between clinicians and patients about prognosis, treatment goals, and end-of-life preferences. Interventions that increase the rate of SICs between oncology clinicians and patients may improve goal-concordant care and patient outcomes. OBJECTIVE: To determine the effect of a clinician-directed intervention integrating machine learning mortality predictions with behavioral nudges on motivating clinician-patient SICs. DESIGN, SETTING, AND PARTICIPANTS: This stepped-wedge cluster randomized clinical trial was conducted across 20 weeks (from June 17 to November 1, 2019) at 9 medical oncology clinics (8 subspecialty oncology and 1 general oncology clinics) within a large academic health system in Pennsylvania. Clinicians at the 2 smallest subspecialty clinics were grouped together, resulting in 8 clinic groups randomly assigned to the 4 intervention wedge periods. Included participants in the intention-to-treat analyses were 78 oncology clinicians who received SIC training and their patients (N = 14 607) who had an outpatient oncology encounter during the study period. INTERVENTIONS: (1) Weekly emails to oncology clinicians with SIC performance feedback and peer comparisons; (2) a list of up to 6 high-risk patients (≥10% predicted risk of 180-day mortality) scheduled for the next week, estimated using a validated machine learning algorithm; and (3) opt-out text message prompts to clinicians on the patient's appointment day to consider an SIC. Clinicians in the control group received usual care consisting of weekly emails with cumulative SIC performance. MAIN OUTCOMES AND MEASURES: Percentage of patient encounters with an SIC in the intervention group vs the usual care (control) group. RESULTS: The sample consisted of 78 clinicians and 14 607 patients. The mean (SD) age of patients was 61.9 (14.2) years, 53.7% were female, and 70.4% were White. For all encounters, SICs were conducted among 1.3% in the control group and 4.6% in the intervention group, a significant difference (adjusted difference in percentage points, 3.3; 95% CI, 2.3-4.5; P < .001). Among 4124 high-risk patient encounters, SICs were conducted among 3.6% in the control group and 15.2% in the intervention group, a significant difference (adjusted difference in percentage points, 11.6; 95% CI, 8.2-12.5; P < .001). CONCLUSIONS AND RELEVANCE: In this stepped-wedge cluster randomized clinical trial, an intervention that delivered machine learning mortality predictions with behavioral nudges to oncology clinicians significantly increased the rate of SICs among all patients and among patients with high mortality risk who were targeted by the intervention. Behavioral nudges combined with machine learning mortality predictions can positively influence clinician behavior and may be applied more broadly to improve care near the end of life. TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT03984773.

Validation of a Machine Learning Algorithm to Predict 180-Day Mortality for Outpatients With Cancer.

IMPORTANCE: Machine learning (ML) algorithms can identify patients with cancer at risk of short-term mortality to inform treatment and advance care planning. However, no ML mortality risk prediction algorithm has been prospectively validated in oncology or compared with routinely used prognostic indices. OBJECTIVE: To validate an electronic health record-embedded ML algorithm that generated real-time predictions of 180-day mortality risk in a general oncology cohort. DESIGN, SETTING, AND PARTICIPANTS: This prognostic study comprised a prospective cohort of patients with outpatient oncology encounters between March 1, 2019, and April 30, 2019. An ML algorithm, trained on retrospective data from a subset of practices, predicted 180-day mortality risk between 4 and 8 days before a patient's encounter. Patient encounters took place in 18 medical or gynecologic oncology practices, including 1 tertiary practice and 17 general oncology practices, within a large US academic health care system. Patients aged 18 years or older with outpatient oncology or hematology and oncology encounters were included in the analysis. Patients were excluded if their appointment was scheduled after weekly predictions were generated and if they were only evaluated in benign hematology, palliative care, or rehabilitation practices. EXPOSURES: Gradient-boosting ML binary classifier. MAIN OUTCOMES AND MEASURES: The primary outcome was the patients' 180-day mortality from the index encounter. The primary performance metric was the area under the receiver operating characteristic curve (AUC). RESULTS: Among 24 582 patients, 1022 (4.2%) died within 180 days of their index encounter. Their median (interquartile range) age was 64.6 (53.6-73.2) years, 15 319 (62.3%) were women, 18 015 (76.0%) were White, and 10 658 (43.4%) were seen in the tertiary practice. The AUC was 0.89 (95% CI, 0.88-0.90) for the full cohort. The AUC varied across disease-specific groups within the tertiary practice (AUC ranging from 0.74 to 0.96) but was similar between the tertiary and general oncology practices. At a prespecified 40% mortality risk threshold used to differentiate high- vs low-risk patients, observed 180-day mortality was 45.2% (95% CI, 41.3%-49.1%) in the high-risk group vs 3.1% (95% CI, 2.9%-3.3%) in the low-risk group. Integrating the algorithm into the Eastern Cooperative Oncology Group and Elixhauser comorbidity index-based classifiers resulted in favorable reclassification (net reclassification index, 0.09 [95% CI, 0.04-0.14] and 0.23 [95% CI, 0.20-0.27], respectively). CONCLUSIONS AND RELEVANCE: In this prognostic study, an ML algorithm was feasibly integrated into the electronic health record to generate real-time, accurate predictions of short-term mortality for patients with cancer and outperformed routinely used prognostic indices. This algorithm may be used to inform behavioral interventions and prompt earlier conversations about goals of care and end-of-life preferences among patients with cancer.