Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study.

BACKGROUND: Machine learning models have the potential to improve diagnostic accuracy and management of acute conditions. Despite growing efforts to evaluate and validate such models, little is known about how to best translate and implement these products as part of routine clinical care. OBJECTIVE: This study aims to explore the factors influencing the integration of a machine learning sepsis early warning system (Sepsis Watch) into clinical workflows. METHODS: We conducted semistructured interviews with 15 frontline emergency department physicians and rapid response team nurses who participated in the Sepsis Watch quality improvement initiative. Interviews were audio recorded and transcribed. We used a modified grounded theory approach to identify key themes and analyze qualitative data. RESULTS: A total of 3 dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations. Participants described their unfamiliarity with machine learning models. As a result, clinician trust was influenced by the perceived accuracy and utility of the model from personal program experience. Implementation of Sepsis Watch was facilitated by the easy-to-use tablet application and communication strategies that were developed by nurses to share model outputs with physicians. Barriers included the flow of information among clinicians and gaps in knowledge about the model itself and broader workflow processes. CONCLUSIONS: This study generated insights into how frontline clinicians perceived machine learning models and the barriers to integrating them into clinical workflows. These findings can inform future efforts to implement machine learning interventions in real-world settings and maximize the adoption of these interventions.

Developing a shared understanding of translational science within CTSA hubs through facilitated retreats: A case study.

Translation of critical and broadly impactful health advancements is stymied by insufficient scientific scrutiny of barriers and roadblocks in the process. The Clinical & Translational Science Award (CTSA) funding opportunity announcement released in July 2021 makes clear the distinction between translational research and translational science (TS) and urges a shift from the former to the latter. This represents a significant shift in the overall scientific direction of the CTSA program and necessitates corresponding shifts in CTSA hub operations. To better support TS, the Team Science Core of the Duke CTSA hub designed and facilitated a virtual retreat for hub personnel that (1) enabled organizational learning about TS and (2) identified anticipated challenges and opportunities. A post-retreat survey was utilized to assess the degree to which the retreat met its stated goals. Our survey received a 62% response rate; 100% of respondents would recommend the session to others. Respondents also reported gains in all areas assessed, with evidence for greater understanding of TS and increased perspective of the value and relevance of TS. In this paper, we provide a roadmap for designing and implementing facilitated TS retreats, which we argue is a key step in TS capacity building through workforce development.

Emergency Medical Service Agency Practices and Cardiac Arrest Survival.

Importance: Survival for out-of-hospital cardiac arrest (OHCA) varies widely across emergency medical service (EMS) agencies in the US. However, little is known about which EMS practices are associated with higher agency-level survival. Objective: To identify resuscitation practices associated with favorable neurological survival for OHCA at EMS agencies. Design, Setting, and Participants: This cohort study surveyed EMS agencies participating in the Cardiac Arrest Registry to Enhance Survival (CARES) with 10 or more OHCAs annually during January 2015 to December 2019; data analyses were performed from April to October 2023. Exposure: Survey of resuscitation practices at EMS agencies. Main Outcomes and Measures: Risk-standardized rates of favorable neurological survival for OHCA at each EMS agency were estimated using hierarchical logistic regression. Multivariable linear regression then examined the association of EMS practices with rates of risk-standardized favorable neurological survival. Results: Of 577 eligible EMS agencies, 470 agencies (81.5%) completed the survey. The mean (SD) rate of risk-standardized favorable neurological survival was 8.1% (1.8%). A total of 7 EMS practices across 3 domains (training, cardiopulmonary resuscitation [CPR], and transport) were associated with higher rates of risk-standardized favorable neurological survival. EMS agencies with higher favorable neurological survival rates were more likely to use simulation to assess CPR competency (ß = 0.54; P = .05), perform frequent reassessment (at least once every 6 months) of CPR competency in new staff (ß = 0.51; P = .04), use full multiperson scenario simulation for ongoing skills training (ß = 0.48; P = .01), perform simulation training at least every 6 months (ß = 0.63; P < .001), and conduct training in the use of mechanical CPR devices at least once annually (ß = 0.43; P = .04). EMS agencies with higher risk-standardized favorable neurological survival were also more likely to use CPR feedback devices (ß = 0.58; P = .007) and to transport patients to a designated cardiac arrest or ST-segment elevation myocardial infarction receiving center (ß = 0.57; P = .003). Adoption of more than half (≥4) of the 7 practices was more common at EMS agencies in the highest quartile of favorable neurological survival rates (70 of 118 agencies [59.3%]) vs the lowest quartile (42 of 118 agencies [35.6%]) (P < .001). Conclusions and Relevance: In a national registry for OHCA, 7 practices associated with higher rates of favorable neurological survival were identified at EMS agencies. Given wide variability in neurological survival across EMS agencies, these findings provide initial insights into EMS practices associated with top-performing EMS agencies in OHCA survival. Future studies are needed to validate these findings and identify best practices for EMS agencies.

Racial Differences in Shared Decision-Making About Critical Illness.

Importance: Shared decision-making is the preferred method for evaluating complex tradeoffs in the care of patients with critical illness. However, it remains unknown whether critical care clinicians engage diverse patients and caregivers equitably in shared decision-making. Objective: To compare critical care clinicians' approaches to shared decision-making in recorded conversations with Black and White caregivers of patients with critical illness. Design, Setting, and Participants: This thematic analysis consisted of unstructured clinician-caregiver meetings audio-recorded during a randomized clinical trial of a decision aid about prolonged mechanical ventilation at 13 intensive care units in the US. Participants in meetings included critical care clinicians and Black or White caregivers of patients who underwent mechanical ventilation. The codebook included components of shared decision-making and known mechanisms of racial disparities in clinical communication. Analysts were blinded to caregiver race during coding. Patterns within and across racial groups were evaluated to identify themes. Data analysis was conducted between August 2021 and April 2023. Main Outcomes and Measures: The main outcomes were themes describing clinician behaviors varying by self-reported race of the caregivers. Results: The overall sample comprised 20 Black and 19 White caregivers for a total of 39 audio-recorded meetings with clinicians. The duration of meetings was similar for both Black and White caregivers (mean [SD], 23.9 [13.7] minutes vs 22.1 [11.2] minutes, respectively). Both Black and White caregivers were generally middle-aged (mean [SD] age, 47.6 [9.9] years vs 51.9 [8.8] years, respectively), female (15 [75.0%] vs 14 [73.7%], respectively), and possessed a high level of self-assessed health literacy, which was scored from 3 to 15 with lower scores indicating increasing health literacy (mean [SD], 5.8 [2.3] vs 5.3 [2.0], respectively). Clinicians conducting meetings with Black and White caregivers were generally young (mean [SD] age, 38.8 [6.6] years vs 37.9 [8.2] years, respectively), male (13 [72.2%] vs 12 [70.6%], respectively), and White (14 [77.8%] vs 17 [100%], respectively). Four variations in clinicians' shared decision-making behaviors by caregiver race were identified: (1) providing limited emotional support for Black caregivers, (2) failing to acknowledge trust and gratitude expressed by Black caregivers, (3) sharing limited medical information with Black caregivers, and (4) challenging Black caregivers' preferences for restorative care. These themes encompass both relational and informational aspects of shared decision-making. Conclusions and Relevance: The results of this thematic analysis showed that critical care clinicians missed opportunities to acknowledge emotions and value the knowledge of Black caregivers compared with White caregivers. These findings may inform future clinician-level interventions aimed at promoting equitable shared decision-making.

Cardiac Arrest Survival at Emergency Medical Service Agencies in Catchment Areas With Primarily Black and Hispanic Populations.

Importance: Black and Hispanic patients are less likely to survive an out-of-hospital cardiac arrest (OHCA) than White patients. Given the central importance of emergency medical service (EMS) agencies in prehospital care, a better understanding of OHCA survival at EMS agencies that work in Black and Hispanic communities and White communities is needed to address OHCA disparities. Objective: To examine whether EMS agencies serving catchment areas with primarily Black and Hispanic populations (Black and Hispanic catchment areas) have different rates of OHCA survival than agencies serving catchment areas with primarily White populations (White catchment areas). Design, Setting, and Participants: A cohort study including adults with nontraumatic OHCA from January 1, 2015, to December 31, 2019, in the Cardiac Arrest Registry to Enhance Survival was conducted. Data analysis was conducted from August 17, 2022, to July 7, 2023. Exposure: Emergency medical service agencies, categorized as working in catchment areas where the combination of Black and Hispanic residents made up more than 50% of the population or where White residents made up more than 50% of the population. Main Outcomes and Measures: The unit of analysis was the EMS agency. The primary outcome was agency-level risk-standardized survival rates (RSSRs) to hospital admission for OHCA at each EMS agency, which were calculated using hierarchical logistic regression and compared between agencies serving Black and Hispanic and White catchment areas. Whether differences in OHCA survival were explained by EMS and first responder measures was evaluated with additional adjustment for these factors. Results: Among 764 EMS agencies representing 258 342 OHCAs, 82 EMS agencies (10.7%) had a Black and Hispanic catchment area. Overall median age of the patients was 63.0 (IQR, 52.0-75.0) years, 36.1% were women, and 63.9% were men. Overall, the mean (SD) RSSR was 27.5% (3.6%), with lower survival at EMS agencies with Black and Hispanic catchment areas (25.8% [3.6%]) compared with agencies with White catchment areas (27.7% [3.5%]; P < .001). Among the 82 EMS agencies with Black and Hispanic catchment areas, a disproportionately higher number (32 [39.0%]) was in the lowest survival quartile, whereas a lower number (12 [14.6%]) was in the highest survival quartile. Additional adjustment for EMS response times, EMS termination of resuscitation rates, and first responder rates of initiating cardiopulmonary resuscitation or applying an automated external defibrillator before EMS arrival did not meaningfully attenuate differences in RSSRs between agencies with Black and Hispanic compared with White catchment areas (mean [SD] RSSRs after adjustment, 25.9% [3.3%] vs 27.7% [3.1%]; P < .001). Conclusions and Relevance: Risk-standardized survival rates for OHCA were 1.9% lower at EMS agencies working in Black and Hispanic catchment areas than in White catchment areas. This difference was not explained by EMS response times, rates of EMS termination of resuscitation, or first responder rates of initiating cardiopulmonary resuscitation or applying an automated external defibrillator. These findings suggest there is a need for further assessment of these discrepancies.

Survey of resuscitation practices at emergency medical service agencies in the U.S.

Background: Survival for out-of-hospital cardiac arrest (OHCA) varies across emergency medical service (EMS) agencies. Yet, little is known about resuscitation response and quality improvement activities at EMS agencies. We describe herein a novel survey to EMS agencies in a U.S. registry for OHCA. Methods: Using data from the Cardiac Arrest Registry to Enhance Survival (CARES), we identified 577 EMS agencies with ≥10 OHCA cases annually between 2015 and 2019 that remained active in CARES. We administered a survey to EMS directors regarding agency characteristics, cardiac arrest response, relationships with first responders and dispatchers, quality improvement activities and perceived barriers in the community. Results: Of eligible EMS agencies, 470 (81.5%) completed the survey. The high completion rate was likely due to frequent personalized emails and phone calls, liaising with CARES state coordinators to encourage survey response, and multiple periodic drawings of an automated external defibrillator during the survey period for participating EMS agencies. The survey examined rates of resuscitation training modalities; use of resuscitation equipment and devices in the field; frequency of simulation; non-EMS stakeholder response to OHCA (dispatchers, fire, police); quality improvement; and community factors affecting bystander response to OHCA. Conclusions: In this study design paper on the RED-CASO survey, we provide summary data on EMS agency characteristics in the U.S. Upon linkage to CARES patient-level data, this survey will provide critical insights into 'best practices' at EMS agencies with the highest OHCA survival rates as well as provide insights into current disparities in outcomes.

Implementation of Electronic Patient-Reported Outcomes in Routine Cancer Care at an Academic Center: Identifying Opportunities and Challenges.

PURPOSE: Electronic patient-reported outcomes (ePROs) can help clinicians proactively assess and manage their patients' symptoms. Despite known benefits, there is limited adoption of ePROs into routine clinical care as a result of workflow and technologic challenges. This study identifies oncologists' perspectives on factors that affect integration of ePROs into clinical workflows. METHODS: We conducted semistructured qualitative interviews with 16 oncologists from a large academic medical center, across diverse subspecialties and cancer types. Oncologists were asked how they currently use or could imagine using ePROs before, during, and after a patient visit. We used an inductive approach to thematically analyze these qualitative data. RESULTS: Results were categorized into the following three main themes: (1) selection and development of ePRO tool, (2) contextual drivers of adoption, and (3) patient-facing concerns. Respondents preferred diagnosis-based ePRO tools over more general symptom screeners. Although they noted information overload as a potential barrier, respondents described strong data visualization and ease of use as facilitators. Contextual drivers of oncologist adoption include identifying target early adopters, incentivizing uptake through use of ePRO data to support billing and documentation, and emphasizing benefits for patient care and efficiency. Respondents also indicated the need to focus on patient-facing issues, such as patient response rate, timing of survey distribution, and validity and reliability of responses. DISCUSSION: Respondents identified several barriers and facilitators to successful uptake of ePROs. Understanding oncologists' perspectives is essential to inform both practice-level implementation strategies and policy-level decisions to include ePROs in alternative payment models for cancer care.

Mobile Complex Care Plans to Enhance Parental Engagement for Children With Medical Complexity.

Care plans can reduce care fragmentation for children with medical complexity (CMC); however, implementation is challenging. Mobile health innovations could improve implementation. This mixed methods study's objectives were to (1) evaluate feasibility of mobile complex care plans (MCCPs) for CMC enrolled in a complex care program and (2) study MCCPs' impact on parent engagement, parent experience, and care coordination. MCCPs were individualized, updated quarterly, integrated within the electronic health record, and visible on parents' mobile devices via an online portal. In 1 year (September 1, 2016, to August 31, 2017), 94% of eligible patients (n = 47) received 162 MCCPs. Seventy-four percent of parents (n = 35) reviewed MCCPs online. Forty-six percent of these parents (n = 16) sent a follow-up message, and the care team responded within 8 hours (median time = 7.2 hours). In interviews, parents identified MCCPs as an important reference and communication tool. MCCPs for CMC in a complex care program were feasible, facilitated parental engagement, and delivered timely communication.