Satisfaction of Older Patients With Emergency Department Care: Psychometric Properties and Construct Validity of the Consumer Emergency Care Satisfaction Scale.

BACKGROUND: Patient satisfaction is an important indicator of quality of care, but its measurement remains challenging. The Consumer Emergency Care Satisfaction Scale (CECSS) was developed to measure patient satisfaction in the emergency department (ED). Although this is a valid and reliable tool, several aspects of the CECSS need to be improved, including the definition, dimension, and scoring of scales. PURPOSE: The purpose of this study was to examine the construct validity of the CECSS and make suggestions on how to improve the tool to measure overall satisfaction with ED care. METHODS: We administered 2 surveys to older adults who presented with a fall to the ED and used electronic health record data to examine construct validity of the CECSS and ceiling effects. RESULTS: Using several criteria, we improved construct validity of the CECSS, reduced ceiling effects, and standardized scoring. CONCLUSION: We addressed several methodological issues with the CECSS and provided recommendations for improvement.

Comparing emergency department use among individuals with varying levels of cognitive impairment.

INTRODUCTION: As the population ages, Alzheimer's disease and related dementias (ADRD) are becoming increasingly common in patients presenting to the emergency department (ED). This study compares the frequency of ED use among a cohort of individuals with well-defined cognitive performance (cognitively intact, mild cognitive impairment (MCI), and ADRD). METHODS: We performed a retrospective cohort study of English-speaking, community-dwelling individuals evaluated at four health system-based multidisciplinary memory clinics from 2014-2016. We obtained demographic and clinical data, including neuropsychological testing results, through chart review and linkage to electronic health record data. We characterized the frequency and quantity of ED use within one year (6 months before and after) of cognitive evaluation and compared ED use between the three groups using bivariate and multivariate approaches. RESULTS: Of the 779 eligible patients, 89 were diagnosed as cognitively intact, 372 as MCI, and 318 as ADRD. The proportion of subjects with any annual ED use did not increase significantly with greater cognitive impairment: cognitively intact (16.9%), MCI (26.1%), and ADRD (28.9%) (p = 0.072). Average number of ED visits increased similarly: cognitively intact (0.27, SD 0.72), MCI (0.41, SD 0.91), and ADRD (0.55, SD 1.25) (p = 0.059). Multivariate logistic regression results showed that patients with MCI (odds ratio (OR) 1.62; CI = 0.87-3.00) and ADRD (OR 1.84; CI = 0.98-3.46) did not significantly differ from cognitively intact adults in any ED use. Multivariate negative binomial regression found patients with MCI (incidence rate ratio (IRR) 1.38; CI = 0.79-2.41) and ADRD (IRR 1.76, CI = 1.00-3.10) had elevated but non-significant risk of an ED visit compared to cognitively intact individuals. CONCLUSION: Though there was no significant difference in ED use in this small sample from one health system, our estimates are comparable to other published work. Results suggested a trend towards higher utilization among adults with MCI or ADRD compared to those who were cognitively intact. We must confirm our findings in other settings to better understand how to optimize systems of acute illness care for individuals with MCI and ADRD.

Utility of bedside assessment to evaluate for cervical-spine fracture post ground-level fall for patients 65 years and older.

OBJECTIVES: The effectiveness of current assessment tools for cervical fracture are mixed with respect to elderly patients. We aim to examine utility of history and physical exam to assess for cervical fracture for elderly patients suffering a ground-level fall. METHODS: Retrospective cohort from a tertiary-care ED for patients ≥65 years, including dementia, after ground-level fall. Logistic regression was used to examine predictability of various clinical factors. Neurologic deficits were considered a hard sign for imaging and were not assessed. RESULTS: Of 1035 patient encounters analyzed, 683 had CT cervical-spine (C-spine) imaging (66.0%) and 16 (1.5%) had cervical fracture. C-spine tenderness (OR 4.7, 95% CI 1.5-14.1), neck pain (OR 10.5, 95% CI 3.4-32.5), altered mental status (AMS) (OR 5.1, 95% CI 1.7-15.6), and external trauma above the clavicles (ETC) (OR 3.8, 95% CI 1.2-12.3) predicted cervical fracture. C-spine tenderness and neck pain were collinear and run-in separate models. Dementia (OR 0.2, 95% CI 0.4-0.9) did not predict cervical fracture in this population. A combination of ETC, C-spine tenderness, and AMS had a sensitivity = 100% and specificity = 40.0% for detection of cervical fracture. ETC was found in all but two fractures requiring intervention with negative predictive value = 99.3%. CONCLUSIONS: Clinical assessment for elderly patients without neurologic signs, together with the absence of ETC, cervical tenderness, and AMS may be reliable in ruling out cervical fracture after a ground-level fall, including patients with history of dementia. Fractures requiring intervention were rare in patients without ETC. However, findings are retrospective and prospective validation is required.

Effect of Timing of and Adherence to Social Distancing Measures on COVID-19 Burden in the United States : A Simulation Modeling Approach.

BACKGROUND: Across the United States, various social distancing measures were implemented to control the spread of coronavirus disease 2019 (COVID-19). However, the effectiveness of such measures for specific regions with varying population demographic characteristics and different levels of adherence to social distancing is uncertain. OBJECTIVE: To determine the effect of social distancing measures in unique regions. DESIGN: An agent-based simulation model. SETTING: Agent-based model applied to Dane County, Wisconsin; the Milwaukee metropolitan (metro) area; and New York City (NYC). PATIENTS: Synthetic population at different ages. INTERVENTION: Different times for implementing and easing social distancing measures at different levels of adherence. MEASUREMENTS: The model represented the social network and interactions among persons in a region, considering population demographic characteristics, limited testing availability, "imported" infections, asymptomatic disease transmission, and age-specific adherence to social distancing measures. The primary outcome was the total number of confirmed COVID-19 cases. RESULTS: The timing of and adherence to social distancing had a major effect on COVID-19 occurrence. In NYC, implementing social distancing measures 1 week earlier would have reduced the total number of confirmed cases from 203 261 to 41 366 as of 31 May 2020, whereas a 1-week delay could have increased the number of confirmed cases to 1 407 600. A delay in implementation had a differential effect on the number of cases in the Milwaukee metro area versus Dane County, indicating that the effect of social distancing measures varies even within the same state. LIMITATION: The effect of weather conditions on transmission dynamics was not considered. CONCLUSION: The timing of implementing and easing social distancing measures has major effects on the number of COVID-19 cases. PRIMARY FUNDING SOURCE: National Institute of Allergy and Infectious Diseases.

Usability of a Human Factors-based Clinical Decision Support in the Emergency Department: Lessons Learned for Design and Implementation.

OBJECTIVE: To evaluate the usability and use of human factors (HF)-based clinical decision support (CDS) implemented in the emergency department (ED). BACKGROUND: Clinical decision support can improve patient safety; however, the acceptance and use of CDS has faced challenges. Following a human-centered design process, we designed a CDS to support pulmonary embolism (PE) diagnosis in the ED. We demonstrated high usability of the CDS during scenario-based usability testing. We implemented the HF-based CDS in one ED in December 2018. METHOD: We conducted a survey of ED physicians to evaluate the usability and use of the HF-based CDS. We distributed the survey via Qualtrics, a web-based survey platform. We compared the computer system usability questionnaire scores of the CDS between those collected in the usability testing to use of the CDS in the real environment. We asked physicians about their acceptance and use of the CDS, barriers to using the CDS, and areas for improvement. RESULTS: Forty-seven physicians (56%) completed the survey. Physicians agreed that diagnosing PE is a major problem and risk scores can support the PE diagnostic process. Usability of the CDS was reported as high, both in the experimental setting and the real clinical setting. However, use of the CDS was low. We identified several barriers to the CDS use in the clinical environment, in particular a lack of workflow integration. CONCLUSION: Design of CDS should be a continuous process and focus on the technology's usability in the context of the broad work system and clinician workflow.

Older Adult Patients and Care Partners as Knowledge Brokers in Fragmented Health Care.

OBJECTIVE: To describe older adult patients' and care partners' knowledge broker roles during emergency department (ED) visits. BACKGROUND: Older adult patients are vulnerable to communication and coordination challenges during an ED visit, which can be exacerbated by the time and resource constrained ED environment. Yet, as a constant throughout the patient journey, patients and care partners can act as an information conduit, or knowledge broker, between fragmented care systems to attain high-quality, safe care. METHODS: Participants included 14 older adult patients (≥ 65 years old) and their care partners (e.g., spouse, adult child) who presented to the ED after having experienced a fall. Human factors researchers collected observation data from patients, care partners and clinician interactions during the patient's ED visit. We used an inductive content analysis to determine the role of patients and care partners as knowledge brokers. RESULTS: We found that patients and care partners act as knowledge brokers by providing information about diagnostic testing, medications, the patient's health history, and care accommodations at the disposition location. Patients and care partners filled the role of knowledge broker proactively (i.e. offer information) and reactively (i.e. are asked to provide information by clinicians or staff), within-ED work system and across work systems (e.g., between the ED and hospital), and in anticipation of future knowledge brokering. CONCLUSION: Patients and care partners, acting as knowledge brokers, often fill gaps in communication and participate in care coordination that assists in mitigating health care fragmentation.

Effect of a split-flow physician in triage model on abdominal CT ordering rate and yield.

OBJECTIVE: The objective of this study was to compare the rate and clinical yield of computed tomography (CT) imaging between patients presenting with abdominal pain initially seen by a physician in triage (PIT) versus those seen only by physicians working in the main emergency department (ED). METHODS: A retrospective study was conducted of all self-arrivals >18 years old presenting to a single ED with abdominal pain. Nine-hundred patients were randomly selected from both the PIT and traditional patient flow groups and rates and yields of CT imaging were compared, both alone and in a model controlling for potential confounders. Predetermined criteria for CT significance included need for admission, consult, or targeted medications. RESULTS: The overall rate of CT imaging (unadjusted) did not differ between the PIT and traditional groups, 48.7% (95% CI 45.4-51.9) vs. 45.1% (95% CI 41.8-48.4), respectively (p = .13). The CT yield for patients seen in in the PIT group was also similar to that of the traditional group: 49.1% (95% CI 44.4-53.8) vs. 50.5% (95% CI 45.6-55.4) (p = .68). In the logistic regression model, when controlling for age, gender, ESI-acuity, race and insurance payor, PIT vs. traditional was not a predictor of CT ordering (OR 1.14, 95% CI 0.94-1.38). CONCLUSIONS: For patients with abdominal pain, we found no significant differences in rates of CT ordering or CT yield for patients seen in a PIT vs. traditional models, suggesting the increased efficiencies offered by PIT models do not come at the cost of increased or decreased imaging utilization.

Comparing the Timeliness of Treatment in Younger vs. Older Patients with ST-Segment Elevation Myocardial Infarction: A Multi-Center Cohort Study.

BACKGROUND: ST-segment elevation myocardial infarction (STEMI) predominantly affects older adults. Lower incidence among younger patients may challenge diagnosis. OBJECTIVES: We hypothesize that among patients ≤ 50 years old, emergent percutaneous coronary intervention (PCI) for STEMI is delayed when compared with patients aged > 50 years. METHODS: This 3-year, 10-center retrospective cohort study included emergency department (ED) STEMI patients ≥ 18 years of age treated with emergent PCI. We excluded patients with an electrocardiogram (ECG) completed prior to ED arrival or a nondiagnostic initial ECG. Our primary outcome was door-to-balloon (D2B) time. We compared characteristics and outcomes among younger vs. older STEMI patients, and among age subgroups. RESULTS: There were 576 ED STEMI PCI patients, of whom 100 were ≤ 50 years old and 476 were > 50 years old. Median age was 44 years in the younger cohort (interquartile range [IQR] 41-47) vs. 62 years (IQR 57-70) among older patients. Median D2B time for the younger cohort was 76.5 min (IQR 67.5-102.5) vs. 81.0 min (IQR 65.0-105.5) in the older cohort (p = 0.91). This outcome did not change when ages 40 or 45 years were used to demarcate younger vs. older. The younger cohort had a higher prevalence of nonwhite races (38% vs. 21%; p < 0.001) and those currently smoking (36% vs. 23%; p = 0.005). The very young (≤30 years; 6/576) and very old (>80 years; 45/576) had 5.51 and 2.2 greater odds of delays. CONCLUSION: We found no statistically significant difference in D2B times between patients ≤ 50 years old and those > 50 years old. Nonwhite patients and those who smoke were disproportionately represented within the younger population. The very young and very old had higher odds of D2B times > 90 min.

Training and Interpreting Machine Learning Algorithms to Evaluate Fall Risk After Emergency Department Visits.

BACKGROUND: Machine learning is increasingly used for risk stratification in health care. Achieving accurate predictive models do not improve outcomes if they cannot be translated into efficacious intervention. Here we examine the potential utility of automated risk stratification and referral intervention to screen older adults for fall risk after emergency department (ED) visits. OBJECTIVE: This study evaluated several machine learning methodologies for the creation of a risk stratification algorithm using electronic health record data and estimated the effects of a resultant intervention based on algorithm performance in test data. METHODS: Data available at the time of ED discharge were retrospectively collected and separated into training and test datasets. Algorithms were developed to predict the outcome of a return visit for fall within 6 months of an ED index visit. Models included random forests, AdaBoost, and regression-based methods. We evaluated models both by the area under the receiver operating characteristic (ROC) curve, also referred to as area under the curve (AUC), and by projected clinical impact, estimating number needed to treat (NNT) and referrals per week for a fall risk intervention. RESULTS: The random forest model achieved an AUC of 0.78, with slightly lower performance in regression-based models. Algorithms with similar performance, when evaluated by AUC, differed when placed into a clinical context with the defined task of estimated NNT in a real-world scenario. CONCLUSION: The ability to translate the results of our analysis to the potential tradeoff between referral numbers and NNT offers decisionmakers the ability to envision the effects of a proposed intervention before implementation.

Scope and Influence of Electronic Health Record-Integrated Clinical Decision Support in the Emergency Department: A Systematic Review.

STUDY OBJECTIVE: As electronic health records evolve, integration of computerized clinical decision support offers the promise of sorting, collecting, and presenting this information to improve patient care. We conducted a systematic review to examine the scope and influence of electronic health record-integrated clinical decision support technologies implemented in the emergency department (ED). METHODS: A literature search was conducted in 4 databases from their inception through January 18, 2018: PubMed, Scopus, the Cumulative Index of Nursing and Allied Health, and Cochrane Central. Studies were included if they examined the effect of a decision support intervention that was implemented in a comprehensive electronic health record in the ED setting. Standardized data collection forms were developed and used to abstract study information and assess risk of bias. RESULTS: A total of 2,558 potential studies were identified after removal of duplicates. Of these, 42 met inclusion criteria. Common targets for clinical decision support intervention included medication and radiology ordering practices, as well as more comprehensive systems supporting diagnosis and treatment for specific disease entities. The majority of studies (83%) reported positive effects on outcomes studied. Most studies (76%) used a pre-post experimental design, with only 3 (7%) randomized controlled trials. CONCLUSION: Numerous studies suggest that clinical decision support interventions are effective in changing physician practice with respect to process outcomes such as guideline adherence; however, many studies are small and poorly controlled. Future studies should consider the inclusion of more specific information in regard to design choices, attempt to improve on uncontrolled before-after designs, and focus on clinically relevant outcomes wherever possible.

Development and validation of a pragmatic natural language processing approach to identifying falls in older adults in the emergency department.

BACKGROUND: Falls among older adults are both a common reason for presentation to the emergency department, and a major source of morbidity and mortality. It is critical to identify fall patients quickly and reliably during, and immediately after, emergency department encounters in order to deliver appropriate care and referrals. Unfortunately, falls are difficult to identify without manual chart review, a time intensive process infeasible for many applications including surveillance and quality reporting. Here we describe a pragmatic NLP approach to automating fall identification. METHODS: In this single center retrospective review, 500 emergency department provider notes from older adult patients (age 65 and older) were randomly selected for analysis. A simple, rules-based NLP algorithm for fall identification was developed and evaluated on a development set of 1084 notes, then compared with identification by consensus of trained abstractors blinded to NLP results. RESULTS: The NLP pipeline demonstrated a recall (sensitivity) of 95.8%, specificity of 97.4%, precision of 92.0%, and F1 score of 0.939 for identifying fall events within emergency physician visit notes, as compared to gold standard manual abstraction by human coders. CONCLUSIONS: Our pragmatic NLP algorithm was able to identify falls in ED notes with excellent precision and recall, comparable to that of more labor-intensive manual abstraction. This finding offers promise not just for improving research methods, but as a potential for identifying patients for targeted interventions, quality measure development and epidemiologic surveillance.

Role network measures to assess healthcare team adaptation to complex situations: the case of venous thromboembolism prophylaxis.

Hospitals are complex environments that rely on clinicians working together to provide appropriate care to patients. These clinical teams adapt their interactions to meet changing situational needs. Venous thromboembolism (VTE) prophylaxis is a complex process that occurs throughout a patient's hospitalisation, presenting five stages with different levels of complexity: admission, interruption, re-initiation, initiation, and transfer. The objective of our study is to understand how the VTE prophylaxis team adapts as the complexity in the process changes; we do this by using social network analysis (SNA) measures. We interviewed 45 clinicians representing 9 different cases, creating 43 role networks. The role networks were analysed using SNA measures to understand team changes between low and high complexity stages. When comparing low and high complexity stages, we found two team adaptation mechanisms: (1) relative increase in the number of people, team activities, and interactions within the team, or (2) relative increase in discussion among the team, reflected by an increase in reciprocity.   Practitioner Summary: The reason for this study was to quantify team adaptation to complexity in a process using social network analysis (SNA). The VTE prophylaxis team adapted to complexity by two different mechanisms, by increasing the roles, activities, and interactions among the team or by increasing the two-way communication and discussion throughout the team. We demonstrated the ability for SNA to identify adaptation within a team.

Effectiveness of Outpatient Antibiotics After Surgical Drainage of Abscesses in Reducing Treatment Failure.

BACKGROUND: The optimal approach to outpatient antibiotic use after surgical drainage of abscesses is unclear given conflicting clinical trial results. OBJECTIVE: Our primary objective was to evaluate the real-world effectiveness of outpatient antibiotic prescribing after surgical drainage of cutaneous abscesses on reducing treatment failure. METHODS: We performed a retrospective observational study using data extracted from the electronic health record of a single academic health care system. All emergency department (ED) visits that resulted in discharge with a surgical drainage of a cutaneous abscess procedure code were included in the sample. All visits were categorized into having received or not having received an antibiotic prescription at the index visit. Outcome frequencies were compared using Pearson's chi-squared test. A multivariable logistic regression model was used to estimate the odds of treatment failure among those who did and did not receive an antibiotic prescription at their index ED visit. RESULTS: The final sample consisted of 421 index ED visits, of which 303 (72%) received an antibiotic prescription. Treatment with antibiotics after drainage did not significantly reduce the odds of composite treatment failure within 30 days when controlling for sociodemographic and clinical encounter variables (odds ratio 0.52, 95% confidence interval 0.23-1.21). CONCLUSIONS: This real-world, comparative effectiveness analysis did not demonstrate any significant reduction in treatment failure with the use of antibiotics after drainage of abscesses in the ED. It is unclear if the clinical benefit observed under controlled trial conditions will carry over to routine clinical practice where varied antibiotic regimens are the norm and local bacterial resistance patterns vary.

Anemia is not a risk factor for developing pulmonary embolism.

OBJECTIVE: Our aim was to validate the previously published claim of a positive relationship between low blood hemoglobin level (anemia) and pulmonary embolism (PE). METHODS: This was a retrospective study of patients undergoing cross-sectional imaging to evaluate for PE at an academic medical center. Patients were identified using billing records for charges attributed to either magnetic resonance angiography or computed tomography angiography of the chest from 2008 to 2013. The main outcome measure was mean hemoglobin levels among those with and without PE. Our reference standard for PE status included index imaging results and a 6-month clinical follow-up for the presence of interval venous thromboembolism, conducted via review of the electronic medical record. Secondarily, we performed a subgroup analysis of only those patients who were seen in the emergency department. Finally, we again compared mean hemoglobin levels when limiting our control population to an age- and sex-matched cohort of the included cases. RESULTS: There were 1294 potentially eligible patients identified, of whom 121 were excluded. Of the remaining 1173 patients, 921 had hemoglobin levels analyzed within 24 hours of their index scan and thus were included in the main analysis. Of those 921 patients, 107 (11.6%; 107/921) were positive for PE. We found no significant difference in mean hemoglobin level between those with and without PE regardless of the control group used (12.4 ± 2.1 g/dL and 12.3 ± 2.0 g/dL [P = .85], respectively). CONCLUSIONS: Our data demonstrated no relationship between anemia and PE.

The Association Between Use of Brain CT for Atraumatic Headache and 30-Day Emergency Department Revisitation.

OBJECTIVE: The purpose of this article is to describe the association between initial CT for atraumatic headache and repeat emergency department (ED) visitation within 30 days of ED discharge. MATERIALS AND METHODS: A retrospective observational study was performed at an academic urban ED with more than 85,000 annual visits. All adult patients with a chief complaint of headache from January through December 2010 who were discharged after ED evaluation were included in the analysis. Patients were excluded if they were transferred, died in the ED, or had a diagnosis indicating a traumatic mechanism. A propensity score-matched logistic regression model was used to determine whether the use of brain CT was associated with the primary outcome of ED revisitation within 30 days, controlling for potential confounding variables. RESULTS: Of 80,619 total patient visits to the ED during the study period, 922 ED discharges with a chief complaint of headache were included. A total of 139 (15.1%) patients revisited within 30 days. The return rate was 11.2% among patients who underwent CT at their initial visit and 21.1% among those who did not. In the adjusted analysis, controlling for age, race, sex, insurance status, triage vital signs, laboratory values, and triage pain level, the odds ratio for revisitation given CT performance was 0.49 (95% CI, 0.27-0.86). CONCLUSION: After adjustment for clinical factors, we found that patients who underwent a brain CT examination for atraumatic headache at an initial ED visit were less likely to return to the ED within 30 days. Future appropriate use quality metrics regarding ED imaging use may need to incorporate downstream health care use.

Automating risk stratification for geriatric syndromes in the emergency department.

BACKGROUND: Geriatric emergency department (GED) guidelines endorse screening older patients for geriatric syndromes in the ED, but there have been significant barriers to widespread implementation. The majority of screening programs require engagement of a clinician, nurse, or social worker, adding to already significant workloads at a time of record-breaking ED patient volumes, staff shortages, and hospital boarding crises. Automated, electronic health record (EHR)-embedded risk stratification approaches may be an alternate solution for extending the reach of the GED mission by directing human actions to a smaller subset of higher risk patients. METHODS: We define the concept of automated risk stratification and screening using existing EHR data. We discuss progress made in three potential use cases in the ED: falls, cognitive impairment, and end-of-life and palliative care, emphasizing the importance of linking automated screening with systems of healthcare delivery. RESULTS: Research progress and operational deployment vary by use case, ranging from deployed solutions in falls screening to algorithmic validation in cognitive impairment and end-of-life care. CONCLUSIONS: Automated risk stratification offers a potential solution to one of the most pressing problems in geriatric emergency care: identifying high-risk populations of older adults most appropriate for specific GED care. Future work is needed to realize the promise of improved care with less provider burden by creating tools suitable for widespread deployment as well as best practices for their implementation and governance.

Call me Dr Ishmael: trends in electronic health record notes available at emergency department visits and admissions.

Objectives: Numerous studies have identified information overload as a key issue for electronic health records (EHRs). This study describes the amount of text data across all notes available to emergency physicians in the EHR, trended over the time since EHR establishment. Materials and Methods: We conducted a retrospective analysis of EHR data from a large healthcare system, examining the number of notes and a corresponding number of total words and total tokens across all notes available to physicians during patient encounters in the emergency department (ED). We assessed the change in these metrics over a 17-year period between 2006 and 2023. Results: The study cohort included 730 968 ED visits made by 293 559 unique patients and a total note count of 132 574 964. The median note count for all encounters in 2006 was 5 (IQR 1-16), accounting for 1735 (IQR 447-5521) words. By the last full year of the study period, 2022, the median number of notes had grown to 359 (IQR 84-943), representing 359 (IQR 84-943) words. Note and word counts were higher for admitted patients. Discussion: The volume of notes available for review by providers has increased by over 30-fold in the 17 years since the implementation of the EHR at a large health system. The task of reviewing these notes has become commensurately more difficult. These data point to the critical need for new strategies and tools for filtering, synthesizing, and summarizing information to achieve the promise of the medical record.

Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions.

BACKGROUND: Existing monitoring of machine-learning-based clinical decision support (ML-CDS) is focused predominantly on the ML outputs and accuracy thereof. Improving patient care requires not only accurate algorithms but also systems of care that enable the output of these algorithms to drive specific actions by care teams, necessitating expanding their monitoring. OBJECTIVES: In this case report, we describe the creation of a dashboard that allows the intervention development team and operational stakeholders to govern and identify potential issues that may require corrective action by bridging the monitoring gap between model outputs and patient outcomes. METHODS: We used an iterative development process to build a dashboard to monitor the performance of our intervention in the broader context of the care system. RESULTS: Our investigation of best practices elsewhere, iterative design, and expert consultation led us to anchor our dashboard on alluvial charts and control charts. Both the development process and the dashboard itself illuminated areas to improve the broader intervention. CONCLUSION: We propose that monitoring ML-CDS algorithms with regular dashboards that allow both a context-level view of the system and a drilled down view of specific components is a critical part of implementing these algorithms to ensure that these tools function appropriately within the broader care system.

Academic Detailing as a Health Information Technology Implementation Method: Supporting the Design and Implementation of an Emergency Department-Based Clinical Decision Support Tool to Prevent Future Falls.

BACKGROUND: Clinical decision support (CDS) tools that incorporate machine learning-derived content have the potential to transform clinical care by augmenting clinicians' expertise. To realize this potential, such tools must be designed to fit the dynamic work systems of the clinicians who use them. We propose the use of academic detailing-personal visits to clinicians by an expert in a specific health IT tool-as a method for both ensuring the correct understanding of that tool and its evidence base and identifying factors influencing the tool's implementation. OBJECTIVE: This study aimed to assess academic detailing as a method for simultaneously ensuring the correct understanding of an emergency department-based CDS tool to prevent future falls and identifying factors impacting clinicians' use of the tool through an analysis of the resultant qualitative data. METHODS: Previously, our team designed a CDS tool to identify patients aged 65 years and older who are at the highest risk of future falls and prompt an interruptive alert to clinicians, suggesting the patient be referred to a mobility and falls clinic for an evidence-based preventative intervention. We conducted 10-minute academic detailing interviews (n=16) with resident emergency medicine physicians and advanced practice providers who had encountered our CDS tool in practice. We conducted an inductive, team-based content analysis to identify factors that influenced clinicians' use of the CDS tool. RESULTS: The following categories of factors that impacted clinicians' use of the CDS were identified: (1) aspects of the CDS tool's design (2) clinicians' understanding (or misunderstanding) of the CDS or referral process, (3) the busy nature of the emergency department environment, (4) clinicians' perceptions of the patient and their associated fall risk, and (5) the opacity of the referral process. Additionally, clinician education was done to address any misconceptions about the CDS tool or referral process, for example, demonstrating how simple it is to place a referral via the CDS and clarifying which clinic the referral goes to. CONCLUSIONS: Our study demonstrates the use of academic detailing for supporting the implementation of health information technologies, allowing us to identify factors that impacted clinicians' use of the CDS while concurrently educating clinicians to ensure the correct understanding of the CDS tool and intervention. Thus, academic detailing can inform both real-time adjustments of a tool's implementation, for example, refinement of the language used to introduce the tool, and larger scale redesign of the CDS tool to better fit the dynamic work environment of clinicians.

Multicenter Development and Prospective Validation of eCARTv5: A Gradient Boosted Machine Learning Early Warning Score.

Rationale: Early detection of clinical deterioration using early warning scores may improve outcomes. However, most implemented scores were developed using logistic regression, only underwent retrospective internal validation, and were not tested in important patient subgroups. Objectives: To develop a gradient boosted machine model (eCARTv5) for identifying clinical deterioration and then validate externally, test prospectively, and evaluate across patient subgroups. Methods: All adult patients hospitalized on the wards in seven hospitals from 2008- 2022 were used to develop eCARTv5, with demographics, vital signs, clinician documentation, and laboratory values utilized to predict intensive care unit transfer or death in the next 24 hours. The model was externally validated retrospectively in 21 hospitals from 2009-2023 and prospectively in 10 hospitals from February to May 2023. eCARTv5 was compared to the Modified Early Warning Score (MEWS) and the National Early Warning Score (NEWS) using the area under the receiver operating characteristic curve (AUROC). Measurements and Main Results: The development cohort included 901,491 admissions, the retrospective validation cohort included 1,769,461 admissions, and the prospective validation cohort included 46,330 admissions. In retrospective validation, eCART had the highest AUROC (0.835; 95%CI 0.834, 0.835), followed by NEWS (0.766 (95%CI 0.766, 0.767)), and MEWS (0.704 (95%CI 0.703, 0.704)). eCART's performance remained high (AUROC ≥0.80) across a range of patient demographics, clinical conditions, and during prospective validation. Conclusions: We developed eCARTv5, which accurately identifies early clinical deterioration in hospitalized ward patients. Our model performed better than the NEWS and MEWS retrospectively, prospectively, and across a range of subgroups.