Governance of Clinical AI applications to facilitate safe and equitable deployment in a large health system: Key elements and early successes.

One of the key challenges in successful deployment and meaningful adoption of AI in healthcare is health system-level governance of AI applications. Such governance is critical not only for patient safety and accountability by a health system, but to foster clinician trust to improve adoption and facilitate meaningful health outcomes. In this case study, we describe the development of such a governance structure at University of Wisconsin Health (UWH) that provides oversight of AI applications from assessment of validity and user acceptability through safe deployment with continuous monitoring for effectiveness. Our structure leverages a multi-disciplinary steering committee along with project specific sub-committees. Members of the committee formulate a multi-stakeholder perspective spanning informatics, data science, clinical operations, ethics, and equity. Our structure includes guiding principles that provide tangible parameters for endorsement of both initial deployment and ongoing usage of AI applications. The committee is tasked with ensuring principles of interpretability, accuracy, and fairness across all applications. To operationalize these principles, we provide a value stream to apply the principles of AI governance at different stages of clinical implementation. This structure has enabled effective clinical adoption of AI applications. Effective governance has provided several outcomes: (1) a clear and institutional structure for oversight and endorsement; (2) a path towards successful deployment that encompasses technologic, clinical, and operational, considerations; (3) a process for ongoing monitoring to ensure the solution remains acceptable as clinical practice and disease prevalence evolve; (4) incorporation of guidelines for the ethical and equitable use of AI applications.

Association between advanced image ordered in the emergency department on subsequent imaging for abdominal pain patients.

BACKGROUND: Abdominal pain is associated with high rates of emergency department (ED) imaging utilization and revisits. While imaging often improves diagnosis, a better understanding is needed on when the decision to image is justified and how it influences subsequent resource utilization and outcomes for patients in the ED presenting with abdominal pain. We evaluated the association between advanced ED imaging on subsequent outpatient imaging and on revisits among abdominal pain patients discharged from the ED. METHODS: A retrospective, observational study was conducted using electronic health record data from an academic ED in the U.S. Midwest. A sample of Medicare patients with a chief complaint of abdominal pain from January 2013 to December 2016 following ED evaluation were included in the analysis. Logistic regression was used to estimate associations between receiving advanced imaging in the ED and subsequent outpatient imaging within 7-, 14-, and 28-day windows after discharge, and 30-day revisit rates to the study ED and to any ED. RESULTS: Of the 1385 ED visits with abdominal pain chief complaint and discharged home from the ED, individuals who were not imaged in the ED had significantly higher adjusted odds of being imaged outside the ED within 7 days (adjusted odds ratio [aOR] 6.65, 95% confidence interval [CI] 3.96-11.17, p < 0.001), 14 days (aOR 4.69, 95% CI 3.11-7.07, p < 0.001), and 28 days (aOR 3.1, 95% CI 2.25-4.27, p < 0.001) of being discharged and had a significantly higher adjusted odds of revisiting the study ED (aOR 1.65, 95% CI 1.29-2.12, p < 0.001) and revisiting any ED (aOR 1.47, 95% CI 1.16-1.86, p = 0.001) within 30 days of being discharged. CONCLUSIONS: Abdominal imaging in the ED was associated with significantly lower imaging utilization after discharge and 30-day revisit rates, suggesting that imaging in the ED may replace downstream outpatient imaging.

Utility of bedside assessment to evaluate for cervical-spine fracture post ground-level fall for patients 65 years and older.

OBJECTIVES: The effectiveness of current assessment tools for cervical fracture are mixed with respect to elderly patients. We aim to examine utility of history and physical exam to assess for cervical fracture for elderly patients suffering a ground-level fall. METHODS: Retrospective cohort from a tertiary-care ED for patients ≥65 years, including dementia, after ground-level fall. Logistic regression was used to examine predictability of various clinical factors. Neurologic deficits were considered a hard sign for imaging and were not assessed. RESULTS: Of 1035 patient encounters analyzed, 683 had CT cervical-spine (C-spine) imaging (66.0%) and 16 (1.5%) had cervical fracture. C-spine tenderness (OR 4.7, 95% CI 1.5-14.1), neck pain (OR 10.5, 95% CI 3.4-32.5), altered mental status (AMS) (OR 5.1, 95% CI 1.7-15.6), and external trauma above the clavicles (ETC) (OR 3.8, 95% CI 1.2-12.3) predicted cervical fracture. C-spine tenderness and neck pain were collinear and run-in separate models. Dementia (OR 0.2, 95% CI 0.4-0.9) did not predict cervical fracture in this population. A combination of ETC, C-spine tenderness, and AMS had a sensitivity = 100% and specificity = 40.0% for detection of cervical fracture. ETC was found in all but two fractures requiring intervention with negative predictive value = 99.3%. CONCLUSIONS: Clinical assessment for elderly patients without neurologic signs, together with the absence of ETC, cervical tenderness, and AMS may be reliable in ruling out cervical fracture after a ground-level fall, including patients with history of dementia. Fractures requiring intervention were rare in patients without ETC. However, findings are retrospective and prospective validation is required.

A Markov Chain Model for Transient Analysis of Handoff Process in Emergency Departments.

Transfer of care between multiple units or facilities is of significant importance for patient safety, care quality, and operation efficiency. Such transfers are often referred to as handoffs in hospitals, which need to be carried out timely, safely, and smoothly with accurate information. This paper introduces a Markov chain model to study the transients of handoff process in hospital emergency departments. The handoff process is modeled by a stochastic process with unavailability of service, which characterizes the constraints in bed capacity, staff shortage, and coordination issues, etc. For systems only allowing one transfer request waiting, the transient performance is obtained through Laplace transform and its inverse transform. Such a result is then used as a building block to study the systems allowing multiple requests waiting through an iteration process, which can reduce the computation complexity substantially. Numerical studies show that such a method can provide estimates of transient performance in the handoff process with acceptable accuracy.

Maggots, Mucous and Monkey Meat: Does Disgust Sensitivity Affect Case Mix Seen During Residency?

INTRODUCTION: Emergency physicians encounter scenarios daily that many would consider "disgusting," including exposure to blood, pus, and stool. Physicians in procedural specialties such as surgery and emergency medicine (EM) have lower disgust sensitivity overall, but the role this plays in clinical practice is unclear. The objective of this study was to determine whether emergency physicians with higher disgust sensitivity see fewer "disgusting" cases during training. METHODS: All EM residents at a midsize urban EM program were eligible to complete the Disgust Scale Revised (DS-R). We preidentified cases as "disgust elicitors" based on diagnoses likely to induce disgust due to physician exposure to bodily fluids, anogenital anatomy, or gross deformity. The "disgust elicitor" case percent was determined by "disgust elicitor" cases seen as the primary resident divided by the number of cases seen thus far in residency. We calculated Pearson's r, t-tests and descriptive statistics on resident and population DS-R scores and "disgust elicitor" cases per month. RESULTS: Mean DS-R for EM residents (n = 40) was 1.20 (standard deviation [SD] 1.24), significantly less than the population mean of 1.67 (SD 0.61, p<0.05). There was no correlation (r = -0.04) between "disgust elicitor" case (n = 2191) percent and DS-R scores. There was no significant difference between DS-R scores for junior residents (31.1, 95% confidence interval [CI], 26.8-35.4) and for senior residents (29.0, 95%CI, 23.4-34.6). CONCLUSION: Higher disgust sensitivity does not appear to be correlated with a lower percentage of "disgust elicitor" cases seen during EM residency.

Cherry Picking Patients: Examining the Interval Between Patient Rooming and Resident Self-assignment.

OBJECTIVE: We aimed to evaluate the association between patient chief complaint and the time interval between patient rooming and resident physician self-assignment ("pickup time"). We hypothesized that significant variation in pickup time would exist based on chief complaint, thereby uncovering resident preferences in patient presentations. METHODS: A retrospective medical record review was performed on consecutive patients at a single, academic, university-based emergency department with over 50,000 visits per year. All patients who presented from August 1, 2012, to July 31, 2013, and were initially seen by a resident were included in the analysis. Patients were excluded if not seen primarily by a resident or if registered with a chief complaint associated with trauma team activation. Data were abstracted from the electronic health record (EHR). The outcome measured was "pickup time," defined as the time interval between room assignment and resident self-assignment. We examined all complaints with >100 visits, with the remaining complaints included in the model in an "other" category. A proportional hazards model was created to control for the following prespecified demographic and clinical factors: age, race, sex, arrival mode, admission vital signs, Emergency Severity Index code, waiting room time before rooming, and waiting room census at time of rooming. RESULTS: Of the 30,382 patients eligible for the study, the median time to pickup was 6 minutes (interquartile range = 2-15 minutes). After controlling for the above factors, we found systematic and significant variation in the pickup time by chief complaint, with the longest times for patients with complaints of abdominal problems, numbness/tingling, and vaginal bleeding and shortest times for patients with ankle injury, allergic reaction, and wrist injury. CONCLUSIONS: A consistent variation in resident pickup time exists for common chief complaints. We suspect that this reflects residents preferentially choosing patients with simpler workups and less perceived diagnostic ambiguity. This work introduces pickup time as a metric that may be useful in the future to uncover and address potential physician bias. Further work is necessary to establish whether practice patterns in this study are carried beyond residency and persist among attendings in the community and how these patterns are shaped by the information presented via the EHR.

Cost-effectiveness of a novel indication of computed tomography of the coronary arteries.

OBJECTIVE: A common strategy for excluding coronary artery disease among patients presenting with low-risk chest pain is observation unit (OU) admission with serial cardiac biomarkers and stress testing for cardiac risk stratification. Patients with positive- or indeterminate-stress tests are often admitted for cardiac catheterization despite a low likelihood of disease. The aim of this study is to estimate the cost-effectiveness of computed tomography of the coronary arteries (CTCA) in the OU for the evaluation of low-risk chest pain patients with indeterminate- or positive-stress test results. METHODS: We conducted a decision analytic study to compare health outcomes and costs between 3 cardiac risk-stratification strategies in a population of patients at low cardiac risk admitted to the OU, who later had indeterminate- or abnormal-stress tests. Our population and test characteristics were based on data obtained both from the published literature and from a retrospective cohort review previously performed at our institution. The 3 strategies compared were (1) A CTCA strategy in which patients with positive- or indeterminate-stress tests subsequently underwent CTCA, and only received catheterization if results were positive, (2) A standard-of-care arm in which all patients with positive- or indeterminate-stress tests were admitted for catheterization, and (3) A do-nothing strategy in which all patients were discharged home after stress testing regardless of outcome. Outcomes measured were cost of care and life expectancy. Sensitivity analysis was performed with a multivariate Monte Carlo methodology. RESULTS: Both the CTCA and standard-of-care strategies dominated the do-nothing strategy in the base case. When comparing the standard-of-care with the CTCA strategy, the incremental cost-effectiveness ratio was $3,423,309 per additional year of life gained. Sensitivity analysis showed that below a willingness to pay of $600,000 per additional year of life, CTCA was the most likely strategy to be cost-effective. CONCLUSIONS: In this computer-modeled analysis, the addition of CTCA following positive- or indeterminate-stress tests to an OU cardiac risk-stratification pathway for low-risk chest pain patients achieved significant cost savings with a small decrease in life expectancy per patient. Adding CTCA after indeterminate- or positive-stress test results is a cost-effective intervention for further risk-stratifying low-risk chest pain patients in the OU setting before proceeding to traditional coronary angiography.

Cost-effectiveness of influenza vaccination of older adults in the ED setting.

OBJECTIVES: Adults older than 50 years are at greater risk for death and severe disability from influenza. Persons in this age group, however, are frequently not vaccinated, despite extensive efforts by physicians to provide this preventive measure in primary care settings. We performed this study to determine if influenza vaccination of older adults in the emergency department (ED) may be cost-effective. METHODS: Using a probabilistic decision model with quasi-Markov modeling of a typical influenza season, we calculated costs and health outcomes for a hypothetical cohort of patients using parameters from the literature. Three ED-based intervention strategies were compared: (1) no vaccination offered, (2) vaccination offered to patients older than 65 years (limited strategy), and (3) vaccination offered to all patients who are 50 years and older (inclusive strategy). Outcomes were measured as costs, lives saved, and incremental costs per life saved. We performed deterministic and probabilistic sensitivity analyses. RESULTS: Vaccination of patients 50 years of age and older results in an incremental cost of $34,610 per life saved when compared with the no-vaccination strategy. Limiting vaccination to only those older than 65 years results in an incremental cost of $13,084 per life saved. Results were sensitive to changes in vaccine cost but were insensitive to changes in other model parameters. CONCLUSIONS: Vaccination of older adults against influenza in the ED setting is cost-effective, especially for those older than 65 years. Emergency departments may be an important setting for providing influenza vaccination to adults who may otherwise have remained unvaccinated.