Effectiveness of an Artificial Intelligence-Enabled Intervention for Detecting Clinical Deterioration.

Importance: Inpatient clinical deterioration is associated with substantial morbidity and mortality but may be easily missed by clinicians. Early warning scores have been developed to alert clinicians to patients at high risk of clinical deterioration, but there is limited evidence for their effectiveness. Objective: To evaluate the effectiveness of an artificial intelligence deterioration model-enabled intervention to reduce the risk of escalations in care among hospitalized patients using a study design that facilitates stronger causal inference. Design, Setting, and Participants: This cohort study used a regression discontinuity design that controlled for confounding and was based on Epic Deterioration Index (EDI; Epic Systems Corporation) prediction model scores. Compared with other observational research, the regression discontinuity design facilitates causal analysis. Hospitalized adults were included from 4 general internal medicine units in 1 academic hospital from January 17, 2021, through November 16, 2022. Exposure: An artificial intelligence deterioration model-enabled intervention, consisting of alerts based on an EDI score threshold with an associated collaborative workflow among nurses and physicians. Main Outcomes and Measures: The primary outcome was escalations in care, including rapid response team activation, transfer to the intensive care unit, or cardiopulmonary arrest during hospitalization. Results: During the study, 9938 patients were admitted to 1 of the 4 units, with 963 patients (median [IQR] age, 76.1 [64.2-86.2] years; 498 males [52.3%]) included within the primary regression discontinuity analysis. The median (IQR) Elixhauser Comorbidity Index score in the primary analysis cohort was 10 (0-24). The intervention was associated with a -10.4-percentage point (95% CI, -20.1 to -0.8 percentage points; P = .03) absolute risk reduction in the primary outcome for patients at the EDI score threshold. There was no evidence of a discontinuity in measured confounders at the EDI score threshold. Conclusions and Relevance: Using a regression discontinuity design, this cohort study found that the implementation of an artificial intelligence deterioration model-enabled intervention was associated with a significantly decreased risk of escalations in care among inpatients. These results provide evidence for the effectiveness of this intervention and support its further expansion and testing in other care settings.

Human Immunodeficiency Virus Status, Tenofovir Exposure, and the Risk of Poor Coronavirus Disease 19 Outcomes: Real-World Analysis From 6 United States Cohorts Before Vaccine Rollout.

BACKGROUND: People with human immunodeficiency virus (HIV) (PWH) may be at increased risk for severe coronavirus disease 2019 (COVID-19) outcomes. We examined HIV status and COVID-19 severity, and whether tenofovir, used by PWH for HIV treatment and people without HIV (PWoH) for HIV prevention, was associated with protection. METHODS: Within 6 cohorts of PWH and PWoH in the United States, we compared the 90-day risk of any hospitalization, COVID-19 hospitalization, and mechanical ventilation or death by HIV status and by prior exposure to tenofovir, among those with severe acute respiratory syndrome coronavirus 2 infection between 1 March and 30 November 2020. Adjusted risk ratios (aRRs) were estimated by targeted maximum likelihood estimation, with adjustment for demographics, cohort, smoking, body mass index, Charlson comorbidity index, calendar period of first infection, and CD4 cell counts and HIV RNA levels (in PWH only). RESULTS: Among PWH (n = 1785), 15% were hospitalized for COVID-19 and 5% received mechanical ventilation or died, compared with 6% and 2%, respectively, for PWoH (n = 189 351). Outcome prevalence was lower for PWH and PWoH with prior tenofovir use. In adjusted analyses, PWH were at increased risk compared with PWoH for any hospitalization (aRR, 1.31 [95% confidence interval, 1.20-1.44]), COVID-19 hospitalizations (1.29 [1.15-1.45]), and mechanical ventilation or death (1.51 [1.19-1.92]). Prior tenofovir use was associated with reduced hospitalizations among PWH (aRR, 0.85 [95% confidence interval, .73-.99]) and PWoH (0.71 [.62-.81]). CONCLUSIONS: Before COVID-19 vaccine availability, PWH were at greater risk for severe outcomes than PWoH. Tenofovir was associated with a significant reduction in clinical events for both PWH and PWoH.

Evaluation of an intervention targeted with predictive analytics to prevent readmissions in an integrated health system: observational study.

OBJECTIVES: To determine the associations between a care coordination intervention (the Transitions Program) targeted to patients after hospital discharge and 30 day readmission and mortality in a large, integrated healthcare system. DESIGN: Observational study. SETTING: 21 hospitals operated by Kaiser Permanente Northern California. PARTICIPANTS: 1 539 285 eligible index hospital admissions corresponding to 739 040 unique patients from June 2010 to December 2018. 411 507 patients were discharged post-implementation of the Transitions Program; 80 424 (19.5%) of these patients were at medium or high predicted risk and were assigned to receive the intervention after discharge. INTERVENTION: Patients admitted to hospital were automatically assigned to be followed by the Transitions Program in the 30 days post-discharge if their predicted risk of 30 day readmission or mortality was greater than 25% on the basis of electronic health record data. MAIN OUTCOME MEASURES: Non-elective hospital readmissions and all cause mortality in the 30 days after hospital discharge. RESULTS: Difference-in-differences estimates indicated that the intervention was associated with significantly reduced odds of 30 day non-elective readmission (adjusted odds ratio 0.91, 95% confidence interval 0.89 to 0.93; absolute risk reduction 95% confidence interval -2.5%, -3.1% to -2.0%) but not with the odds of 30 day post-discharge mortality (1.00, 0.95 to 1.04). Based on the regression discontinuity estimate, the association with readmission was of similar magnitude (absolute risk reduction -2.7%, -3.2% to -2.2%) among patients at medium risk near the risk threshold used for enrollment. However, the regression discontinuity estimate of the association with post-discharge mortality (-0.7% -1.4% to -0.0%) was significant and suggested benefit in this subgroup of patients. CONCLUSIONS: In an integrated health system, the implementation of a comprehensive readmissions prevention intervention was associated with a reduction in 30 day readmission rates. Moreover, there was no association with 30 day post-discharge mortality, except among medium risk patients, where some evidence for benefit was found. Altogether, the study provides evidence to suggest the effectiveness of readmission prevention interventions in community settings, but further research might be required to confirm the findings beyond this setting.

Developing machine learning models to personalize care levels among emergency room patients for hospital admission.

OBJECTIVE: To develop prediction models for intensive care unit (ICU) vs non-ICU level-of-care need within 24 hours of inpatient admission for emergency department (ED) patients using electronic health record data. MATERIALS AND METHODS: Using records of 41 654 ED visits to a tertiary academic center from 2015 to 2019, we tested 4 algorithms-feed-forward neural networks, regularized regression, random forests, and gradient-boosted trees-to predict ICU vs non-ICU level-of-care within 24 hours and at the 24th hour following admission. Simple-feature models included patient demographics, Emergency Severity Index (ESI), and vital sign summary. Complex-feature models added all vital signs, lab results, and counts of diagnosis, imaging, procedures, medications, and lab orders. RESULTS: The best-performing model, a gradient-boosted tree using a full feature set, achieved an AUROC of 0.88 (95%CI: 0.87-0.89) and AUPRC of 0.65 (95%CI: 0.63-0.68) for predicting ICU care need within 24 hours of admission. The logistic regression model using ESI achieved an AUROC of 0.67 (95%CI: 0.65-0.70) and AUPRC of 0.37 (95%CI: 0.35-0.40). Using a discrimination threshold, such as 0.6, the positive predictive value, negative predictive value, sensitivity, and specificity were 85%, 89%, 30%, and 99%, respectively. Vital signs were the most important predictors. DISCUSSION AND CONCLUSIONS: Undertriaging admitted ED patients who subsequently require ICU care is common and associated with poorer outcomes. Machine learning models using readily available electronic health record data predict subsequent need for ICU admission with good discrimination, substantially better than the benchmarking ESI system. The results could be used in a multitiered clinical decision-support system to improve ED triage.

Rib Fracture Frailty Index: A risk stratification tool for geriatric patients with multiple rib fractures.

BACKGROUND: Rib fractures are consequential injuries for geriatric patients (age, ≥65 years). Although age and injury patterns drive many rib fracture management decisions, the impact of frailty-which baseline conditions affect rib fracture-specific outcomes-remains unclear for geriatric patients. We aimed to develop and validate the Rib Fracture Frailty (RFF) Index, a practical risk stratification tool specific for geriatric patients with rib fractures. We hypothesized that a compact list of frailty markers can accurately risk stratify clinical outcomes after rib fractures. METHODS: We queried nationwide US admission encounters of geriatric patients admitted with multiple rib fractures from 2016 to 2017. Partitioning around medoids clustering identified a development subcohort with previously validated frailty characteristics. Ridge regression with penalty for multicollinearity aggregated baseline conditions most prevalent in this frail subcohort into RFF scores. Regression models with adjustment for injury severity, sex, and age assessed associations between frailty risk categories (low, medium, and high) and inpatient outcomes among validation cohorts (odds ratio [95% confidence interval]). We report results according to Transparent Reporting of Multivariable Prediction Model for Individual Prognosis guidelines. RESULTS: Development cohort (n = 55,540) cluster analysis delineated 13 baseline conditions constituting the RFF Index. Among external validation cohort (n = 77,710), increasing frailty risk (low [reference group], moderate, high) was associated with stepwise worsening adjusted odds of mortality (1.5 [1.2-1.7], 3.5 [3.0-4.0]), intubation (2.4 [1.5-3.9], 4.7 [3.1-7.5]), hospitalization ≥5 days (1.4 [1.3-1.5], 1.8 [1.7-2.0]), and disposition to home (0.6 [0.5-0.6], 0.4 [0.3-0.4]). Locally weighted scatterplot smoothing showed correlations between increasing RFF scores and worse outcomes. CONCLUSION: The RFF Index is a practical frailty risk stratification tool for geriatric patients with multiple rib fractures. The mobile app we developed may facilitate rapid implementation and further validation of RFF Index at the bedside. LEVEL OF EVIDENCE: Prognostic study, level III.

Predicting preventable hospital readmissions with causal machine learning.

OBJECTIVE: To assess both the feasibility and potential impact of predicting preventable hospital readmissions using causal machine learning applied to data from the implementation of a readmissions prevention intervention (the Transitions Program). DATA SOURCES: Electronic health records maintained by Kaiser Permanente Northern California (KPNC). STUDY DESIGN: Retrospective causal forest analysis of postdischarge outcomes among KPNC inpatients. Using data from both before and after implementation, we apply causal forests to estimate individual-level treatment effects of the Transitions Program intervention on 30-day readmission. These estimates are used to characterize treatment effect heterogeneity and to assess the notional impacts of alternative targeting strategies in terms of the number of readmissions prevented. DATA COLLECTION: 1 539 285 index hospitalizations meeting the inclusion criteria and occurring between June 2010 and December 2018 at 21 KPNC hospitals. PRINCIPAL FINDINGS: There appears to be substantial heterogeneity in patients' responses to the intervention (omnibus test for heterogeneity p = 2.23 × 10-7 ), particularly across levels of predicted risk. Notably, predicted treatment effects become more positive as predicted risk increases; patients at somewhat lower risk appear to have the largest predicted effects. Moreover, these estimates appear to be well calibrated, yielding the same estimate of annual readmissions prevented in the actual treatment subgroup (1246, 95% confidence interval [CI] 1110-1381) as did a formal evaluation of the Transitions Program (1210, 95% CI 990-1430). Estimates of the impacts of alternative targeting strategies suggest that as many as 4458 (95% CI 3925-4990) readmissions could be prevented annually, while decreasing the number needed to treat from 33 to 23, by targeting patients with the largest predicted effects rather than those at highest risk. CONCLUSIONS: Causal machine learning can be used to identify preventable hospital readmissions, if the requisite interventional data are available. Moreover, our results suggest a mismatch between risk and treatment effects.

Accurate and interpretable intensive care risk adjustment for fused clinical data with generalized additive models.

Risk adjustment models for intensive care outcomes have yet to realize the full potential of data unlocked by the increasing adoption of EHRs. In particular, they fail to fully leverage the information present in longitudinal, structured clinical data - including laboratory test results and vital signs - nor can they infer patient state from unstructured clinical narratives without lengthy manual abstraction. A fully electronic ICU risk model fusing these two types of data sources may yield improved accuracy and more personalized risk estimates, and in obviating manual abstraction, could also be used for real-time decision-making. As a first step towards fully "electronic" ICU models based on fused data, we present results of generalized additive modeling applied to a sample of over 36,000 ICU patients. Our approach outperforms those based on the SAPS and OASIS systems (A UC: 0.908 vs. 0.794 and 0.874), and appears to yield more granular and easily visualized risk estimates.

Validation of Prediction Models for Critical Care Outcomes Using Natural Language Processing of Electronic Health Record Data.

Importance: Accurate prediction of outcomes among patients in intensive care units (ICUs) is important for clinical research and monitoring care quality. Most existing prediction models do not take full advantage of the electronic health record, using only the single worst value of laboratory tests and vital signs and largely ignoring information present in free-text notes. Whether capturing more of the available data and applying machine learning and natural language processing (NLP) can improve and automate the prediction of outcomes among patients in the ICU remains unknown. Objectives: To evaluate the change in power for a mortality prediction model among patients in the ICU achieved by incorporating measures of clinical trajectory together with NLP of clinical text and to assess the generalizability of this approach. Design, Setting, and Participants: This retrospective cohort study included 101 196 patients with a first-time admission to the ICU and a length of stay of at least 4 hours. Twenty ICUs at 2 academic medical centers (University of California, San Francisco [UCSF], and Beth Israel Deaconess Medical Center [BIDMC], Boston, Massachusetts) and 1 community hospital (Mills-Peninsula Medical Center [MPMC], Burlingame, California) contributed data from January 1, 2001, through June 1, 2017. Data were analyzed from July 1, 2017, through August 1, 2018. Main Outcomes and Measures: In-hospital mortality and model discrimination as assessed by the area under the receiver operating characteristic curve (AUC) and model calibration as assessed by the modified Hosmer-Lemeshow statistic. Results: Among 101 196 patients included in the analysis, 51.3% (n = 51 899) were male, with a mean (SD) age of 61.3 (17.1) years; their in-hospital mortality rate was 10.4% (n = 10 505). A baseline model using only the highest and lowest observed values for each laboratory test result or vital sign achieved a cross-validated AUC of 0.831 (95% CI, 0.830-0.832). In contrast, that model augmented with measures of clinical trajectory achieved an AUC of 0.899 (95% CI, 0.896-0.902; P < .001 for AUC difference). Further augmenting this model with NLP-derived terms associated with mortality further increased the AUC to 0.922 (95% CI, 0.916-0.924; P < .001). These NLP-derived terms were associated with improved model performance even when applied across sites (AUC difference for UCSF: 0.077 to 0.021; AUC difference for MPMC: 0.071 to 0.051; AUC difference for BIDMC: 0.035 to 0.043; P < .001) when augmenting with NLP at each site. Conclusions and Relevance: Intensive care unit mortality prediction models incorporating measures of clinical trajectory and NLP-derived terms yielded excellent predictive performance and generalized well in this sample of hospitals. The role of these automated algorithms, particularly those using unstructured data from notes and other sources, in clinical research and quality improvement seems to merit additional investigation.

Variation in Quality of Urgent Health Care Provided During Commercial Virtual Visits.

IMPORTANCE: Commercial virtual visits are an increasingly popular model of health care for the management of common acute illnesses. In commercial virtual visits, patients access a website to be connected synchronously-via videoconference, telephone, or webchat-to a physician with whom they have no prior relationship. To date, whether the care delivered through those websites is similar or quality varies among the sites has not been assessed. OBJECTIVE: To assess the variation in the quality of urgent health care among virtual visit companies. DESIGN, SETTING, AND PARTICIPANTS: This audit study used 67 trained standardized patients who presented to commercial virtual visit companies with the following 6 common acute illnesses: ankle pain, streptococcal pharyngitis, viral pharyngitis, acute rhinosinusitis, low back pain, and recurrent female urinary tract infection. The 8 commercial virtual visit websites with the highest web traffic were selected for audit, for a total of 599 visits. Data were collected from May 1, 2013, to July 30, 2014, and analyzed from July 1, 2014, to September 1, 2015. MAIN OUTCOMES AND MEASURES: Completeness of histories and physical examinations, the correct diagnosis (vs an incorrect or no diagnosis), and adherence to guidelines of key management decisions. RESULTS: Sixty-seven standardized patients completed 599 commercial virtual visits during the study period. Histories and physical examinations were complete in 417 visits (69.6%; 95% CI, 67.7%-71.6%); diagnoses were correctly named in 458 visits (76.5%; 95% CI, 72.9%-79.9%), and key management decisions were adherent to guidelines in 325 visits (54.3%; 95% CI, 50.2%-58.3%). Rates of guideline-adherent care ranged from 206 visits (34.4%) to 396 visits (66.1%) across the 8 websites. Variation across websites was significantly greater for viral pharyngitis and acute rhinosinusitis (adjusted rates, 12.8% to 82.1%) than for streptococcal pharyngitis and low back pain (adjusted rates, 74.6% to 96.5%) or ankle pain and recurrent urinary tract infection (adjusted rates, 3.4% to 40.4%). No statistically significant variation in guideline adherence by mode of communication (videoconference vs telephone vs webchat) was found. CONCLUSIONS AND RELEVANCE: Significant variation in quality was found among companies providing virtual visits for management of common acute illnesses. More variation was found in performance for some conditions than for others, but no variation by mode of communication.

Receipt of Intravenous Iron and Clinical Outcomes among Hemodialysis Patients Hospitalized for Infection.

BACKGROUND AND OBJECTIVES: Anemia guidelines for CKD recommend withholding intravenous iron in the setting of active infection, although no data specifically support this recommendation. This study aimed to examine the association between intravenous iron and clinical outcomes among hemodialysis patients hospitalized for infection. DESIGN, SETTING, PARTICIPANTS, & MEASUREMENTS: This was a retrospective observational cohort study using data from the US Renal Data System of 22,820 adult Medicare beneficiaries on in-center hemodialysis who had received intravenous iron in the 14 days preceding their first hospitalization for bacterial infection in 2010. In multivariable analyses, the association between receipt of intravenous iron at any point from the day of hospital admission to discharge and all-cause 30-day mortality, mortality in 2010, length of hospital stay, and readmission for infection or death within 30 days of discharge was evaluated. RESULTS: There were 2463 patients (10.8%) who received intravenous iron at any point from the day of admission to discharge. Receipt of intravenous iron was not associated with age, dialysis vintage, or comorbidities. There were 2618 deaths within 30 days of admission and 6921 deaths in 2010 (median follow-up 173 days; 25th and 75th percentiles, 78-271 days). The median length of stay was 7 days (25th and 75th percentiles, 5-12 days). Receipt of intravenous iron was not associated with higher 30-day mortality (odds ratio, 0.86; 95% confidence interval [95% CI], 0.74 to 1.00), higher mortality in 2010 (hazard ratio, 0.92; 95% CI, 0.85 to 1.00), longer mean length of stay (10.1 days [95% CI, 9.7 to 10.5] versus 10.5 days [95% CI, 10.3 to 10.7]; P=0.05), or readmission for infection or death within 30 days of discharge (odds ratio, 1.08; 95% CI, 0.96 to 1.22) compared with no receipt of intravenous iron. CONCLUSIONS: This analysis does not support withholding intravenous iron upon admission for bacterial infection in hemodialysis patients, although clinical trials are required to make definitive recommendations.

Efficient and sparse feature selection for biomedical text classification via the elastic net: Application to ICU risk stratification from nursing notes.

BACKGROUND AND SIGNIFICANCE: Sparsity is often a desirable property of statistical models, and various feature selection methods exist so as to yield sparser and interpretable models. However, their application to biomedical text classification, particularly to mortality risk stratification among intensive care unit (ICU) patients, has not been thoroughly studied. OBJECTIVE: To develop and characterize sparse classifiers based on the free text of nursing notes in order to predict ICU mortality risk and to discover text features most strongly associated with mortality. METHODS: We selected nursing notes from the first 24h of ICU admission for 25,826 adult ICU patients from the MIMIC-II database. We then developed a pair of stochastic gradient descent-based classifiers with elastic-net regularization. We also studied the performance-sparsity tradeoffs of both classifiers as their regularization parameters were varied. RESULTS: The best-performing classifier achieved a 10-fold cross-validated AUC of 0.897 under the log loss function and full L2 regularization, while full L1 regularization used just 0.00025% of candidate input features and resulted in an AUC of 0.889. Using the log loss (range of AUCs 0.889-0.897) yielded better performance compared to the hinge loss (0.850-0.876), but the latter yielded even sparser models. DISCUSSION: Most features selected by both classifiers appear clinically relevant and correspond to predictors already present in existing ICU mortality models. The sparser classifiers were also able to discover a number of informative - albeit nonclinical - features. CONCLUSION: The elastic-net-regularized classifiers perform reasonably well and are capable of reducing the number of features required by over a thousandfold, with only a modest impact on performance.

Thirty-day hospital readmissions in systemic lupus erythematosus: predictors and hospital- and state-level variation.

OBJECTIVE: Systemic lupus erythematosus (SLE) has one of the highest hospital readmission rates among chronic conditions. This study was undertaken to identify patient-level, hospital-level, and geographic predictors of 30-day hospital readmissions associated with SLE. METHODS: Using hospital discharge databases from 5 geographically dispersed states, we studied all-cause readmission of SLE patients between 2008 and 2009. We evaluated each hospitalization as a possible index event leading up to a readmission, our primary outcome. We accounted for clustering of hospitalizations within patients and within hospitals and adjusted for hospital case mix. Using multilevel mixed-effects logistic regression, we examined factors associated with 30-day readmission and calculated risk-standardized hospital-level and state-level readmission rates. RESULTS: We examined 55,936 hospitalizations among 31,903 patients with SLE. Of these hospitalizations, 9,244 (16.5%) resulted in readmission within 30 days. In adjusted analyses, age was inversely related to risk of readmission. African American and Hispanic patients were more likely to be readmitted than white patients, as were those with Medicare or Medicaid insurance (versus private insurance). Several clinical characteristics of lupus, including nephritis, serositis, and thrombocytopenia, were associated with readmission. Readmission rates varied significantly between hospitals after accounting for patient-level clustering and hospital case mix. We also found geographic variation, with risk-adjusted readmission rates lower in New York and higher in Florida as compared to California. CONCLUSION: We found that ~1 in 6 hospitalized patients with SLE were readmitted within 30 days of discharge, with higher rates among historically underserved populations. Significant geographic and hospital-level variation in risk-adjusted readmission rates suggests potential for quality improvement.

N-gram support vector machines for scalable procedure and diagnosis classification, with applications to clinical free text data from the intensive care unit.

BACKGROUND: Existing risk adjustment models for intensive care unit (ICU) outcomes rely on manual abstraction of patient-level predictors from medical charts. Developing an automated method for abstracting these data from free text might reduce cost and data collection times. OBJECTIVE: To develop a support vector machine (SVM) classifier capable of identifying a range of procedures and diagnoses in ICU clinical notes for use in risk adjustment. MATERIALS AND METHODS: We selected notes from 2001-2008 for 4191 neonatal ICU (NICU) and 2198 adult ICU patients from the MIMIC-II database from the Beth Israel Deaconess Medical Center. Using these notes, we developed an implementation of the SVM classifier to identify procedures (mechanical ventilation and phototherapy in NICU notes) and diagnoses (jaundice in NICU and intracranial hemorrhage (ICH) in adult ICU). On the jaundice classification task, we also compared classifier performance using n-gram features to unigrams with application of a negation algorithm (NegEx). RESULTS: Our classifier accurately identified mechanical ventilation (accuracy=0.982, F1=0.954) and phototherapy use (accuracy=0.940, F1=0.912), as well as jaundice (accuracy=0.898, F1=0.884) and ICH diagnoses (accuracy=0.938, F1=0.943). Including bigram features improved performance on the jaundice (accuracy=0.898 vs 0.865) and ICH (0.938 vs 0.927) tasks, and outperformed NegEx-derived unigram features (accuracy=0.898 vs 0.863) on the jaundice task. DISCUSSION: Overall, a classifier using n-gram support vectors displayed excellent performance characteristics. The classifier generalizes to diverse patient populations, diagnoses, and procedures. CONCLUSIONS: SVM-based classifiers can accurately identify procedure status and diagnoses among ICU patients, and including n-gram features improves performance, compared to existing methods.