The Impact of Health Care Algorithms on Racial and Ethnic Disparities : A Systematic Review.

BACKGROUND: There is increasing concern for the potential impact of health care algorithms on racial and ethnic disparities. PURPOSE: To examine the evidence on how health care algorithms and associated mitigation strategies affect racial and ethnic disparities. DATA SOURCES: Several databases were searched for relevant studies published from 1 January 2011 to 30 September 2023. STUDY SELECTION: Using predefined criteria and dual review, studies were screened and selected to determine: 1) the effect of algorithms on racial and ethnic disparities in health and health care outcomes and 2) the effect of strategies or approaches to mitigate racial and ethnic bias in the development, validation, dissemination, and implementation of algorithms. DATA EXTRACTION: Outcomes of interest (that is, access to health care, quality of care, and health outcomes) were extracted with risk-of-bias assessment using the ROBINS-I (Risk Of Bias In Non-randomised Studies - of Interventions) tool and adapted CARE-CPM (Critical Appraisal for Racial and Ethnic Equity in Clinical Prediction Models) equity extension. DATA SYNTHESIS: Sixty-three studies (51 modeling, 4 retrospective, 2 prospective, 5 prepost studies, and 1 randomized controlled trial) were included. Heterogenous evidence on algorithms was found to: a) reduce disparities (for example, the revised kidney allocation system), b) perpetuate or exacerbate disparities (for example, severity-of-illness scores applied to critical care resource allocation), and/or c) have no statistically significant effect on select outcomes (for example, the HEART Pathway [history, electrocardiogram, age, risk factors, and troponin]). To mitigate disparities, 7 strategies were identified: removing an input variable, replacing a variable, adding race, adding a non-race-based variable, changing the racial and ethnic composition of the population used in model development, creating separate thresholds for subpopulations, and modifying algorithmic analytic techniques. LIMITATION: Results are mostly based on modeling studies and may be highly context-specific. CONCLUSION: Algorithms can mitigate, perpetuate, and exacerbate racial and ethnic disparities, regardless of the explicit use of race and ethnicity, but evidence is heterogeneous. Intentionality and implementation of the algorithm can impact the effect on disparities, and there may be tradeoffs in outcomes. PRIMARY FUNDING SOURCE: Agency for Healthcare Quality and Research.

Two Weeks Versus One Week of Maximal Patient-Intensivist Continuity for Adult Medical Intensive Care Patients: A Two-Center Target Trial Emulation.

OBJECTIVES: To compare outcomes for 2 weeks vs. 1 week of maximal patient-intensivist continuity in the ICU. DESIGN: Retrospective cohort study. SETTING: Two U.S. urban, teaching, medical ICUs where intensivists were scheduled for 2-week service blocks: site A was in the Midwest and site B was in the Northeast. PATIENTS: Patients 18 years old or older admitted to a study ICU between March 1, 2017, and February 28, 2020. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: We applied target trial emulation to compare admission during an intensivist's first week (as a proxy for 2 wk of maximal continuity) vs. admission during their second week (as a proxy for 1 wk of maximal continuity). Outcomes included hospital mortality, ICU length of stay, and, for mechanically ventilated patients, duration of ventilation. Exploratory outcomes included imaging, echocardiogram, and consultation orders. We used inverse probability weighting to adjust for baseline differences and random-effects meta-analysis to calculate overall effect estimates. Among 2571 patients, 1254 were admitted during an intensivist's first week and 1317 were admitted during a second week. At sites A and B, hospital mortality rates were 25.8% and 24.2%, median ICU length of stay were 4 and 2 days, and median mechanical ventilation durations were 3 and 3 days, respectively. There were no differences in adjusted mortality (odds ratio [OR], 1.01 [95% CI, 0.96-1.06]) or ICU length of stay (-0.25 d [-0.82 d to +0.32 d]) for 2 weeks vs. 1 week of maximal continuity. Among mechanically ventilated patients, there were no differences in adjusted mortality (OR, 1.00 [0.87-1.16]), ICU length of stay (+0.06 d [-0.78 d to +0.91 d]), or duration of mechanical ventilation (+0.37 d [-0.46 d to +1.21 d]) for 2 weeks vs. 1 week of maximal continuity. CONCLUSIONS: Two weeks of maximal patient-intensivist continuity was not associated with differences in clinical outcomes compared with 1 week in two medical ICUs.

Prediction of In-hospital Mortality Among Intensive Care Unit Patients Using Modified Daily Laboratory-based Acute Physiology Score, Version 2.

BACKGROUND: Mortality prediction for intensive care unit (ICU) patients frequently relies on single ICU admission acuity measures without accounting for subsequent clinical changes. OBJECTIVE: Evaluate novel models incorporating modified admission and daily, time-updating Laboratory-based Acute Physiology Score, version 2 (LAPS2) to predict in-hospital mortality among ICU patients. RESEARCH DESIGN: Retrospective cohort study. PATIENTS: ICU patients in 5 hospitals from October 2017 through September 2019. MEASURES: We used logistic regression, penalized logistic regression, and random forest models to predict in-hospital mortality within 30 days of ICU admission using admission LAPS2 alone in patient-level and patient-day-level models, or admission and daily LAPS2 at the patient-day level. Multivariable models included patient and admission characteristics. We performed internal-external validation using 4 hospitals for training and the fifth for validation, repeating analyses for each hospital as the validation set. We assessed performance using scaled Brier scores (SBS), c -statistics, and calibration plots. RESULTS: The cohort included 13,993 patients and 107,699 ICU days. Across validation hospitals, patient-day-level models including daily LAPS2 (SBS: 0.119-0.235; c -statistic: 0.772-0.878) consistently outperformed models with admission LAPS2 alone in patient-level (SBS: 0.109-0.175; c -statistic: 0.768-0.867) and patient-day-level (SBS: 0.064-0.153; c -statistic: 0.714-0.861) models. Across all predicted mortalities, daily models were better calibrated than models with admission LAPS2 alone. CONCLUSIONS: Patient-day-level models incorporating daily, time-updating LAPS2 to predict mortality among an ICU population performs as well or better than models incorporating modified admission LAPS2 alone. The use of daily LAPS2 may offer an improved tool for clinical prognostication and risk adjustment in research in this population.

A Data-Driven Analysis of Ward Capacity Strain Metrics That Predict Clinical Outcomes Among Survivors of Acute Respiratory Failure.

Supply-demand mismatch of ward resources ("ward capacity strain") alters care and outcomes. Narrow strain definitions and heterogeneous populations limit strain literature. Evaluate the predictive utility of a large set of candidate strain variables for in-hospital mortality and discharge destination among acute respiratory failure (ARF) survivors. In a retrospective cohort of ARF survivors transferred from intensive care units (ICUs) to wards in five hospitals from 4/2017-12/2019, we applied 11 machine learning (ML) models to identify ward strain measures during the first 24 hours after transfer most predictive of outcomes. Measures spanned patient volume (census, admissions, discharges), staff workload (medications administered, off-ward transports, transfusions, isolation precautions, patients per respiratory therapist and nurse), and average patient acuity (Laboratory Acute Physiology Score version 2, ICU transfers) domains. The cohort included 5,052 visits in 43 wards. Median age was 65 years (IQR 56-73); 2,865 (57%) were male; and 2,865 (57%) were white. 770 (15%) patients died in the hospital or had hospice discharges, and 2,628 (61%) were discharged home and 964 (23%) to skilled nursing facilities (SNFs). Ward admissions, isolation precautions, and hospital admissions most consistently predicted in-hospital mortality across ML models. Patients per nurse most consistently predicted discharge to home and SNF, and medications administered predicted SNF discharge. In this hypothesis-generating analysis of candidate ward strain variables' prediction of outcomes among ARF survivors, several variables emerged as consistently predictive of key outcomes across ML models. These findings suggest targets for future inferential studies to elucidate mechanisms of ward strain's adverse effects.

Word embeddings trained on published case reports are lightweight, effective for clinical tasks, and free of protected health information.

OBJECTIVE: Quantify tradeoffs in performance, reproducibility, and resource demands across several strategies for developing clinically relevant word embeddings. MATERIALS AND METHODS: We trained separate embeddings on all full-text manuscripts in the Pubmed Central (PMC) Open Access subset, case reports therein, the English Wikipedia corpus, the Medical Information Mart for Intensive Care (MIMIC) III dataset, and all notes in the University of Pennsylvania Health System (UPHS) electronic health record. We tested embeddings in six clinically relevant tasks including mortality prediction and de-identification, and assessed performance using the scaled Brier score (SBS) and the proportion of notes successfully de-identified, respectively. RESULTS: Embeddings from UPHS notes best predicted mortality (SBS 0.30, 95% CI 0.15 to 0.45) while Wikipedia embeddings performed worst (SBS 0.12, 95% CI -0.05 to 0.28). Wikipedia embeddings most consistently (78% of notes) and the full PMC corpus embeddings least consistently (48%) de-identified notes. Across all six tasks, the full PMC corpus demonstrated the most consistent performance, and the Wikipedia corpus the least. Corpus size ranged from 49 million tokens (PMC case reports) to 10 billion (UPHS). DISCUSSION: Embeddings trained on published case reports performed as least as well as embeddings trained on other corpora in most tasks, and clinical corpora consistently outperformed non-clinical corpora. No single corpus produced a strictly dominant set of embeddings across all tasks and so the optimal training corpus depends on intended use. CONCLUSION: Embeddings trained on published case reports performed comparably on most clinical tasks to embeddings trained on larger corpora. Open access corpora allow training of clinically relevant, effective, and reproducible embeddings.

Equitably Allocating Resources during Crises: Racial Differences in Mortality Prediction Models.

Rationale: Crisis standards of care (CSCs) guide critical care resource allocation during crises. Most recommend ranking patients on the basis of their expected in-hospital mortality using the Sequential Organ Failure Assessment (SOFA) score, but it is unknown how SOFA or other acuity scores perform among patients of different races. Objectives: To test the prognostic accuracy of the SOFA score and version 2 of the Laboratory-based Acute Physiology Score (LAPS2) among Black and white patients. Methods: We included Black and white patients admitted for sepsis or acute respiratory failure at 27 hospitals. We calculated the discrimination and calibration for in-hospital mortality of SOFA, LAPS2, and modified versions of each, including categorical SOFA groups recommended in a popular CSC and a SOFA score without creatinine to reduce the influence of race. Measurements and Main Results: Of 113,158 patients, 27,644 (24.4%) identified as Black. The LAPS2 demonstrated higher discrimination (area under the receiver operating characteristic curve [AUC], 0.76; 95% confidence interval [CI], 0.76-0.77) than the SOFA score (AUC, 0.68; 95% CI, 0.68-0.69). The LAPS2 was also better calibrated than the SOFA score, but both underestimated in-hospital mortality for white patients and overestimated in-hospital mortality for Black patients. Thus, in a simulation using observed mortality, 81.6% of Black patients were included in lower-priority CSC categories, and 9.4% of all Black patients were erroneously excluded from receiving the highest prioritization. The SOFA score without creatinine reduced racial miscalibration. Conclusions: Using SOFA in CSCs may lead to racial disparities in resource allocation. More equitable mortality prediction scores are needed.

A Simulated Prospective Evaluation of a Deep Learning Model for Real-Time Prediction of Clinical Deterioration Among Ward Patients.

OBJECTIVES: The National Early Warning Score, Modified Early Warning Score, and quick Sepsis-related Organ Failure Assessment can predict clinical deterioration. These scores exhibit only moderate performance and are often evaluated using aggregated measures over time. A simulated prospective validation strategy that assesses multiple predictions per patient-day would provide the best pragmatic evaluation. We developed a deep recurrent neural network deterioration model and conducted a simulated prospective evaluation. DESIGN: Retrospective cohort study. SETTING: Four hospitals in Pennsylvania. PATIENTS: Inpatient adults discharged between July 1, 2017, and June 30, 2019. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: We trained a deep recurrent neural network and logistic regression model using data from electronic health records to predict hourly the 24-hour composite outcome of transfer to ICU or death. We analyzed 146,446 hospitalizations with 16.75 million patient-hours. The hourly event rate was 1.6% (12,842 transfers or deaths, corresponding to 260,295 patient-hours within the predictive horizon). On a hold-out dataset, the deep recurrent neural network achieved an area under the precision-recall curve of 0.042 (95% CI, 0.04-0.043), comparable with logistic regression model (0.043; 95% CI 0.041 to 0.045), and outperformed National Early Warning Score (0.034; 95% CI, 0.032-0.035), Modified Early Warning Score (0.028; 95% CI, 0.027- 0.03), and quick Sepsis-related Organ Failure Assessment (0.021; 95% CI, 0.021-0.022). For a fixed sensitivity of 50%, the deep recurrent neural network achieved a positive predictive value of 3.4% (95% CI, 3.4-3.5) and outperformed logistic regression model (3.1%; 95% CI 3.1-3.2), National Early Warning Score (2.0%; 95% CI, 2.0-2.0), Modified Early Warning Score (1.5%; 95% CI, 1.5-1.5), and quick Sepsis-related Organ Failure Assessment (1.5%; 95% CI, 1.5-1.5). CONCLUSIONS: Commonly used early warning scores for clinical decompensation, along with a logistic regression model and a deep recurrent neural network model, show very poor performance characteristics when assessed using a simulated prospective validation. None of these models may be suitable for real-time deployment.

Effects of Neighborhood-level Data on Performance and Algorithmic Equity of a Model That Predicts 30-day Heart Failure Readmissions at an Urban Academic Medical Center.

BACKGROUND: Socioeconomic data may improve predictions of clinical events. However, owing to structural racism, algorithms may not perform equitably across racial subgroups. Therefore, we sought to compare the predictive performance overall, and by racial subgroup, of commonly used predictor variables for heart failure readmission with and without the area deprivation index (ADI), a neighborhood-level socioeconomic measure. METHODS AND RESULTS: We conducted a retrospective cohort study of 1316 Philadelphia residents discharged with a primary diagnosis of congestive heart failure from the University of Pennsylvania Health System between April 1, 2015, and March 31, 2017. We trained a regression model to predict the probability of a 30-day readmission using clinical and demographic variables. A second model also included the ADI as a predictor variable. We measured predictive performance with the Brier Score (BS) in a held-out test set. The baseline model had moderate performance overall (BS 0.13, 95% CI 0.13-0.14), and among White (BS 0.12, 95% CI 0.12-0.13) and non-White (BS 0.13, 95% CI 0.13-0.14) patients. Neither performance nor algorithmic equity were significantly changed with the addition of the ADI. CONCLUSIONS: The inclusion of neighborhood-level data may not reliably improve performance or algorithmic equity.

Algorithmic prognostication in critical care: a promising but unproven technology for supporting difficult decisions.

PURPOSE OF REVIEW: Patients, surrogate decision makers, and clinicians face weighty and urgent decisions under uncertainty in the ICU, which could be aided by risk prediction. Although emerging artificial intelligence/machine learning (AI/ML) algorithms could reduce uncertainty surrounding these life and death decisions, certain criteria must be met to ensure their bedside value. RECENT FINDINGS: Although ICU severity of illness scores have existed for decades, these tools have not been shown to predict well or to improve outcomes for individual patients. Novel AI/ML tools offer the promise of personalized ICU care but remain untested in clinical trials. Ensuring that these predictive models account for heterogeneity in patient characteristics and treatments, are not only specific to a clinical action but also consider the longitudinal course of critical illness, and address patient-centered outcomes related to equity, transparency, and shared decision-making will increase the likelihood that these tools improve outcomes. Improved clarity around standards and contributions from institutions and critical care departments will be essential. SUMMARY: Improved ICU prognostication, enabled by advanced ML/AI methods, offer a promising approach to inform difficult and urgent decisions under uncertainty. However, critical knowledge gaps around performance, equity, safety, and effectiveness must be filled and prospective, randomized testing of predictive interventions are still needed.

Locally Informed Simulation to Predict Hospital Capacity Needs During the COVID-19 Pandemic.

BACKGROUND: The coronavirus disease 2019 (COVID-19) pandemic challenges hospital leaders to make time-sensitive, critical decisions about clinical operations and resource allocations. OBJECTIVE: To estimate the timing of surges in clinical demand and the best- and worst-case scenarios of local COVID-19-induced strain on hospital capacity, and thus inform clinical operations and staffing demands and identify when hospital capacity would be saturated. DESIGN: Monte Carlo simulation instantiation of a susceptible, infected, removed (SIR) model with a 1-day cycle. SETTING: 3 hospitals in an academic health system. PATIENTS: All people living in the greater Philadelphia region. MEASUREMENTS: The COVID-19 Hospital Impact Model (CHIME) (http://penn-chime.phl.io) SIR model was used to estimate the time from 23 March 2020 until hospital capacity would probably be exceeded, and the intensity of the surge, including for intensive care unit (ICU) beds and ventilators. RESULTS: Using patients with COVID-19 alone, CHIME estimated that it would be 31 to 53 days before demand exceeds existing hospital capacity. In best- and worst-case scenarios of surges in the number of patients with COVID-19, the needed total capacity for hospital beds would reach 3131 to 12 650 across the 3 hospitals, including 338 to 1608 ICU beds and 118 to 599 ventilators. LIMITATIONS: Model parameters were taken directly or derived from published data across heterogeneous populations and practice environments and from the health system's historical data. CHIME does not incorporate more transition states to model infection severity, social networks to model transmission dynamics, or geographic information to account for spatial patterns of human interaction. CONCLUSION: Publicly available and designed for hospital operations leaders, this modeling tool can inform preparations for capacity strain during the early days of a pandemic. PRIMARY FUNDING SOURCE: University of Pennsylvania Health System and the Palliative and Advanced Illness Research Center.

Intensive Care Unit Capacity Strain and Outcomes of Critical Illness in a Resource-Limited Setting: A 2-Hospital Study in South Africa.

OBJECTIVE: To measure the association of intensive care unit (ICU) capacity strain with processes of care and outcomes of critical illness in a resource-limited setting. METHODS: We performed a retrospective cohort study of 5332 patients referred to the ICUs at 2 public hospitals in South Africa using the country's first published multicenter electronic critical care database. We assessed the association between multiple ICU capacity strain metrics (ICU occupancy, turnover, census acuity, and referral burden) at different exposure time points (ICU referral, admission, and/or discharge) with clinical and process of care outcomes. The association of ICU capacity strain at the time of ICU admission with ICU length of stay (LOS), the primary outcome, was analyzed with a multivariable Cox proportional hazard model. Secondary outcomes of ICU triage decision (with strain at ICU referral), ICU mortality (with strain at ICU admission), and ICU LOS (with strain at ICU discharge), were analyzed with linear and logistic multivariable regression. RESULTS: No measure of ICU capacity strain at the time of ICU admission was associated with ICU LOS, the primary outcome. The ICU occupancy at the time of ICU admission was associated with increased odds of ICU mortality (odds ratio = 1.07, 95% confidence interval: 1.02-1.11; P = .004), a secondary outcome, such that a 10% increase in ICU occupancy would be associated with a 7% increase in the odds of ICU mortality. CONCLUSIONS: In a resource-limited setting in South Africa, ICU capacity strain at the time of ICU admission was not associated with ICU LOS. In secondary analyses, higher ICU occupancy at the time of ICU admission, but not other measures of capacity strain, was associated with increased odds of ICU mortality.

Construct validity of six sentiment analysis methods in the text of encounter notes of patients with critical illness.

Sentiment analysis may offer insights into patient outcomes through the subjective expressions made by clinicians in the text of encounter notes. We analyzed the predictive, concurrent, convergent, and content validity of six sentiment methods in a sample of 793,725 multidisciplinary clinical notes among 41,283 hospitalizations associated with an intensive care unit stay. None of these approaches improved early prediction of in-hospital mortality using logistic regression models, but did improve both discrimination and calibration when using random forests. Additionally, positive sentiment measured by the CoreNLP (OR 0.04, 95% CI 0.002-0.55), Pattern (OR 0.09, 95% CI 0.04-0.17), sentimentr (OR 0.37, 95% CI 0.25-0.63), and Opinion (OR 0.25, 95% CI 0.07-0.89) methods were inversely associated with death on the concurrent day after adjustment for demographic characteristics and illness severity. Median daily lexical coverage ranged from 5.4% to 20.1%. While sentiment between all methods was positively correlated, their agreement was weak. Sentiment analysis holds promise for clinical applications but will require a novel domain-specific method applicable to clinical text.

Inclusion of Unstructured Clinical Text Improves Early Prediction of Death or Prolonged ICU Stay.

OBJECTIVES: Early prediction of undesired outcomes among newly hospitalized patients could improve patient triage and prompt conversations about patients' goals of care. We evaluated the performance of logistic regression, gradient boosting machine, random forest, and elastic net regression models, with and without unstructured clinical text data, to predict a binary composite outcome of in-hospital death or ICU length of stay greater than or equal to 7 days using data from the first 48 hours of hospitalization. DESIGN: Retrospective cohort study with split sampling for model training and testing. SETTING: A single urban academic hospital. PATIENTS: All hospitalized patients who required ICU care at the Beth Israel Deaconess Medical Center in Boston, MA, from 2001 to 2012. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Among eligible 25,947 hospital admissions, we observed 5,504 (21.2%) in which patients died or had ICU length of stay greater than or equal to 7 days. The gradient boosting machine model had the highest discrimination without (area under the receiver operating characteristic curve, 0.83; 95% CI, 0.81-0.84) and with (area under the receiver operating characteristic curve, 0.89; 95% CI, 0.88-0.90) text-derived variables. Both gradient boosting machines and random forests outperformed logistic regression without text data (p < 0.001), whereas all models outperformed logistic regression with text data (p < 0.02). The inclusion of text data increased the discrimination of all four model types (p < 0.001). Among those models using text data, the increasing presence of terms "intubated" and "poor prognosis" were positively associated with mortality and ICU length of stay, whereas the term "extubated" was inversely associated with them. CONCLUSIONS: Variables extracted from unstructured clinical text from the first 48 hours of hospital admission using natural language processing techniques significantly improved the abilities of logistic regression and other machine learning models to predict which patients died or had long ICU stays. Learning health systems may adapt such models using open-source approaches to capture local variation in care patterns.

Eliminating Gender-Based Bias in Academic Medicine: More Than Naming the "Elephant in the Room".

Gender-based discrimination and bias are widespread in professional settings, including academic medicine. Overt manifestations such as sexual harassment have long been identified but attention is only more recently turning towards subtler forms of bias, including inequity in promotion and compensation. Barriers to progress vary across institutions and include lack of awareness, inadequate training, poor informational transparency, and challenging power dynamics. We propose five solutions that the academic medical community can adopt to not only name, but also address, gender-based bias as the proverbial elephant in the room: definitively identify the systemic nature of the problem, prompt those with influence and power to advance a culture of equity, broadly incorporate evidence-based explicit anti-sexist training, increase transparency of information related to professional development and compensation, and use robust research methods to study the drivers and potential solutions of gender inequity within academic medicine. While implementing these proposals is no small task, doing so is an important step in helping the academic medical community become more just.

Guiding Risk Adjustment Models Toward Machine Learning Methods.

This Viewpoint reviews the history of administrative risk adjustment models used in health care and provides recommendations for modernizing these models to promote their safe, transparent, equitable, and efficient use.

Validation of an Administrative Definition of ICU Admission Using Revenue Center Codes.

OBJECTIVES: Describe the operating characteristics of a proposed set of revenue center codes to correctly identify ICU stays among hospitalized patients. DESIGN: Retrospective cohort study. We report the operating characteristics of all ICU-related revenue center codes for intensive and coronary care, excluding nursery, intermediate, and incremental care, to identify ICU stays. We use a classification and regression tree model to further refine identification of ICU stays using administrative data. The gold standard for classifying ICU admission was an electronic patient location tracking system. SETTING: The University of Pennsylvania Health System in Philadelphia, PA, United States. PATIENTS: All adult inpatient hospital admissions between July 1, 2013, and June 30, 2015. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Among 127,680 hospital admissions, the proposed combination of revenue center codes had 94.6% sensitivity (95% CI, 94.3-94.9%) and 96.1% specificity (95% CI, 96.0-96.3%) for correctly identifying hospital admissions with an ICU stay. The classification and regression tree algorithm had 92.3% sensitivity (95% CI, 91.6-93.1%) and 97.4% specificity (95% CI, 97.2-97.6%), with an overall improved accuracy (χ = 398; p < 0.001). CONCLUSIONS: Use of the proposed combination of revenue center codes has excellent sensitivity and specificity for identifying true ICU admission. A classification and regression tree algorithm with additional administrative variables offers further improvements to accuracy.