Sustainable deployment of clinical prediction tools-a 360° approach to model maintenance.

BACKGROUND: As the enthusiasm for integrating artificial intelligence (AI) into clinical care grows, so has our understanding of the challenges associated with deploying impactful and sustainable clinical AI models. Complex dataset shifts resulting from evolving clinical environments strain the longevity of AI models as predictive accuracy and associated utility deteriorate over time. OBJECTIVE: Responsible practice thus necessitates the lifecycle of AI models be extended to include ongoing monitoring and maintenance strategies within health system algorithmovigilance programs. We describe a framework encompassing a 360° continuum of preventive, preemptive, responsive, and reactive approaches to address model monitoring and maintenance from critically different angles. DISCUSSION: We describe the complementary advantages and limitations of these four approaches and highlight the importance of such a coordinated strategy to help ensure the promise of clinical AI is not short-lived.

A framework for understanding label leakage in machine learning for health care.

INTRODUCTION: The pitfalls of label leakage, contamination of model input features with outcome information, are well established. Unfortunately, avoiding label leakage in clinical prediction models requires more nuance than the common advice of applying "no time machine rule." FRAMEWORK: We provide a framework for contemplating whether and when model features pose leakage concerns by considering the cadence, perspective, and applicability of predictions. To ground these concepts, we use real-world clinical models to highlight examples of appropriate and inappropriate label leakage in practice. RECOMMENDATIONS: Finally, we provide recommendations to support clinical and technical stakeholders as they evaluate the leakage tradeoffs associated with model design, development, and implementation decisions. By providing common language and dimensions to consider when designing models, we hope the clinical prediction community will be better prepared to develop statistically valid and clinically useful machine learning models.

Use of Electronic Health Record Data for Drug Safety Signal Identification: A Scoping Review.

INTRODUCTION: Pharmacovigilance programs protect patient health and safety by identifying adverse event signals through postmarketing surveillance of claims data and spontaneous reports. Electronic health records (EHRs) provide new opportunities to address limitations of traditional approaches and promote discovery-oriented pharmacovigilance. METHODS: To evaluate the current state of EHR-based medication safety signal identification, we conducted a scoping literature review of studies aimed at identifying safety signals from routinely collected patient-level EHR data. We extracted information on study design, EHR data elements utilized, analytic methods employed, drugs and outcomes evaluated, and key statistical and data analysis choices. RESULTS: We identified 81 eligible studies. Disproportionality methods were the predominant analytic approach, followed by data mining and regression. Variability in study design makes direct comparisons difficult. Studies varied widely in terms of data, confounding adjustment, and statistical considerations. CONCLUSION: Despite broad interest in utilizing EHRs for safety signal identification, current efforts fail to leverage the full breadth and depth of available data or to rigorously control for confounding. The development of best practices and application of common data models would promote the expansion of EHR-based pharmacovigilance.

Simulating complex patient populations with hierarchical learning effects to support methods development for post-market surveillance.

BACKGROUND: Validating new algorithms, such as methods to disentangle intrinsic treatment risk from risk associated with experiential learning of novel treatments, often requires knowing the ground truth for data characteristics under investigation. Since the ground truth is inaccessible in real world data, simulation studies using synthetic datasets that mimic complex clinical environments are essential. We describe and evaluate a generalizable framework for injecting hierarchical learning effects within a robust data generation process that incorporates the magnitude of intrinsic risk and accounts for known critical elements in clinical data relationships. METHODS: We present a multi-step data generating process with customizable options and flexible modules to support a variety of simulation requirements. Synthetic patients with nonlinear and correlated features are assigned to provider and institution case series. The probability of treatment and outcome assignment are associated with patient features based on user definitions. Risk due to experiential learning by providers and/or institutions when novel treatments are introduced is injected at various speeds and magnitudes. To further reflect real-world complexity, users can request missing values and omitted variables. We illustrate an implementation of our method in a case study using MIMIC-III data for reference patient feature distributions. RESULTS: Realized data characteristics in the simulated data reflected specified values. Apparent deviations in treatment effects and feature distributions, though not statistically significant, were most common in small datasets (n < 3000) and attributable to random noise and variability in estimating realized values in small samples. When learning effects were specified, synthetic datasets exhibited changes in the probability of an adverse outcomes as cases accrued for the treatment group impacted by learning and stable probabilities as cases accrued for the treatment group not affected by learning. CONCLUSIONS: Our framework extends clinical data simulation techniques beyond generation of patient features to incorporate hierarchical learning effects. This enables the complex simulation studies required to develop and rigorously test algorithms developed to disentangle treatment safety signals from the effects of experiential learning. By supporting such efforts, this work can help identify training opportunities, avoid unwarranted restriction of access to medical advances, and hasten treatment improvements.

Integrating Clinical and Air Quality Data to Improve Prediction of COPD Exacerbations.

Several studies have found associations between air pollution and respiratory disease outcomes. However, there is minimal prognostic research exploring whether integrating air quality into clinical prediction models can improve accuracy and utility. In this study, we built models using both logistic regression and random forests to determine the benefits of including air quality data with meteorological and clinical data in prediction of COPD exacerbations requiring medical care. Logistic models were not improved by inclusion of air quality. However, the net benefit curves of random forest models showed greater clinical utility with the addition of air quality data. These models demonstrate a practical and relatively low-cost way to include environmental information into clinical prediction tools to improve the clinical utility of COPD prediction. Findings could be used to provide population level health warnings as well as individual-patient risk assessments.

Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings.

As the implementation of artificial intelligence (AI)-enabled tools is realized across diverse clinical environments, there is a growing understanding of the need for ongoing monitoring and updating of prediction models. Dataset shift-temporal changes in clinical practice, patient populations, and information systems-is now well-documented as a source of deteriorating model accuracy and a challenge to the sustainability of AI-enabled tools in clinical care. While best practices are well-established for training and validating new models, there has been limited work developing best practices for prospective validation and model maintenance. In this paper, we highlight the need for updating clinical prediction models and discuss open questions regarding this critical aspect of the AI modeling lifecycle in three focus areas: model maintenance policies, performance monitoring perspectives, and model updating strategies. With the increasing adoption of AI-enabled tools, the need for such best practices must be addressed and incorporated into new and existing implementations. This commentary aims to encourage conversation and motivate additional research across clinical and data science stakeholders.

Maintaining a National Acute Kidney Injury Risk Prediction Model to Support Local Quality Benchmarking.

BACKGROUND: The utility of quality dashboards to inform decision-making and improve clinical outcomes is tightly linked to the accuracy of the information they provide and, in turn, accuracy of underlying prediction models. Despite recognition of the need to update prediction models to maintain accuracy over time, there is limited guidance on updating strategies. We compare predefined and surveillance-based updating strategies applied to a model supporting quality evaluations among US veterans. METHODS: We evaluated the performance of a US Department of Veterans Affairs-specific model for postcardiac catheterization acute kidney injury using routinely collected observational data over the 6 years following model development (n=90 295 procedures in 2013-2019). Predicted probabilities were generated from the original model, an annually retrained model, and a surveillance-based approach that monitored performance to inform the timing and method of updates. We evaluated how updating the national model impacted regional quality profiles. We compared observed-to-expected outcome ratios, where values above and below 1 indicated more and fewer adverse outcomes than expected, respectively. RESULTS: The original model overpredicted risk at the national level (observed-to-expected outcome ratio, 0.75 [0.74-0.77]). Annual retraining updated the model 5×; surveillance-based updating retrained once and recalibrated twice. While both strategies improved performance, the surveillance-based approach provided superior calibration (observed-to-expected outcome ratio, 1.01 [0.99-1.03] versus 0.94 [0.92-0.96]). Overprediction by the original model led to optimistic quality assessments, incorrectly indicating most of the US Department of Veterans Affairs' 18 regions observed fewer acute kidney injury events than predicted. Both updating strategies revealed 16 regions performed as expected and 2 regions increasingly underperformed, having more acute kidney injury events than predicted. CONCLUSIONS: Miscalibrated clinical prediction models provide inaccurate pictures of performance across clinical units, and degrading calibration further complicates our understanding of quality. Updating strategies tailored to health system needs and capacity should be incorporated into model implementation plans to promote the utility and longevity of quality reporting tools.

Detection of calibration drift in clinical prediction models to inform model updating.

Model calibration, critical to the success and safety of clinical prediction models, deteriorates over time in response to the dynamic nature of clinical environments. To support informed, data-driven model updating strategies, we present and evaluate a calibration drift detection system. Methods are developed for maintaining dynamic calibration curves with optimized online stochastic gradient descent and for detecting increasing miscalibration with adaptive sliding windows. These methods are generalizable to support diverse prediction models developed using a variety of learning algorithms and customizable to address the unique needs of clinical use cases. In both simulation and case studies, our system accurately detected calibration drift. When drift is detected, our system further provides actionable alerts by including information on a window of recent data that may be appropriate for model updating. Simulations showed these windows were primarily composed of data accruing after drift onset, supporting the potential utility of the windows for model updating. By promoting model updating as calibration deteriorates rather than on pre-determined schedules, implementations of our drift detection system may minimize interim periods of insufficient model accuracy and focus analytic resources on those models most in need of attention.

Predicting 30-Day Hospital Readmission Risk in a National Cohort of Patients with Cirrhosis.

BACKGROUND: Early hospital readmission for patients with cirrhosis continues to challenge the healthcare system. Risk stratification may help tailor resources, but existing models were designed using small, single-institution cohorts or had modest performance. AIMS: We leveraged a large clinical database from the Department of Veterans Affairs (VA) to design a readmission risk model for patients hospitalized with cirrhosis. Additionally, we analyzed potentially modifiable or unexplored readmission risk factors. METHODS: A national VA retrospective cohort of patients with a history of cirrhosis hospitalized for any reason from January 1, 2006, to November 30, 2013, was developed from 123 centers. Using 174 candidate variables within demographics, laboratory results, vital signs, medications, diagnoses and procedures, and healthcare utilization, we built a 47-variable penalized logistic regression model with the outcome of all-cause 30-day readmission. We excluded patients who left against medical advice, transferred to a non-VA facility, or if the hospital length of stay was greater than 30 days. We evaluated calibration and discrimination across variable volume and compared the performance to recalibrated preexisting risk models for readmission. RESULTS: We analyzed 67,749 patients and 179,298 index hospitalizations. The 30-day readmission rate was 23%. Ascites was the most common cirrhosis-related cause of index hospitalization and readmission. The AUC of the model was 0.670 compared to existing models (0.649, 0.566, 0.577). The Brier score of 0.165 showed good calibration. CONCLUSION: Our model achieved better discrimination and calibration compared to existing models, even after local recalibration. Assessment of calibration by variable parsimony revealed performance improvements for increasing variable inclusion well beyond those detectable for discrimination.

Development of a national Department of Veterans Affairs mortality risk prediction model among patients with cirrhosis.

OBJECTIVE: Cirrhotic patients are at high hospitalisation risk with subsequent high mortality. Current risk prediction models have varied performances with methodological room for improvement. We used current analytical techniques using automatically extractable variables from the electronic health record (EHR) to develop and validate a posthospitalisation mortality risk score for cirrhotic patients and compared performance with the model for end-stage liver disease (MELD), model for end-stage liver disease with sodium (MELD-Na), and the CLIF Consortium Acute Decompensation (CLIF-C AD) models. DESIGN: We analysed a retrospective cohort of 73 976 patients comprising 247 650 hospitalisations between 2006 and 2013 at any of 123 Department of Veterans Affairs hospitals. Using 45 predictor variables, we built a time-dependent Cox proportional hazards model with all-cause mortality as the outcome. We compared performance to the three extant models and reported discrimination and calibration using bootstrapping. Furthermore, we analysed differential utility using the net reclassification index (NRI). RESULTS: The C-statistic for the final model was 0.863, representing a significant improvement over the MELD, MELD-Na, and the CLIF-C AD, which had C-statistics of 0.655, 0.675, and 0.679, respectively. Multiple risk factors were significant in our model, including variables reflecting disease severity and haemodynamic compromise. The NRI showed a 24% improvement in predicting survival of low-risk patients and a 30% improvement in predicting death of high-risk patients. CONCLUSION: We developed a more accurate mortality risk prediction score using variables automatically extractable from an EHR that may be used to risk stratify patients with cirrhosis for targeted postdischarge management.

A nonparametric updating method to correct clinical prediction model drift.

OBJECTIVE: Clinical prediction models require updating as performance deteriorates over time. We developed a testing procedure to select updating methods that minimizes overfitting, incorporates uncertainty associated with updating sample sizes, and is applicable to both parametric and nonparametric models. MATERIALS AND METHODS: We describe a procedure to select an updating method for dichotomous outcome models by balancing simplicity against accuracy. We illustrate the test's properties on simulated scenarios of population shift and 2 models based on Department of Veterans Affairs inpatient admissions. RESULTS: In simulations, the test generally recommended no update under no population shift, no update or modest recalibration under case mix shifts, intercept correction under changing outcome rates, and refitting under shifted predictor-outcome associations. The recommended updates provided superior or similar calibration to that achieved with more complex updating. In the case study, however, small update sets lead the test to recommend simpler updates than may have been ideal based on subsequent performance. DISCUSSION: Our test's recommendations highlighted the benefits of simple updating as opposed to systematic refitting in response to performance drift. The complexity of recommended updating methods reflected sample size and magnitude of performance drift, as anticipated. The case study highlights the conservative nature of our test. CONCLUSIONS: This new test supports data-driven updating of models developed with both biostatistical and machine learning approaches, promoting the transportability and maintenance of a wide array of clinical prediction models and, in turn, a variety of applications relying on modern prediction tools.

Comparison of Prediction Model Performance Updating Protocols: Using a Data-Driven Testing Procedure to Guide Updating.

In evolving clinical environments, the accuracy of prediction models deteriorates over time. Guidance on the design of model updating policies is limited, and there is limited exploration of the impact of different policies on future model performance and across different model types. We implemented a new data-driven updating strategy based on a nonparametric testing procedure and compared this strategy to two baseline approaches in which models are never updated or fully refit annually. The test-based strategy generally recommended intermittent recalibration and delivered more highly calibrated predictions than either of the baseline strategies. The test-based strategy highlighted differences in the updating requirements between logistic regression, L1-regularized logistic regression, random forest, and neural network models, both in terms of the extent and timing of updates. These findings underscore the potential improvements in using a data-driven maintenance approach over "one-size fits all" to sustain more stable and accurate model performance over time.

Development of an automated phenotyping algorithm for hepatorenal syndrome.

OBJECTIVE: Hepatorenal Syndrome (HRS) is a devastating form of acute kidney injury (AKI) in advanced liver disease patients with high morbidity and mortality, but phenotyping algorithms have not yet been developed using large electronic health record (EHR) databases. We evaluated and compared multiple phenotyping methods to achieve an accurate algorithm for HRS identification. MATERIALS AND METHODS: A national retrospective cohort of patients with cirrhosis and AKI admitted to 124 Veterans Affairs hospitals was assembled from electronic health record data collected from 2005 to 2013. AKI was defined by the Kidney Disease: Improving Global Outcomes criteria. Five hundred and four hospitalizations were selected for manual chart review and served as the gold standard. Electronic Health Record based predictors were identified using structured and free text clinical data, subjected through NLP from the clinical Text Analysis Knowledge Extraction System. We explored several dimension reduction techniques for the NLP data, including newer high-throughput phenotyping and word embedding methods, and ascertained their effectiveness in identifying the phenotype without structured predictor variables. With the combined structured and NLP variables, we analyzed five phenotyping algorithms: penalized logistic regression, naïve Bayes, support vector machines, random forest, and gradient boosting. Calibration and discrimination metrics were calculated using 100 bootstrap iterations. In the final model, we report odds ratios and 95% confidence intervals. RESULTS: The area under the receiver operating characteristic curve (AUC) for the different models ranged from 0.73 to 0.93; with penalized logistic regression having the best discriminatory performance. Calibration for logistic regression was modest, but gradient boosting and support vector machines were superior. NLP identified 6985 variables; a priori variable selection performed similarly to dimensionality reduction using high-throughput phenotyping and semantic similarity informed clustering (AUC of 0.81 - 0.82). CONCLUSION: This study demonstrated improved phenotyping of a challenging AKI etiology, HRS, over ICD-9 coding. We also compared performance among multiple approaches to EHR-derived phenotyping, and found similar results between methods. Lastly, we showed that automated NLP dimension reduction is viable for acute illness.

Long-term Patterns of Patient Portal Use for Pediatric Patients at an Academic Medical Center.

BACKGROUND: Patient portal adoption has increased over the last two decades. Most research about patient portals has focused on adult populations in the primary care and medical specialty settings. OBJECTIVE: We describe initial and long-term portal use by pediatric patients and their caregivers in a broadly deployed patient portal at an academic medical center. METHODS: We analyzed portal usage for pediatric patients and their caregivers from 2008 to 2014. We recorded usage events with time stamps; user role defined as self, surrogate (i.e., parent or guardian), or delegate; and functions accessed. Usage events were grouped into sessions to calculate descriptive statistics by patient age, user role, and active use over time. RESULTS: From 2008 to 2014, the number of portal accounts increased from 633 to 17,128. 15.9% of pediatric patients had their own account; 93.6%, a surrogate account; and 2.2% a delegate account. During the study period, 15,711 unique users initiated 493,753 sessions and accessed 1,491,237 functions. Most commonly used functions were secure messaging (accessed in 309,204 sessions; 62.6%); test results (174,239; 35.3%) and appointments (104,830; 21.2%). Function usage was greatest for patients ages 0-2 years (136,245 functions accessed; 23.1%) and 15-17 years (109,241;18.5%). Surrogate users conducted 83.2% of logins for adolescent patients. Portal accounts were actively used for < 1 year for 9,551 patients (55.8%), 1-2 years for 2,826 patients (16.5%), 2-3 years for 1,968 patients (11.5%) and over 3 years for 2,783 patients (16.3%). CONCLUSION: Pediatric patients and caregivers have avidly used messaging, test result, and appointment functions. The majority of access was done by surrogates. Adolescent portal usage increased with age. Most accounts for pediatric patients were only used actively for a few years, with peak usage for patients in early childhood and late adolescence.

Calibration drift in regression and machine learning models for acute kidney injury.

OBJECTIVE: Predictive analytics create opportunities to incorporate personalized risk estimates into clinical decision support. Models must be well calibrated to support decision-making, yet calibration deteriorates over time. This study explored the influence of modeling methods on performance drift and connected observed drift with data shifts in the patient population. MATERIALS AND METHODS: Using 2003 admissions to Department of Veterans Affairs hospitals nationwide, we developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years. RESULTS: Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration across ranges of probability, capturing more admissions than did the regression models. The magnitude of overprediction increased over time for the regression models while remaining stable and small for the machine learning models. Changes in the rate of acute kidney injury were strongly linked to increasing overprediction, while changes in predictor-outcome associations corresponded with diverging patterns of calibration drift across methods. CONCLUSIONS: Efficient and effective updating protocols will be essential for maintaining accuracy of, user confidence in, and safety of personalized risk predictions to support decision-making. Model updating protocols should be tailored to account for variations in calibration drift across methods and respond to periods of rapid performance drift rather than be limited to regularly scheduled annual or biannual intervals.

Calibration Drift Among Regression and Machine Learning Models for Hospital Mortality.

Advanced regression and machine learning models can provide personalized risk predictions to support clinical decision-making. We aimed to understand whether modeling methods impact the tendency of calibration to deteriorate as patient populations shift over time, with the goal of informing model updating practices. We developed models for 30-day hospital mortality using seven common regression and machine learning methods. Models were developed on 2006 admissions to Department of Veterans Affairs hospitals and validated on admissions in 2007-2013. All models maintained discrimination. Calibration was stable for the neural network model and declined for all other models. The L-2 penalized logistic regression and random forest models experienced smaller magnitudes of calibration drift than the other regression models. Calibration drift was linked with a changing case mix rather than shifts in predictoroutcome associations or outcome rate. Model updating protocols will need to be tailored to variations in calibration drift across methods.

Adoption of Secure Messaging in a Patient Portal across Pediatric Specialties.

Few studies have explored adoption of patient portals for pediatric patients outside primary care or disease-specific applications. We examined use of patient-provider messaging in a patient portal across pediatric specialties during the three years after implementation of pediatric portal accounts at Vanderbilt University Medical Center. We determined the number of patient-initiated message threads and clinic visits for pediatric specialties and percentage of these outpatient interactions (i.e., message threads + clinic visits) done through messaging. Generalized estimating equations measured the likelihood of message-based interaction. During the study period, pediatric families initiated 33,503 messages and participated in 318,386 clinic visits. The number of messages sent (and messaging percentage of outpatient interaction) increased each year from 2,860 (2.7%) to 18,772 (17%). Primary care received 4,368 messages (3.4% of outpatient interactions); pediatric subspecialties, 29,135 (13.0%). Rapid growth in messaging volume over time was seen in primary care and most pediatric specialties (OR>1.0; p<0.05).

Use of a Patient Portal During Hospital Admissions to Surgical Services.

Patient portal research has focused on medical outpatient settings, with little known about portal use during hospitalizations or by surgical patients. We measured portal adoption among patients admitted to surgical services over two years. Surgical services managed 37,025 admissions of 31,310 unique patients. One-fourth of admissions (9,362, 25.3%) involved patients registered for the portal. Registration rates were highest for admissions to laparoscopic/gastrointestinal (55%) and oncology/endocrine (50%) services. Portal use occurred during 1,486 surgical admissions, 4% of all and 16% of those registered at admission. Inpatient portal use was associated with patients who were white, male, and had longer lengths of stay (p < 0.01). Viewing health record data and secure messaging were the most commonly used functions, accessed in 4,836 (72.9%) and 1,626 (24.5%) user sessions. Without specific encouragement, hospitalized surgical patients are using our patient portal. The surgical inpatient setting may provide opportunities for patient engagement using patient portals.

Rapid growth in surgeons' use of secure messaging in a patient portal.

BACKGROUND: Use of secure messaging through patient portals has risen substantially in recent years due to provider incentives and consumer demand. Secure messaging may increase patient satisfaction and improve outcomes, but also adds to physician workload. Most prior studies of secure messaging focused on primary care and medical specialties. We examined surgeons' use of secure messaging and the contribution of messaging to outpatient interactions in a broadly deployed patient portal. METHODS: We determined the number of clinic visits and secure messages for surgical providers in the first 3 years (2008-2010) after patient portal deployment at an academic medical center. We calculated the proportion of outpatient interaction conducted through messaging for each specialty. Logistic regression models compared the likelihood of message-based versus clinic outpatient interaction across surgical specialties. RESULTS: Over the study period, surgical providers delivered care in 648,200 clinic visits and received 83,912 messages, with more than 200% growth in monthly message volume. Surgical specialties receiving the most messages were orthopedics/podiatry (25.1%), otolaryngology (20.1%), urology (10.8%), and general surgery (9.6%); vascular surgery (0.8%) and pediatric general surgery (0.2%) received the fewest. The proportion of outpatient interactions conducted through secure messaging increased significantly from 5.4% in 2008 to 15.3% in 2010 (p < 0.001) with all specialties experiencing growth. Heart/lung transplantation (74.9%), liver/kidney/pancreas transplantation (69.5%), and general surgery (48.7%) had the highest proportion of message-based outpatient interaction by the end of the study. CONCLUSIONS: This study demonstrates rapid adoption of online secure messaging across surgical specialties with significant growth in its use for outpatient interaction. Some specialties, particularly those with long-term follow-up, interacted with patients more through secure messaging than in person. As surgeons devote more time to secure messaging, additional research will be needed to understand the care delivered through online interactions and to develop models for reimbursement.

Health Literacy, Education Levels, and Patient Portal Usage During Hospitalizations.

Patient portal adoption has rapidly increased, and portal usage has been associated with patients' sociodemographics, health literacy, and education. Research on patient portals has primarily focused on the outpatient setting. We explored whether health literacy and education were associated with portal usage in an inpatient population. Among 60,159 admissions in 2012-2013, 23.3% of patients reported limited health literacy; 50.4% reported some post-secondary education; 34.4% were registered for the portal; and 23.4% of registered patients used the portal during hospitalization. Probability of registration and inpatient portal use increased with educational attainment. Health literacy was associated with registration but not inpatient use. Among admissions with inpatient use, educational attainment was associated with viewing health record data, and health literacy was associated use of appointment and health education tools. The inpatient setting may provide an opportunity to overcome barriers to patient portal adoption and reduce disparities in use of health information technologies.