Simulating complex patient populations with hierarchical learning effects to support methods development for post-market surveillance.

BACKGROUND: Validating new algorithms, such as methods to disentangle intrinsic treatment risk from risk associated with experiential learning of novel treatments, often requires knowing the ground truth for data characteristics under investigation. Since the ground truth is inaccessible in real world data, simulation studies using synthetic datasets that mimic complex clinical environments are essential. We describe and evaluate a generalizable framework for injecting hierarchical learning effects within a robust data generation process that incorporates the magnitude of intrinsic risk and accounts for known critical elements in clinical data relationships. METHODS: We present a multi-step data generating process with customizable options and flexible modules to support a variety of simulation requirements. Synthetic patients with nonlinear and correlated features are assigned to provider and institution case series. The probability of treatment and outcome assignment are associated with patient features based on user definitions. Risk due to experiential learning by providers and/or institutions when novel treatments are introduced is injected at various speeds and magnitudes. To further reflect real-world complexity, users can request missing values and omitted variables. We illustrate an implementation of our method in a case study using MIMIC-III data for reference patient feature distributions. RESULTS: Realized data characteristics in the simulated data reflected specified values. Apparent deviations in treatment effects and feature distributions, though not statistically significant, were most common in small datasets (n < 3000) and attributable to random noise and variability in estimating realized values in small samples. When learning effects were specified, synthetic datasets exhibited changes in the probability of an adverse outcomes as cases accrued for the treatment group impacted by learning and stable probabilities as cases accrued for the treatment group not affected by learning. CONCLUSIONS: Our framework extends clinical data simulation techniques beyond generation of patient features to incorporate hierarchical learning effects. This enables the complex simulation studies required to develop and rigorously test algorithms developed to disentangle treatment safety signals from the effects of experiential learning. By supporting such efforts, this work can help identify training opportunities, avoid unwarranted restriction of access to medical advances, and hasten treatment improvements.

Predicting 30-Day Hospital Readmission Risk in a National Cohort of Patients with Cirrhosis.

BACKGROUND: Early hospital readmission for patients with cirrhosis continues to challenge the healthcare system. Risk stratification may help tailor resources, but existing models were designed using small, single-institution cohorts or had modest performance. AIMS: We leveraged a large clinical database from the Department of Veterans Affairs (VA) to design a readmission risk model for patients hospitalized with cirrhosis. Additionally, we analyzed potentially modifiable or unexplored readmission risk factors. METHODS: A national VA retrospective cohort of patients with a history of cirrhosis hospitalized for any reason from January 1, 2006, to November 30, 2013, was developed from 123 centers. Using 174 candidate variables within demographics, laboratory results, vital signs, medications, diagnoses and procedures, and healthcare utilization, we built a 47-variable penalized logistic regression model with the outcome of all-cause 30-day readmission. We excluded patients who left against medical advice, transferred to a non-VA facility, or if the hospital length of stay was greater than 30 days. We evaluated calibration and discrimination across variable volume and compared the performance to recalibrated preexisting risk models for readmission. RESULTS: We analyzed 67,749 patients and 179,298 index hospitalizations. The 30-day readmission rate was 23%. Ascites was the most common cirrhosis-related cause of index hospitalization and readmission. The AUC of the model was 0.670 compared to existing models (0.649, 0.566, 0.577). The Brier score of 0.165 showed good calibration. CONCLUSION: Our model achieved better discrimination and calibration compared to existing models, even after local recalibration. Assessment of calibration by variable parsimony revealed performance improvements for increasing variable inclusion well beyond those detectable for discrimination.

A clinical risk prediction model to identify patients with hepatorenal syndrome at hospital admission.

BACKGROUND: Hepatorenal syndrome (HRS) is a life-threatening complication of cirrhosis and early detection of evolving HRS may provide opportunities for early intervention. We developed a HRS risk model to assist early recognition of inpatient HRS. METHODS: We analysed a retrospective cohort of patients hospitalised from among 122 medical centres in the US Department of Veterans Affairs between 1 January 2005 and 31 December 2013. We included cirrhotic patients who had Kidney Disease Improving Global Outcomes criteria based acute kidney injury on admission. We developed a logistic regression risk prediction model to detect HRS on admission using 10 variables. We calculated 95% confidence intervals on the model building dataset and, subsequently, calculated performance on a 1000 sample holdout test set. We report model performance with area under the curve (AUC) for discrimination and several calibration measures. RESULTS: The cohort included 19 368 patients comprising 32 047 inpatient admissions. The event rate for hospitalised HRS was 2810/31 047 (9.1%) and 79/1000 (7.9%) in the model building and validation datasets, respectively. The variable selection procedure designed a parsimonious model involving ten predictor variables. Final model performance in the validation dataset had an AUC of 0.87, Brier score of 0.05, slope of 1.10 and intercept of 0.04. CONCLUSIONS: We developed a probabilistic risk model to diagnose HRS within 24 hours of hospital admission using routine clinical variables in the largest ever published HRS cohort. The performance was excellent and this model may help identify high-risk patients for HRS and promote early intervention.

Development of an automated phenotyping algorithm for hepatorenal syndrome.

OBJECTIVE: Hepatorenal Syndrome (HRS) is a devastating form of acute kidney injury (AKI) in advanced liver disease patients with high morbidity and mortality, but phenotyping algorithms have not yet been developed using large electronic health record (EHR) databases. We evaluated and compared multiple phenotyping methods to achieve an accurate algorithm for HRS identification. MATERIALS AND METHODS: A national retrospective cohort of patients with cirrhosis and AKI admitted to 124 Veterans Affairs hospitals was assembled from electronic health record data collected from 2005 to 2013. AKI was defined by the Kidney Disease: Improving Global Outcomes criteria. Five hundred and four hospitalizations were selected for manual chart review and served as the gold standard. Electronic Health Record based predictors were identified using structured and free text clinical data, subjected through NLP from the clinical Text Analysis Knowledge Extraction System. We explored several dimension reduction techniques for the NLP data, including newer high-throughput phenotyping and word embedding methods, and ascertained their effectiveness in identifying the phenotype without structured predictor variables. With the combined structured and NLP variables, we analyzed five phenotyping algorithms: penalized logistic regression, naïve Bayes, support vector machines, random forest, and gradient boosting. Calibration and discrimination metrics were calculated using 100 bootstrap iterations. In the final model, we report odds ratios and 95% confidence intervals. RESULTS: The area under the receiver operating characteristic curve (AUC) for the different models ranged from 0.73 to 0.93; with penalized logistic regression having the best discriminatory performance. Calibration for logistic regression was modest, but gradient boosting and support vector machines were superior. NLP identified 6985 variables; a priori variable selection performed similarly to dimensionality reduction using high-throughput phenotyping and semantic similarity informed clustering (AUC of 0.81 - 0.82). CONCLUSION: This study demonstrated improved phenotyping of a challenging AKI etiology, HRS, over ICD-9 coding. We also compared performance among multiple approaches to EHR-derived phenotyping, and found similar results between methods. Lastly, we showed that automated NLP dimension reduction is viable for acute illness.

Clinical Characteristics and Outcomes of Drug-Induced Acute Kidney Injury Cases.

Introduction: Drug-induced acute kidney injury (DI-AKI) is a frequent adverse event. The identification of DI-AKI is challenged by competing etiologies, clinical heterogeneity among patients, and a lack of accurate diagnostic tools. Our research aims to describe the clinical characteristics and predictive variables of DI-AKI. Methods: We analyzed data from the Drug-Induced Renal Injury Consortium (DIRECT) study (NCT02159209), an international, multicenter, observational cohort study of enriched clinically adjudicated DI-AKI cases. Cases met the primary inclusion criteria if the patient was exposed to at least 1 nephrotoxic drug for a minimum of 24 hours prior to AKI onset. Cases were clinically adjudicated, and inter-rater reliability (IRR) was measured using Krippendorff's alpha. Variables associated with DI-AKI were identified using L1 regularized multivariable logistic regression. Model performance was assessed using the area under the receiver operating characteristic curve (ROC AUC). Results: A total of 314 AKI cases met the eligibility criteria for this analysis, and 271 (86%) cases were adjudicated as DI-AKI. The majority of the AKI cases were recruited from the United States (68%). The most frequent causal nephrotoxic drugs were vancomycin (48.7%), nonsteroidal antiinflammatory drugs (18.2%), and piperacillin/tazobactam (17.8%). The IRR for DI-AKI adjudication was 0.309. The multivariable model identified age, vascular capacity, hyperglycemia, infections, pyuria, serum creatinine (SCr) trends, and contrast media as significant predictors of DI-AKI with good performance (ROC AUC 0.86). Conclusion: The identification of DI-AKI is challenging even with comprehensive adjudication by experienced nephrologists. Our analysis identified key clinical characteristics and outcomes of DI-AKI compared to other AKI etiologies.

Application of contextual design methods to inform targeted clinical decision support interventions in sub-specialty care environments.

BACKGROUND & OBJECTIVES: In healthcare, the routine use of evidence-based specialty care management plans is mixed. Targeted computerized clinical decision support (CCDS) interventions can improve physician adherence, but adoption depends on CCDS' 'fit' within clinical work. We analyzed clinical work in outpatient and inpatient settings as a basis for developing guidelines for optimizing CCDS design. METHODS: The contextual design approach guided data collection, collation and analysis. Forty (40) consenting physicians were observed and interviewed in general internal medicine inpatient units and gastroenterology (GI) outpatient clinics at two academic medical centers. Data were collated using interpretive debriefing, and consolidated using thematic analysis and three work modeling approaches (communication flow, sequence and artifact models). RESULTS: Twenty-six consenting physicians were observed at Site A and 14 at Site B. Observations included attending (33%) and resident physicians. During research team debriefings, 220 of 341 unique topics were categorized into 5 CCDS-relevant themes. Resident physicians relied on patient assessment & planning processes to support their roles as communication and coordination hubs within the medical team. Artifact analysis further elucidated the evolution of assessment and planning over work shifts. CONCLUSIONS: The usefulness of CCDS tools may be enhanced in clinical care if the design: 1) accounts for clinical work that is distributed across people, space, and time; 2) targets communication and coordination hubs (specific roles) that can amplify the usefulness of CCDS interventions; 3) integrates CCDS with early clinical assessment & planning processes; and 4) provides CCDS in both electronic & hardcopy formats. These requirements provide a research agenda for future research in clinician-CCDS integration.