Towards global model generalizability: independent cross-site feature evaluation for patient-level risk prediction models using the OHDSI network.

BACKGROUND: Predictive models show promise in healthcare, but their successful deployment is challenging due to limited generalizability. Current external validation often focuses on model performance with restricted feature use from the original training data, lacking insights into their suitability at external sites. Our study introduces an innovative methodology for evaluating features during both the development phase and the validation, focusing on creating and validating predictive models for post-surgery patient outcomes with improved generalizability. METHODS: Electronic health records (EHRs) from 4 countries (United States, United Kingdom, Finland, and Korea) were mapped to the OMOP Common Data Model (CDM), 2008-2019. Machine learning (ML) models were developed to predict post-surgery prolonged opioid use (POU) risks using data collected 6 months before surgery. Both local and cross-site feature selection methods were applied in the development and external validation datasets. Models were developed using Observational Health Data Sciences and Informatics (OHDSI) tools and validated on separate patient cohorts. RESULTS: Model development included 41 929 patients, 14.6% with POU. The external validation included 31 932 (UK), 23 100 (US), 7295 (Korea), and 3934 (Finland) patients with POU of 44.2%, 22.0%, 15.8%, and 21.8%, respectively. The top-performing model, Lasso logistic regression, achieved an area under the receiver operating characteristic curve (AUROC) of 0.75 during local validation and 0.69 (SD = 0.02) (averaged) in external validation. Models trained with cross-site feature selection significantly outperformed those using only features from the development site through external validation (P < .05). CONCLUSIONS: Using EHRs across four countries mapped to the OMOP CDM, we developed generalizable predictive models for POU. Our approach demonstrates the significant impact of cross-site feature selection in improving model performance, underscoring the importance of incorporating diverse feature sets from various clinical settings to enhance the generalizability and utility of predictive healthcare models.

Machine learning for improving high-dimensional proxy confounder adjustment in healthcare database studies: An overview of the current literature.

PURPOSE: Supplementing investigator-specified variables with large numbers of empirically identified features that collectively serve as 'proxies' for unspecified or unmeasured factors can often improve confounding control in studies utilizing administrative healthcare databases. Consequently, there has been a recent focus on the development of data-driven methods for high-dimensional proxy confounder adjustment in pharmacoepidemiologic research. In this paper, we survey current approaches and recent advancements for high-dimensional proxy confounder adjustment in healthcare database studies. METHODS: We discuss considerations underpinning three areas for high-dimensional proxy confounder adjustment: (1) feature generation-transforming raw data into covariates (or features) to be used for proxy adjustment; (2) covariate prioritization, selection, and adjustment; and (3) diagnostic assessment. We discuss challenges and avenues of future development within each area. RESULTS: There is a large literature on methods for high-dimensional confounder prioritization/selection, but relatively little has been written on best practices for feature generation and diagnostic assessment. Consequently, these areas have particular limitations and challenges. CONCLUSIONS: There is a growing body of evidence showing that machine-learning algorithms for high-dimensional proxy-confounder adjustment can supplement investigator-specified variables to improve confounding control compared to adjustment based on investigator-specified variables alone. However, more research is needed on best practices for feature generation and diagnostic assessment when applying methods for high-dimensional proxy confounder adjustment in pharmacoepidemiologic studies.

How the weather affects the pain of citizen scientists using a smartphone app.

Patients with chronic pain commonly believe their pain is related to the weather. Scientific evidence to support their beliefs is inconclusive, in part due to difficulties in getting a large dataset of patients frequently recording their pain symptoms during a variety of weather conditions. Smartphones allow the opportunity to collect data to overcome these difficulties. Our study Cloudy with a Chance of Pain analysed daily data from 2658 patients collected over a 15-month period. The analysis demonstrated significant yet modest relationships between pain and relative humidity, pressure and wind speed, with correlations remaining even when accounting for mood and physical activity. This research highlights how citizen-science experiments can collect large datasets on real-world populations to address long-standing health questions. These results will act as a starting point for a future system for patients to better manage their health through pain forecasts.

Integrated multisystem analysis in a mental health and criminal justice ecosystem.

BACKGROUND: Patients with a serious mental illness often receive care that is fragmented due to reduced availability of or access to resources, and inadequate, discontinuous, and uncoordinated care across health, social services, and criminal justice organizations. This article describes the creation of a multisystem analysis that derives insights from an integrated dataset including patient access to case management services, medical services, and interactions with the criminal justice system. METHODS: Data were combined from electronic systems within a US mental health ecosystem that included mental health and substance abuse services, as well as data from the criminal justice system. Cox models were applied to test the associations between delivery of services and re-incarceration. Additionally, machine learning was used to train and validate a predictive model to examine effects of non-modifiable risk factors (age, past arrests, mental health diagnosis) and modifiable risk factors (outpatient, medical and case management services, and use of a jail diversion program) on re-arrest outcome. RESULTS: An association was found between past arrests and admission to crisis stabilization services in this population (N = 10,307). Delivery of case management or medical services provided after release from jail was associated with a reduced risk for re-arrest. Predictive models linked non-modifiable and modifiable risk factors and outcomes and predicted the probability of re-arrests with fair accuracy (area under the receiver operating characteristic curve of 0.67). CONCLUSIONS: By modeling the complex interactions between risk factors, service delivery, and outcomes, systems of care might be better enabled to meet patient needs and improve outcomes.

Paradoxical Hypersusceptibility of Drug-resistant Mycobacteriumtuberculosis to ß-lactam Antibiotics.

Mycobacterium tuberculosis (M. tuberculosis) is considered innately resistant to ß-lactam antibiotics. However, there is evidence that susceptibility to ß-lactam antibiotics in combination with ß-lactamase inhibitors is variable among clinical isolates, and these may present therapeutic options for drug-resistant cases. Here we report our investigation of susceptibility to ß-lactam/ß-lactamase inhibitor combinations among clinical isolates of M. tuberculosis, and the use of comparative genomics to understand the observed heterogeneity in susceptibility. Eighty-nine South African clinical isolates of varying first and second-line drug susceptibility patterns and two reference strains of M. tuberculosis underwent minimum inhibitory concentration (MIC) determination to two ß-lactams: amoxicillin and meropenem, both alone and in combination with clavulanate, a ß-lactamase inhibitor. 41/91 (45%) of tested isolates were found to be hypersusceptible to amoxicillin/clavulanate relative to reference strains, including 14/24 (58%) of multiple drug-resistant (MDR) and 22/38 (58%) of extensively drug-resistant (XDR) isolates. Genome-wide polymorphisms identified using whole-genome sequencing were used in a phylogenetically-aware linear mixed model to identify polymorphisms associated with amoxicillin/clavulanate susceptibility. Susceptibility to amoxicillin/clavulanate was over-represented among isolates within a specific clade (LAM4), in particular among XDR strains. Twelve sets of polymorphisms were identified as putative markers of amoxicillin/clavulanate susceptibility, five of which were confined solely to LAM4. Within the LAM4 clade, 'paradoxical hypersusceptibility' to amoxicillin/clavulanate has evolved in parallel to first and second-line drug resistance. Given the high prevalence of LAM4 among XDR TB in South Africa, our data support an expanded role for ß-lactam/ß-lactamase inhibitor combinations for treatment of drug-resistant M. tuberculosis.

Integrated multisystem analysis in a mental health and criminal justice ecosystem.

Patients with a serious mental illness often receive care that is fragmented due to reduced availability of or access to resources, and inadequate, discontinuous, and uncoordinated care across health, social services, and criminal justice organizations. These gaps in care may lead to increased mental health disease burden and relapse, as well as repeated incarcerations. Further, the complex health, social service, and criminal justice ecosystem within which the patient may be embedded makes it difficult to examine the role of modifiable risk factors and delivered services on patient outcomes, particularly given that agencies often maintain isolated sets of relevant data. Here we describe an approach to creating a multisystem analysis that derives insights from an integrated data set including patient access to case management services, medical services, and interactions with the criminal justice system. We combined data from electronic systems within a US mental health ecosystem that included mental health and substance abuse services, as well as data from the criminal justice system. We applied Cox models to test the associations between delivery of services and re-incarceration. Using this approach, we found an association between arrests and crisis stabilization services in this population. We also found that delivery of case management or medical services provided after release from jail was associated with a reduced risk for re-arrest. Additionally, we used machine learning to train and validate a predictive model linking non-modifiable and modifiable risk factors and outcomes. A predictive model, constructed using elastic net regularized logistic regression, and considering age, past arrests, mental health diagnosis, as well as use of a jail diversion program, outpatient, medical and case management services predicted the probability of re-arrests with fair accuracy (AUC=.67). By modeling the complex interactions between risk factors, service delivery and outcomes, we may better enable systems of care to meet patient needs and improve outcomes.

Predictive models for type 2 diabetes onset in middle-aged subjects with the metabolic syndrome.

OBJECTIVE: To investigate the predictive value of different biomarkers for the incidence of type 2 diabetes mellitus (T2DM) in subjects with metabolic syndrome. METHODS: A prospective study of 525 non-diabetic, middle-aged Lithuanian men and women with metabolic syndrome but without overt atherosclerotic diseases during a follow-up period of two to four years. We used logistic regression to develop predictive models for incident cases and to investigate the association between various markers and the onset of T2DM. RESULTS: Fasting plasma glucose (FPG), body mass index (BMI), and glycosylated haemoglobin can be used to predict diabetes onset with a high level of accuracy and each was shown to have a cumulative predictive value. The estimated area under the receiver-operating characteristic curve (AUC) for this combination was 0.92. The oral glucose tolerance test (OGTT) did not show cumulative predictive value. Additionally, progression to diabetes was associated with high values of aortic pulse-wave velocity (aPWV). CONCLUSION: T2DM onset in middle-aged metabolic syndrome subjects can be predicted with remarkable accuracy using the combination of FPG, BMI, and HbA1c, and is related to elevated aPWV measurements.

Information-based sequential selection of clinical tests in risk assessment.

We present a new framework for supporting decisions in sequential clinical risk assessment examinations. In this framework, the decision whether to perform a test depends on its expected contribution to risk assessment, given results of previous tests, and the contribution is quantified using information theory. In many cases adding an additional examination clearly improves the predictive model. However, there are cases in which the improvement is not constant for all values of previous tests, and quantification of possible improvement can support decision on further examinations. Using this approach can prevent many expensive, unpleasant or risky examinations. We demonstrate the use of this method on an example of type 2 diabetes onset study. The results show that reducing a considerable percent of the blood tests does not decrease the model's prediction power.