Privacy-Preserving Federated Model Predicting Bipolar Transition in Patients With Depression: Prediction Model Development Study.

BACKGROUND: Mood disorder has emerged as a serious concern for public health; in particular, bipolar disorder has a less favorable prognosis than depression. Although prompt recognition of depression conversion to bipolar disorder is needed, early prediction is challenging due to overlapping symptoms. Recently, there have been attempts to develop a prediction model by using federated learning. Federated learning in medical fields is a method for training multi-institutional machine learning models without patient-level data sharing. OBJECTIVE: This study aims to develop and validate a federated, differentially private multi-institutional bipolar transition prediction model. METHODS: This retrospective study enrolled patients diagnosed with the first depressive episode at 5 tertiary hospitals in South Korea. We developed models for predicting bipolar transition by using data from 17,631 patients in 4 institutions. Further, we used data from 4541 patients for external validation from 1 institution. We created standardized pipelines to extract large-scale clinical features from the 4 institutions without any code modification. Moreover, we performed feature selection in a federated environment for computational efficiency and applied differential privacy to gradient updates. Finally, we compared the federated and the 4 local models developed with each hospital's data on internal and external validation data sets. RESULTS: In the internal data set, 279 out of 17,631 patients showed bipolar disorder transition. In the external data set, 39 out of 4541 patients showed bipolar disorder transition. The average performance of the federated model in the internal test (area under the curve [AUC] 0.726) and external validation (AUC 0.719) data sets was higher than that of the other locally developed models (AUC 0.642-0.707 and AUC 0.642-0.699, respectively). In the federated model, classifications were driven by several predictors such as the Charlson index (low scores were associated with bipolar transition, which may be due to younger age), severe depression, anxiolytics, young age, and visiting months (the bipolar transition was associated with seasonality, especially during the spring and summer months). CONCLUSIONS: We developed and validated a differentially private federated model by using distributed multi-institutional psychiatric data with standardized pipelines in a real-world environment. The federated model performed better than models using local data only.

External validation of existing dementia prediction models on observational health data.

BACKGROUND: Many dementia prediction models have been developed, but only few have been externally validated, which hinders clinical uptake and may pose a risk if models are applied to actual patients regardless. Externally validating an existing prediction model is a difficult task, where we mostly rely on the completeness of model reporting in a published article. In this study, we aim to externally validate existing dementia prediction models. To that end, we define model reporting criteria, review published studies, and externally validate three well reported models using routinely collected health data from administrative claims and electronic health records. METHODS: We identified dementia prediction models that were developed between 2011 and 2020 and assessed if they could be externally validated given a set of model criteria. In addition, we externally validated three of these models (Walters' Dementia Risk Score, Mehta's RxDx-Dementia Risk Index, and Nori's ADRD dementia prediction model) on a network of six observational health databases from the United States, United Kingdom, Germany and the Netherlands, including the original development databases of the models. RESULTS: We reviewed 59 dementia prediction models. All models reported the prediction method, development database, and target and outcome definitions. Less frequently reported by these 59 prediction models were predictor definitions (52 models) including the time window in which a predictor is assessed (21 models), predictor coefficients (20 models), and the time-at-risk (42 models). The validation of the model by Walters (development c-statistic: 0.84) showed moderate transportability (0.67-0.76 c-statistic). The Mehta model (development c-statistic: 0.81) transported well to some of the external databases (0.69-0.79 c-statistic). The Nori model (development AUROC: 0.69) transported well (0.62-0.68 AUROC) but performed modestly overall. Recalibration showed improvements for the Walters and Nori models, while recalibration could not be assessed for the Mehta model due to unreported baseline hazard. CONCLUSION: We observed that reporting is mostly insufficient to fully externally validate published dementia prediction models, and therefore, it is uncertain how well these models would work in other clinical settings. We emphasize the importance of following established guidelines for reporting clinical prediction models. We recommend that reporting should be more explicit and have external validation in mind if the model is meant to be applied in different settings.

Seek COVER: using a disease proxy to rapidly develop and validate a personalized risk calculator for COVID-19 outcomes in an international network.

BACKGROUND: We investigated whether we could use influenza data to develop prediction models for COVID-19 to increase the speed at which prediction models can reliably be developed and validated early in a pandemic. We developed COVID-19 Estimated Risk (COVER) scores that quantify a patient's risk of hospital admission with pneumonia (COVER-H), hospitalization with pneumonia requiring intensive services or death (COVER-I), or fatality (COVER-F) in the 30-days following COVID-19 diagnosis using historical data from patients with influenza or flu-like symptoms and tested this in COVID-19 patients. METHODS: We analyzed a federated network of electronic medical records and administrative claims data from 14 data sources and 6 countries containing data collected on or before 4/27/2020. We used a 2-step process to develop 3 scores using historical data from patients with influenza or flu-like symptoms any time prior to 2020. The first step was to create a data-driven model using LASSO regularized logistic regression, the covariates of which were used to develop aggregate covariates for the second step where the COVER scores were developed using a smaller set of features. These 3 COVER scores were then externally validated on patients with 1) influenza or flu-like symptoms and 2) confirmed or suspected COVID-19 diagnosis across 5 databases from South Korea, Spain, and the United States. Outcomes included i) hospitalization with pneumonia, ii) hospitalization with pneumonia requiring intensive services or death, and iii) death in the 30 days after index date. RESULTS: Overall, 44,507 COVID-19 patients were included for model validation. We identified 7 predictors (history of cancer, chronic obstructive pulmonary disease, diabetes, heart disease, hypertension, hyperlipidemia, kidney disease) which combined with age and sex discriminated which patients would experience any of our three outcomes. The models achieved good performance in influenza and COVID-19 cohorts. For COVID-19 the AUC ranges were, COVER-H: 0.69-0.81, COVER-I: 0.73-0.91, and COVER-F: 0.72-0.90. Calibration varied across the validations with some of the COVID-19 validations being less well calibrated than the influenza validations. CONCLUSIONS: This research demonstrated the utility of using a proxy disease to develop a prediction model. The 3 COVER models with 9-predictors that were developed using influenza data perform well for COVID-19 patients for predicting hospitalization, intensive services, and fatality. The scores showed good discriminatory performance which transferred well to the COVID-19 population. There was some miscalibration in the COVID-19 validations, which is potentially due to the difference in symptom severity between the two diseases. A possible solution for this is to recalibrate the models in each location before use.

Development of multivariable models to predict perinatal depression before and after delivery using patient reported survey responses at weeks 4-10 of pregnancy.

BACKGROUND: Perinatal depression is estimated to affect ~ 12% of pregnancies and is linked to numerous negative outcomes. There is currently no model to predict perinatal depression at multiple time-points during and after pregnancy using variables ascertained early into pregnancy. METHODS: A prospective cohort design where 858 participants filled in a baseline self-reported survey at week 4-10 of pregnancy (that included social economics, health history, various psychiatric measures), with follow-up until 3 months after delivery. Our primary outcome was an Edinburgh Postnatal Depression Score (EPDS) score of 12 or more (a proxy for perinatal depression) assessed during each trimester and again at two time periods after delivery. Five gradient boosting machines were trained to predict the risk of having EPDS score > = 12 at each of the five follow-up periods. The predictors consisted of 21 variables from 3 validated psychometric scales. As a sensitivity analysis, we also investigated different predictor sets that contained: i) 17 of the 21 variables predictors by only including two of the psychometric scales and ii) including 143 additional social economics and health history predictors, resulting in 164 predictors. RESULTS: We developed five prognostic models: PND-T1 (trimester 1), PND-T2 (trimester 2), PND-T3 (trimester 3), PND-A1 (after delivery 1) and PND-A2 (delayed onset after delivery) that calculate personalised risks while only requiring that women be asked 21 questions from 3 validated psychometric scales at weeks 4-10 of pregnancy. C-statistics (also known as AUC) ranged between 0.69 (95% CI 0.65-0.73) and 0.77 (95% CI 0.74-0.80). At 50% sensitivity the positive predictive value ranged between 30%-50% across the models, generally identifying groups of patients with double the average risk. Models trained using the 17 predictors and 164 predictors did not improve model performance compared to the models trained using 21 predictors. CONCLUSIONS: The five models can predict risk of perinatal depression within each trimester and in two post-natal periods using survey responses as early as week 4 of pregnancy with modest performance. The models need to be externally validated and prospectively tested to ensure generalizability to any pregnant patient.

Adaptation and validation of a coding algorithm for the Charlson Comorbidity Index in administrative claims data using the SNOMED CT standardized vocabulary.

OBJECTIVES: The Charlson comorbidity index (CCI), the most ubiquitous comorbid risk score, predicts one-year mortality among hospitalized patients and provides a single aggregate measure of patient comorbidity. The Quan adaptation of the CCI revised the CCI coding algorithm for applications to administrative claims data using the International Classification of Diseases (ICD). The purpose of the current study is to adapt and validate a coding algorithm for the CCI using the SNOMED CT standardized vocabulary, one of the most commonly used vocabularies for data collection in healthcare databases in the U.S. METHODS: The SNOMED CT coding algorithm for the CCI was adapted through the direct translation of the Quan coding algorithms followed by manual curation by clinical experts. The performance of the SNOMED CT and Quan coding algorithms were compared in the context of a retrospective cohort study of inpatient visits occurring during the calendar years of 2013 and 2018 contained in two U.S. administrative claims databases. Differences in the CCI or frequency of individual comorbid conditions were assessed using standardized mean differences (SMD). Performance in predicting one-year mortality among hospitalized patients was measured based on the c-statistic of logistic regression models. RESULTS: For each database and calendar year combination, no significant differences in the CCI or frequency of individual comorbid conditions were observed between vocabularies (SMD ≤ 0.10). Specifically, the difference in CCI measured using the SNOMED CT vs. Quan coding algorithms was highest in MDCD in 2013 (3.75 vs. 3.6; SMD = 0.03) and lowest in DOD in 2018 (3.93 vs. 3.86; SMD = 0.02). Similarly, as indicated by the c-statistic, there was no evidence of a difference in the performance between coding algorithms in predicting one-year mortality (SNOMED CT vs. Quan coding algorithms, range: 0.725-0.789 vs. 0.723-0.787, respectively). A total of 700 of 5,348 (13.1%) ICD code mappings were inconsistent between coding algorithms. The most common cause of discrepant codes was multiple ICD codes mapping to a SNOMED CT code (n = 560) of which 213 were deemed clinically relevant thereby leading to information gain. CONCLUSION: The current study repurposed an important tool for conducting observational research to use the SNOMED CT standardized vocabulary.

Learning patient-level prediction models across multiple healthcare databases: evaluation of ensembles for increasing model transportability.

BACKGROUND: Prognostic models that are accurate could help aid medical decision making. Large observational databases often contain temporal medical data for large and diverse populations of patients. It may be possible to learn prognostic models using the large observational data. Often the performance of a prognostic model undesirably worsens when transported to a different database (or into a clinical setting). In this study we investigate different ensemble approaches that combine prognostic models independently developed using different databases (a simple federated learning approach) to determine whether ensembles that combine models developed across databases can improve model transportability (perform better in new data than single database models)? METHODS: For a given prediction question we independently trained five single database models each using a different observational healthcare database. We then developed and investigated numerous ensemble models (fusion, stacking and mixture of experts) that combined the different database models. Performance of each model was investigated via discrimination and calibration using a leave one dataset out technique, i.e., hold out one database to use for validation and use the remaining four datasets for model development. The internal validation of a model developed using the hold out database was calculated and presented as the 'internal benchmark' for comparison. RESULTS: In this study the fusion ensembles generally outperformed the single database models when transported to a previously unseen database and the performances were more consistent across unseen databases. Stacking ensembles performed poorly in terms of discrimination when the labels in the unseen database were limited. Calibration was consistently poor when both ensembles and single database models were applied to previously unseen databases. CONCLUSION: A simple federated learning approach that implements ensemble techniques to combine models independently developed across different databases for the same prediction question may improve the discriminative performance in new data (new database or clinical setting) but will need to be recalibrated using the new data. This could help medical decision making by improving prognostic model performance.

90-Day all-cause mortality can be predicted following a total knee replacement: an international, network study to develop and validate a prediction model.

PURPOSE: The purpose of this study was to develop and validate a prediction model for 90-day mortality following a total knee replacement (TKR). TKR is a safe and cost-effective surgical procedure for treating severe knee osteoarthritis (OA). Although complications following surgery are rare, prediction tools could help identify high-risk patients who could be targeted with preventative interventions. The aim was to develop and validate a simple model to help inform treatment choices. METHODS: A mortality prediction model for knee OA patients following TKR was developed and externally validated using a US claims database and a UK general practice database. The target population consisted of patients undergoing a primary TKR for knee OA, aged ≥ 40 years and registered for ≥ 1 year before surgery. LASSO logistic regression models were developed for post-operative (90-day) mortality. A second mortality model was developed with a reduced feature set to increase interpretability and usability. RESULTS: A total of 193,615 patients were included, with 40,950 in The Health Improvement Network (THIN) database and 152,665 in Optum. The full model predicting 90-day mortality yielded AUROC of 0.78 when trained in OPTUM and 0.70 when externally validated on THIN. The 12 variable model achieved internal AUROC of 0.77 and external AUROC of 0.71 in THIN. CONCLUSIONS: A simple prediction model based on sex, age, and 10 comorbidities that can identify patients at high risk of short-term mortality following TKR was developed that demonstrated good, robust performance. The 12-feature mortality model is easily implemented and the performance suggests it could be used to inform evidence based shared decision-making prior to surgery and targeting prophylaxis for those at high risk. LEVEL OF EVIDENCE: III.

Evaluating the impact of covariate lookback times on performance of patient-level prediction models.

BACKGROUND: The goal of our study is to examine the impact of the lookback length when engineering features to use in developing predictive models using observational healthcare data. Using a longer lookback for feature engineering gives more insight about patients but increases the issue of left-censoring. METHODS: We used five US observational databases to develop patient-level prediction models. A target cohort of subjects with hypertensive drug exposures and outcome cohorts of subjects with acute (stroke and gastrointestinal bleeding) and chronic outcomes (diabetes and chronic kidney disease) were developed. Candidate predictors that exist on or prior to the target index date were derived within the following lookback periods: 14, 30, 90, 180, 365, 730, and all days prior to index were evaluated. We predicted the risk of outcomes occurring 1 day until 365 days after index. Ten lasso logistic models for each lookback period were generated to create a distribution of area under the curve (AUC) metrics to evaluate the discriminative performance of the models. Calibration intercept and slope were also calculated. Impact on external validation performance was investigated across five databases. RESULTS: The maximum differences in AUCs for the models developed using different lookback periods within a database was < 0.04 for diabetes (in MDCR AUC of 0.593 with 14-day lookback vs. AUC of 0.631 with all-time lookback) and 0.012 for renal impairment (in MDCR AUC of 0.675 with 30-day lookback vs. AUC of 0.687 with 365-day lookback ). For the acute outcomes, the max difference in AUC across lookbacks within a database was 0.015 (in MDCD AUC of 0.767 with 14-day lookback vs. AUC 0.782 with 365-day lookback) for stroke and < 0.03 for gastrointestinal bleeding (in CCAE AUC of 0.631 with 14-day lookback vs. AUC of 0.660 with 730-day lookback). CONCLUSIONS: In general the choice of covariate lookback had only a small impact on discrimination and calibration, with a short lookback (< 180 days) occasionally decreasing discrimination. Based on the results, if training a logistic regression model for prediction then using covariates with a 365 day lookback appear to be a good tradeoff between performance and interpretation.

Perinatal depressive symptoms often start in the prenatal rather than postpartum period: results from a longitudinal study.

Depressive symptoms during and after pregnancy confer risks for adverse outcomes in both the mother and child. Postpartum depression is traditionally diagnosed after birth of the child. Perinatal depression is a serious, prevalent heterogeneous syndrome that can occur during the period from conception through several months after childbirth. Onset and course are not well understood. There is a paucity of longitudinal studies of the disorder that include the antenatal period in population-based samples. We used an Internet panel of pregnant women recruited in 2 cohorts; 858 ascertained in the first and 322 ascertained in the third trimesters of pregnancy. We recruited the second cohort in order to assure sufficient sample to examine depressive symptoms later in pregnancy and in the postpartum period. Assessments included standard psychometric measures, health history, and pregnancy experience. The Edinburgh Postnatal Depression Scale was used for the assessment of depressive symptoms. Nearly 10% of women entered the pregnancy with depressive symptoms. Prevalence was about the same at 4 weeks and 3 months postpartum. During pregnancy, prevalence increased to 16% in the third trimester. Among incident cases, 80% occurred during pregnancy, with 1/3 occurring in the first trimester. We describe predictors of incident depressive symptoms and covariates associated with time-to-onset which include health history (psychiatric and medical) and social support covariates. The majority of incident depressive symptoms occur during pregnancy rather than afterward. This finding underscores the mandate for mental health screening early in pregnancy and throughout gestation. It will be important to find safe and effective interventions that prevent, mitigate, or delay the onset of depressive symptoms that can be implemented during pregnancy.

An empirical analysis of dealing with patients who are lost to follow-up when developing prognostic models using a cohort design.

BACKGROUND: Researchers developing prediction models are faced with numerous design choices that may impact model performance. One key decision is how to include patients who are lost to follow-up. In this paper we perform a large-scale empirical evaluation investigating the impact of this decision. In addition, we aim to provide guidelines for how to deal with loss to follow-up. METHODS: We generate a partially synthetic dataset with complete follow-up and simulate loss to follow-up based either on random selection or on selection based on comorbidity. In addition to our synthetic data study we investigate 21 real-world data prediction problems. We compare four simple strategies for developing models when using a cohort design that encounters loss to follow-up. Three strategies employ a binary classifier with data that: (1) include all patients (including those lost to follow-up), (2) exclude all patients lost to follow-up or (3) only exclude patients lost to follow-up who do not have the outcome before being lost to follow-up. The fourth strategy uses a survival model with data that include all patients. We empirically evaluate the discrimination and calibration performance. RESULTS: The partially synthetic data study results show that excluding patients who are lost to follow-up can introduce bias when loss to follow-up is common and does not occur at random. However, when loss to follow-up was completely at random, the choice of addressing it had negligible impact on model discrimination performance. Our empirical real-world data results showed that the four design choices investigated to deal with loss to follow-up resulted in comparable performance when the time-at-risk was 1-year but demonstrated differential bias when we looked into 3-year time-at-risk. Removing patients who are lost to follow-up before experiencing the outcome but keeping patients who are lost to follow-up after the outcome can bias a model and should be avoided. CONCLUSION: Based on this study we therefore recommend (1) developing models using data that includes patients that are lost to follow-up and (2) evaluate the discrimination and calibration of models twice: on a test set including patients lost to follow-up and a test set excluding patients lost to follow-up.

Feasibility and evaluation of a large-scale external validation approach for patient-level prediction in an international data network: validation of models predicting stroke in female patients newly diagnosed with atrial fibrillation.

BACKGROUND: To demonstrate how the Observational Healthcare Data Science and Informatics (OHDSI) collaborative network and standardization can be utilized to scale-up external validation of patient-level prediction models by enabling validation across a large number of heterogeneous observational healthcare datasets. METHODS: Five previously published prognostic models (ATRIA, CHADS2, CHADS2VASC, Q-Stroke and Framingham) that predict future risk of stroke in patients with atrial fibrillation were replicated using the OHDSI frameworks. A network study was run that enabled the five models to be externally validated across nine observational healthcare datasets spanning three countries and five independent sites. RESULTS: The five existing models were able to be integrated into the OHDSI framework for patient-level prediction and they obtained mean c-statistics ranging between 0.57-0.63 across the 6 databases with sufficient data to predict stroke within 1 year of initial atrial fibrillation diagnosis for females with atrial fibrillation. This was comparable with existing validation studies. The validation network study was run across nine datasets within 60 days once the models were replicated. An R package for the study was published at https://github.com/OHDSI/StudyProtocolSandbox/tree/master/ExistingStrokeRiskExternalValidation. CONCLUSION: This study demonstrates the ability to scale up external validation of patient-level prediction models using a collaboration of researchers and a data standardization that enable models to be readily shared across data sites. External validation is necessary to understand the transportability or reproducibility of a prediction model, but without collaborative approaches it can take three or more years for a model to be validated by one independent researcher. In this paper we show it is possible to both scale-up and speed-up external validation by showing how validation can be done across multiple databases in less than 2 months. We recommend that researchers developing new prediction models use the OHDSI network to externally validate their models.

Using Machine Learning Applied to Real-World Healthcare Data for Predictive Analytics: An Applied Example in Bariatric Surgery.

OBJECTIVES: Laparoscopic metabolic surgery (MxS) can lead to remission of type 2 diabetes (T2D); however, treatment response to MxS can be heterogeneous. Here, we demonstrate an open-source predictive analytics platform that applies machine-learning techniques to a common data model; we develop and validate a predictive model of antihyperglycemic medication cessation (validated proxy for A1c control) in patients with treated T2D who underwent MxS. METHODS: We selected patients meeting the following criteria in 2 large US healthcare claims databases (Truven Health MarketScan Commercial [CCAE]; Optum Clinformatics [Optum]): underwent MxS between January 1, 2007, to October 1, 2013 (first = index); aged ≥18 years; continuous enrollment 180 days pre-index (baseline) to 730 days postindex; baseline T2D diagnosis and treatment. The outcome was no antihyperglycemic medication treatment from 365 to 730 days after MxS. A regularized logistic regression model was trained using the following candidate predictor categories measured at baseline: demographics, conditions, medications, measurements, and procedures. A 75% to 25% split of the CCAE group was used for model training and testing; the Optum group was used for external validation. RESULTS: 13 050 (CCAE) and 3477 (Optum) patients met the study inclusion criteria. Antihyperglycemic medication cessation rates were 72.9% (CCAE) and 70.8% (Optum). The model possessed good internal discriminative accuracy (area under the curve [AUC] = 0.778 [95% CI = 0.761-0.795] in CCAE test set N = 3527) and transportability (external AUC = 0.759 [95% CI = 0.741-0.777] in Optum N = 3477). CONCLUSION: The application of machine learning techniques to real-world healthcare data can yield useful predictive models to assist patient selection. In future practice, establishment of prerequisite technological infrastructure will be needed to implement such models for real-world decision support.

Supplementing claims data analysis using self-reported data to develop a probabilistic phenotype model for current smoking status.

OBJECTIVES: Smoking status is poorly record in US claims data. IBM MarketScan Commercial is a claims database that can be linked to an additional health risk assessment with self-reported smoking status for a subset of 1,966,174 patients. We investigate whether this subset could be used to learn a smoking status phenotype model generalizable to all US claims data that calculates the probability of being a current smoker. METHODS: 251,643 (12.8%) had self-reported their smoking status as 'current smoker'. A regularized logistic regression model, the Current Risk of Smoking Status (CROSS), was trained using the subset of patients with self-reported smoking status. CROSS considered 53,027 candidate covariates including demographics and conditions/drugs/measurements/procedures/observations recorded in the prior 365 days, The CROSS phenotype model was validated across multiple other claims data. RESULTS: The internal validation showed the CROSS model achieved an area under the receiver operating characteristic curve (AUC) of 0.76 and the calibration plots indicated it was well calibrated. The external validation across three US claims databases obtained AUCs ranging between 0.82 and 0.87 showing the model appears to be transportable across Claims data. CONCLUSION: CROSS predicts current smoking status based on the claims records in the prior year. CROSS can be readily implemented to any US insurance claims mapped to the OMOP common data model and will be a useful way to impute smoking status when conducting epidemiology studies where smoking is a known confounder but smoking status is not recorded. CROSS is available from https://github.com/OHDSI/StudyProtocolSandbox/tree/master/SmokingModel.

Finding factors that predict treatment-resistant depression: Results of a cohort study.

BACKGROUND: Treatment for depressive disorders often requires subsequent interventions. Patients who do not respond to antidepressants have treatment-resistant depression (TRD). Predicting who will develop TRD may help healthcare providers make more effective treatment decisions. We sought to identify factors that predict TRD in a real-world setting using claims databases. METHODS: A retrospective cohort study was conducted in a US claims database of adult subjects with newly diagnosed and treated depression with no mania, dementia, and psychosis. The index date was the date of antidepressant dispensing. The outcome was TRD, defined as having at least three distinct antidepressants or one antidepressant and one antipsychotic within 1 year after the index date. Predictors were age, gender, medical conditions, medications, and procedures 1 year before the index date. RESULTS: Of 230,801 included patients, 10.4% developed TRD within 1 year. TRD patients at baseline were younger; 10.87% were between 18 and 19 years old versus 7.64% in the no-TRD group, risk ratio (RR) = 1.42 (95% confidence interval [CI] 1.37-1.48). TRD patients were more likely to have an anxiety disorder at baseline than non-TRD patients, RR = 1.38 (95% CI 1.35-1.14). At 3.68, fatigue had the highest RR (95% CI 3.18-4.25). TRD patients had substance use disorders, psychiatric conditions, insomnia, and pain more often at baseline than non-TRD patients. CONCLUSION: Ten percent of subjects newly diagnosed and treated for depression developed TRD within a year. They were younger and suffered more frequently from fatigue, substance use disorders, anxiety, psychiatric conditions, insomnia, and pain than non-TRD patients.

Finding treatment-resistant depression in real-world data: How a data-driven approach compares with expert-based heuristics.

BACKGROUND: Depression that does not respond to antidepressants is treatment-resistant depression (TRD). TRD definitions include assessments of treatment response, dose and duration, and implementing these definitions in claims databases can be challenging. We built a data-driven TRD definition and evaluated its performance. METHODS: We included adults with depression, ≥1 antidepressant, and no diagnosis of mania, dementia, or psychosis. Subjects were stratified into those with and without proxy for TRD. Proxies for TRD were electroconvulsive therapy, deep brain, or vagus nerve stimulation. The index date for subjects with proxy for TRD was the procedure date, and for subjects without, the date of a randomly selected visit. We used three databases. We fit decision tree predictive models. We included number of distinct antidepressants, with and without adequate doses and duration, number of antipsychotics and psychotherapies, and expert-based definitions, 3, 6, and 12 months before index date. To assess performance, we calculated area under the curve (AUC) and transportability. RESULTS: We analyzed 33,336 subjects with no proxy for TRD, and 3,566 with the proxy. Number of antidepressants and antipsychotics were selected in all periods. The best model was at 12 months with an AUC = 0.81. The rule transported well and states that a subject with ≥1 antipsychotic or ≥3 antidepressants in the last year has TRD. Applying this rule, 15.8% of subjects treated for depression had TRD. CONCLUSION: The definition that best discriminates between subjects with and without TRD considers number of distinct antidepressants (≥3) or antipsychotics (≥1) in the last year.

A signalome screening approach in the autoinflammatory disease TNF receptor associated periodic syndrome (TRAPS) highlights the anti-inflammatory properties of drugs for repurposing.

TNF receptor associated periodic syndrome (TRAPS) is an autoinflammatory disease caused by mutations in TNF Receptor 1 (TNFR1). Current therapies for TRAPS are limited and do not target the pro-inflammatory signalling pathways that are central to the disease mechanism. Our aim was to identify drugs for repurposing as anti-inflammatories based on their ability to down-regulate molecules associated with inflammatory signalling pathways that are activated in TRAPS. This was achieved using rigorously optimized, high through-put cell culture and reverse phase protein microarray systems to screen compounds for their effects on the TRAPS-associated inflammatory signalome. 1360 approved, publically available, pharmacologically active substances were investigated for their effects on 40 signalling molecules associated with pro-inflammatory signalling pathways that are constitutively upregulated in TRAPS. The drugs were screened at four 10-fold concentrations on cell lines expressing both wild-type (WT) TNFR1 and TRAPS-associated C33Y mutant TNFR1, or WT TNFR1 alone; signalling molecule levels were then determined in cell lysates by the reverse-phase protein microarray. A novel mathematical methodology was developed to rank the compounds for their ability to reduce the expression of signalling molecules in the C33Y-TNFR1 transfectants towards the level seen in the WT-TNFR1 transfectants. Seven high-ranking drugs were selected and tested by RPPA for effects on the same 40 signalling molecules in lysates of peripheral blood mononuclear cells (PBMCs) from C33Y-TRAPS patients compared to PBMCs from normal controls. The fluoroquinolone antibiotic lomefloxacin, as well as others from this class of compounds, showed the most significant effects on multiple pro-inflammatory signalling pathways that are constitutively activated in TRAPS; lomefloxacin dose-dependently significantly reduced expression of 7/40 signalling molecules across the Jak/Stat, MAPK, NF-κB and PI3K/AKT pathways. This study demonstrates the power of signalome screening for identifying candidates for drug repurposing.

A supervised adverse drug reaction signalling framework imitating Bradford Hill's causality considerations.

Big longitudinal observational medical data potentially hold a wealth of information and have been recognised as potential sources for gaining new drug safety knowledge. Unfortunately there are many complexities and underlying issues when analysing longitudinal observational data. Due to these complexities, existing methods for large-scale detection of negative side effects using observational data all tend to have issues distinguishing between association and causality. New methods that can better discriminate causal and non-causal relationships need to be developed to fully utilise the data. In this paper we propose using a set of causality considerations developed by the epidemiologist Bradford Hill as a basis for engineering features that enable the application of supervised learning for the problem of detecting negative side effects. The Bradford Hill considerations look at various perspectives of a drug and outcome relationship to determine whether it shows causal traits. We taught a classifier to find patterns within these perspectives and it learned to discriminate between association and causality. The novelty of this research is the combination of supervised learning and Bradford Hill's causality considerations to automate the Bradford Hill's causality assessment. We evaluated the framework on a drug safety gold standard known as the observational medical outcomes partnership's non-specified association reference set. The methodology obtained excellent discrimination ability with area under the curves ranging between 0.792 and 0.940 (existing method optimal: 0.73) and a mean average precision of 0.640 (existing method optimal: 0.141). The proposed features can be calculated efficiently and be readily updated, making the framework suitable for big observational data.

Comparing penalization methods for linear models on large observational health data.

OBJECTIVE: This study evaluates regularization variants in logistic regression (L1, L2, ElasticNet, Adaptive L1, Adaptive ElasticNet, Broken adaptive ridge [BAR], and Iterative hard thresholding [IHT]) for discrimination and calibration performance, focusing on both internal and external validation. MATERIALS AND METHODS: We use data from 5 US claims and electronic health record databases and develop models for various outcomes in a major depressive disorder patient population. We externally validate all models in the other databases. We use a train-test split of 75%/25% and evaluate performance with discrimination and calibration. Statistical analysis for difference in performance uses Friedman's test and critical difference diagrams. RESULTS: Of the 840 models we develop, L1 and ElasticNet emerge as superior in both internal and external discrimination, with a notable AUC difference. BAR and IHT show the best internal calibration, without a clear external calibration leader. ElasticNet typically has larger model sizes than L1. Methods like IHT and BAR, while slightly less discriminative, significantly reduce model complexity. CONCLUSION: L1 and ElasticNet offer the best discriminative performance in logistic regression for healthcare predictions, maintaining robustness across validations. For simpler, more interpretable models, L0-based methods (IHT and BAR) are advantageous, providing greater parsimony and calibration with fewer features. This study aids in selecting suitable regularization techniques for healthcare prediction models, balancing performance, complexity, and interpretability.

Health-Analytics Data to Evidence Suite (HADES): Open-Source Software for Observational Research.

The Health-Analytics Data to Evidence Suite (HADES) is an open-source software collection developed by Observational Health Data Sciences and Informatics (OHDSI). It executes directly against healthcare data such as electronic health records and administrative claims, that have been converted to the Observational Medical Outcomes Partnership (OMOP) Common Data Model. Using advanced analytics, HADES performs characterization, population-level causal effect estimation, and patient-level prediction, potentially across a federated data network, allowing patient-level data to remain locally while only aggregated statistics are shared. Designed to run across a wide array of technical environments, including different operating systems and database platforms, HADES uses continuous integration with a large set of unit tests to maintain reliability. HADES implements OHDSI best practices, and is used in almost all published OHDSI studies, including some that have directly informed regulatory decisions.