Tree-based scan statistics to generate drug repurposing hypotheses: a test case using sodium-glucose cotransporter-2 inhibitors.

Most drug repurposing studies using real-world data focused on validating, instead of generating, hypotheses. We used tree-based scan statistics to generate repurposing hypotheses for sodium-glucose cotransporter-2 inhibitors (SGLT2i). We used an active-comparator, new-user design to create a 1:1 propensity-score matched cohort of SGLT2i and dipeptidyl peptidase-4 inhibitors (DPP4i) initiators in the MerativeTM MarketScan® Research Databases. Tree-based scan statistics were estimated across an ICD-10-CM-based hierarchical outcome tree using incident outcomes identified from hospital and outpatient diagnoses. We used an adjusted P≤0.01 as the threshold for statistical alert to prioritize associations for evaluation as repurposing signals. We varied the analyses by tree size, scanning level, and clinical settings for outcomes. There were 80,510 matched SGLT2i-DPP4i initiator pairs with 215,333 outcomes among SGLT2i initiators and 223,428 outcomes among DPP4i initiators. There were 18 prioritized associations, which included chronic kidney disease (P=0.0001), an expected signal, and anemia (P=0.0001). Heart failure (P=0.0167), another expected signal, was identified slightly beyond the statistical alert threshold. Narrowing the outcome tree, scanning at different tree levels, and including outcomes from different clinical settings influenced the scan statistics. We identified signals aligning with recently approved indications of SGLT2i, plus potential repurposing signals supported by existing evidence but requiring future validation.

Sharing Is Caring? International Society for Pharmacoepidemiology Review and Recommendations for Sharing Programming Code.

PURPOSE: There is increasing recognition of the importance of transparency and reproducibility in scientific research. This study aimed to quantify the extent to which programming code is publicly shared in pharmacoepidemiology, and to develop a set of recommendations on this topic. METHODS: We conducted a literature review identifying all studies published in Pharmacoepidemiology and Drug Safety (PDS) between 2017 and 2022. Data were extracted on the frequency and types of programming code shared, and other key open science practices (clinical codelist sharing, data sharing, study preregistration, and stated use of reporting guidelines and preprinting). We developed six recommendations for investigators who choose to share code and gathered feedback from members of the International Society for Pharmacoepidemiology (ISPE). RESULTS: Programming code sharing by articles published in PDS ranged from 1.8% in 2017 to 9.5% in 2022. It was more prevalent among articles with a methodological focus, simulation studies, and papers which also shared record-level data. CONCLUSION: Programming code sharing is rare but increasing in pharmacoepidemiology studies published in PDS. We recommend improved reporting of whether code is shared and how available code can be accessed. When sharing programming code, we recommend the use of permanent digital identifiers, appropriate licenses, and, where possible, adherence to good software practices around the provision of metadata and documentation, computational reproducibility, and data privacy.

External Controls to Study Treatment Effects in Rare Diseases: Challenges and Future Directions.

Regulators increasingly rely on real-world evidence generated from routine-care health data to evaluate novel therapies. Particularly, external control arms are increasingly used to supplement and contextualize efficacy and safety claims of single arm clinical trials for rare disease therapies. However, there are a number of methodological issues that may affect the validity of results derived from such comparisons. In this mini-review, we briefly summarize frequently used approaches and outline some of the most important criticisms and paths forward.

Efficacy versus effectiveness: The HORIZON Pivotal Fracture Trial and its emulation in Claims Data.

OBJECTIVES: To evaluate the concordance of results between the HORIZON-Pivotal Fracture Trial (PFT) and a non-randomized database study designed to emulate the trial. METHODS: HORIZON-PFT evaluated the efficacy of zoledronic acid vs placebo in reducing the risk of hip fractures and found a 41% risk reduction over a 3-year treatment period (HR = 0.59; 95% CI 0.42 to 0.83). Using two U.S. claims databases from 08/2007 to 12/2020 or 06/2021 we applied eligibility criteria from HORIZON-PFT and identified women with osteoporosis who initiated zoledronic acid or raloxifene as a proxy for placebo. The study protocol was registered on ClinicalTrials.gov (NCT04736693) before inferential analyses. We compared HORIZON-PFT and database study results using prespecified metrics. RESULTS: Due to low adherence in clinical practice, on-treatment follow up was truncated at 18 months in the database study. The hip fracture risk after 18 months was 9.3/1000 in the trial and 8.3/1000 in the database analysis. In the database study, zoledronic acid was associated with a 28% reduction in hip fractures risk compared to raloxifene (HR = 0.72; 95% CI 0.51 to 0.92). The attenuated effect of zoledronic acid in the database study may be explained by its shorter follow-up, as the interpolated estimate of the effect in HORIZON-PFT at 18 months was HRRCT 0.74, nearly identical to the observational estimate HRdatabase 0.72. CONCLUSION: Real-world emulation of the HORIZON-PFT found that zoledronic acid reduced hip fractures risk over an 18-month follow-up period. Limited adherence in clinical practice diminished the magnitude of its preventive effect and precluded long-term estimation of effectiveness in this setting.

Hierarchical Clustering Analysis to Inform Classification of Congenital Malformations for Surveillance of Medication Safety in Pregnancy.

There is growing interest in the secondary use of healthcare data to evaluate medication safety in pregnancy. Tree-based scan statistics (TBSS) offer an innovative approach to help identify potential safety signals. TBSS utilize hierarchically organized outcomes, generally based on existing clinical coding systems that group outcomes by organ system. When assessing teratogenicity, such groupings often lack a sound embryologic basis given the etiologic heterogeneity of congenital malformations. The study objective was to enhance the grouping of congenital malformations to be used in scanning approaches through implementation of hierarchical clustering analysis (HCA) and to pilot test an HCA-enhanced TBSS approach for medication safety surveillance in pregnancy in two test cases using >4.2 million mother-child dyads from two US-nationwide databases. HCA identified (1) malformation combinations belonging to the same organ system already grouped in existing classifications, (2) known combinations across different organ systems not previously grouped, (3) unknown combinations not previously grouped, and (4) malformations seemingly standing on their own. Testing the approach with valproate and topiramate identified expected signals, and a signal for an HCA-cluster missed by traditional classification. Augmenting existing classifications with clusters identified through large data exploration may be promising when defining phenotypes for surveillance and causal inference studies.

A Dynamic Prognostic Model for Identifying Vulnerable COVID-19 Patients at High Risk of Rapid Deterioration.

PURPOSE: We aimed to validate and, if performance was unsatisfactory, update the previously published prognostic model to predict clinical deterioration in patients hospitalized for COVID-19, using data following vaccine availability. METHODS: Using electronic health records of patients ≥18 years, with laboratory-confirmed COVID-19, from a large care-delivery network in Massachusetts, USA, from March 2020 to November 2021, we tested the performance of the previously developed prediction model and updated the prediction model by incorporating data after availability of COVID-19 vaccines. We randomly divided data into development (70%) and validation (30%) cohorts. We built a model predicting worsening in a published severity scale in 24 h by LASSO regression and evaluated performance by c-statistic and Brier score. RESULTS: Our study cohort consisted of 8185 patients (Development: 5730 patients [mean age: 62; 44% female] and Validation: 2455 patients [mean age: 62; 45% female]). The previously published model had suboptimal performance using data after November 2020 (N = 4973, c-statistic = 0.60. Brier score = 0.11). After retraining with the new data, the updated model included 38 predictors including 18 changing biomarkers. Patients hospitalized after Jun 1st, 2021 (when COVID-19 vaccines became widely available in Massachusetts) were younger and had fewer comorbidities than those hospitalized before. The c-statistic and Brier score were 0.77 and 0.13 in the development cohort, and 0.73 and 0.14 in the validation cohort. CONCLUSION: The characteristics of patients hospitalized for COVID-19 differed substantially over time. We developed a new dynamic model for rapid progression with satisfactory performance in the validation set.

Individualized Treatment Effect Prediction with Machine Learning - Salient Considerations.

BACKGROUND: Machine learning-based approaches that seek to accomplish individualized treatment effect prediction have gained traction; however, some salient challenges lack wider recognition. METHODS: We describe key methodologic considerations for individualized treatment effect prediction models using data from the Treatment of Preserved Cardiac Function Heart Failure with an Aldosterone Antagonist Trial for spironolactone in heart failure with preserved ejection fraction. The causal survival forest algorithm was used for model development. Calibration and discrimination were evaluated using a bootstrapping-based internal validation procedure. Observed benefits were described for predicted benefit quartiles and quartiles of a known effect modifier: ejection fraction. A negative control analysis with noncardiovascular death as the outcome was implemented to detect confounding. RESULTS: Among 3445 participants, 671 events occurred over a median of 3.3 years of follow-up. In internal validation, a higher average observed benefit was noted among patients in the highest quartile of predicted benefit. The median (interquartile range) of the observed restricted mean survival time difference at 3.3 years at the highest quartile of model-predicted benefit was 62 days (32 to 83) and was 47 days (26 to 67) at the lowest quartile of ejection fraction. Body-mass index had higher contribution to prediction of benefit relative to other included measures (33.7% vs. glomerular filtration rate [27.3%], ejection fraction [15.1%], and younger age [12.8%]) No benefit was observed for noncardiovascular death at higher model-predicted benefit quartiles, although benefit for noncardiovascular death was observed at lower quartiles. CONCLUSIONS: Carefully applied and validated predictive models hold promise in identifying heterogeneous treatment effects and are useful for hypothesis generation regarding the role of phenotypic characteristics in modifying the benefit of experimental interventions in clinical trials. (Funded by the National Heart, Lung, and Blood Institute; ClinicalTrials.gov number, NCT00094302.).

Building transparency and reproducibility into the practice of pharmacoepidemiology and outcomes research.

Real-world evidence (RWE) studies are increasingly used to inform policy and clinical decisions. However, there remain concerns about the credibility and reproducibility of RWE studies. Observational researchers should highlight the level of transparency of their studies by providing a succinct statement addressing study transparency with the publication of every paper, poster, or presentation that reports on a RWE study. In this paper, we propose a framework for an explicit transparency statement that declares the level of transparency a given RWE study has achieved across five key domains: 1) protocol, 2) pre-registration, 3) data, 4) code sharing, and 5) reporting checklists.

Emulation of randomized trials of direct oral anticoagulants with claims data and implications for new Factor XI inhibitors.

Direct oral anticoagulants (DOACs) revolutionized the management of thromboembolic disorders. Clinical care may be further improved as Factor XIs undergo large-scale outcome trials. What role can non-randomized database studies play in expediting understanding of these drugs in clinical practice? The RCT-DUPLICATIVE Initiative emulated the design of eight DOAC randomized clinical trials (RCT) using non-randomized claims database studies. RCT study design parameters and measurements were closely emulated by the database studies and produced highly concordant results. The results of the single database study that did not meet all agreement metrics with the specific RCT it was emulating were aligned with a meta-analysis of six trials studying similar questions, suggesting the trial result was an outlier. Well-designed database studies using fit-for-purpose data came to the same conclusions as DOAC trials, illustrating how database studies could complement RCTs for Factor XI inhibitors-by accelerating insights in underrepresented populations, demonstrating effectiveness and safety in clinical practice, and testing broader indications.

Data Checks Before Registering Study Protocols for Health Care Database Analyses.

This Viewpoint discusses the challenges involved with secondary health care data collection vs primary data collection and provides a list of suggested data checks before registration of a study protocol using secondary data.

Targeted Learning with an Undersmoothed Lasso Propensity Score Model for Large-Scale Covariate Adjustment in Healthcare Database Studies.

Lasso regression is widely used for large-scale propensity score (PS) estimation in healthcare database studies. In these settings, previous work has shown that undersmoothing (overfitting) Lasso PS models can improve confounding control, but it can also cause problems of non-overlap in covariate distributions. It remains unclear how to select the degree of undersmoothing when fitting large-scale Lasso PS models to improve confounding control while avoiding issues that can result from reduced covariate overlap. Here, we used simulations to evaluate the performance of using collaborative-controlled targeted learning to data-adaptively select the degree of undersmoothing when fitting large-scale PS models within both singly and doubly robust frameworks to reduce bias in causal estimators. Simulations showed that collaborative learning can data-adaptively select the degree of undersmoothing to reduce bias in estimated treatment effects. Results further showed that when fitting undersmoothed Lasso PS-models, the use of cross-fitting was important for avoiding non-overlap in covariate distributions and reducing bias in causal estimates.

An open-source implementation of tree-based scan statistics.

PURPOSE: We develop an open-source R package to implement tree-based scan statistics (TBSS) analyses. METHODS: TBSS are data mining methods used by the United States Food and Drug Administration and the Centers for Disease Control. They simultaneously screen thousands of hierarchically aggregated outcomes to identify unsuspected adverse effects of drugs or vaccines, accounting for multiple comparisons. The general structure of TBSS is highly adaptable, with four essential components: (1) a hierarchical outcome structure, (2) a test statistic to be computed for each element of the hierarchy, (3) an algorithm to generate data replicates under a null distribution, and (4) observed outcomes at the lower level of the hierarchy. We encode the general TBSS framework in a convenient R package that offers user-friendly functions for the most used TBSS methods. To illustrate the performance of our software, we evaluated two examples of archetypical TBSS analyses previously analyzed using proprietary, closed-source TreeScan™ software. The first considers the risk of congenital malformations associated with first-trimester exposure to valproate, and the second compares exposure to newly prescribed canagliflozin with a dipeptidyl peptidase 4 inhibitor in adults affected by type 2 diabetes. RESULTS: The results of the original studies are replicated. CONCLUSIONS: The diffusion of an open-source implementation of TBSS can enhance innovation of TBSS methods and foster collaborations. We offer an intuitive R package implementing standard TBSS methods with accompanying tutorials. Our unified object-oriented implementation allows expert users to extend the framework, introduce new features, or enhance existing ones.

Design differences and variation in results between randomised trials and non-randomised emulations: meta-analysis of RCT-DUPLICATE data.

Objective: To explore how design emulation and population differences relate to variation in results between randomised controlled trials (RCT) and non-randomised real world evidence (RWE) studies, based on the RCT-DUPLICATE initiative (Randomised, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology). Design: Meta-analysis of RCT-DUPLICATE data. Data sources: Trials included in RCT-DUPLICATE, a demonstration project that emulated 32 randomised controlled trials using three real world data sources: Optum Clinformatics Data Mart, 2004-19; IBM MarketScan, 2003-17; and subsets of Medicare parts A, B, and D, 2009-17. Eligibility criteria for selecting studies: Trials where the primary analysis resulted in a hazard ratio; 29 RCT-RWE study pairs from RCT-DUPLICATE. Results: Differences and variation in effect sizes between the results from randomised controlled trials and real world evidence studies were investigated. Most of the heterogeneity in effect estimates between the RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: treatment started in hospital (which does not appear in health insurance claims data), discontinuation of some baseline treatments at randomisation (which would have been an unusual care decision in clinical practice), and delayed onset of drug effects (which would be under-reported in real world clinical practice because of the relatively short persistence of the treatment). Adding the three emulation differences to the meta-regression reduced heterogeneity from 1.9 to almost 1 (absence of heterogeneity). Conclusions: This analysis suggests that a substantial proportion of the observed variation between results from randomised controlled trials and real world evidence studies can be attributed to differences in design emulation.

The FAIRification of research in real-world evidence: A practical introduction to reproducible analytic workflows using Git and R.

Transparency and reproducibility are major prerequisites for conducting meaningful real-world evidence (RWE) studies that are fit for decision-making. Many advances have been made in the documentation and reporting of study protocols and results, but the principles for version control and sharing of analytic code in RWE are not yet as established as in other quantitative disciplines like computational biology and health informatics. In this practical tutorial, we aim to give an introduction to distributed version control systems (VCS) tailored toward the FAIR (Findable, Accessible, Interoperable, and Reproducible) implementation of RWE studies. To ease adoption, we provide detailed step-by-step instructions with practical examples on how the Git VCS and R programming language can be implemented into RWE study workflows to facilitate reproducible analyzes. We further discuss and showcase how these tools can be used to track changes, collaborate, disseminate, and archive RWE studies through dedicated project repositories that maintain a complete audit trail of all relevant study documents.

High-Dose vs. Standard-Dose Influenza Vaccine and Cardiopulmonary Hospitalization or Mortality: Emulating the INVESTED Trial Using Insurance Claims Data.

The INVESTED trial did not show benefits of high-dose (HD) vaccine vs. standard-dose (SD) for a primary composite outcome of cardiopulmonary hospitalization or all-cause mortality (hazard ratio (HR) = 1.05, 95% confidence interval (CI) = 0.96-1.15) and its components (all-cause mortality HR = 1.01, 95% CI = 0.84-1.21, cardiopulmonary hospitalization HR = 1.05, 95% CI = 0.96-1.16) during three influenza seasons (2016-2019) among participants with recent myocardial infarction or hospitalization for heart failure (HHF). We emulated INVESTED using Medicare claims data to assess whether the real-world evidence (RWE) study reached similar conclusions. We identified 1:1 propensity score (PS)-matched trial-eligible Medicare beneficiaries aged > 65 years and with prior HHF who received an HD or SD vaccine for the 2016-2019 seasons. We also re-analyzed the INVESTED trial data restricting to participants > 65 years with prior HHF to align eligibility criteria more closely with the RWE study. We compared HRs from the trial and RWE study for the main outcomes. Among 53,393 pairs of PS-matched Medicare beneficiaries, the HD vaccine group showed lower risk of the primary composite outcome (HR = 0.96, 95% CI = 0.95-0.98) and all-cause mortality (HR = 0.93, 95% CI = 0.91-0.95), and similar risk of cardiopulmonary hospitalization (HR = 0.98, 95% CI = 0.96-1.00), compared with SD. The RWE and trial results were closely concordant after the trial population was limited to participants > 65 years with prior HHF: trial-based results for the primary composite outcome (HR = 1.02, 95% CI = 0.89-1.17), all-cause mortality (HR = 0.92, 95% CI = 0.72-1.16), and cardiopulmonary hospitalization (HR = 1.02, 95% CI = 0.88-1.18). Although similar to the main trial results, the RWE was closer to the results from trial participants with aligned eligibility criteria. This study affirms the importance of considering different distributions of baseline patient characteristics when comparing trial findings to RWE.