Building transparency and reproducibility into the practice of pharmacoepidemiology and outcomes research.

Real-world evidence (RWE) studies are increasingly used to inform policy and clinical decisions. However, there remain concerns about the credibility and reproducibility of RWE studies. Observational researchers should highlight the level of transparency of their studies by providing a succinct statement addressing study transparency with the publication of every paper, poster, or presentation that reports on a RWE study. In this paper, we propose a framework for an explicit transparency statement that declares the level of transparency a given RWE study has achieved across five key domains: 1) protocol, 2) pre-registration, 3) data, 4) code sharing, and 5) reporting checklists.

Targeted Learning with an Undersmoothed Lasso Propensity Score Model for Large-Scale Covariate Adjustment in Healthcare Database Studies.

Lasso regression is widely used for large-scale propensity score (PS) estimation in healthcare database studies. In these settings, previous work has shown that undersmoothing (overfitting) Lasso PS models can improve confounding control, but it can also cause problems of non-overlap in covariate distributions. It remains unclear how to select the degree of undersmoothing when fitting large-scale Lasso PS models to improve confounding control while avoiding issues that can result from reduced covariate overlap. Here, we used simulations to evaluate the performance of using collaborative-controlled targeted learning to data-adaptively select the degree of undersmoothing when fitting large-scale PS models within both singly and doubly robust frameworks to reduce bias in causal estimators. Simulations showed that collaborative learning can data-adaptively select the degree of undersmoothing to reduce bias in estimated treatment effects. Results further showed that when fitting undersmoothed Lasso PS-models, the use of cross-fitting was important for avoiding non-overlap in covariate distributions and reducing bias in causal estimates.

The FAIRification of research in real-world evidence: A practical introduction to reproducible analytic workflows using Git and R.

Transparency and reproducibility are major prerequisites for conducting meaningful real-world evidence (RWE) studies that are fit for decision-making. Many advances have been made in the documentation and reporting of study protocols and results, but the principles for version control and sharing of analytic code in RWE are not yet as established as in other quantitative disciplines like computational biology and health informatics. In this practical tutorial, we aim to give an introduction to distributed version control systems (VCS) tailored toward the FAIR (Findable, Accessible, Interoperable, and Reproducible) implementation of RWE studies. To ease adoption, we provide detailed step-by-step instructions with practical examples on how the Git VCS and R programming language can be implemented into RWE study workflows to facilitate reproducible analyzes. We further discuss and showcase how these tools can be used to track changes, collaborate, disseminate, and archive RWE studies through dedicated project repositories that maintain a complete audit trail of all relevant study documents.

An open-source implementation of tree-based scan statistics.

PURPOSE: We develop an open-source R package to implement tree-based scan statistics (TBSS) analyses. METHODS: TBSS are data mining methods used by the United States Food and Drug Administration and the Centers for Disease Control. They simultaneously screen thousands of hierarchically aggregated outcomes to identify unsuspected adverse effects of drugs or vaccines, accounting for multiple comparisons. The general structure of TBSS is highly adaptable, with four essential components: (1) a hierarchical outcome structure, (2) a test statistic to be computed for each element of the hierarchy, (3) an algorithm to generate data replicates under a null distribution, and (4) observed outcomes at the lower level of the hierarchy. We encode the general TBSS framework in a convenient R package that offers user-friendly functions for the most used TBSS methods. To illustrate the performance of our software, we evaluated two examples of archetypical TBSS analyses previously analyzed using proprietary, closed-source TreeScan™ software. The first considers the risk of congenital malformations associated with first-trimester exposure to valproate, and the second compares exposure to newly prescribed canagliflozin with a dipeptidyl peptidase 4 inhibitor in adults affected by type 2 diabetes. RESULTS: The results of the original studies are replicated. CONCLUSIONS: The diffusion of an open-source implementation of TBSS can enhance innovation of TBSS methods and foster collaborations. We offer an intuitive R package implementing standard TBSS methods with accompanying tutorials. Our unified object-oriented implementation allows expert users to extend the framework, introduce new features, or enhance existing ones.

Emulation of randomized trials of direct oral anticoagulants with claims data and implications for new Factor XI inhibitors.

Direct oral anticoagulants (DOACs) revolutionized the management of thromboembolic disorders. Clinical care may be further improved as Factor XIs undergo large-scale outcome trials. What role can non-randomized database studies play in expediting understanding of these drugs in clinical practice? The RCT-DUPLICATIVE Initiative emulated the design of eight DOAC randomized clinical trials (RCT) using non-randomized claims database studies. RCT study design parameters and measurements were closely emulated by the database studies and produced highly concordant results. The results of the single database study that did not meet all agreement metrics with the specific RCT it was emulating were aligned with a meta-analysis of six trials studying similar questions, suggesting the trial result was an outlier. Well-designed database studies using fit-for-purpose data came to the same conclusions as DOAC trials, illustrating how database studies could complement RCTs for Factor XI inhibitors-by accelerating insights in underrepresented populations, demonstrating effectiveness and safety in clinical practice, and testing broader indications.

Stratified analysis in comparative effectiveness studies that emulate randomized trials.

PURPOSE: For observational cohort studies that employ matching by propensity scores (PS), preliminary stratification by consequential predictors of outcome better emulates stratified randomization and potentially reduces variance and bias through relaxed dependence on modeling assumptions. We assessed the impact of pre-stratification in two real-life examples. For both, prior evidence from placebo-controlled randomized clinical trials (RCTs) suggested small or no risk reduction, but observational analysis suggested protection, presumably the result of confounding bias. STUDY DESIGN AND SETTING: The study populations consisted of Medicare beneficiaries (2014-18) with type 2 diabetes initiating either (i) empagliflozin versus dipeptidyl peptidase-4 inhibitors (DPP-4i) or (ii) empagliflozin versus glucagon-like peptide-1 receptor agonists (GLP-1RA). The outcome was myocardial infarction or stroke. We estimated hazard ratios (HR) and rate differences (RD) after controlling for 143 pre-exposure covariates via 1:1 PS matching after (1) PS estimation in the total cohort (total-cohort PS-matching) and (2) PS estimation separately by baseline cardiovascular disease (stratified PS matching). RESULTS: Stratified PS matching resulted in HRs that exceeded those from total-cohort PS-matching by 13% and 9%, respectively, for the comparisons of empagliflozin to DPP-4i and GLP-1RA. Against both comparators, HRs and RDs after stratified PS matching were closer to the null, with slightly higher variances (2%-3%) than those after total-cohort PS matching. CONCLUSION: Stratified PS matching produced effect estimates closer to the expected trial findings than total-cohort PS matching. The price paid in increased variance was minimal.

Comparative Effectiveness and Safety of Generic Versus Brand-Name Fluticasone-Salmeterol to Treat Chronic Obstructive Pulmonary Disease.

BACKGROUND: In 2019, the U.S. Food and Drug Administration (FDA) approved the first generic maintenance inhaler for asthma and chronic obstructive pulmonary disease (COPD). The inhaler, Wixela Inhub (fluticasone-salmeterol; Viatris), is a substitutable version of the dry powder inhaler Advair Diskus (fluticasone-salmeterol; GlaxoSmithKline). When approving complex generic products like inhalers, the FDA applies a special "weight-of-evidence" approach. In this case, manufacturers were required to perform a randomized controlled trial in patients with asthma but not COPD, although the product received approval for both indications. OBJECTIVE: To compare the effectiveness and safety of generic (Wixela Inhub) and brand-name (Advair Diskus) fluticasone-salmeterol among patients with COPD treated in routine care. DESIGN: A 1:1 propensity score-matched cohort study. SETTING: A large, longitudinal health care database. PATIENTS: Adults older than 40 years with a diagnosis of COPD. MEASUREMENTS: Incidence of first moderate or severe COPD exacerbation (effectiveness outcome) and incidence of first pneumonia hospitalization (safety outcome) in the 365 days after cohort entry. RESULTS: Among 45 369 patients (27 305 Advair Diskus users and 18 064 Wixela Inhub users), 10 012 matched pairs were identified for the primary analysis. Compared with Advair Diskus use, Wixela Inhub use was associated with a nearly identical incidence of first moderate or severe COPD exacerbation (hazard ratio [HR], 0.97 [95% CI, 0.90 to 1.04]) and first pneumonia hospitalization (HR, 0.99 [CI, 0.86 to 1.15]). LIMITATIONS: Follow-up times were short, reflecting real-world clinical practice. The possibility of residual confounding cannot be completely excluded. CONCLUSION: Use of generic and brand-name fluticasone-salmeterol was associated with similar outcomes among patients with COPD treated in routine practice. PRIMARY FUNDING SOURCE: National Heart, Lung, and Blood Institute.

Separation of Plasmid DNA Topological Forms, Messenger RNA, and Lipid Nanoparticle Aggregates Using an Ultrawide Pore Size Exclusion Chromatography Column.

Health authorities have highlighted the need to determine oligonucleotide aggregates. However, existing technologies have limitations that have prevented the reliable analysis of size variants for large nucleic acids and lipid nanoparticles (LNPs). In this work, nucleic acid and LNP aggregation was examined using prototype, low adsorption ultrawide pore size exclusion chromatography (SEC) columns. A preliminary study was conducted to determine the column's physicochemical properties. A large difference in aggregate content (17.8 vs 59.7 %) was found for a model messenger RNA (mRNA) produced by different manufacturers. We further investigated the nature of the aggregates via a heat treatment. Interestingly, thermal stress irreversibly decreased the amount of aggregates from 59.7 to 4.1% and increased the main peak area 3.3-fold. To the best of our knowledge, for the first time, plasmid DNA topological forms and multimers were separated by analytical SEC. The degradation trends were compared to the data obtained with an anion exchange chromatography method. Finally, unconjugated and fragment antigen-binding (Fab)-guided LNPs were analyzed and their elution times were plotted against their sizes as measured by DLS. Multi-angle light scattering (MALS) was coupled to SEC in order to gain further insights on large species eluting before the LNPs, which were later identified as self-associating LNPs. This study demonstrated the utility of ultrawide pore SEC columns in characterizing the size variants of large nucleic acid therapeutics and LNPs.

Process Robustness in Lipid Nanoparticle Production: A Comparison of Microfluidic and Turbulent Jet Mixing.

The recent clinical and commercial success of lipid nanoparticles (LNPs) for nucleic acid delivery has incentivized the development of new technologies to manufacture LNPs. As new technologies emerge, researchers must determine which technologies to assess and how to perform comparative evaluations. In this article, we use a quality-by-design approach to systematically investigate how the mixer technology used to form LNPs influences LNPstructure. Specifically, a coaxial turbulent jet mixer and a staggered herringbone microfluidic mixer were systematically compared via matched formulation and process conditions. A full-factorial design-of-experiments study with three factors and three levels was executed for each mixer to compare process robustness in the production of antisense oligonucleotide (ASO) LNPs. ASO-LNPs generated with the coaxial turbulent jet mixer were consistently smaller, had a narrower particle size distribution, and had a higher ASO encapsulation as compared to the microfluidic mixer, but had a greater variation in internal structure with less ordered cores. A subset of the study was replicated for mRNA-LNPs with comparable trends in particle size and encapsulation, but more frequent bleb features for LNPs produced by the coaxial turbulent jet mixer. The study design used here provides a road map for how researchers may compare different mixer technologies (or process changes more broadly) and how such studies can inform process robustness and manufacturing control strategies.

Klotho Deficiency Causes Heart Aging via Impairing the Nrf2-GR Pathway.

RATIONALE: Cardiac aging is an important contributing factor for heart failure, which affects a large population but remains poorly understood. OBJECTIVE: The purpose of this study is to investigate whether Klotho plays a role in cardiac aging. METHODS AND RESULTS: Heart function declined in old mice (24 months), as evidenced by decreases in fractional shortening, ejection fraction, and cardiac output. Heart size and weight, cardiomyocyte size, and cardiac fibrosis were increased in old mice, indicating that aging causes cardiac hypertrophy and remodeling. Circulating Klotho levels were dramatically decreased in old mice, which prompted us to investigate whether the Klotho decline may cause heart aging. We found that Klotho gene mutation (KL-/-) largely decreased serum klotho levels and impaired heart function. Interestingly, supplement of exogenous secreted Klotho prevented heart failure, hypertrophy, and remodeling in both old mice and KL (-/-) mice. Secreted Klotho treatment inhibited excessive cardiac oxidative stress, senescence and apoptosis in old mice and KL (-/-) mice. Serum phosphate levels in KL (-/-) mice were kept in the normal range, suggesting that Klotho deficiency-induced heart aging is independent of phosphate metabolism. Mechanistically, Klotho deficiency suppressed GR (glutathione reductase) expression and activity in the heart via inhibition of transcription factor Nrf2 (nuclear factor-erythroid 2 p45-related factor 2). Furthermore, cardiac-specific overexpression of GR prevented excessive oxidative stress, apoptosis, and heart failure in both old and KL (-/-) mice. CONCLUSIONS: Klotho deficiency causes cardiac aging via impairing the Nrf2-GR pathway. Supplement of exogenous secreted Klotho represents a promising therapeutic strategy for aging-associated cardiomyopathy and heart failure.

Prospective validation of a dynamic prognostic model for identifying COVID-19 patients at high risk of rapid deterioration.

BACKGROUND: We sought to develop and prospectively validate a dynamic model that incorporates changes in biomarkers to predict rapid clinical deterioration in patients hospitalized for COVID-19. METHODS: We established a retrospective cohort of hospitalized patients aged ≥18 years with laboratory-confirmed COVID-19 using electronic health records (EHR) from a large integrated care delivery network in Massachusetts including >40 facilities from March to November 2020. A total of 71 factors, including time-varying vital signs and laboratory findings during hospitalization were screened. We used elastic net regression and tree-based scan statistics for variable selection to predict rapid deterioration, defined as progression by two levels of a published severity scale in the next 24 h. The development cohort included the first 70% of patients identified chronologically in calendar time; the latter 30% served as the validation cohort. A cut-off point was estimated to alert clinicians of high risk of imminent clinical deterioration. RESULTS: Overall, 3706 patients (2587 in the development and 1119 in the validation cohort) met the eligibility criteria with a median of 6 days of follow-up. Twenty-four variables were selected in the final model, including 16 dynamic changes of laboratory results or vital signs. Area under the ROC curve was 0.81 (95% CI, 0.79-0.82) in the development set and 0.74 (95% CI, 0.71-0.78) in the validation set. The model was well calibrated (slope = 0.84 and intercept = -0.07 on the calibration plot in the validation set). The estimated cut-off point, with a positive predictive value of 83%, was 0.78. CONCLUSIONS: Our prospectively validated dynamic prognostic model demonstrated temporal generalizability in a rapidly evolving pandemic and can be used to inform day-to-day treatment and resource allocation decisions based on dynamic changes in biophysiological factors.

HARmonized Protocol Template to Enhance Reproducibility of hypothesis evaluating real-world evidence studies on treatment effects: A good practices report of a joint ISPE/ISPOR task force.

PROBLEM: Ambiguity in communication of key study parameters limits the utility of real-world evidence (RWE) studies in healthcare decision-making. Clear communication about data provenance, design, analysis, and implementation is needed. This would facilitate reproducibility, replication in independent data, and assessment of potential sources of bias. WHAT WE DID: The International Society for Pharmacoepidemiology (ISPE) and ISPOR-The Professional Society for Health Economics and Outcomes Research (ISPOR) convened a joint task force, including representation from key international stakeholders, to create a harmonized protocol template for RWE studies that evaluate a treatment effect and are intended to inform decision-making. The template builds on existing efforts to improve transparency and incorporates recent insights regarding the level of detail needed to enable RWE study reproducibility. The overarching principle was to reach for sufficient clarity regarding data, design, analysis, and implementation to achieve 3 main goals. One, to help investigators thoroughly consider, then document their choices and rationale for key study parameters that define the causal question (e.g., target estimand), two, to facilitate decision-making by enabling reviewers to readily assess potential for biases related to these choices, and three, to facilitate reproducibility. STRATEGIES TO DISSEMINATE AND FACILITATE USE: Recognizing that the impact of this harmonized template relies on uptake, we have outlined a plan to introduce and pilot the template with key international stakeholders over the next 2 years. CONCLUSION: The HARmonized Protocol Template to Enhance Reproducibility (HARPER) helps to create a shared understanding of intended scientific decisions through a common text, tabular and visual structure. The template provides a set of core recommendations for clear and reproducible RWE study protocols and is intended to be used as a backbone throughout the research process from developing a valid study protocol, to registration, through implementation and reporting on those implementation decisions.

Body dissatisfaction, ideals, and identity in the development of disordered eating among adolescent ballet dancers.

OBJECTIVE: Little is known about how female adolescent ballet dancers-a group at high-risk for the development of body dissatisfaction and eating disorders-construct body ideals, and how their social identities interact with body ideals to confer risk for disordered eating. Using a novel body figure behavioral task, this study investigated (1) whether degree of body dissatisfaction corresponded to severity of disordered eating thoughts and behaviors, and (2) how ballet identity corresponded with ideal body figure size among adolescent ballet dancers. METHODS: Participants were 188 female ballet dancers ages 13-18 years who completed self-report measures of study constructs and the behavioral task. RESULTS: Linear regression models indicated that more severe body dissatisfaction was positively associated with increased disordered eating thoughts and behaviors (p < .19), except for muscle building (p = .32). We also found that identifying more strongly as a ballet dancer was correlated with having a smaller ideal body size (p = .017). CONCLUSION: Findings from this study suggest desire to achieve smaller body sizes is correlated with more severe disordered eating endorsement and stronger ballet identity. Instructors and clinicians may consider assessing the extent to which individuals identify as a ballet dancer as a risk factor for disordered eating and encourage adolescent dancers to build and nurture other identities beyond ballet. PUBLIC SIGNIFICANCE: Eating disorders are debilitating conditions that can lead to malnutrition, social isolation, and even premature death. Though disordered eating thoughts and behaviors can affect anyone, adolescents in physically demanding and body image-driven activities including ballet dance are particularly vulnerable. Investigating how factors like body dissatisfaction and strength of identity are associated with disordered eating among high-risk groups is crucial for developing effective prevention and intervention methods that minimize harm.

Comparing self-harming intentions underlying eating disordered behaviors and nonsuicidal self-injury: Replication and extension in adolescents.

OBJECTIVE: Eating disorder (ED) behaviors are often characterized as indirect forms of self-harm. However, recent research has found less clear demarcations between direct self-harming behaviors (e.g., nonsuicidal self-injury [NSSI], suicidal behaviors) than previously assumed. The aim of this study was to replicate findings of this prior research on adult populations in adolescents with a history of restrictive eating. METHOD: A total of 117 adolescents between ages 12-14 were included in the study. Participants reported the presence and frequency of binge eating, compensatory, restrictive eating, and NSSI. Participants also reported thoughts of and intentions to hurt and kill themselves when engaging in each behavior on average. The t-tests and linear effects models were conducted to compare self-harming thoughts and intentions across behaviors. RESULTS: Participants reported at least some intent to hurt themselves physically in the moment and in the long-term when engaging in all ED behaviors and NSSI, and reported engaging in these behaviors while thinking about suicide. Direct self-harming knowledge and intentions were most frequently reported with NSSI and longer-term knowledge and intentions via NSSI and restrictive eating. Additionally, participants reported some suicidal thoughts and intentions across behaviors. DISCUSSION: This study replicates prior research, suggesting that adolescents engage in ED behaviors and NSSI with non-zero self-harming and suicidal thoughts and intentions. ED behaviors and NSSI may better be explained on a continuum. Implications include the recommendation of safety planning during ED treatment. PUBLIC SIGNIFICANCE STATEMENT: This study highlights the overlap between eating disorder (ED) behaviors, nonsuicidal self-injury (NSSI), and suicide. Though clear distinctions typically exist for motives of self-harming behavior between ED behaviors (i.e., indirect, in the long run) and NSSI (i.e., direct, in the moment), this research suggests that intentions for self-harming and suicide may exist on a continuum. Clinical ED treatment should consider safety planning as part of routine interventions.

Development of transdiagnostic clinical risk prediction models for 12-month onset and course of eating disorders among adolescents in the community.

OBJECTIVE: To develop and internally validate risk prediction models for adolescent onset and persistence of eating disorders. METHODS: N = 963 Australian adolescents (11-19 years) in the EveryBODY Study cohort completed online surveys in 2018 and 2019. Models were built to predict 12-month risk of (1) onset, and (2) persistence of a DSM-5 eating disorder. RESULTS: Onset Model. Of the n = 687 adolescents without an eating disorder at baseline, 16.9% were identified with an eating disorder after 12 months. The prediction model was based on evidence-based risk factors for eating disorder onset available within the dataset (sex, body mass index percentile, strict weight loss dieting, history of bullying, psychological distress, weight/shape concerns). This model showed fair discriminative performance (mean AUC = .75). The most important factors were psychological distress, weight and shape concerns, and female sex. Diagnostic Persistence Model. Of the n = 276 adolescents with an eating disorder at baseline, 74.6% were identified as continuing to meet criteria for an eating disorder after 12 months. The prediction model for diagnostic persistence was based on available evidence-based risk factors for eating disorder persistence (purging, distress, social impairment). This model showed poor discriminative performance (mean AUC = .65). The most important factors were psychological distress and self-induced vomiting for weight control. DISCUSSION: We found preliminary evidence for the utility of a parsimonious model for 12-month onset of an eating disorder among adolescents in the community. Future research should include additional evidence-based risk factors and validate models beyond the original sample. PUBLIC SIGNIFICANCE: This study demonstrated the feasibility of developing parsimonious and accurate models for the prediction of future onset of an eating disorder among adolescents. The most important predictors in this model included psychological distress and weight and shape concerns. This study has laid the ground work for future research to build and test more accurate prediction models in diverse samples, prior to translation into a clinical tool for use in real world settings to aid decisions about referral to early intervention.

Aging impairs arterial compliance via Klotho-mediated downregulation of B-cell population and IgG levels.

OBJECTIVE: Aging is associated with compromised immune function and arterial remodeling and stiffness. The purpose of this study is to investigate whether in vivo AAV-based delivery of secreted Klotho (SKL) gene (AAV-SKL) improves aging- and senescence-associated immune dysfunction and arterial stiffness. METHODS AND RESULTS: Senescence-accelerated mice prone strain 1 (SAMP1, 10 months) and old mice (20 months) were used. Serum SKL levels, B-cell population and serum IgG levels were markedly decreased in SAMP1 and old mice. Rescue of downregulation of serum SKL levels by in vivo AAV2-based delivery of SKL gene (AAV-SKL) increased B-cell population and serum IgG levels and attenuated arterial stiffness in SAMP1 and old mice. Thus, Klotho deficiency may play a role in senescence- and aging-associated humoral immune dysfunction and arterial stiffness. Vascular infiltration of inflammatory cells and expression of TGFß1, collagen 1, scleraxis, MMP-2 and MMP-9 were increased while the elastin level was decreased in aortas of SAMP1 and old mice which can be rescued by AAV-SKL. Interestingly, treatment with IgG effectively rescued arterial inflammation and remodeling and attenuated arterial stiffness and hypertension in aging mice. In cultured B-lymphoblast cells, we further showed that SKL regulates B-cell proliferation and maturation partly via the NFkB pathway. CONCLUSION: Aging-associated arterial stiffening may be largely attributed to downregulation of B-cell population and serum IgG levels. AAV-SKL attenuates arterial stiffness in aging mice partly via restoring B-cell population and serum IgG levels which attenuates aging-associated vascular inflammation and arterial remodeling.

Research on the intrinsic mechanism of the darkening of liquid foundation.

BACKGROUND: To investigate the intrinsic mechanism that causes the darkening of liquid foundations. MATERIALS AND METHOD: A total of 36 commercial liquid foundations were firstly studied for preliminary screening of influencing factors. A basic liquid foundation was developed for controlling variables to study the influence of each single factor. These samples were evenly spread on the standard opacity charts with the thickness of 100 µm and applied onto human inner forearm skin with the dosage of 2 mg/cm2 . The discoloration of each sample was continuously recorded using spectrophotometers and reported in the CIE 1976 L*a*b* color space for at least 120 min, and ΔE was calculated to describe the severity of darkening. RESULTS: One hundred twenty-minute ΔE of all commercial foundations was highly negatively correlated with their 120-min ΔITA° (R2  = 0.88, p < 0.01). A strong positive correlation was found between the severity of darkening and the volatilization of the basic foundations (R2  = 0.83, p < 0.01). And the darkening of silicone-based basic foundations using pigment coating with silicon is weaker than those without silicon (p < 0.05). CONCLUSION: The process of the discoloration of liquid foundation is accompanied by the decrease of ITA° and manifested as darkening. The volatilization rate of the product and the coating method of the pigments used in the formula can noticeably affect the darkening of the liquid foundation.

Emulation of Randomized Clinical Trials With Nonrandomized Database Analyses: Results of 32 Clinical Trials.

Importance: Nonrandomized studies using insurance claims databases can be analyzed to produce real-world evidence on the effectiveness of medical products. Given the lack of baseline randomization and measurement issues, concerns exist about whether such studies produce unbiased treatment effect estimates. Objective: To emulate the design of 30 completed and 2 ongoing randomized clinical trials (RCTs) of medications with database studies using observational analogues of the RCT design parameters (population, intervention, comparator, outcome, time [PICOT]) and to quantify agreement in RCT-database study pairs. Design, Setting, and Participants: New-user cohort studies with propensity score matching using 3 US claims databases (Optum Clinformatics, MarketScan, and Medicare). Inclusion-exclusion criteria for each database study were prespecified to emulate the corresponding RCT. RCTs were explicitly selected based on feasibility, including power, key confounders, and end points more likely to be emulated with real-world data. All 32 protocols were registered on ClinicalTrials.gov before conducting analyses. Emulations were conducted from 2017 through 2022. Exposures: Therapies for multiple clinical conditions were included. Main Outcomes and Measures: Database study emulations focused on the primary outcome of the corresponding RCT. Findings of database studies were compared with RCTs using predefined metrics, including Pearson correlation coefficients and binary metrics based on statistical significance agreement, estimate agreement, and standardized difference. Results: In these highly selected RCTs, the overall observed agreement between the RCT and the database emulation results was a Pearson correlation of 0.82 (95% CI, 0.64-0.91), with 75% meeting statistical significance, 66% estimate agreement, and 75% standardized difference agreement. In a post hoc analysis limited to 16 RCTs with closer emulation of trial design and measurements, concordance was higher (Pearson r, 0.93; 95% CI, 0.79-0.97; 94% meeting statistical significance, 88% estimate agreement, 88% standardized difference agreement). Weaker concordance occurred among 16 RCTs for which close emulation of certain design elements that define the research question (PICOT) with data from insurance claims was not possible (Pearson r, 0.53; 95% CI, 0.00-0.83; 56% meeting statistical significance, 50% estimate agreement, 69% standardized difference agreement). Conclusions and Relevance: Real-world evidence studies can reach similar conclusions as RCTs when design and measurements can be closely emulated, but this may be difficult to achieve. Concordance in results varied depending on the agreement metric. Emulation differences, chance, and residual confounding can contribute to divergence in results and are difficult to disentangle.

HARmonized Protocol Template to Enhance Reproducibility of Hypothesis Evaluating Real-World Evidence Studies on Treatment Effects: A Good Practices Report of a Joint ISPE/ISPOR Task Force.

OBJECTIVES: Ambiguity in communication of key study parameters limits the utility of real-world evidence (RWE) studies in healthcare decision-making. Clear communication about data provenance, design, analysis, and implementation is needed. This would facilitate reproducibility, replication in independent data, and assessment of potential sources of bias. METHODS: The International Society for Pharmacoepidemiology (ISPE) and ISPOR-The Professional Society for Health Economics and Outcomes Research (ISPOR) convened a joint task force, including representation from key international stakeholders, to create a harmonized protocol template for RWE studies that evaluate a treatment effect and are intended to inform decision-making. The template builds on existing efforts to improve transparency and incorporates recent insights regarding the level of detail needed to enable RWE study reproducibility. The over-arching principle was to reach for sufficient clarity regarding data, design, analysis, and implementation to achieve 3 main goals. One, to help investigators thoroughly consider, then document their choices and rationale for key study parameters that define the causal question (e.g., target estimand), two, to facilitate decision-making by enabling reviewers to readily assess potential for biases related to these choices, and three, to facilitate reproducibility. STRATEGIES TO DISSEMINATE AND FACILITATE USE: Recognizing that the impact of this harmonized template relies on uptake, we have outlined a plan to introduce and pilot the template with key international stakeholders over the next 2 years. CONCLUSION: The HARmonized Protocol Template to Enhance Reproducibility (HARPER) helps to create a shared understanding of intended scientific decisions through a common text, tabular and visual structure. The template provides a set of core recommendations for clear and reproducible RWE study protocols and is intended to be used as a backbone throughout the research process from developing a valid study protocol, to registration, through implementation and reporting on those implementation decisions.

Visualizations throughout pharmacoepidemiology study planning, implementation, and reporting.

Transparency is increasingly promoted to instill trust in nonrandomized studies using real-world data. Graphics and data visualizations support transparency by aiding communication and understanding, and can inform study design and analysis decisions. However, other than graphical representation of a study design and flow diagrams (e.g., a Consolidated Standards of Reporting Trials [CONSORT] like diagram), specific standards on how to maximize validity and transparency with visualization are needed. This paper provides guidance on how to use visualizations throughout the life cycle of a pharmacoepidemiology study-from initial study design to final report-to facilitate rationalized and transparent decision-making about study design and implementation, and clear communication of study findings. Our intent is to help researchers align their practices with current consensus statements on transparency.