High-Dose vs. Standard-Dose Influenza Vaccine and Cardiopulmonary Hospitalization or Mortality: Emulating the INVESTED Trial Using Insurance Claims Data.

The INVESTED trial did not show benefits of high-dose (HD) vaccine vs. standard-dose (SD) for a primary composite outcome of cardiopulmonary hospitalization or all-cause mortality (hazard ratio (HR) = 1.05, 95% confidence interval (CI) = 0.96-1.15) and its components (all-cause mortality HR = 1.01, 95% CI = 0.84-1.21, cardiopulmonary hospitalization HR = 1.05, 95% CI = 0.96-1.16) during three influenza seasons (2016-2019) among participants with recent myocardial infarction or hospitalization for heart failure (HHF). We emulated INVESTED using Medicare claims data to assess whether the real-world evidence (RWE) study reached similar conclusions. We identified 1:1 propensity score (PS)-matched trial-eligible Medicare beneficiaries aged > 65 years and with prior HHF who received an HD or SD vaccine for the 2016-2019 seasons. We also re-analyzed the INVESTED trial data restricting to participants > 65 years with prior HHF to align eligibility criteria more closely with the RWE study. We compared HRs from the trial and RWE study for the main outcomes. Among 53,393 pairs of PS-matched Medicare beneficiaries, the HD vaccine group showed lower risk of the primary composite outcome (HR = 0.96, 95% CI = 0.95-0.98) and all-cause mortality (HR = 0.93, 95% CI = 0.91-0.95), and similar risk of cardiopulmonary hospitalization (HR = 0.98, 95% CI = 0.96-1.00), compared with SD. The RWE and trial results were closely concordant after the trial population was limited to participants > 65 years with prior HHF: trial-based results for the primary composite outcome (HR = 1.02, 95% CI = 0.89-1.17), all-cause mortality (HR = 0.92, 95% CI = 0.72-1.16), and cardiopulmonary hospitalization (HR = 1.02, 95% CI = 0.88-1.18). Although similar to the main trial results, the RWE was closer to the results from trial participants with aligned eligibility criteria. This study affirms the importance of considering different distributions of baseline patient characteristics when comparing trial findings to RWE.

Comparison of EHR Data-Completeness in Patients with Different Types of Medical Insurance Coverage in the United States.

Prior studies have demonstrated that misclassification of study variables due to electronic health record (EHR)-discontinuity can be mitigated by restricting EHR-based analyses to subjects with high predicted EHR-continuity based on a simple algorithm. In this study, we compared EHR continuity in populations covered by Medicare, Medicaid, or commercial insurance. Using claims-linked EHRs from a multicenter network in Massachusetts, including Medicare (MA EHR-Medicare cohort) and Medicaid (MA EHR-Medicaid cohort) claims data; and TriNetX (TriNetX cohort) claims-linked EHR data from 11 US-based healthcare organizations, we assessed (1) EHR-continuity quantified by proportion of encounters captured by EHR (capture proportion (CP)); (2) area under receiver operating curve (AUROC) of previously validated model to identify patients with high EHR-continuity (CP ≥ 0.6); (3) misclassification of 40 patient characteristics, quantified by average standardized absolute mean difference (ASAMD). Study participants were ≥ 65 years (Medicare) or ≥ 18 years (Medicaid, TriNetX) with ≥ 365 days of continuous insurance enrollment overlapping with an EHR encounter. We found that the mean CP was 0.30, 0.18, and 0.19 and AUROC of the prediction model to identify patients with high EHR-continuity was 0.92, 0.89, and 0.77 in the MA EHR-Medicare, MA EHR-Medicaid, and TriNetX cohorts, respectively. Restricting to patients with predicted EHR-continuity percentile of top 20%, 50%, and 50% in MA EHR-Medicare, MA EHR-Medicaid, and TriNetX cohorts resulted in acceptable levels of misclassification (ASAMD < 0.1). Using a prediction model to identify cohorts with high EHR-continuity can improve validity, but cutoffs to achieve this goal vary by population.

Predicting Treatment Effects of a New-to-Market Drug in Clinical Practice Based on Phase III Randomized Trial Results.

Trial results may not be generalizable to target populations treated in clinical practice with different distributions of baseline characteristics that modify the treatment effect. We used outcome models developed with trial data to predict treatment effects in Medicare populations. We used data from the Randomized Evaluation of Long-Term Anticoagulation Therapy trial (RE-LY), which investigated the effect of dabigatran vs. warfarin on stroke or systemic embolism (stroke/SE) among patients with atrial fibrillation. We developed outcome models by fitting proportional hazards models in trial data. Target populations were trial-eligible Medicare beneficiaries who initiated dabigatran or warfarin in 2010-2011 ("early") and 2010-2017 ("extended"). We predicted 2-year risk ratios (RRs) and risk differences (RDs) for stroke/SE, major bleeding, and all-cause death in the Medicare populations using the observed baseline characteristics. The trial and early target populations had similar mean (SD) CHADS2 scores (2.15 (SD 1.13) vs. 2.15 (SD 0.91)) but different mean ages (71 vs. 79 years). Compared with RE-LY, the early Medicare population had similar predicted benefit of dabigatran vs. warfarin for stroke/SE (trial RR = 0.63, 95% confidence interval (CI) = 0.50 to 0.76 and RD = -1.37%, -1.96% to -0.77%, Medicare RR = 0.73, 0.65 to 0.82 and RD = -0.92%, -1.26% to -0.59%) and risks for major bleeding and all-cause death. The time-extended target population showed similar results. Outcome model-based prediction facilitates estimating the average treatment effects of a drug in different target populations when treatment and outcome data are unreliable or unavailable. The predicted effects may inform payers' coverage decisions for patients, especially shortly after a drug's launch when observational data are scarce.

The Impact of Longitudinal Data-Completeness of Electronic Health Record Data on the Prediction Performance of Clinical Risk Scores.

The impact of electronic health record (EHR) discontinuity (i.e., receiving care outside of a given EHR system) on EHR-based risk prediction is unknown. We aimed to assess the impact of EHR-continuity on the performance of clinical risk scores. The study cohort consisted of patients aged ≥ 65 years with ≥ 1 EHR encounter in the 2 networks in Massachusetts (MA; 2007/1/1-2017/12/31, internal training and validation dataset), and one network in North Carolina (NC; 2007/1/1-2016/12/31, external validation dataset) that were linked with Medicare claims data. Risk scores were calculated using EHR data alone vs. linked EHR-claims data (not subject to misclassification due to EHR-discontinuity): (i) combined comorbidity score (CCS), (ii) claim-based frailty score (CFI), (iii) CHAD2 DS2 -VASc, and (iv) Hypertension, Abnormal renal/liver function, Stroke, Bleeding, Labile, Elderly, and Drugs (HAS-BLED). We assessed the performance of CCS and CFI predicting death, CHAD2 DS2 -VASc predicting ischemic stroke, and HAS-BLED predicting bleeding by area under receiver operating characteristic curve (AUROC), stratified by quartiles of predicted EHR-continuity (Q1-4). There were 319,740 patients in the MA systems and 125,380 in the NC system. In the external validation dataset, AUROC for EHR-based CCS predicting 1-year risk of death was 0.583 in Q1 (lowest) EHR-continuity group, which increased to 0.739 in Q4 (highest) EHR-continuity group. The corresponding improvement in AUROC was 0.539 to 0.647 for CFI, 0.556 to 0.637 for CHAD2 DS2 -VASc, and 0.517 to 0.556 for HAS-BLED. The AUROC in Q4 EHR-continuity group based on EHR alone approximates that based on EHR-claims data. The prediction performance of four clinical risk scores was substantially worse in patients with lower vs. high EHR-continuity.

Active Surveillance of the Safety of Medications Used During Pregnancy.

The scientific community relies on postmarketing approaches to define the risk of using medications in pregnancy because information available at the time of drug approval is limited. Most studies carried out in pregnancy focus on a single outcome or selected outcomes. However, women must balance the benefit of treatment against all possible adverse effects. We aimed to apply and evaluate a tree-based scan statistic data-mining method (TreeScan; Martin Kulldorff, Harvard Medical School, Boston, Massachusetts) as a safety surveillance approach that allows for simultaneous evaluation of a comprehensive range of adverse pregnancy outcomes, while preserving the overall rate of false-positive alerts. We evaluated TreeScan with a cohort design and adjustment via propensity score techniques, using 2 test cases: 1) opioids and neonatal opioid withdrawal syndrome and 2) valproate and congenital malformations, implemented in pregnancy cohorts nested within the Medicaid Analytic eXtract (January 1, 2000-December 31, 2014) and the IBM MarketScan Research Database (IBM, Armonk, New York) (January 1, 2003-September 30, 2015). In both cases, we identified known safety concerns, with only 1 previously unreported alert at the preset statistical alerting threshold. This evaluation shows the promise of TreeScan-based approaches for systematic drug safety monitoring in pregnancy. A targeted screening approach followed by deeper investigation to refine understanding of potential signals will ensure that pregnant women and their physicians have access to the best available evidence to inform treatment decisions.

Screening Medications for Association with Progression to Wet Age-Related Macular Degeneration.

PURPOSE: There is an urgent need for treatments that prevent or delay development to advanced age-related macular degeneration (AMD). Drugs already on the market for other conditions could affect progression to neovascular AMD (nAMD). If identified, these drugs could provide insights for drug development targets. The objective of this study was to use a novel data mining method that can simultaneously evaluate thousands of correlated hypotheses, while adjusting for multiple testing, to screen for associations between drugs and delayed progression to nAMD. DESIGN: We applied a nested case-control study to administrative insurance claims data to identify cases with nAMD and risk-set sampled controls that were 1:4 variable ratio matched on age, gender, and recent healthcare use. PARTICIPANTS: The study population included cases with nAMD and risk set matched controls. METHODS: We used a tree-based scanning method to evaluate associations between hierarchical classifications of drugs that patients were exposed to within 6 months, 7 to 24 months, or ever before their index date. The index date was the date of first nAMD diagnosis in cases. Risk-set sampled controls were assigned the same index date as the case to which they were matched. The study was implemented using Medicare data from New Jersey and Pennsylvania, and national data from IBM MarketScan Research Database. We set an a priori threshold for statistical alerting at P ≤ 0.01 and focused on associations with large magnitude (relative risks ≥ 2.0). MAIN OUTCOME MEASURES: Progression to nAMD. RESULTS: Of approximately 4000 generic drugs and drug classes evaluated, the method detected 19 distinct drug exposures with statistically significant, large relative risks indicating that cases were less frequently exposed than controls. These included (1) drugs with prior evidence for a causal relationship (e.g., megestrol); (2) drugs without prior evidence for a causal relationship, but potentially worth further exploration (e.g., donepezil, epoetin alfa); (3) drugs with alternative biologic explanations for the association (e.g., sevelamer); and (4) drugs that may have resulted in statistical alerts due to their correlation with drugs that alerted for other reasons. CONCLUSIONS: This exploratory drug-screening study identified several potential targets for follow-up studies to further evaluate and determine if they may prevent or delay progression to advanced AMD.

Improving Transparency to Build Trust in Real-World Secondary Data Studies for Hypothesis Testing-Why, What, and How: Recommendations and a Road Map from the Real-World Evidence Transparency Initiative.

Real-world data (RWD) and the derivations of these data into real-world evidence (RWE) are rapidly expanding from informing healthcare decisions at the patient and health system level to influencing major health policy decisions, including regulatory approvals and coverage. Recent examples include the approval of palbociclib in combination with endocrine therapy for male breast cancer and the inclusion of RWE in the label of paliperidone palmitate for schizophrenia. This interest has created an urgency to develop processes that promote trust in the evidence-generation process. Key stakeholders and decision-makers include patients and their healthcare providers; learning health systems; health technology assessment bodies and payers; pharmacoepidemiologists and other clinical reseachers, and policy makers interested in bioethical and regulatory issues. A key to optimal uptake of RWE is transparency of the research process to enable decision-makers to evaluate the quality of the methods used and the applicability of the evidence that results from the RWE studies. Registration of RWE studies-particularly for hypothesis evaluating treatment effectiveness (HETE) studies-has been proposed to improve transparency, trust, and research replicability. Although registration would not guarantee better RWE studies would be conducted, it would encourage the prospective disclosure of study plans, timing, and rationale for modifications. A joint task force of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) recommended that investigators preregister their RWE studies and post their study protocols in a publicly available forum before starting studies to reduce publication bias and improve the transparency of research methods. Recognizing that published recommendations alone are insufficient, especially without accessible registration options and with no incentives, a group of experts gathered on February 25 and 26, 2019, in National Harbor, Maryland, to explore the structural and practical challenges to the successful implementation of the recommendations of the ISPOR/ISPE task force for preregistration. This positioning article describes a plan for making registration of HETE RWE studies routine. The plan includes specifying the rationale for registering HETE RWE studies, the studies that should be registered, where and when these studies should be registered, how and when analytic deviations from protocols should be reported, how and when to publish results, and incentives to encourage registration. Table 1 summarizes the rationale, goals, and potential solutions that increase transparency, in addition to unique concerns about secondary data studies. Definitions of terms used throughout this report are provided in Table 2.

Improving transparency to build trust in real-world secondary data studies for hypothesis testing-Why, what, and how: recommendations and a road map from the real-world evidence transparency initiative.

Real-world data (RWD) and the derivations of these data into real-world evidence (RWE) are rapidly expanding from informing healthcare decisions at the patient and health system level to influencing major health policy decisions, including regulatory approvals and coverage. Recent examples include the approval of palbociclib in combination with endocrine therapy for male breast cancer and the inclusion of RWE in the label of paliperidone palmitate for schizophrenia. This interest has created an urgency to develop processes that promote trust in the evidence-generation process. Key stakeholders and decision-makers include patients and their healthcare providers; learning health systems; health technology assessment bodies and payers; pharmacoepidemiologists and other clinical reseachers, and policy makers interested in bioethical and regulatory issues. A key to optimal uptake of RWE is transparency of the research process to enable decision-makers to evaluate the quality of the methods used and the applicability of the evidence that results from the RWE studies. Registration of RWE studies-particularly for hypothesis evaluating treatment effectiveness (HETE) studies-has been proposed to improve transparency, trust, and research replicability. Although registration would not guarantee better RWE studies would be conducted, it would encourage the prospective disclosure of study plans, timing, and rationale for modifications. A joint task force of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) recommended that investigators preregister their RWE studies and post their study protocols in a publicly available forum before starting studies to reduce publication bias and improve the transparency of research methods. Recognizing that published recommendations alone are insufficient, especially without accessible registration options and with no incentives, a group of experts gathered on February 25 and 26, 2019, in National Harbor, Maryland, to explore the structural and practical challenges to the successful implementation of the recommendations of the ISPOR/ISPE task force for preregistration. This positioning article describes a plan for making registration of HETE RWE studies routine. The plan includes specifying the rationale for registering HETE RWE studies, the studies that should be registered, where and when these studies should be registered, how and when analytic deviations from protocols should be reported, how and when to publish results, and incentives to encourage registration. Table 1 summarizes the rationale, goals, and potential solutions that increase transparency, in addition to unique concerns about secondary data studies. Definitions of terms used throughout this report are provided in Table 2.

Comparison of Machine Learning Methods With Traditional Models for Use of Administrative Claims With Electronic Medical Records to Predict Heart Failure Outcomes.

Importance: Accurate risk stratification of patients with heart failure (HF) is critical to deploy targeted interventions aimed at improving patients' quality of life and outcomes. Objectives: To compare machine learning approaches with traditional logistic regression in predicting key outcomes in patients with HF and evaluate the added value of augmenting claims-based predictive models with electronic medical record (EMR)-derived information. Design, Setting, and Participants: A prognostic study with a 1-year follow-up period was conducted including 9502 Medicare-enrolled patients with HF from 2 health care provider networks in Boston, Massachusetts ("providers" includes physicians, clinicians, other health care professionals, and their institutions that comprise the networks). The study was performed from January 1, 2007, to December 31, 2014; data were analyzed from January 1 to December 31, 2018. Main Outcomes and Measures: All-cause mortality, HF hospitalization, top cost decile, and home days loss greater than 25% were modeled using logistic regression, least absolute shrinkage and selection operation regression, classification and regression trees, random forests, and gradient-boosted modeling (GBM). All models were trained using data from network 1 and tested in network 2. After selecting the most efficient modeling approach based on discrimination, Brier score, and calibration, area under precision-recall curves (AUPRCs) and net benefit estimates from decision curves were calculated to focus on the differences when using claims-only vs claims + EMR predictors. Results: A total of 9502 patients with HF with a mean (SD) age of 78 (8) years were included: 6113 from network 1 (training set) and 3389 from network 2 (testing set). Gradient-boosted modeling consistently provided the highest discrimination, lowest Brier scores, and good calibration across all 4 outcomes; however, logistic regression had generally similar performance (C statistics for logistic regression based on claims-only predictors: mortality, 0.724; 95% CI, 0.705-0.744; HF hospitalization, 0.707; 95% CI, 0.676-0.737; high cost, 0.734; 95% CI, 0.703-0.764; and home days loss claims only, 0.781; 95% CI, 0.764-0.798; C statistics for GBM: mortality, 0.727; 95% CI, 0.708-0.747; HF hospitalization, 0.745; 95% CI, 0.718-0.772; high cost, 0.733; 95% CI, 0.703-0.763; and home days loss, 0.790; 95% CI, 0.773-0.807). Higher AUPRCs were obtained for claims + EMR vs claims-only GBMs predicting mortality (0.484 vs 0.423), HF hospitalization (0.413 vs 0.403), and home time loss (0.575 vs 0.521) but not cost (0.249 vs 0.252). The net benefit for claims + EMR vs claims-only GBMs was higher at various threshold probabilities for mortality and home time loss outcomes but similar for the other 2 outcomes. Conclusions and Relevance: Machine learning methods offered only limited improvement over traditional logistic regression in predicting key HF outcomes. Inclusion of additional predictors from EMRs to claims-based models appeared to improve prediction for some, but not all, outcomes.

Using Healthcare Databases to Refine Understanding of Exploratory Associations Between Drugs and Progression of Open-Angle Glaucoma.

We sought to refine understanding about associations identified in prior studies between angiotensin-II receptor blockers, metformin, selective serotonin reuptake inhibitors, fibric-acid derivatives, or calcium channel blockers and progression to glaucoma filtration surgery for open-angle glaucoma (OAG). We used new-initiator, active-comparator cohort designs to investigate these drugs in two data sources. We adjusted for confounders using stabilized inverse-probability-of-treatment weights and evaluated results using "intention-to-treat" and "as-treated" follow-up approaches. In both data sources, Kaplan-Meier curves showed trends for more rapid progression to glaucoma filtration surgery in patients taking calcium channel blockers compared with thiazides with as-treated (MarketScan P = 0.15; Medicare P = 0.03) and intention-to-treat follow-up (MarketScan P < 0.01; Medicare P = 0.10). There was suggestion of delayed progression for selective serotonin reuptake inhibitor compared with tricyclic antidepressants in Medicare, which was not observed in MarketScan. Our study provided support for a relationship between calcium channel blockers and OAG progression but not for other investigated drugs.

Evaluating the use of bootstrapping in cohort studies conducted with 1:1 propensity score matching-A plasmode simulation study.

PURPOSE: Bootstrapping can account for uncertainty in propensity score (PS) estimation and matching processes in 1:1 PS-matched cohort studies. While theory suggests that the classical bootstrap can fail to produce proper coverage, practical impact of this theoretical limitation in settings typical to pharmacoepidemiology is not well studied. METHODS: In a plasmode-based simulation study, we compared performance of the standard parametric approach, which ignores uncertainty in PS estimation and matching, with two bootstrapping methods. The first method only accounted for uncertainty introduced during the matching process (the observation resampling approach). The second method accounted for uncertainty introduced during both PS estimation and matching processes (the PS reestimation approach). Variance was estimated based on percentile and empirical standard errors, and treatment effect estimation was based on median and mean of the estimated treatment effects across 1000 bootstrap resamples. Two treatment prevalence scenarios (5% and 29%) across two treatment effect scenarios (hazard ratio of 1.0 and 2.0) were evaluated in 500 simulated cohorts of 10 000 patients each. RESULTS: We observed that 95% confidence intervals from the bootstrapping approaches but not the standard approach, resulted in inaccurate coverage rates (98%-100% for the observation resampling approach, 99%-100% for the PS reestimation approach, and 95%-96% for standard approach). Treatment effect estimation based on bootstrapping approaches resulted in lower bias than the standard approach (less than 1.4% vs 4.1%) at 5% treatment prevalence; however, the performance was equivalent at 29% treatment prevalence. CONCLUSION: Use of bootstrapping led to variance overestimation and inconsistent coverage, while coverage remained more consistent with parametric estimation.

Using Real-World Data to Extrapolate Evidence From Randomized Controlled Trials.

Randomized controlled trials (RCTs) provide evidence for regulatory agencies, shape clinical practice, influence formulary decisions, and have important implications for patients. However, many patient groups that are major consumers of drugs are under-represented in randomized trials. We review three methods to extrapolate evidence from trial participants to different target populations following market approval and discuss how these could be implemented in practice to support regulatory and health technology assessment decisions. Although these methods are not a substitute for less restrictive pre-approval RCTs or rigorous observational studies when sufficient data are available in the post-approval setting, they can help to fill the evidence gap that exists in the early marketing period. Early evidence using real-world data and methods for extrapolating evidence should be reported with clear explanation of assumptions and limitations especially when used to support regulatory and health technology assessment decisions.

Development and Preliminary Validation of a Medicare Claims-Based Model to Predict Left Ventricular Ejection Fraction Class in Patients With Heart Failure.

BACKGROUND: Ejection fraction (EF) class is an important predictor of treatment response in heart failure (HF); however, administrative claims databases lack information on EF, limiting their usefulness in clinical and health services research of HF. METHODS AND RESULTS: We linked Medicare claims data to electronic medical records containing EF measurements for a cohort of 11 073 patients with HF from 2 academic medical centers. A a claims-based model predicting EF class was constructed using data from center 1 ("training sample") and validated using data from center 2 ("testing sample). Linear and logistic regression models with least absolute square shrinkage operator and Bayesian information criteria were developed to select the relevant predictor variables out of the total 57 candidate variables in the training sample. Higher accuracy was noted in the testing sample with models classifying patients into 2 EF classes (reduced EF <0.45) versus preserved EF (≥0.45) when compared with classifying patients into 3 EF classes (reduced, <0.40, moderately reduced, 0.40-0.49, or preserved, ≥0.50). In the testing sample, the most efficient model had 35 predictors and resulted in 83% of patients being correctly classified (95% CI, 82%-84%). The model had positive predictive value of 0.73 (95% CI, 0.68-0.78) and 0.84 (95% CI, 0.83-0.86) and sensitivity of 0.29 (95% CI, 0.25-0.32) and 0.97 (95% CI, 0.97-0.98) for reduced and preserved EF, respectively. In addition to HF-specific diagnosis codes, other factors including age, sex, medication use, and comorbidities, such as myocardial infarction and valve disorders, were important discriminators between EF classes. CONCLUSIONS: The claims-based model developed in this study may be used to identify patient subgroups with specific EF class in studies evaluating the health outcomes, utilization patterns, and cost, of HF patients in routine care when EF measurements are not available.

Good Practices for Real-World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR-ISPE Special Task Force on Real-World Evidence in Health Care Decision Making.

PURPOSE: Real-world evidence (RWE) includes data from retrospective or prospective observational studies and observational registries and provides insights beyond those addressed by randomized controlled trials. RWE studies aim to improve health care decision making. METHODS: The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) created a task force to make recommendations regarding good procedural practices that would enhance decision makers' confidence in evidence derived from RWD studies. Peer review by ISPOR/ISPE members and task force participants provided a consensus-building iterative process for the topics and framing of recommendations. RESULTS: The ISPOR/ISPE Task Force recommendations cover seven topics such as study registration, replicability, and stakeholder involvement in RWE studies. These recommendations, in concert with earlier recommendations about study methodology, provide a trustworthy foundation for the expanded use of RWE in health care decision making. CONCLUSION: The focus of these recommendations is good procedural practices for studies that test a specific hypothesis in a specific population. We recognize that some of the recommendations in this report may not be widely adopted without appropriate incentives from decision makers, journal editors, and other key stakeholders.

Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0.

PURPOSE: Defining a study population and creating an analytic dataset from longitudinal healthcare databases involves many decisions. Our objective was to catalogue scientific decisions underpinning study execution that should be reported to facilitate replication and enable assessment of validity of studies conducted in large healthcare databases. METHODS: We reviewed key investigator decisions required to operate a sample of macros and software tools designed to create and analyze analytic cohorts from longitudinal streams of healthcare data. A panel of academic, regulatory, and industry experts in healthcare database analytics discussed and added to this list. CONCLUSION: Evidence generated from large healthcare encounter and reimbursement databases is increasingly being sought by decision-makers. Varied terminology is used around the world for the same concepts. Agreeing on terminology and which parameters from a large catalogue are the most essential to report for replicable research would improve transparency and facilitate assessment of validity. At a minimum, reporting for a database study should provide clarity regarding operational definitions for key temporal anchors and their relation to each other when creating the analytic dataset, accompanied by an attrition table and a design diagram. A substantial improvement in reproducibility, rigor and confidence in real world evidence generated from healthcare databases could be achieved with greater transparency about operational study parameters used to create analytic datasets from longitudinal healthcare databases.

A review of the performance of different methods for propensity score matched subgroup analyses and a summary of their application in peer-reviewed research studies.

PURPOSE: When evaluating safety signals, there is often interest in understanding safety in all patients for whom compared treatments are reasonable alternatives, as well as in specific subgroups of interest. There are numerous ways that propensity score (PS) matching can be implemented for subgroup analyses. METHODS: We conducted a systematic literature review of methods papers that compared the performance of alternative methods to implement PS matched subgroup analyses and examined how frequently different PS matching methods have been used for subgroup analyses in applied studies. RESULTS: We identified 5 methods papers reporting small improvements in covariate balance and bias with use of a subgroup-specific PS instead of a mis-specified overall PS within subgroups. Applied research papers frequently used PS for subgroups in ways not evaluated in methods papers. Thirty three percent used PS to match in the overall cohort and broke the matched sets for subgroup analysis without further adjustment. CONCLUSIONS: While the performance of several alternative ways to use PS matching in subgroup analyses has been evaluated in methods literature, these evaluations do not include the most commonly used methods to implement PS matched subgroup analyses in applied studies. There is a need to better understand the relative performance of commonly used methods for PS matching in subgroup analyses, particularly within settings encountered during active surveillance, where there may be low exposure, infrequent outcomes, and multiple subgroups of interest.

Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0.

PURPOSE: Defining a study population and creating an analytic dataset from longitudinal healthcare databases involves many decisions. Our objective was to catalogue scientific decisions underpinning study execution that should be reported to facilitate replication and enable assessment of validity of studies conducted in large healthcare databases. METHODS: We reviewed key investigator decisions required to operate a sample of macros and software tools designed to create and analyze analytic cohorts from longitudinal streams of healthcare data. A panel of academic, regulatory, and industry experts in healthcare database analytics discussed and added to this list. CONCLUSION: Evidence generated from large healthcare encounter and reimbursement databases is increasingly being sought by decision-makers. Varied terminology is used around the world for the same concepts. Agreeing on terminology and which parameters from a large catalogue are the most essential to report for replicable research would improve transparency and facilitate assessment of validity. At a minimum, reporting for a database study should provide clarity regarding operational definitions for key temporal anchors and their relation to each other when creating the analytic dataset, accompanied by an attrition table and a design diagram. A substantial improvement in reproducibility, rigor and confidence in real world evidence generated from healthcare databases could be achieved with greater transparency about operational study parameters used to create analytic datasets from longitudinal healthcare databases.

Good practices for real-world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making.

PURPOSE: Real-world evidence (RWE) includes data from retrospective or prospective observational studies and observational registries and provides insights beyond those addressed by randomized controlled trials. RWE studies aim to improve health care decision making. METHODS: The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) created a task force to make recommendations regarding good procedural practices that would enhance decision makers' confidence in evidence derived from RWD studies. Peer review by ISPOR/ISPE members and task force participants provided a consensus-building iterative process for the topics and framing of recommendations. RESULTS: The ISPOR/ISPE Task Force recommendations cover seven topics such as study registration, replicability, and stakeholder involvement in RWE studies. These recommendations, in concert with earlier recommendations about study methodology, provide a trustworthy foundation for the expanded use of RWE in health care decision making. CONCLUSION: The focus of these recommendations is good procedural practices for studies that test a specific hypothesis in a specific population. We recognize that some of the recommendations in this report may not be widely adopted without appropriate incentives from decision makers, journal editors, and other key stakeholders.

Switching generic antiepileptic drug manufacturer not linked to seizures: A case-crossover study.

OBJECTIVE: With more antiepileptic drugs (AED) becoming available in generic form, we estimated the risk of seizure-related events associated with refilling generic AEDs and the effect of switching between different manufacturers of the same generic drug. METHODS: We designed a population-based case-crossover study using the Medicaid Analytic eXtract and a US commercial health insurance database. We identified 83,001 generic AED users who experienced a seizure-related hospital admission or emergency room visit between 2000 and 2013 and assessed whether they received a refill of the same AED from the same manufacturer or a different manufacturer. Patients served as their own controls and conditional logistic regression was used to compare exposure to a refill during the hazard period, defined as days 2-36 preceding the seizure-related event, to exposure during the control period, defined as days 51-85 preceding the seizure-related event. RESULTS: Generic AED refilling was associated with an 8% increase in the odds of seizure-related events (odds ratio [OR] 1.08; 95% confidence interval [CI] 1.06-1.11). The OR following a switch to a different manufacturer of the same AED was 1.09 (95% CI 1.03-1.15); however, after adjusting for the process of refilling, there was no association between switching and seizure-related hospital visits (OR 1.00; 95% CI 0.94-1.07). CONCLUSIONS: Among patients on a generic AED, refilling the same AED was associated with an elevated risk of seizure-related event; however, there was no additional risk from switching during that refill to a different manufacturer. Generic AEDs available to US patients, with Food and Drug Administration-validated bioequivalence, appear to be safe clinical choices.