Triple challenges - Small sample size in both exposure and control groups to scan rare maternal outcomes in a signal identification approach: A simulation study.

There is a dearth of safety data on maternal outcomes after perinatal medication exposure. Data-mining for unexpected adverse event occurrence in existing datasets is a potentially useful approach. One method, the Poisson tree-based scan statistic (TBSS), assumes that the expected outcome counts, based on incidence of outcomes in the control group, are estimated without error. This assumption may be difficult to satisfy with a small control group. Our simulation study evaluated the effect of imprecise incidence proportions from the control group on TBSS' ability to identify maternal outcomes in pregnancy research. We simulated base case analyses with "true" expected incidence proportions and compared these to imprecise incidence proportions derived from sparse control samples. We varied parameters impacting Type I error and statistical power (exposure group size, outcome's incidence proportion, and effect size). We found that imprecise incidence proportions generated by a small control group resulted in inaccurate alerting, inflation of Type I error, and removal of very rare outcomes for TBSS analysis due to "zero" background counts. Ideally, the control size should be at least several times larger than the exposure size to limit the number of false positive alerts and retain statistical power for true alerts.

Transparency, reproducibility, and replicability of pharmacoepidemiology studies in a distributed network environment.

PURPOSE: Our objective is to describe how the U.S. Food and Drug Administration (FDA)'s Sentinel System implements best practices to ensure trust in drug safety studies using real-world data from disparate sources. METHODS: We present a stepwise schematic for Sentinel's data harmonization, data quality check, query design and implementation, and reporting practices, and describe approaches to enhancing the transparency, reproducibility, and replicability of studies at each step. CONCLUSIONS: Each Sentinel data partner converts its source data into the Sentinel Common Data Model. The transformed data undergoes rigorous quality checks before it can be used for Sentinel queries. The Sentinel Common Data Model framework, data transformation codes for several data sources, and data quality assurance packages are publicly available. Designed to run against the Sentinel Common Data Model, Sentinel's querying system comprises a suite of pre-tested, parametrizable computer programs that allow users to perform sophisticated descriptive and inferential analysis without having to exchange individual-level data across sites. Detailed documentation of capabilities of the programs as well as the codes and information required to execute them are publicly available on the Sentinel website. Sentinel also provides public trainings and online resources to facilitate use of its data model and querying system. Its study specifications conform to established reporting frameworks aimed at facilitating reproducibility and replicability of real-world data studies. Reports from Sentinel queries and associated design and analytic specifications are available for download on the Sentinel website. Sentinel is an example of how real-world data can be used to generate regulatory-grade evidence at scale using a transparent, reproducible, and replicable process.

Sequential Data-Mining for Adverse Events After Recombinant Herpes Zoster Vaccination Using the Tree-Based Scan Statistic.

Tree-based scan statistics have been successfully used to study the safety of several vaccines without prespecifying health outcomes of concern. In this study, the binomial tree-based scan statistic was applied sequentially to detect adverse events in days 1-28 compared with days 29-56 after recombinant herpes zoster (RZV) vaccination, with 5 looks at the data and formal adjustment for the repeated analyses over time. IBM MarketScan data on commercially insured persons ≥50 years of age receiving RZV during January 1, 2018, to May 5, 2020, were used. With 999,876 doses of RZV included, statistically significant signals were detected only for unspecified adverse effects/complications following immunization, with attributable risks as low as 2 excess cases per 100,000 vaccinations. Ninety percent of cases in the signals occurred in the week after vaccination and, based on previous studies, likely represent nonserious events like fever, fatigue, and headache. Strengths of our study include its untargeted nature, self-controlled design, and formal adjustment for repeated testing. Although the method requires prespecification of the risk window of interest and may miss some true signals detectable using the tree-temporal variant of the method, it allows for early detection of potential safety problems through early initiation of ongoing monitoring.

Monitoring Drug Safety in Pregnancy with Scan Statistics: A Comparison of Two Study Designs.

BACKGROUND: Traditional surveillance of adverse infant outcomes following maternal medication exposures relies on pregnancy exposure registries, which are often underpowered. We characterize the statistical power of TreeScan, a data mining tool, to identify potential signals in the setting of perinatal medication exposures and infant outcomes. METHODS: We used empirical data to inform background incidence of major congenital malformations and other birth conditions. Statistical power was calculated using two probability models compatible with TreeScan, Bernoulli and Poisson, while varying the sample size, magnitude of the risk increase, and incidence of a specified outcome. We also simulated larger referent to exposure matching ratios when using the Bernoulli model in the setting of fixed N:1 propensity score matching. Finally, we assessed the impact of outcome misclassification on power. RESULTS: The Poisson model demonstrated greater power to detect signals than the Bernoulli model across all scenarios and suggested a sample size of 4,000 exposed pregnancies is needed to detect a twofold increase in risk of a common outcome (approximately 8 per 1,000) with 85% power. Increasing the fixed matching ratio with the Bernoulli model did not reliably increase power. An outcome definition with high sensitivity is expected to have somewhat greater power to detect signals than an outcome definition with high positive predictive value. CONCLUSIONS: Use of the Poisson model with an outcome definition that prioritizes sensitivity may be optimal for signal detection. TreeScan is a viable method for surveillance of adverse infant outcomes following maternal medication use.

Assessing medical product safety during pregnancy using parameterizable tools in the sentinel distributed database.

PURPOSE: The US Food and Drug Administration established the Sentinel System to monitor the safety of medical products. A component of this system includes parameterizable analytic tools to identify mother-infant pairs and evaluate infant outcomes to enable the routine monitoring of the utilization and safety of drugs used in pregnancy. We assessed the feasibility of using the data and tools in the Sentinel System by assessing a known association between topiramate use during pregnancy and oral clefts in the infant. METHODS: We identified mother-infant pairs using the mother-infant linkage table from six data partners contributing to the Sentinel Distributed Database from January 1, 2000, to September 30, 2015. We compared mother-infant pairs with first-trimester exposure to topiramate to mother-infant pairs that were topiramate-unexposed or lamotrigine-exposed and used a validated algorithm to identify oral clefts in the infant. We estimated adjusted risk ratios through propensity score stratification. RESULTS: There were 2007 topiramate-exposed and 1 066 086 unexposed mother-infant pairs in the main comparison. In the active-comparator analysis, there were 1996 topiramate-exposed and 2859 lamotrigine-exposed mother-infant pairs. After propensity score stratification, the odds ratio for oral clefts was 2.92 (95% CI: 1.43, 5.93) comparing the topiramate-exposed to unexposed groups and 2.72 (95% CI: 0.75, 9.93) comparing the topiramate-exposed to lamotrigine-exposed groups. CONCLUSIONS: We found an increased risk of oral clefts after topiramate exposure in the first trimester in the Sentinel database. These results are similar to prior published observational study results and demonstrate the ability of Sentinel's data and analytic tools to assess medical product safety in cohorts of mother-infant pairs in a timely manner.

Novel methods for pregnancy drug safety surveillance in the FDA Sentinel System.

PURPOSE: It is a priority of the US Food and Drug Administration (FDA) to monitor the safety of medications used during pregnancy. Pregnancy exposure registries and cohort studies utilizing electronic health record data are primary sources of information but are limited by small sample sizes and limited outcome assessment. TreeScan™, a statistical data mining tool, can be applied within the FDA Sentinel System to simultaneously identify multiple potential adverse neonatal and infant outcomes after maternal medication exposure. METHODS: We implemented TreeScan using the Sentinel analytic tools in a cohort of linked live birth deliveries and infants nested in the IBM MarketScan® Research Database. As a case study, we compared first trimester fluoroquinolone use and cephalosporin use. We used the Bernoulli and Poisson TreeScan statistics with compatible propensity score-based study designs for confounding control (matching and stratification) and used multiple propensity score models with various strategies for confounding control to inform best practices. We developed a hierarchical outcome tree including major congenital malformations and outcomes of gestational length and birth weight. RESULTS: A total of 1791 fluoroquinolone-exposed and 8739 cephalosporin-exposed mother-infant pairs were eligible for analysis. Both TreeScan analysis methods resulted in single alerts that were deemed to be due to uncontrolled confounding or otherwise not warranting follow-up. CONCLUSIONS: In this implementation of TreeScan using Sentinel analytic tools, we did not observe any new safety signals for fluoroquinolone use in the first trimester. TreeScan, with tailored or high-dimensional propensity scores for confounding control, is a valuable tool in addition to current safety surveillance methods for medications used during pregnancy.

Invited Commentary: Go BIG and Go Global-Executing Large-Scale, Multisite Pharmacoepidemiologic Studies Using Real-World Data.

At the time medical products are approved, we rarely know enough about their comparative safety and effectiveness vis-à-vis alternative therapies to advise patients and providers. Postmarket generation of evidence on rare adverse events following medical product exposure increasingly requires analysis of millions of longitudinal patient records that can provide complete capture of data on patient experiences. In the accompanying article by Pradhan et al. (Am J Epidemiology. 2022;191(8):1352-1367), the authors demonstrate how observational database studies are often the most practical approach, provided these databases are carefully chosen to be "fit for purpose." Distributed data networks with common data models have proliferated in the last 2 decades in pharmacoepidemiology, allowing efficient capture of patient data in a standardized and structured format across disparate real-world data sources. Use of common data models facilitates transparency by allowing standardized programming approaches that can be easily reproduced. The distributed data network architecture, combined with a common data approach, supports not only multisite observational studies but also pragmatic clinical trials. It also helps bridge international boundaries and further increases the sample size and diversity of study populations.

A Broad Safety Assessment of the Recombinant Herpes Zoster Vaccine.

The recombinant herpes zoster vaccine (RZV), approved as a 2-dose series in the United States in October 2017, has proven highly effective and generally safe. However, a small risk of Guillain-Barré syndrome after vaccination was identified after approval, and questions remain about other possible adverse events. This data-mining study assessed RZV safety in the United States using the self-controlled tree-temporal scan statistic, scanning data on thousands of diagnoses recorded during follow-up to detect any statistically unusual temporal clustering of cases within a large hierarchy of diagnoses. IBM MarketScan data on commercially insured persons at least 50 years of age receiving RZV between January 1, 2018, and May 5, 2020, were used, including 56 days of follow-up; 1,014,329 doses were included. Statistically significant clustering was found within a few days of vaccination for unspecified adverse effects, complications, or reactions to immunization or other medical substances/care; fever; unspecified allergy; syncope/collapse; cellulitis; myalgia; and dizziness/giddiness. These findings are consistent with the known safety profile of this and other injected vaccines. No cluster of Guillain-Barré syndrome was detected, possibly due to insufficient sample size. This signal-detection method has now been applied to 5 vaccines, with consistently plausible results, and seems a promising addition to vaccine-safety evaluation methods.

Utility of fertility procedures and prenatal tests to estimate gestational age for live-births and stillbirths in electronic health plan databases.

PURPOSE: Current algorithms to evaluate gestational age (GA) during pregnancy rely on hospital coding at delivery and are not applicable to non-live births. We developed an algorithm using fertility procedures and fertility tests, without relying on delivery coding, to develop a novel GA algorithm in live-births and stillbirths. METHODS: Three pregnancy cohorts were identified from 16 health-plans in the Sentinel System: 1) hospital admissions for live-birth, 2) hospital admissions for stillbirth, and 3) medical chart-confirmed stillbirths. Fertility procedures and prenatal tests, recommended within specific GA windows were evaluated for inclusion in our GA algorithm. Our GA algorithm was developed against a validated delivery-based GA algorithm in live-births, implemented within a sample of chart-confirmed stillbirths, and compared to national estimates of GA at stillbirth. RESULTS: Our algorithm, including fertility procedures and 11 prenatal tests, assigned a GA at delivery to 97.9% of live-births and 92.6% of stillbirths. For live-births (n = 4 701 207), it estimated GA within 2 weeks of a reference delivery-based GA algorithm in 82.5% of pregnancies, with a mean difference of 3.7 days. In chart-confirmed stillbirths (n = 49), it estimated GA within 2 weeks of the clinically recorded GA at delivery for 80% of pregnancies, with a mean difference of 11.1 days. Implementation of the algorithm in a cohort of stillbirths (n = 40 484) had an increased percentage of deliveries after 36 weeks compared to national estimates. CONCLUSIONS: In a population of primarily commercially-insured pregnant women, fertility procedures and prenatal tests can estimate GA with sufficient sensitivity and accuracy for utility in pregnancy studies.

A Broad Safety Assessment of the 9-Valent Human Papillomavirus Vaccine.

Parents indicate that safety is their top concern about human papillomavirus (HPV) vaccination. A data-mining method not requiring prespecification of health outcome(s) or postexposure period(s) of potentially increased risk can be used to identify possible associations between an exposure and any of thousands of medically attended health outcomes; this method was applied to data on the 9-valent HPV vaccine (HPV9) to detect potential safety problems. Data on 9- to 26-year-olds who had received HPV9 vaccine between November 4, 2016, and August 5, 2018, inclusive, were extracted from the MarketScan database and analyzed for statistically significant clustering of incident diagnoses within the hierarchy of diagnoses coded using the International Classification of Diseases and temporally within the 1 year after vaccination, using the self-controlled tree-temporal scan statistic and TreeScan software. Only 56 days of postvaccination enrollment was required; subsequent follow-up was censored at disenrollment. Multiple testing was adjusted for. The analysis included 493,089 doses of HPV9. Almost all signals resulted from temporal confounding, not unexpected with a 1-year follow-up period. The only plausible signals were for nonspecific adverse events (e.g., injection-site reactions, headache) on days 1-2 after vaccination, with attributable risks as low as 1 per 100,000 vaccinees. Considering the broad scope of the evaluation and the high statistical power, the findings of no specific serious adverse events should provide reassurance about this vaccine's safety.

A General Propensity Score for Signal Identification Using Tree-Based Scan Statistics.

The tree-based scan statistic (TreeScan; Martin Kulldorff, Harvard Medical School, Boston, Massachusetts) is a data-mining method that adjusts for multiple testing of correlated hypotheses when screening thousands of potential adverse events for signal identification. Simulation has demonstrated the promise of TreeScan with a propensity score (PS)-matched cohort design. However, it is unclear which variables to include in a PS for applied signal identification studies to simultaneously adjust for confounding across potential outcomes. We selected 4 pairs of medications with well-understood safety profiles. For each pair, we evaluated 5 candidate PSs with different combinations of 1) predefined general covariates (comorbidity, frailty, utilization), 2) empirically selected (data-driven) covariates, and 3) covariates tailored to the drug pair. For each pair, statistical alerting patterns were similar with alternative PSs (≤11 alerts in 7,996 outcomes scanned). Inclusion of covariates tailored to exposure did not appreciably affect screening results. Inclusion of empirically selected covariates can provide better proxy coverage for confounders but can also decrease statistical power. Unlike tailored covariates, empirical and predefined general covariates can be applied "out of the box" for signal identification. The choice of PS depends on the level of concern about residual confounding versus loss of power. Potential signals should be followed by pharmacoepidemiologic assessment where confounding control is tailored to the specific outcome(s) under investigation.

Health outcomes coding trends in the US Food and Drug Administration's Sentinel System during transition to International Classification of Diseases-10 coding system: A brief review.

BACKGROUND AND PURPOSE: The transition from International Classification of Diseases, 9th revision, clinical modification (ICD-9-CM) to ICD-10-CM poses a challenge to epidemiologic studies that use diagnostic codes to identify health outcomes and covariates. We evaluated coding trends in health outcomes in the US Food and Drug Administration's Sentinel System during the transition. METHODS: We reviewed all health outcomes coding trends reports on the Sentinel website through November 30, 2019 and analyzed trends in incidence and prevalence across the ICD-9-CM and ICD-10-CM eras by visual inspection. RESULTS: We identified 78 unique health outcomes (22 acute, 32 chronic, and 24 acute or chronic) and 140 time-series graphs of incidence and prevalence. The reports also included code lists and code mapping methods used. Of the 140 graphs reviewed, 81 (57.9%) showed consistent trends across the ICD-9-CM and ICD-10-CM eras, while 51 (36.4%) and 8 (5.7%) graphs showed inconsistent and uncertain trends, respectively. Chronic HOIs and acute/chronic HOIs had higher proportions of consistent trends in prevalence definitions (83.9% and 78.3%, respectively) than acute HOIs (28.6%). For incidence, 55.6% of acute HOIs showed consistent trends, while 41.2% of chronic HOIs and 39.3% of acute/chronic HOIs showed consistency. CONCLUSIONS: Researchers using ICD-10-CM algorithms obtained by standardized mappings from ICD-9-CM algorithms should assess the mapping performance before use. The Sentinel reports provide a valuable resource for researchers who need to develop and assess mapping strategies. The reports could benefit from additional information about the algorithm selection process and additional details on monthly incidence and prevalence rates. KEY POINTS: We reviewed health outcomes coding trends reports on the US FDA Sentinel website through November 30, 2019 and analyzed trends in incidence and prevalence across the International Classification of Diseases, 9th revision, Clinical Modification (ICD-9-CM) and ICD-10-CM eras by code mapping method and the type of health outcomes of interest (acute, chronic, acute or chronic). More than a third of the 140 time-series graphs of incidence and prevalence of health outcomes showed inconsistent or uncertain trends. Consistency in trends varied by code mapping method, type of health outcomes of interest, and whether the measurement was incidence or prevalence. Studies using ICD-9-CM-based algorithms mapped to ICD-10-CM codes need to assess the performance of the mappings and conduct manual refinement of the algorithms as needed before using them.

A COVID-19-ready public health surveillance system: The Food and Drug Administration's Sentinel System.

The US Food and Drug Administration's Sentinel System was established in 2009 to use routinely collected electronic health data for improving the national capability to assess post-market medical product safety. Over more than a decade, Sentinel has become an integral part of FDA's surveillance capabilities and has been used to conduct analyses that have contributed to regulatory decisions. FDA's role in the COVID-19 pandemic response has necessitated an expansion and enhancement of Sentinel. Here we describe how the Sentinel System has supported FDA's response to the COVID-19 pandemic. We highlight new capabilities developed, key data generated to date, and lessons learned, particularly with respect to working with inpatient electronic health record data. Early in the pandemic, Sentinel developed a multi-pronged approach to support FDA's anticipated data and analytic needs. It incorporated new data sources, created a rapidly refreshed database, developed protocols to assess the natural history of COVID-19, validated a diagnosis-code based algorithm for identifying patients with COVID-19 in administrative claims data, and coordinated with other national and international initiatives. Sentinel is poised to answer important questions about the natural history of COVID-19 and is positioned to use this information to study the use, safety, and potentially the effectiveness of medical products used for COVID-19 prevention and treatment.

Validity of ICD-10-CM diagnoses to identify hospitalizations for serious infections among patients treated with biologic therapies.

PURPOSE: Identifying hospitalizations for serious infections among patients dispensed biologic therapies within healthcare databases is important for post-marketing surveillance of these drugs. We determined the positive predictive value (PPV) of an ICD-10-CM-based diagnostic coding algorithm to identify hospitalization for serious infection among patients dispensed biologic therapy within the FDA's Sentinel Distributed Database. METHODS: We identified health plan members who met the following algorithm criteria: (1) hospital ICD-10-CM discharge diagnosis of serious infection between July 1, 2016 and August 31, 2018; (2) either outpatient/emergency department infection diagnosis or outpatient antimicrobial treatment within 7 days prior to hospitalization; (3) inflammatory bowel disease, psoriasis, or rheumatological diagnosis within 1 year prior to hospitalization, and (4) were dispensed outpatient biologic therapy within 90 days prior to admission. Medical records were reviewed by infectious disease clinicians to adjudicate hospitalizations for serious infection. The PPV (95% confidence interval [CI]) for confirmed events was determined after further weighting by the prevalence of the type of serious infection in the database. RESULTS: Among 223 selected health plan members who met the algorithm, 209 (93.7% [95% CI, 90.1%-96.9%]) were confirmed to have a hospitalization for serious infection. After weighting by the prevalence of the type of serious infection, the PPV of the ICD-10-CM algorithm identifying a hospitalization for serious infection was 80.2% (95% CI, 75.3%-84.7%). CONCLUSIONS: The ICD-10-CM-based algorithm for hospitalization for serious infection among patients dispensed biologic therapies within the Sentinel Distributed Database had 80% PPV for confirmed events and could be considered for use within pharmacoepidemiologic studies.

Validation of an electronic algorithm for Hodgkin and non-Hodgkin lymphoma in ICD-10-CM.

PURPOSE: Lymphoma is a health outcome of interest for drug safety studies. Studies using administrative claims data require the accurate identification of lymphoma cases. We developed and validated an International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM)-based algorithm to identify lymphoma in healthcare claims data. METHODS: We developed a three-component algorithm to identify patients aged ≥15 years who were newly diagnosed with Hodgkin (HL) or non-Hodgkin (NHL) lymphoma from January 2016 through July 2018 among members of four Data Partners within the FDA's Sentinel System. The algorithm identified potential cases as patients with ≥2 ICD-10-CM lymphoma diagnosis codes on different dates within 183 days; ≥1 procedure code for a diagnostic procedure (e.g., biopsy, flow cytometry) and ≥1 procedure code for a relevant imaging study within 90 days of the first lymphoma diagnosis code. Cases identified by the algorithm were adjudicated via chart review and a positive predictive value (PPV) was calculated. RESULTS: We identified 8723 potential lymphoma cases via the algorithm and randomly sampled 213 for validation. We retrieved 138 charts (65%) and adjudicated 134 (63%). The overall PPV was 77% (95% confidence interval: 69%-84%). Most cases also had subtype information available, with 88% of cases identified as NHL and 11% as HL. CONCLUSIONS: Seventy-seven percent of lymphoma cases identified by an algorithm based on ICD-10-CM diagnosis and procedure codes and applied to claims data were true cases. This novel algorithm represents an efficient, cost-effective way to target an important health outcome of interest for large-scale drug safety and public health surveillance studies.

Leveraging the Capabilities of the FDA's Sentinel System To Improve Kidney Care.

The Sentinel System is a national electronic postmarketing resource established by the US Food and Drug Administration to support assessment of the safety and effectiveness of marketed medical products. It has built a large, multi-institutional, distributed data network that contains comprehensive electronic health data, covering about 700 million person-years of longitudinal observation time nationwide. With its sophisticated infrastructure and a large selection of flexible analytic tools, the Sentinel System permits rapid and secure analyses, while preserving patient privacy and health-system autonomy. The Sentinel System also offers enhanced capabilities, including accessing full-text medical records, supporting randomized clinical trials embedded in healthcare delivery systems, and facilitating effective collection of patient-reported data using mobile devices, among many other research programs. The nephrology research community can use the infrastructure, tools, and data that this national resource offers for evidence generation. This review summarizes the Sentinel System and its ability to rapidly generate high-quality, real-world evidence; discusses the program's experience in, and potential for, addressing gaps in kidney care; and outlines avenues for conducting research, leveraging this national resource in collaboration with Sentinel investigators.

Using electronic health records to identify candidates for human immunodeficiency virus pre-exposure prophylaxis: An application of super learning to risk prediction when the outcome is rare.

Human immunodeficiency virus (HIV) pre-exposure prophylaxis (PrEP) protects high risk patients from becoming infected with HIV. Clinicians need help to identify candidates for PrEP based on information routinely collected in electronic health records (EHRs). The greatest statistical challenge in developing a risk prediction model is that acquisition is extremely rare. METHODS: Data consisted of 180 covariates (demographic, diagnoses, treatments, prescriptions) extracted from records on 399 385 patient (150 cases) seen at Atrius Health (2007-2015), a clinical network in Massachusetts. Super learner is an ensemble machine learning algorithm that uses k-fold cross validation to evaluate and combine predictions from a collection of algorithms. We trained 42 variants of sophisticated algorithms, using different sampling schemes that more evenly balanced the ratio of cases to controls. We compared super learner's cross validated area under the receiver operating curve (cv-AUC) with that of each individual algorithm. RESULTS: The least absolute shrinkage and selection operator (lasso) using a 1:20 class ratio outperformed the super learner (cv-AUC = 0.86 vs 0.84). A traditional logistic regression model restricted to 23 clinician-selected main terms was slightly inferior (cv-AUC = 0.81). CONCLUSION: Machine learning was successful at developing a model to predict 1-year risk of acquiring HIV based on a physician-curated set of predictors extracted from EHRs.

Quantifying how small variations in design elements affect risk in an incident cohort study in claims.

BACKGROUND: Epidemiological study reporting is improving but is not transparent enough for easy evaluation or replication. One barrier is insufficient details about design elements in published studies. METHODS: Using a previously conducted drug safety evaluation in claims as a test case, we investigated the impact of small changes in five key design elements on risk estimation. These elements are index day of incident exposure's determination of look-back or follow-up periods, exposure duration algorithms, heparin exposure exclusion, propensity score model variables, and Cox proportional hazard model stratification. We covaried these elements using a fractional factorial design, resulting in 24 risk estimates for one outcome. We repeated eight of these combinations for two additional outcomes. We measured design effects on cohort sizes, follow-up time, and risk estimates. RESULTS: Small changes in specifications of index day and exposure algorithm affected the risk estimation process the most. They affected cohort size on average by 8 to 10%, follow-up time by up to 31%, and magnitude of log hazard ratios by up to 0.22. Other elements affected cohort before matching or risk estimate's precision but not its magnitude. Any change in design substantially altered the matched control-group subjects in 1:1 matching. CONCLUSIONS: Exposure-related design elements require attention from investigators initiating, evaluating, or wishing to replicate a study or from analysts standardizing definitions. The methods we developed, using factorial design and mapping design effect on causal estimation process, are applicable to planning of sensitivity analyses in similar studies.

Using the Self-Controlled Tree-Temporal Scan Statistic to Assess the Safety of Live Attenuated Herpes Zoster Vaccine.

The self-controlled tree-temporal scan statistic allows detection of potential vaccine- or drug-associated adverse events without prespecifying the specific events or postexposure risk intervals of concern. It thus opens a promising new avenue for safety studies. The method has been successfully used to evaluate the safety of 2 vaccines for adolescents and young adults, but its suitability to study vaccines for older adults had not been established. The present study applied the method to assess the safety of live attenuated herpes zoster vaccination during 2011-2017 in US adults aged ≥60 years, using claims data from Truven Health MarketScan Research Databases. Counts of International Classification of Diseases diagnosis codes recorded in emergency department or hospital settings were scanned for any statistically unusual clustering within a hierarchical tree structure of diagnoses and within 42 days after vaccination. Among 1.24 million vaccinations, 4 clusters were found: cellulitis on days 1-3, nonspecific erythematous condition on days 2-4, "other complications . . ." on days 1-3, and nonspecific allergy on days 1-6. These results are consistent with local injection-site reactions and other known, generally mild, vaccine-associated adverse events and a favorable safety profile. This method might be useful for assessing the safety of other vaccines for older adults.

The Devil's in the details: Reports on reproducibility in pharmacoepidemiologic studies.

PURPOSE: The U.S. Food and Drug Administration's Sentinel Initiative "modular programs" have been shown to replicate findings from conventional protocol-driven, custom-programmed studies. One such parallel assessment-dabigatran and warfarin and selected outcomes-produced concordant findings for three of four study outcomes. The effect estimates and confidence intervals for the fourth-acute myocardial infarction-had more variability as compared with other outcomes. This paper evaluates the potential sources of that variability that led to unexpected divergence in findings. METHODS: We systematically compared the two studies and evaluated programming differences and their potential impact using a different dataset that allowed more granular data access for investigation. We reviewed the output at each of five main processing steps common in both study programs: cohort identification, propensity score estimation, propensity score matching, patient follow-up, and risk estimation. RESULTS: Our findings point to several design features that warrant greater investigator attention when performing observational database studies: (a) treatment of recorded events (eg, diagnoses, procedures, and dispensings) co-occurring on the index date of study drug dispensing in cohort eligibility criteria and propensity score estimation and (b) construction of treatment episodes for study drugs of interest that have more complex dispensing patterns. CONCLUSIONS: More precise and unambiguous operational definitions of all study parameters will increase transparency and reproducibility in observational database studies.