Characterizing Imbalance in the Tails of the Propensity Score Distribution.

Understanding characteristics of patients with propensity scores in the tails of the propensity score (PS) distribution has relevance for inverse-probability-of-treatment-weighted and PS-based estimation in observational studies. Here we outline a method for identifying variables most responsible for extreme propensity scores. The approach is illustrated in 3 scenarios: 1) a plasmode simulation of adult patients in the National Ambulatory Medical Care Survey (2011-2015) and 2) timing of dexamethasone initiation and 3) timing of remdesivir initiation in patients hospitalized for coronavirus disease 2019 from February 2020 through January 2021. PS models were fitted using relevant baseline covariates, and tails of the PS distribution were defined using asymmetric first and 99th percentiles. After fitting of the PS model in each original data set, values of each key covariate were permuted and model-agnostic variable importance measures were examined. Visualization and variable importance techniques were helpful in identifying variables most responsible for extreme propensity scores and may help identify individual characteristics that might make patients inappropriate for inclusion in a study (e.g., off-label use). Subsetting or restricting the study sample based on variables identified using this approach may help investigators avoid the need for trimming or overlap weights in studies.

Risk of acute pancreatitis among new users of empagliflozin compared to sulfonylureas in patients with type 2 diabetes: A post-authorization safety study.

PURPOSE: This study was undertaken to evaluate the potential risk of acute pancreatitis with empagliflozin in patients with type 2 diabetes (T2D) newly initiating empagliflozin. METHODS: Data from two large US claims databases were analyzed in an observational study of patients with T2D receiving metformin who were newly prescribed empagliflozin versus sulfonylurea (SU). Because dipeptidyl peptidase-4 inhibitors and glucagon-like peptide-1 receptor agonists have been associated with the risk of acute pancreatitis in some studies, patients on these agents were excluded. Using pooled analyses of data from the two databases (2014-2021), patients initiating empagliflozin were matched 1:1 within database to patients initiating SU using propensity scores (PS) that incorporated relevant demographic and clinical characteristics. Prespecified sensitivity analyses were performed for design parameters. RESULTS: The analyses identified 72 661 new users of empagliflozin and 422 018 new users of SUs, with both patient groups on concurrent metformin therapy. Baseline characteristics within treatment groups appeared to be similar across the 72 621 matched pairs. After mean follow-up of ~6 months, incidence rates of acute pancreatitis in the pooled matched cohort were 10.30 (95% confidence interval [CI] 9.29-11.39) events per 1000 patient-years (PY) for empagliflozin and 11.65 (95% CI 10.59-12.77) events per 1000 PY for SUs. On a background of metformin, patients newly initiating empagliflozin did not have an increased risk of acute pancreatitis compared with those initiating an SU (pooled PS matched hazard ratio 0.88 [0.76-1.02]) across 75621.42 PY of follow-up. CONCLUSIONS: The results of this voluntary post-approval safety study provide additional evidence that the use of empagliflozin for the treatment of T2D is not associated with an increased risk of acute pancreatitis.

Clarifying the causal contrast: An empirical example applying the prevalent new user study design.

PURPOSE: The prevalent new user design extends the active comparator new user design to include patients switching to a treatment of interest from a comparator. We examined the impact of adding "switchers" to incident new users on the estimated hazard ratio (HR) of hospitalized heart failure. METHODS: Using MarketScan claims data (2000-2014), we estimated HRs of hospitalized heart failure between patients initiating GLP-1 receptor agonists (GLP-1 RA) and sulfonylureas (SU). We considered three estimands: (1) the effect of incident new use; (2) the effect of switching; and (3) the effect of incident new use or switching, combining the two population. We used time-conditional propensity scores (TCPS) and time-stratified standardized morbidity ratio (SMR) weighting to adjust for confounding. RESULTS: We identified 76 179 GLP-1 RA new users, of which 12% were direct switchers (within 30 days) from SU. Among incident new users, GLP-1 RA was protective against heart failure (adjHRSMR = 0.74 [0.69, 0.80]). Among switchers, GLP-1 RA was not protective (adjHRSMR = 0.99 [0.83, 1.18]). Results in the combined population were largely driven by the incident new users, with GLP-1 RA having a protective effect (adjHRSMR = 0.77 [0.72, 0.83]). Results using TCPS were consistent with those estimated using SMR weighting. CONCLUSIONS: When analyses were conducted only among incident new users, GLP-1 RA had a protective effect. However, among switchers from SU to GLP-1 RA, the effect estimates substantially shifted toward the null. Combining patients with varying treatment histories can result in poor confounding control and camouflage important heterogeneity.

Alternative analytic and matching approaches for the prevalent new-user design: A simulation study.

PURPOSE: To describe the creation of prevalent new user (PNU) cohorts and compare the relative bias and computational efficiency of several alternative analytic and matching approaches in PNU studies. METHODS: In a simulated cohort, we estimated the effect of a treatment of interest vs a comparator among those who switched to the treatment of interest using the originally proposed time-conditional propensity score (TCPS) matching, standardized morbidity ratio weighting (SMRW), disease risk scores (DRS), and several alternative propensity score matching approaches. For each analytic method, we compared the average RR (across 2000 replicates) to the known risk ratio (RR) of 1.00. RESULTS: SMRW and DRS yielded unbiased results (RR = 0.998 and 0.997, respectively). TCPS matching with replacement was also unbiased (RR = 0.999). TCPS matching without replacement was unbiased when matches were identified starting with patients with the shortest treatment history as initially proposed (RR = 0.999), but it resulted in very slight bias (RR = 0.983) when starting with patients with the longest treatment history. Similarly, creating a match pool without replacement starting with patients with the shortest treatment history yielded an unbiased estimate (RR = 0.997), but matching with the longest treatment history first resulted in substantial bias (RR = 0.903). The most biased strategy was matching after selecting one random comparator observation per individual that continued on the comparator (RR = 0.802). CONCLUSIONS: Multiple analytic methods can estimate treatment effects without bias in a PNU cohort. Still, researchers should be wary of introducing bias when selecting controls for complex matching strategies beyond the initially proposed TCPS.

Using propensity scores to estimate effects of treatment initiation decisions: State of the science.

Confounding can cause substantial bias in nonexperimental studies that aim to estimate causal effects. Propensity score methods allow researchers to reduce bias from measured confounding by summarizing the distributions of many measured confounders in a single score based on the probability of receiving treatment. This score can then be used to mitigate imbalances in the distributions of these measured confounders between those who received the treatment of interest and those in the comparator population, resulting in less biased treatment effect estimates. This methodology was formalized by Rosenbaum and Rubin in 1983 and, since then, has been used increasingly often across a wide variety of scientific disciplines. In this review article, we provide an overview of propensity scores in the context of real-world evidence generation with a focus on their use in the setting of single treatment decisions, that is, choosing between two therapeutic options. We describe five aspects of propensity score analysis: alignment with the potential outcomes framework, implications for study design, estimation procedures, implementation options, and reporting. We add context to these concepts by highlighting how the types of comparator used, the implementation method, and balance assessment techniques have changed over time. Finally, we discuss evolving applications of propensity scores.

Increased risk of incident dementia following use of anticholinergic agents: A systematic literature review and meta-analysis.

BACKGROUND/RATIONALE: Long-term treatment with anticholinergic agents may increase the risk of cognitive impairment or dementia. This systematic literature review and meta-analysis aimed to assess the impact of ≥3 months of exposure to anticholinergics as a class on the risk of dementia, mild cognitive impairment, and change in cognitive function. The impact of anticholinergic agents specifically used to treat overactive bladder was also evaluated. MATERIALS AND METHODS: A systematic literature review was conducted to identify English language articles evaluating the impact of anticholinergic use for ≥3 months on dementia or cognitive function in adult patients. Databases searched included PubMed, Embase, and the Cochrane Library. Meta-analyses were conducted using random-effects models; 95% confidence intervals (CIs) and 95% prediction intervals (PIs) were reported. RESULTS: A total of 2122 records were identified. Out of those, 21 studies underwent qualitative synthesis and 6 reported endpoints relevant for inclusion in a meta-analysis assessing the risk of incident dementia. The overall rate ratio for incident dementia was 1.46 (95% CI: 1.17-1.81; 95% PI: 0.70-3.04; n = 6). The risk of incident dementia increased with increasing exposure (n = 3). In addition, two studies from the meta-analysis reported an increased risk of dementia with ≥3 months of use of bladder antimuscarinics (adjusted odds ratios ranged from 1.21 to 1.65, depending on exposure category). CONCLUSION: Anticholinergic use for ≥3 months increased the risk of dementia on average by an estimated 46% versus nonuse. This relationship was consistent in studies assessing overactive bladder medications. The risk of developing dementia should be carefully considered in the context of potential benefit before prescribing anticholinergics.

Evaluation of the modified FINDRISC to identify individuals at high risk for diabetes among middle-aged white and black ARIC study participants.

OBJECTIVE: To evaluate a modified Finnish Diabetes Risk Score (FINDRISC) for predicting the risk of incident diabetes among white and black middle-aged participants from the Atherosclerosis Risk in Communities (ARIC) study. RESEARCH DESIGN AND METHODS: We assessed 9754 ARIC cohort participants who were free of diabetes at baseline. Logistic regression and receiver operator characteristic (ROC) curves were used to evaluate a modified FINDRISC for predicting incident diabetes after 9 years of follow-up, overall and by race/gender group. The modified FINDRISC used comprised age, body mass index, waist circumference, blood pressure medication and family history. RESULTS: The mean FINDRISC (range, 2 [lowest risk] to 17 [highest risk]) for black women was higher (9.9 ± 3.6) than that for black men (7.6 ± 3.9), white women (8.0 ± 3.6) and white men (7.6 ± 3.5). The incidence of diabetes increased generally across deciles of FINDRISC for all 4 race/gender groups. ROC curve statistics for the FINDRISC showed the highest area under the curve for white women (0.77) and the lowest for black men (0.70). CONCLUSIONS: We used a modified FINDRISC to predict the 9-year risk of incident diabetes in a biracial US population. The modified risk score can be useful for early screening of incident diabetes in biracial populations, which may be helpful for early interventions to delay or prevent diabetes.

Comparison of diagnostic evaluations for cough among initiators of angiotensin converting enzyme inhibitors and angiotensin receptor blockers.

PURPOSE: Differential diagnostic evaluation associated with a drug may bias effect estimates because of an increased detection of preclinical outcomes. Persistent cough is a common side effect with angiotensin-converting enzyme inhibitors (ACEI), and we hypothesized that ACEI initiators would undergo more diagnostic evaluations, potentially leading to diagnosis of preclinical lung cancer. We compared the incidence of cough-related diagnostic evaluations and lung cancer among ACEI versus angiotensin receptor blockers (ARB) initiators. METHODS: Using a 20% sample of Medicare claims 2007-2012, we identified initiators of ACEI or ARB, age 66-99 years. Incidence of diagnostic evaluation and lung cancer were compared using adjusted Cox models. Monthly probabilities of workup were compared using proportion differences. RESULTS: There were 342 611 and 108 116 ACEI and ARB initiators, respectively. Monthly probability of chest X-rays ranged from minimum 4.7% to maximum 21.2% in the 6 months pre and post-initiation. Differences in incidence of diagnostic procedures in the 6 months after initiation were only minimal (chest X-rays hazard ratio (HR) = 1.12; 95% CI: 1.10-1.14), chest-MRI (0.86, 95% CI: 0.74-0.99), CT-scans (1.09, 95% CI: 0.99-1.18) or bronchoscopies (1.03, 95% CI: 0.83-1.29). Proportion differences for chest X-rays peaked in the month pre-initiation (8.4%, 95% CI: 8.1-8.6) but negligible thereafter. There was no difference in the incidence of lung cancer among ACEI versus ARB initiators (HR = 0.99, 95% CI: 0.84-1.16). CONCLUSION: Results indicate minimal differential chest workup after ACEI versus ARB initiation and no difference in lung cancer incidence, but suggest differential workup in the month before the first recorded prescription. The latter may reflect drug use before the first observed pharmacy claim or increased workup before initiation of ACEI therapy. Copyright © 2016 John Wiley & Sons, Ltd.

Sensitivity analysis of methods for active surveillance of acute myocardial infarction using electronic databases.

BACKGROUND: The validity of conclusions from observational studies depends on decisions regarding design, analysis, data quality, and implementation. Through sensitivity analyses, we explored the impact of such decisions on balance control and risk estimates. METHODS: Using as a template the Mini-Sentinel protocol for the active surveillance of acute myocardial infarction (MI) in association with use of antidiabetic agents, we defined cohorts of new users of metformin and second-generation sulfonylureas, baseline covariates and acute MI events using three combinations of washout and baseline periods. Using propensity-score matching, we assessed balance control and risk estimates using cumulative data for matching all patients compared with not rematching prior matches in quarterly analyses over the follow-up period. RESULTS: A longer washout period increased the confidence in new-user status, but at the expense of sample size; a longer baseline period improved capture of covariates related to pre-existing chronic conditions. When all patients were matched each quarter, balance was improved and risk estimates were more robust, especially in the later quarters. CONCLUSIONS: Durations of washout and baseline periods influence the likelihood of new-user status and sample size. Matching all patients tends to result in better covariate balance than matching only new patients. Decisions regarding the durations of washout and baseline periods depend on the specific research question and availability of longitudinal patient data within the database. This paper demonstrates the importance and utility of sensitivity analysis of methods for evaluating the robustness of results in observational studies.

Matching on the disease risk score in comparative effectiveness research of new treatments.

PURPOSE: We use simulations and an empirical example to evaluate the performance of disease risk score (DRS) matching compared with propensity score (PS) matching when controlling large numbers of covariates in settings involving newly introduced treatments. METHODS: We simulated a dichotomous treatment, a dichotomous outcome, and 100 baseline covariates that included both continuous and dichotomous random variables. For the empirical example, we evaluated the comparative effectiveness of dabigatran versus warfarin in preventing combined ischemic stroke and all-cause mortality. We matched treatment groups on a historically estimated DRS and again on the PS. We controlled for a high-dimensional set of covariates using 20% and 1% samples of Medicare claims data from October 2010 through December 2012. RESULTS: In simulations, matching on the DRS versus the PS generally yielded matches for more treated individuals and improved precision of the effect estimate. For the empirical example, PS and DRS matching in the 20% sample resulted in similar hazard ratios (0.88 and 0.87) and standard errors (0.04 for both methods). In the 1% sample, PS matching resulted in matches for only 92.0% of the treated population and a hazard ratio and standard error of 0.89 and 0.19, respectively, while DRS matching resulted in matches for 98.5% and a hazard ratio and standard error of 0.85 and 0.16, respectively. CONCLUSIONS: When PS distributions are separated, DRS matching can improve the precision of effect estimates and allow researchers to evaluate the treatment effect in a larger proportion of the treated population. However, accurately modeling the DRS can be challenging compared with the PS.

The role of prediction modeling in propensity score estimation: an evaluation of logistic regression, bCART, and the covariate-balancing propensity score.

The covariate-balancing propensity score (CBPS) extends logistic regression to simultaneously optimize covariate balance and treatment prediction. Although the CBPS has been shown to perform well in certain settings, its performance has not been evaluated in settings specific to pharmacoepidemiology and large database research. In this study, we use both simulations and empirical data to compare the performance of the CBPS with logistic regression and boosted classification and regression trees. We simulated various degrees of model misspecification to evaluate the robustness of each propensity score (PS) estimation method. We then applied these methods to compare the effect of initiating glucagonlike peptide-1 agonists versus sulfonylureas on cardiovascular events and all-cause mortality in the US Medicare population in 2007-2009. In simulations, the CBPS was generally more robust in terms of balancing covariates and reducing bias compared with misspecified logistic PS models and boosted classification and regression trees. All PS estimation methods performed similarly in the empirical example. For settings common to pharmacoepidemiology, logistic regression with balance checks to assess model specification is a valid method for PS estimation, but it can require refitting multiple models until covariate balance is achieved. The CBPS is a promising method to improve the robustness of PS models.

Assessing the impact of propensity score estimation and implementation on covariate balance and confounding control within and across important subgroups in comparative effectiveness research.

PURPOSE: Researchers are often interested in estimating treatment effects in subgroups controlling for confounding based on a propensity score (PS) estimated in the overall study population. OBJECTIVE: To evaluate covariate balance and confounding control in sulfonylurea versus metformin initiators within subgroups defined by cardiovascular disease (CVD) history comparing an overall PS with subgroup-specific PSs implemented by 1:1 matching and stratification. METHODS: We analyzed younger patients from a US insurance claims database and older patients from 2 Medicare (Humana Medicare Advantage, fee-for-service Medicare Parts A, B, and D) datasets. Confounders and risk factors for acute myocardial infarction were included in an overall PS and subgroup PSs with and without CVD. Covariate balance was assessed using the average standardized absolute mean difference (ASAMD). RESULTS: Compared with crude estimates, ASAMD across covariates was improved 70%-94% for stratification for Medicare cohorts and 44%-99% for the younger cohort, with minimal differences between overall and subgroup-specific PSs. With matching, 75%-99% balance improvement was achieved regardless of cohort and PS, but with smaller sample size. Hazard ratios within each CVD subgroup differed minimally among PS and cohorts. CONCLUSIONS: Both overall PSs and CVD subgroup-specific PSs achieved good balance on measured covariates when assessing the relative association of diabetes monotherapy with nonfatal myocardial infarction. PS matching generally led to better balance than stratification, but with smaller sample size. Our study is limited insofar as crude differences were minimal, suggesting that the new user, active comparator design identified patients with some equipoise between treatments.

A Framework for the Use and Likelihood of Regulatory Acceptance of Single-Arm Trials.

BACKGROUND: Single-arm clinical trials (SAT) are common in drug and biologic submissions for rare or life-threatening conditions, especially when no therapeutic options exist. External control arms (ECAs) improve interpretation of SATs but pose methodological and regulatory challenges. OBJECTIVE: Through narrative reviews and expert input, we developed a framework for considerations that might influence regulatory use and likelihood of regulatory acceptance of an SAT, identifying non-oncology first indication approvals as an area of interest. We systematically analyzed FDA and EMA approvals using SATs as pivotal evidence. The framework guided outcome abstraction on regulatory responses. METHODS: We examined all non-oncology FDA and EMA drug and biologic approvals for first indications from 2019 to 2022 to identify those with SAT as pivotal safety or efficacy evidence. We abstracted outcomes, key study design features, regulator responses to SAT and (where applicable) ECA design, and product label content. RESULTS: Among 20 SAT-based FDA approvals and 17 SAT-based EMA approvals, most common indications were progressive rare diseases with high unmet need/limited therapeutic options and a natural history without spontaneous improvement. Of the types of comparators, most were natural history cohorts (45% FDA; 47% EMA) and baseline controls (40% FDA; 47% EMA). Common critiques were of non-contemporaneous ECAs, subjective endpoints, and baseline covariate imbalance between arms. CONCLUSION: Based on recent FDA and EMA approvals, the likelihood of regulatory success for SATs with ECAs depends on many design, analytic, and data quality considerations. Our framework is useful in early drug development when considering SAT strategies for evidence generation.

Current Practices and Challenges When Submitting Patient Experience Data for Regulatory Decisions by the US Food and Drug Administration: An Industry Survey.

OBJECTIVES: To understand industry practices and challenges when submitting patient experience data (PED) for regulatory decisions by the US Food and Drug Administration (FDA). METHODS: A two-part online survey related to collection, submission, and use of PED by FDA in regulatory decision-making (part 1) and a best-worst exercise for prioritizing potential PED initiatives (part 2) was completed by industry and contract research organization (CRO) members with ≥ 2 years of recent experience with patient-reported outcome (PRO), natural history study (NHS), or patient preference (PP) data; and direct experience with FDA filings including PED. RESULTS: A total of 50 eligible respondents (84% industry) completed part 1 of the survey, among which 46 completed part 2. Respondents mostly had PRO (86%) and PP (50%) experience. All indicated that FDA meetings should have a standing agenda item to discuss PED. Most (78%) reported meetings should occur before pivotal trials. A common challenge was justifying inclusion without knowing if and how data will be used. Most agreed that FDA and industry should co-develop the PED table in the FDA clinical review (74%), and the table should report reason(s) for not using PED (96%) in regulatory decision-making. Most important efforts to advance PED use in decision-making were a dedicated meeting pathway and expanded FDA guidance (51% each). CONCLUSIONS: FDA has policy targets expanding PED use, but challenges remain regarding pathways for PED submission and transparency in regulatory decision-making. Alignment on the use of existing meeting opportunities to discuss PED, co-development of the PED table, and expanded guidance are encouraged.

Variable selection for propensity score models when estimating treatment effects on multiple outcomes: a simulation study.

PURPOSE: It is often preferable to simplify the estimation of treatment effects on multiple outcomes by using a single propensity score (PS) model. Variable selection in PS models impacts the efficiency and validity of treatment effects. However, the impact of different variable selection strategies on the estimated treatment effects in settings involving multiple outcomes is not well understood. The authors use simulations to evaluate the impact of different variable selection strategies on the bias and precision of effect estimates to provide insight into the performance of various PS models in settings with multiple outcomes. METHODS: Simulated studies consisted of dichotomous treatment, two Poisson outcomes, and eight standard-normal covariates. Covariates were selected for the PS models based on their effects on treatment, a specific outcome, or both outcomes. The PSs were implemented using stratification, matching, and weighting (inverse probability treatment weighting). RESULTS: PS models including only covariates affecting a specific outcome (outcome-specific models) resulted in the most efficient effect estimates. The PS model that only included covariates affecting either outcome (generic-outcome model) performed best among the models that simultaneously controlled measured confounding for both outcomes. Similar patterns were observed over the range of parameter values assessed and all PS implementation methods. CONCLUSIONS: A single, generic-outcome model performed well compared with separate outcome-specific models in most scenarios considered. The results emphasize the benefit of using prior knowledge to identify covariates that affect the outcome when constructing PS models and support the potential to use a single, generic-outcome PS model when multiple outcomes are being examined.

Can Weight of Evidence, Quantitative Bias, and Bounding Methods Evaluate Robustness of Real-world Evidence for Regulator and Health Technology Assessment Decisions on Medical Interventions?

PURPOSE: High-quality evidence is crucial for health care intervention decision-making. These decisions frequently use nonrandomized data, which can be more vulnerable to biases than randomized trials. Accordingly, methods to quantify biases and weigh available evidence could elucidate the robustness of findings, giving regulators more confidence in making approval and reimbursement decisions. METHODS: We conducted an integrative literature review to identify methods for determining probability of causation, evaluating weight of evidence, and conducting quantitative bias analysis as related to health care interventions. Eligible studies were published from 2012 to 2021, applicable to pharmacoepidemiology, and presented a method that met our objective. FINDINGS: Twenty-two eligible studies were classified into 4 categories: (1) quantitative bias analysis; (2) weight of evidence methods; (3) Bayesian networks; and (4) miscellaneous. All of the methods have strengths, limitations, and situations in which they are more well suited than others. Some methods seem to lend themselves more to applications of health care evidence on medical interventions than others. IMPLICATIONS: To provide robust evidence for and improve confidence in regulatory or reimbursement decisions, we recommend applying multiple methods to triangulate associations of medical interventions, accounting for biases in different ways. This approach could lead to well-defined robustness assessments of study findings and appropriate science-driven decisions by regulators and payers for public health.

Strategy for Generating Blinded Evidence for Single-Arm Trials with External Controls Using Expert Review of Home Video.

INTRODUCTION: Neurodegenerative diseases cause developmental delays and loss of milestones in infants and children. However, scalable outcome measures that quantify features meaningful to parents/caregivers (P/CGs) and have regulatory precedence are lacking for assessing the effectiveness of treatments in clinical trials of neurodegenerative disorders. To address this gap, we developed an innovative, blinded strategy for single-arm trials with external controls using expert panel review of home video. METHOD: We identified meaningful, observable, and objective developmental milestones from iterative interviews with P/CGs and clinical experts. Subsequently, we standardized video recording procedures and instructions to ensure consistency in how P/CGs solicited each activity. In practice, videos would be graded by an expert panel blinded to treatment. To ensure blinding and quality control, video recordings from interim time points would be randomly interspersed. We conducted a pilot study and a pretest of grading to test feasibility and improve the final strategy. RESULTS: The five P/CGs participating in the pilot study found the instructions clear, selected activities important and reflective of their children's abilities, and recordings at-home preferrable to in-clinic assessments. The three grading experts found the videos easy to grade and the milestones clinically meaningful. CONCLUSION: Our standardized strategy enables expert panel grading of developmental milestone achievements using at-home recordings, blinded to treatment and post-baseline time points. This rigorous and objective scoring system has broad applicability in various disease contexts, with or without external controls. Moreover, our strategy facilitates flexible, continued data collection and the videos can be archived for future analyses.