Development and Validation of a Claims-Based Model to Predict Categories of Obesity.

We developed and validated a claims-based algorithm that classifies patients into obesity categories. Using Medicare (2007-2017) and Medicaid (2000-2014) claims data linked to 2 electronic health record (EHR) systems in Boston, Massachusetts, we identified a cohort of patients with an EHR-based body mass index (BMI) measurement (calculated as weight (kg)/height (m)2). We used regularized regression to select from 137 variables and built generalized linear models to classify patients with BMIs of ≥25, ≥30, and ≥40. We developed the prediction model using EHR system 1 (training set) and validated it in EHR system 2 (validation set). The cohort contained 123,432 patients in the Medicare population and 40,736 patients in the Medicaid population. The model comprised 97 variables in the Medicare set and 95 in the Medicaid set, including BMI-related diagnosis codes, cardiovascular and antidiabetic drugs, and obesity-related comorbidities. The areas under the receiver-operating-characteristic curve in the validation set were 0.72, 0.75, and 0.83 (Medicare) and 0.66, 0.66, and 0.70 (Medicaid) for BMIs of ≥25, ≥30, and ≥40, respectively. The positive predictive values were 81.5%, 80.6%, and 64.7% (Medicare) and 81.6%, 77.5%, and 62.5% (Medicaid), for BMIs of ≥25, ≥30, and ≥40, respectively. The proposed model can identify obesity categories in claims databases when BMI measurements are missing and can be used for confounding adjustment, defining subgroups, or probabilistic bias analysis.

An application of the Causal Roadmap in two safety monitoring case studies: Causal inference and outcome prediction using electronic health record data.

Background: Real-world data, such as administrative claims and electronic health records, are increasingly used for safety monitoring and to help guide regulatory decision-making. In these settings, it is important to document analytic decisions transparently and objectively to assess and ensure that analyses meet their intended goals. Methods: The Causal Roadmap is an established framework that can guide and document analytic decisions through each step of the analytic pipeline, which will help investigators generate high-quality real-world evidence. Results: In this paper, we illustrate the utility of the Causal Roadmap using two case studies previously led by workgroups sponsored by the Sentinel Initiative - a program for actively monitoring the safety of regulated medical products. Each case example focuses on different aspects of the analytic pipeline for drug safety monitoring. The first case study shows how the Causal Roadmap encourages transparency, reproducibility, and objective decision-making for causal analyses. The second case study highlights how this framework can guide analytic decisions beyond inference on causal parameters, improving outcome ascertainment in clinical phenotyping. Conclusion: These examples provide a structured framework for implementing the Causal Roadmap in safety surveillance and guide transparent, reproducible, and objective analysis.

The impact of electronic health record discontinuity on prediction modeling.

BACKGROUND: To determine the impact of electronic health record (EHR)-discontinuity on the performance of prediction models. METHODS: The study population consisted of patients with a history of cardiovascular (CV) comorbidities identified using US Medicare claims data from 2007 to 2017, linked to EHR from two networks (used as model training and validation set, respectively). We built models predicting one-year risk of mortality, major CV events, and major bleeding events, stratified by high vs. low algorithm-predicted EHR-continuity. The best-performing models for each outcome were chosen among 5 commonly used machine-learning models. We compared model performance by Area under the ROC curve (AUROC) and Area under the precision-recall curve (AUPRC). RESULTS: Based on 180,950 in the training and 103,061 in the validation set, we found EHR captured only 21.0-28.1% of all the non-fatal outcomes in the low EHR-continuity cohort but 55.4-66.1% of that in the high EHR-continuity cohort. In the validation set, the best-performing model developed among high EHR-continuity patients had consistently higher AUROC than that based on low-continuity patients: AUROC was 0.849 vs. 0.743 when predicting mortality; AUROC was 0.802 vs. 0.659 predicting the CV events; AUROC was 0.635 vs. 0.567 predicting major bleeding. We observed a similar pattern when using AUPRC as the outcome metric. CONCLUSIONS: Among patients with CV comorbidities, when predicting mortality, major CV events, and bleeding outcomes, the prediction models developed in datasets with low EHR-continuity consistently had worse performance compared to models developed with high EHR-continuity.

Development and Validation of Algorithms to Estimate Live Birth Gestational Age in Medicaid Analytic eXtract Data.

BACKGROUND: While healthcare utilization data are useful for postmarketing surveillance of drug safety in pregnancy, the start of pregnancy and gestational age at birth are often incompletely recorded or missing. Our objective was to develop and validate a claims-based live birth gestational age algorithm. METHODS: Using the Medicaid Analytic eXtract (MAX) linked to birth certificates in three states, we developed four candidate algorithms based on: preterm codes; preterm or postterm codes; timing of prenatal care; and prediction models - using conventional regression and machine-learning approaches with a broad range of prespecified and empirically selected predictors. We assessed algorithm performance based on mean squared error (MSE) and proportion of pregnancies with estimated gestational age within 1 and 2 weeks of the gold standard, defined as the clinical or obstetric estimate of gestation on the birth certificate. We validated the best-performing algorithms against medical records in a nationwide sample. We quantified misclassification of select drug exposure scenarios due to estimated gestational age as positive predictive value (PPV), sensitivity, and specificity. RESULTS: Among 114,117 eligible pregnancies, the random forest model with all predictors emerged as the best performing algorithm: MSE 1.5; 84.8% within 1 week and 96.3% within 2 weeks, with similar performance in the nationwide validation cohort. For all exposure scenarios, PPVs were >93.8%, sensitivities >94.3%, and specificities >99.4%. CONCLUSIONS: We developed a highly accurate algorithm for estimating gestational age among live births in the nationwide MAX data, further supporting the value of these data for drug safety surveillance in pregnancy. See video abstract at, http://links.lww.com/EDE/B989 .

Scalable Feature Engineering from Electronic Free Text Notes to Supplement Confounding Adjustment of Claims-Based Pharmacoepidemiologic Studies.

Natural language processing (NLP) tools turn free-text notes (FTNs) from electronic health records (EHRs) into data features that can supplement confounding adjustment in pharmacoepidemiologic studies. However, current applications are difficult to scale. We used unsupervised NLP to generate high-dimensional feature spaces from FTNs to improve prediction of drug exposure and outcomes compared with claims-based analyses. We linked Medicare claims with EHR data to generate three cohort studies comparing different classes of medications on the risk of various clinical outcomes. We used "bag-of-words" to generate features for the top 20,000 most prevalent terms from FTNs. We compared machine learning (ML) prediction algorithms using different sets of candidate predictors: Set1 (39 researcher-specified variables), Set2 (Set1 + ML-selected claims codes), and Set3 (Set1 + ML-selected NLP-generated features), vs. Set4 (Set1 + 2 + 3). When modeling treatment choice, we observed a consistent pattern across the examples: ML models utilizing Set4 performed best followed by Set2, Set3, then Set1. When modeling the outcome risk, there was little to no improvement beyond models based on Set1. Supplementing claims data with NLP-generated features from free text notes improved prediction of prescribing choices but had little or no improvement on clinical risk prediction. These findings have implications for strategies to improve confounding using EHR data in pharmacoepidemiologic studies.

Greedy caliper propensity score matching can yield variable estimates of the treatment-outcome association-A simulation study.

PURPOSE: Greedy caliper propensity score (PS) matching is dependent on randomness, which can ultimately affect causal estimates. We sought to investigate the variation introduced by this randomness. METHODS: Based on a literature search to define the simulation parameters, we simulated 36 cohorts of different sizes, treatment prevalence, outcome prevalence, treatment-outcome-association. We performed 1:1 caliper and nearest neighbor (NN) caliper PS-matching and repeated this 1000 times in the same cohort, before calculating the treatment-outcome association. RESULTS: Repeating caliper and NN caliper matching in the same cohort yielded large variations in effect estimates, in all 36 scenarios, with both types of matching. The largest variation was found in smaller cohorts, where the odds ratio (OR) ranged from 0.53 to 10.00 (IQR of ORs: 1.11-1.67). The 95% confidence interval was not consistently overlapping a neutral association after repeating the matching with both algorithms. We confirmed these findings in a noninterventional example study. CONCLUSION: Caliper PS-matching can yield highly variable estimates of the treatment-outcome association if the analysis is repeated.

Using Real-World Data to Predict Clinical and Economic Benefits of a Future Drug Based on its Target Product Profile.

INTRODUCTION: For a new drug to be developed, the desired properties are described in a target product profile. OBJECTIVE: We propose a framework for using real-world data to measure the disease-specific costs of the current standard of care and then to project the costs of the proposed new product for early data-driven portfolio decisions to select drug candidates for development. METHODS: We sampled from a cohort of patients representing the current standard of care to generate a hypothetical cohort of patients that fits a given target product profile for a new (hypothetical) treatment. The healthcare costs were determined and compared between standard of care and the new treatment. The approach differed according to the number of outcomes defined in the target product profile, and the cases for one, two, and three outcome variables are described. RESULTS: Based on assumed hypothetical treatment effect, absolute risk and cost reductions were estimated in a worked example. The median costs per day for one patient were estimated to be $10.37 and $8.39 in the original and hypothetical cohorts, respectively. This means that the assumed target product profile would result in cost savings of $1.98 per day and patient-not accounting for any additional drug costs. CONCLUSIONS: We present a simple approach to assess the potential absolute clinical and economic benefit of a new drug based on real-world data and its target product profile. The approach allows for early data-driven portfolio decisions to select drug candidates based on their expected cost savings.

Serious infection risk in children with psoriasis on systemic treatment: A propensity score-matched population-based study.

BACKGROUND: Psoriasis is increasingly treated with systemic medications, yet their safety is not well characterized in children. OBJECTIVE: We sought to estimate the 6-month risk of serious infections in children with psoriasis treated with biologics, systemic nonbiologics, and phototherapy. METHODS: Using insurance claims data, we identified children aged <18 years with psoriasis and compared the frequency of serious infections in those initiating biologics, systemic nonbiologics, and phototherapy. Relative risks were estimated before and after 1:1 propensity score matching. RESULTS: Among 57,323 children with psoriasis, the 6-month risk of infection was 4.2 per 1000 patient-years in 722 biologic initiators, 5.1 in 988 systemic nonbiologic initiators, and 1.1 in 2657 phototherapy initiators. The relative risk (95% confidence interval) of infection in biologics vs nonbiologics was 0.67 (0.11-3.98), in biologics vs phototherapy was 1.50 (0.25-8.95), and in nonbiologics vs phototherapy was 5.00 (0.59-42.71). The background risk of infection in children with psoriasis was 1 per 1000, almost double the risk compared with children without psoriasis (relative risk, 1.84; 95% confidence interval, 1.15-1.97). CONCLUSIONS: We found no meaningful difference in infection risk between biologics vs nonbiologics and no robust difference between systemic users vs phototherapy. Independent of treatment, children with psoriasis had a higher risk of infection than those without psoriasis.

Evaluating the use of bootstrapping in cohort studies conducted with 1:1 propensity score matching-A plasmode simulation study.

PURPOSE: Bootstrapping can account for uncertainty in propensity score (PS) estimation and matching processes in 1:1 PS-matched cohort studies. While theory suggests that the classical bootstrap can fail to produce proper coverage, practical impact of this theoretical limitation in settings typical to pharmacoepidemiology is not well studied. METHODS: In a plasmode-based simulation study, we compared performance of the standard parametric approach, which ignores uncertainty in PS estimation and matching, with two bootstrapping methods. The first method only accounted for uncertainty introduced during the matching process (the observation resampling approach). The second method accounted for uncertainty introduced during both PS estimation and matching processes (the PS reestimation approach). Variance was estimated based on percentile and empirical standard errors, and treatment effect estimation was based on median and mean of the estimated treatment effects across 1000 bootstrap resamples. Two treatment prevalence scenarios (5% and 29%) across two treatment effect scenarios (hazard ratio of 1.0 and 2.0) were evaluated in 500 simulated cohorts of 10 000 patients each. RESULTS: We observed that 95% confidence intervals from the bootstrapping approaches but not the standard approach, resulted in inaccurate coverage rates (98%-100% for the observation resampling approach, 99%-100% for the PS reestimation approach, and 95%-96% for standard approach). Treatment effect estimation based on bootstrapping approaches resulted in lower bias than the standard approach (less than 1.4% vs 4.1%) at 5% treatment prevalence; however, the performance was equivalent at 29% treatment prevalence. CONCLUSION: Use of bootstrapping led to variance overestimation and inconsistent coverage, while coverage remained more consistent with parametric estimation.

Restriction of Pharmacoepidemiologic Cohorts to Initiators of Medications in Unrelated Preventive Drug Classes to Reduce Confounding by Frailty in Older Adults.

Nonexperimental studies of the effectiveness of seasonal influenza vaccine in older adults have found 40%-60% reductions in all-cause mortality associated with vaccination, potentially due to confounding by frailty. We restricted our cohort to initiators of medications in preventive drug classes (statins, antiglaucoma drugs, and ß blockers) as an approach to reducing confounding by frailty by excluding frail older adults who would not initiate use of these drugs. Using a random 20% sample of US Medicare beneficiaries, we framed our study as a series of nonrandomized "trials" comparing vaccinated beneficiaries with unvaccinated beneficiaries who had an outpatient health-care visit during the 5 influenza seasons occurring in 2010-2015. We pooled data across trials and used standardized-mortality-ratio-weighted Cox proportional hazards models to estimate the association between influenza vaccination and all-cause mortality before influenza season, expecting a null association. Weighted hazard ratios among preventive drug initiators were generally closer to the null than those in the nonrestricted cohort. Restriction of the study population to statin initiators with an uncensored approach resulted in a weighted hazard ratio of 1.00 (95% confidence interval: 0.84, 1.19), and several other hazard ratios were above 0.95. Restricting the cohort to initiators of medications in preventive drug classes can reduce confounding by frailty in this setting, but further work is required to determine the most appropriate criteria to use.

A review of the performance of different methods for propensity score matched subgroup analyses and a summary of their application in peer-reviewed research studies.

PURPOSE: When evaluating safety signals, there is often interest in understanding safety in all patients for whom compared treatments are reasonable alternatives, as well as in specific subgroups of interest. There are numerous ways that propensity score (PS) matching can be implemented for subgroup analyses. METHODS: We conducted a systematic literature review of methods papers that compared the performance of alternative methods to implement PS matched subgroup analyses and examined how frequently different PS matching methods have been used for subgroup analyses in applied studies. RESULTS: We identified 5 methods papers reporting small improvements in covariate balance and bias with use of a subgroup-specific PS instead of a mis-specified overall PS within subgroups. Applied research papers frequently used PS for subgroups in ways not evaluated in methods papers. Thirty three percent used PS to match in the overall cohort and broke the matched sets for subgroup analysis without further adjustment. CONCLUSIONS: While the performance of several alternative ways to use PS matching in subgroup analyses has been evaluated in methods literature, these evaluations do not include the most commonly used methods to implement PS matched subgroup analyses in applied studies. There is a need to better understand the relative performance of commonly used methods for PS matching in subgroup analyses, particularly within settings encountered during active surveillance, where there may be low exposure, infrequent outcomes, and multiple subgroups of interest.

Controlling confounding by frailty when estimating influenza vaccine effectiveness using predictors of dependency in activities of daily living.

PURPOSE: To improve control of confounding by frailty when estimating the effect of influenza vaccination on all-cause mortality by controlling for a published set of claims-based predictors of dependency in activities of daily living (ADL). METHODS: Using Medicare claims data, a cohort of beneficiaries >65 years of age was followed from September 1, 2007, to April 12, 2008, with covariates assessed in the 6 months before follow-up. We estimated Cox proportional hazards models of all-cause mortality, with influenza vaccination as a time-varying exposure. We controlled for common demographics, comorbidities, and health care utilization variables and then added 20 ADL dependency predictors. To gauge residual confounding, we estimated pre-influenza season hazard ratios (HRs) between September 1, 2007 and January 5, 2008, which should be 1.0 in the absence of bias. RESULTS: A cohort of 2 235 140 beneficiaries was created, with a median follow-up of 224 days. Overall, 52% were vaccinated and 4% died during follow-up. During the pre-influenza season period, controlling for demographics, comorbidities, and health care use resulted in a HR of 0.66 (0.64, 0.67). Adding the ADL dependency predictors moved the HR to 0.68 (0.67, 0.70). Controlling for demographics and ADL dependency predictors alone resulted in a HR of 0.68 (0.66, 0.70). CONCLUSIONS: Results were consistent with those in the literature, with significant uncontrolled confounding after adjustment for demographics, comorbidities, and health care use. Adding ADL dependency predictors moved HRs slightly closer to the null. Of the comorbidities, health care use variables, and ADL dependency predictors, the last set reduced confounding most. However, substantial uncontrolled confounding remained.

Assessing the impact of propensity score estimation and implementation on covariate balance and confounding control within and across important subgroups in comparative effectiveness research.

PURPOSE: Researchers are often interested in estimating treatment effects in subgroups controlling for confounding based on a propensity score (PS) estimated in the overall study population. OBJECTIVE: To evaluate covariate balance and confounding control in sulfonylurea versus metformin initiators within subgroups defined by cardiovascular disease (CVD) history comparing an overall PS with subgroup-specific PSs implemented by 1:1 matching and stratification. METHODS: We analyzed younger patients from a US insurance claims database and older patients from 2 Medicare (Humana Medicare Advantage, fee-for-service Medicare Parts A, B, and D) datasets. Confounders and risk factors for acute myocardial infarction were included in an overall PS and subgroup PSs with and without CVD. Covariate balance was assessed using the average standardized absolute mean difference (ASAMD). RESULTS: Compared with crude estimates, ASAMD across covariates was improved 70%-94% for stratification for Medicare cohorts and 44%-99% for the younger cohort, with minimal differences between overall and subgroup-specific PSs. With matching, 75%-99% balance improvement was achieved regardless of cohort and PS, but with smaller sample size. Hazard ratios within each CVD subgroup differed minimally among PS and cohorts. CONCLUSIONS: Both overall PSs and CVD subgroup-specific PSs achieved good balance on measured covariates when assessing the relative association of diabetes monotherapy with nonfatal myocardial infarction. PS matching generally led to better balance than stratification, but with smaller sample size. Our study is limited insofar as crude differences were minimal, suggesting that the new user, active comparator design identified patients with some equipoise between treatments.

Variable selection for propensity score models when estimating treatment effects on multiple outcomes: a simulation study.

PURPOSE: It is often preferable to simplify the estimation of treatment effects on multiple outcomes by using a single propensity score (PS) model. Variable selection in PS models impacts the efficiency and validity of treatment effects. However, the impact of different variable selection strategies on the estimated treatment effects in settings involving multiple outcomes is not well understood. The authors use simulations to evaluate the impact of different variable selection strategies on the bias and precision of effect estimates to provide insight into the performance of various PS models in settings with multiple outcomes. METHODS: Simulated studies consisted of dichotomous treatment, two Poisson outcomes, and eight standard-normal covariates. Covariates were selected for the PS models based on their effects on treatment, a specific outcome, or both outcomes. The PSs were implemented using stratification, matching, and weighting (inverse probability treatment weighting). RESULTS: PS models including only covariates affecting a specific outcome (outcome-specific models) resulted in the most efficient effect estimates. The PS model that only included covariates affecting either outcome (generic-outcome model) performed best among the models that simultaneously controlled measured confounding for both outcomes. Similar patterns were observed over the range of parameter values assessed and all PS implementation methods. CONCLUSIONS: A single, generic-outcome model performed well compared with separate outcome-specific models in most scenarios considered. The results emphasize the benefit of using prior knowledge to identify covariates that affect the outcome when constructing PS models and support the potential to use a single, generic-outcome PS model when multiple outcomes are being examined.