Targeted Learning with an Undersmoothed Lasso Propensity Score Model for Large-Scale Covariate Adjustment in Healthcare Database Studies.

Lasso regression is widely used for large-scale propensity score (PS) estimation in healthcare database studies. In these settings, previous work has shown that undersmoothing (overfitting) Lasso PS models can improve confounding control, but it can also cause problems of non-overlap in covariate distributions. It remains unclear how to select the degree of undersmoothing when fitting large-scale Lasso PS models to improve confounding control while avoiding issues that can result from reduced covariate overlap. Here, we used simulations to evaluate the performance of using collaborative-controlled targeted learning to data-adaptively select the degree of undersmoothing when fitting large-scale PS models within both singly and doubly robust frameworks to reduce bias in causal estimators. Simulations showed that collaborative learning can data-adaptively select the degree of undersmoothing to reduce bias in estimated treatment effects. Results further showed that when fitting undersmoothed Lasso PS-models, the use of cross-fitting was important for avoiding non-overlap in covariate distributions and reducing bias in causal estimates.

Improving Methods of Identifying Anaphylaxis for Medical Product Safety Surveillance Using Natural Language Processing and Machine Learning.

We sought to determine whether machine learning and natural language processing (NLP) applied to electronic medical records could improve performance of automated health-care claims-based algorithms to identify anaphylaxis events using data on 516 patients with outpatient, emergency department, or inpatient anaphylaxis diagnosis codes during 2015-2019 in 2 integrated health-care institutions in the Northwest United States. We used one site's manually reviewed gold-standard outcomes data for model development and the other's for external validation based on cross-validated area under the receiver operating characteristic curve (AUC), positive predictive value (PPV), and sensitivity. In the development site 154 (64%) of 239 potential events met adjudication criteria for anaphylaxis compared with 180 (65%) of 277 in the validation site. Logistic regression models using only structured claims data achieved a cross-validated AUC of 0.58 (95% CI: 0.54, 0.63). Machine learning improved cross-validated AUC to 0.62 (0.58, 0.66); incorporating NLP-derived covariates further increased cross-validated AUCs to 0.70 (0.66, 0.75) in development and 0.67 (0.63, 0.71) in external validation data. A classification threshold with cross-validated PPV of 79% and cross-validated sensitivity of 66% in development data had cross-validated PPV of 78% and cross-validated sensitivity of 56% in external data. Machine learning and NLP-derived data improved identification of validated anaphylaxis events.

Active Postlicensure Safety Surveillance for Recombinant Zoster Vaccine Using Electronic Health Record Data.

Recombinant zoster vaccine (RZV) (Shingrix; GlaxoSmithKline, Brentford, United Kingdom) is an adjuvanted glycoprotein vaccine that was licensed in 2017 to prevent herpes zoster (shingles) and its complications in older adults. In this prospective, postlicensure Vaccine Safety Datalink study using electronic health records, we sequentially monitored a real-world population of adults aged ≥50 years who received care in multiple US Vaccine Safety Datalink health systems to identify potentially increased risks of 10 prespecified health outcomes, including stroke, anaphylaxis, and Guillain-Barré syndrome (GBS). Among 647,833 RZV doses administered from January 2018 through December 2019, we did not detect a sustained increased risk of any monitored outcome for RZV recipients relative to either historical (2013-2017) recipients of zoster vaccine live, a live attenuated virus vaccine (Zostavax; Merck & Co., Inc., Kenilworth, New Jersey), or contemporary non-RZV vaccine recipients who had an annual well-person visit during the 2018-2019 study period. We confirmed prelicensure trial findings of increased risks of systemic and local reactions following RZV. Our study provides additional reassurance about the overall safety of RZV. Despite a large sample, uncertainty remains regarding potential associations with GBS due to the limited number of confirmed GBS cases that were observed.

COVID-19 Vaccine Safety Surveillance in Early Pregnancy in the United States: Design Factors Affecting the Association Between Vaccine and Spontaneous Abortion.

In the Vaccine Safety Datalink (VSD), we previously reported no association between coronavirus disease 2019 (COVID-19) vaccination in early pregnancy and spontaneous abortion (SAB). The present study aims to understand how time since vaccine rollout or other methodological factors could affect results. Using a case-control design and generalized estimating equations, we estimated the odds ratios (ORs) of COVID-19 vaccination in the 28 days before a SAB or last date of the surveillance period (index date) in ongoing pregnancies and occurrence of SAB, across cumulative 4-week periods from December 2020 through June 2021. Using data from a single site, we evaluated alternative methodological approaches: increasing the exposure window to 42 days, modifying the index date from the last day to the midpoint of the surveillance period, and constructing a cohort design with a time-dependent exposure model. A protective effect (OR = 0.78, 95% confidence interval: 0.69, 0.89), observed with 3-cumulative periods ending March 8, 2021, was attenuated when surveillance extended to June 28, 2021 (OR = 1.02, 95% confidence interval: 0.96, 1.08). We observed a lower OR for a 42-day window compared with a 28-day window. The time-dependent model showed no association. Timing of the surveillance appears to be an important factor affecting the observed vaccine-SAB association.

Validation of Acute Pancreatitis Among Adults in an Integrated Healthcare System.

BACKGROUND: Acute pancreatitis is a serious gastrointestinal disease that is an important target for drug safety surveillance. Little is known about the accuracy of ICD-10 codes for acute pancreatitis in the United States, or their performance in specific clinical settings. We conducted a validation study to assess the accuracy of acute pancreatitis ICD-10 diagnosis codes in inpatient, emergency department (ED), and outpatient settings. METHODS: We reviewed electronic medical records for encounters with acute pancreatitis diagnosis codes in an integrated healthcare system from October 2015 to December 2019. Trained abstractors and physician adjudicators determined whether events met criteria for acute pancreatitis. RESULTS: Out of 1,844 eligible events, we randomly sampled 300 for review. Across all clinical settings, 182 events met validation criteria for an overall positive predictive value (PPV) of 61% (95% confidence intervals [CI] = 55, 66). The PPV was 87% (95% CI = 79, 92%) for inpatient codes, but only 45% for ED (95% CI = 35, 54%) and outpatient (95% CI = 34, 55%) codes. ED and outpatient encounters accounted for 43% of validated events. Acute pancreatitis codes from any encounter type with lipase >3 times the upper limit of normal had a PPV of 92% (95% CI = 86, 95%) and identified 85% of validated events (95% CI = 79, 89%), while codes with lipase <3 times the upper limit of normal had a PPV of only 22% (95% CI = 16, 30%). CONCLUSIONS: These results suggest that ICD-10 codes accurately identified acute pancreatitis in the inpatient setting, but not in the ED and outpatient settings. Laboratory data substantially improved algorithm performance.

Racial and ethnic disparities in influenza vaccination coverage among pregnant women in the United States: The contribution of vaccine-related attitudes.

OBJECTIVE: Racial and ethnic disparities in influenza vaccination coverage among pregnant women in the United States have been documented. This study assessed the contribution of vaccine-related attitudes to coverage disparities. METHODS: Surveys were conducted following the 2019-2020 and 2020-2021 influenza seasons in a US research network. Using electronic health record data to identify pregnant women, random samples were selected for surveying; non-Hispanic Black women and influenza-unvaccinated women were oversampled. Regression-based decomposition analyses were used to assess the contribution of vaccine-related attitudes to racial and ethnic differences in influenza vaccination. Data were combined across survey years, and analyses were weighted and accounted for survey design. RESULTS: Survey response rate was 41.2% (721 of 1748) for 2019-2020 and 39.3% (706 of 1798) for 2020-2021. Self-reported influenza vaccination was higher among non-Hispanic White respondents (79.4% coverage, 95% CI 73.1%-85.7%) than Hispanic (66.2% coverage, 95% CI 52.5%-79.9%) and non-Hispanic Black (55.8% coverage, 95% CI 50.2%-61.4%) respondents. For all racial and ethnic groups, a high proportion (generally >80%) reported being seen for care, recommended for influenza vaccination, and offered vaccination. In decomposition analyses, vaccine-related attitudes (e.g., worry about vaccination causing influenza; concern about vaccine safety and effectiveness) explained a statistically significant portion of the observed racial and ethnic disparities in vaccination. Maternal age, education, and health status were not significant contributors after controlling for vaccine-related attitudes. CONCLUSIONS: In a setting with relatively high influenza vaccination coverage among pregnant women, racial and ethnic disparities in coverage were identified. Vaccine-related attitudes were associated with the disparities observed.

In older adults, use of a recombinant zoster vaccine was associated with Guillain-Barré syndrome.

SOURCE CITATION: Goud R, Lufkin B, Duffy J, et al. Risk of Guillain-Barré syndrome following recombinant zoster vaccine in Medicare beneficiaries. JAMA Intern Med. 2021;181:1623-30. 34724025.

Identification and Validation of Anaphylaxis Using Electronic Health Data in a Population-based Setting.

BACKGROUND: Anaphylaxis is a life-threatening allergic reaction that is difficult to identify accurately with administrative data. We conducted a population-based validation study to assess the accuracy of ICD-10 diagnosis codes for anaphylaxis in outpatient, emergency department, and inpatient settings. METHODS: In an integrated healthcare system in Washington State, we obtained medical records from healthcare encounters with anaphylaxis diagnosis codes (potential events) from October 2015 to December 2018. To capture events missed by anaphylaxis diagnosis codes, we also obtained records on a sample of serious allergic and drug reactions. Two physicians determined whether potential events met established clinical criteria for anaphylaxis (validated events). RESULTS: Out of 239 potential events with anaphylaxis diagnosis codes, the overall positive predictive value (PPV) for validated events was 64% (95% CI = 58 to 70). The PPV decreased with increasing age. Common precipitants for anaphylaxis were food (39%), medications (35%), and insect bite or sting (12%). The sensitivity of emergency department and inpatient anaphylaxis diagnosis codes for all validated events was 58% (95% CI = 51 to 65), but sensitivity increased to 95% (95% CI = 74 to 99) when outpatient diagnosis codes were included. Using information from all validated events and sampling weights, the incidence rate for anaphylaxis was 3.6 events per 10,000 person-years (95% CI = 3.1 to 4.0). CONCLUSIONS: In this population-based setting, ICD-10 diagnosis codes for anaphylaxis from emergency department and inpatient settings had moderate PPV and sensitivity for validated events. These findings have implications for epidemiologic studies that seek to estimate risks of anaphylaxis using electronic health data.

Safety surveillance and the estimation of risk in select populations: Flexible methods to control for confounding while targeting marginal comparisons via standardization.

We consider the critical problem of pharmacosurveillance for adverse events once a drug or medical product is incorporated into routine clinical care. When making inference on comparative safety using large-scale electronic health records, we often encounter an extremely rare binary adverse outcome with a large number of potential confounders. In this context, it is challenging to offer flexible methods to adjust for high-dimensional confounders, whereas use of the propensity score (PS) can help address this challenge by providing both confounding control and dimension reduction. Among PS methods, regression adjustment using the PS as a covariate in an outcome model has been incompletely studied and potentially misused. Previous studies have suggested that simple linear adjustment may not provide sufficient control of confounding. Moreover, no formal representation of the statistical procedure and associated inference has been detailed. In this paper, we characterize a three-step procedure, which performs flexible regression adjustment of the estimated PS followed by standardization to estimate the causal effect in a select population. We also propose a simple variance estimation method for performing inference. Through a realistic simulation mimicking data from the Food and Drugs Administration's Sentinel Initiative comparing the effect of angiotensin-converting enzyme inhibitors and beta blockers on incidence of angioedema, we show that flexible regression on the PS resulted in less bias without loss of efficiency, and can outperform other methods when the PS model is correctly specified. In addition, the direct variance estimation method is a computationally fast and reliable approach for inference.

Challenges and Opportunities for Using Big Health Care Data to Advance Medical Science and Public Health.

Methodological advancements in epidemiology, biostatistics, and data science have strengthened the research world's ability to use data captured from electronic health records (EHRs) to address pressing medical questions, but gaps remain. We describe methods investments that are needed to curate EHR data toward research quality and to integrate complementary data sources when EHR data alone are insufficient for research goals. We highlight new methods and directions for improving the integrity of medical evidence generated from pragmatic trials, observational studies, and predictive modeling. We also discuss needed methods contributions to further ease data sharing across multisite EHR data networks. Throughout, we identify opportunities for training and for bolstering collaboration among subject matter experts, methodologists, practicing clinicians, and health system leaders to help ensure that methods problems are identified and resulting advances are translated into mainstream research practice more quickly.

Safety of repeated doses of tetanus toxoid, reduced diphtheria toxoid, and acellular pertussis vaccine in adults and adolescents.

In light of waning immunity to pertussis following receipt of tetanus toxoid, reduced diphtheria toxoid and acellular pertussis (Tdap) vaccine, maintaining protection may require repeated Tdap vaccination. We evaluated the safety of repeated doses of tetanus-containing vaccine in 68 915 nonpregnant adolescents and adults in the Vaccine Safety Datalink population who had received an initial dose of Tdap. Compared with 7521 subjects who received a subsequent dose of tetanus toxoid, reduced diphtheria (Td) vaccine, the 61 394 subjects who received a subsequent dose of Tdap did not have significantly elevated risk of medical visits for seizure, cranial nerve disorders, limb swelling, pain in limb, cellulitis, paralytic syndromes, or encephalopathy/encephalitis/meningitis. These results suggest that repeated Tdap vaccination has acceptable safety relative to Tdap vaccination followed by Td vaccination.

Intersections of poverty, race/ethnicity, and sex: alcohol consumption and adverse outcomes in the United States.

We examine whether intersectionality theory-which formalizes the notion that adverse health outcomes owing to having a marginalized social status, identity, or characteristic, may be magnified for individuals with an additional marginalized social status, identity, or characteristic-can be applied using quantitative methods to describe the differential effects of poverty on alcohol consumption across sex and race/ethnicity. Using the National Epidemiologic Survey on Alcohol and Related Conditions, we analyze longitudinal data from Black, Hispanic, and White drinkers (n = 21,140) to assess multiplicative interactions between poverty, as defined by the US Census Bureau, sex, and race/ethnicity, on adverse alcohol outcomes. Findings indicated that the effect of poverty on the past-year incidence of heavy episodic drinking was stronger among Black men and Black women in comparison to men and women of other racial/ethnic groups. Poverty reduction programs that are culturally informed may help reduce racial/ethnic disparities in the adverse outcomes of alcohol consumption.

Design and analysis choices for safety surveillance evaluations need to be tuned to the specifics of the hypothesized drug-outcome association.

BACKGROUND: We reviewed the results of the Observational Medical Outcomes Research Partnership (OMOP) 2010 Experiment in hopes of finding examples where apparently well-designed drug studies repeatedly produce anomalous findings. OMOP had applied thousands of designs and design parameters to 53 drug-outcome pairs across 10 electronic data resources. Our intent was to use this repository to elucidate some sources of error in observational studies. METHOD: From the 2010 OMOP Experiment, we sought drug-outcome-method combinations (DOMCs) that met consensus design criteria, yet repeatedly produced results contrary to expectation. We set aside DOMCs for which we could not agree on the suitability of the designs, then selected for an in-depth scrutiny one drug-outcome pair analyzed by a seemingly plausible methodological approach, whose results consistently disagreed with the a priori expectation. RESULTS: The OMOP "all-by-all" assessment of possible DOMCs yielded many combinations that would not be chosen by researchers as actual study options. Among those that passed a first level of scrutiny, two of seven drug-outcome pairs for which there were plausible research designs had anomalous results. The use of benzodiazepines was unexpectedly associated with acute renal failure and upper gastrointestinal bleeding. We chose the latter as an example for in-depth study. The factitious appearance of a bleeding risk may have been partly driven by an excess of procedures on the first day of treatment. A risk window definition that excluded the first day largely removed the spurious association. CONCLUSION: One cause of reproducible "error" may be repeated failure to tie design choices closely enough to the research question at hand. Copyright © 2016 John Wiley & Sons, Ltd.

Simulation study comparing exposure matching with regression adjustment in an observational safety setting with group sequential monitoring.

Sequential methods are well established for randomized clinical trials (RCTs), and their use in observational settings has increased with the development of national vaccine and drug safety surveillance systems that monitor large healthcare databases. Observational safety monitoring requires that sequential testing methods be better equipped to incorporate confounder adjustment and accommodate rare adverse events. New methods designed specifically for observational surveillance include a group sequential likelihood ratio test that uses exposure matching and generalized estimating equations approach that involves regression adjustment. However, little is known about the statistical performance of these methods or how they compare to RCT methods in both observational and rare outcome settings. We conducted a simulation study to determine the type I error, power and time-to-surveillance-end of group sequential likelihood ratio test, generalized estimating equations and RCT methods that construct group sequential Lan-DeMets boundaries using data from a matched (group sequential Lan-DeMets-matching) or unmatched regression (group sequential Lan-DeMets-regression) setting. We also compared the methods using data from a multisite vaccine safety study. All methods had acceptable type I error, but regression methods were more powerful, faster at detecting true safety signals and less prone to implementation difficulties with rare events than exposure matching methods. Method performance also depended on the distribution of information and extent of confounding by site. Our results suggest that choice of sequential method, especially the confounder control strategy, is critical in rare event observational settings. These findings provide guidance for choosing methods in this context and, in particular, suggest caution when conducting exposure matching.

A general framework for developing computable clinical phenotype algorithms.

OBJECTIVE: To present a general framework providing high-level guidance to developers of computable algorithms for identifying patients with specific clinical conditions (phenotypes) through a variety of approaches, including but not limited to machine learning and natural language processing methods to incorporate rich electronic health record data. MATERIALS AND METHODS: Drawing on extensive prior phenotyping experiences and insights derived from 3 algorithm development projects conducted specifically for this purpose, our team with expertise in clinical medicine, statistics, informatics, pharmacoepidemiology, and healthcare data science methods conceptualized stages of development and corresponding sets of principles, strategies, and practical guidelines for improving the algorithm development process. RESULTS: We propose 5 stages of algorithm development and corresponding principles, strategies, and guidelines: (1) assessing fitness-for-purpose, (2) creating gold standard data, (3) feature engineering, (4) model development, and (5) model evaluation. DISCUSSION AND CONCLUSION: This framework is intended to provide practical guidance and serve as a basis for future elaboration and extension.

Impact of the COVID-19 Pandemic on Health Care Utilization in the Vaccine Safety Datalink: Retrospective Cohort Study.

BACKGROUND: Understanding the long-term impact of the COVID-19 pandemic on health care utilization is important to health care organizations and policy makers for strategic planning, as well as to researchers when designing studies that use observational electronic health record data during the pandemic period. OBJECTIVE: This study aimed to evaluate the changes in health care utilization across all care settings among a large, diverse, and insured population in the United States during the COVID-19 pandemic. METHODS: We conducted a retrospective cohort study within 8 health care organizations participating in the Vaccine Safety Datalink Project using electronic health record data from members of all ages from January 1, 2017, to December 31, 2021. The visit rates per person-year were calculated monthly during the study period for 4 health care settings combined as well as by inpatient, emergency department (ED), outpatient, and telehealth settings, both among all members and members without COVID-19. Difference-in-difference analysis and interrupted time series analysis were performed to assess the changes in visit rates from the prepandemic period (January 2017 to February 2020) to the early pandemic period (April-December 2020) and the later pandemic period (July-December 2021), respectively. An exploratory analysis was also conducted to assess trends through June 2023 at one of the largest sites, Kaiser Permanente Southern California. RESULTS: The study included more than 11 million members from 2017 to 2021. Compared with the prepandemic period, we found reductions in visit rates during the early pandemic period for all in-person care settings. During the later pandemic period, overall use reached 8.36 visits per person-year, exceeding the prepandemic level of 7.49 visits per person-year in 2019 (adjusted percent change 5.1%, 95% CI 0.6%-9.9%); inpatient and ED visits returned to prepandemic levels among all members, although they remained low at 0.095 and 0.241 visits per person-year, indicating a 7.5% and 8% decrease compared to pre-pandemic levels among members without COVID-19, respectively. Telehealth visits, which were approximately 42% of the volume of outpatient visits during the later pandemic period, were increased by 97.5% (95% CI 86.0%-109.7%) from 0.865 visits per person-year in 2019 to 2.35 visits per person-year in the later pandemic period. The trends in Kaiser Permanente Southern California were similar to those of the entire study population. Visit rates from January 2022 to June 2023 were stable and appeared to be a continuation of the use levels observed at the end of 2021. CONCLUSIONS: Telehealth services became a mainstay of the health care system during the late COVID-19 pandemic period. Inpatient and ED visits returned to prepandemic levels, although they remained low among members without evidence of COVID-19. Our findings provide valuable information for strategic resource allocation for postpandemic patient care and for designing observational studies involving the pandemic period.

Influenza vaccination accuracy among adults: Self-report compared with electronic health record data.

OBJECTIVE: To assess the validity of electronic health record (EHR)-based influenza vaccination data among adults in a multistate network. METHODS: Following the 2018-2019 and 2019-2020 influenza seasons, surveys were conducted among a random sample of adults who did or did not appear influenza-vaccinated (per EHR data) during the influenza season. Participants were asked to report their influenza vaccination status; self-report was treated as the criterion standard. Results were combined across survey years. RESULTS: Survey response rate was 44.7% (777 of 1740) for the 2018-2019 influenza season and 40.5% (505 of 1246) for the 2019-2020 influenza season. The sensitivity of EHR-based influenza vaccination data was 75.0% (95% confidence interval [CI] 68.1, 81.1), specificity 98.4% (95% CI 92.9, 99.9), and negative predictive value 73.9% (95% CI 68.0, 79.3). CONCLUSIONS: In a multistate research network across two recent influenza seasons, there was moderate concordance between EHR-based vaccination data and self-report.

Post-COVID conditions following COVID-19 vaccination: a retrospective matched cohort study of patients with SARS-CoV-2 infection.

COVID-19 vaccinations protect against severe illness and death, but associations with post-COVID conditions (PCC) are less clear. We aimed to evaluate the association between prior COVID-19 vaccination and new-onset PCC among individuals with SARS-CoV-2 infection across eight large healthcare systems in the United States. This retrospective matched cohort study used electronic health records (EHR) from patients with SARS-CoV-2 positive tests during March 2021-February 2022. Vaccinated and unvaccinated COVID-19 cases were matched on location, test date, severity of acute infection, age, and sex. Vaccination status was ascertained using EHR and integrated data on externally administered vaccines. Adjusted relative risks (RRs) were obtained from Poisson regression. PCC was defined as a new diagnosis in one of 13 PCC categories 30 days to 6 months following a positive SARS-CoV-2 test. The study included 161,531 vaccinated COVID-19 cases and 161,531 matched unvaccinated cases. Compared to unvaccinated cases, vaccinated cases had a similar or lower risk of all PCC categories except mental health disorders (RR: 1.06, 95% CI: 1.02-1.10). Vaccination was associated with ≥10% lower risk of sensory (RR: 0.90, 0.86-0.95), circulatory (RR: 0.88, 0.83-0.94), blood and hematologic (RR: 0.79, 0.71-0.89), skin and subcutaneous (RR: 0.69, 0.66-0.72), and non-specific COVID-19 related disorders (RR: 0.53, 0.51-0.56). In general, associations were stronger at younger ages but mostly persisted regardless of SARS-CoV-2 variant period, receipt of ≥3 vs. 1-2 vaccine doses, or time since vaccination. Pre-infection vaccination was associated with reduced risk of several PCC outcomes and hence may decrease the long-term consequences of COVID-19.