A standardized framework for risk-based assessment of treatment effect heterogeneity in observational healthcare databases.

Treatment effects are often anticipated to vary across groups of patients with different baseline risk. The Predictive Approaches to Treatment Effect Heterogeneity (PATH) statement focused on baseline risk as a robust predictor of treatment effect and provided guidance on risk-based assessment of treatment effect heterogeneity in a randomized controlled trial. The aim of this study is to extend this approach to the observational setting using a standardized scalable framework. The proposed framework consists of five steps: (1) definition of the research aim, i.e., the population, the treatment, the comparator and the outcome(s) of interest; (2) identification of relevant databases; (3) development of a prediction model for the outcome(s) of interest; (4) estimation of relative and absolute treatment effect within strata of predicted risk, after adjusting for observed confounding; (5) presentation of the results. We demonstrate our framework by evaluating heterogeneity of the effect of thiazide or thiazide-like diuretics versus angiotensin-converting enzyme inhibitors on three efficacy and nine safety outcomes across three observational databases. We provide a publicly available R software package for applying this framework to any database mapped to the Observational Medical Outcomes Partnership Common Data Model. In our demonstration, patients at low risk of acute myocardial infarction receive negligible absolute benefits for all three efficacy outcomes, though they are more pronounced in the highest risk group, especially for acute myocardial infarction. Our framework allows for the evaluation of differential treatment effects across risk strata, which offers the opportunity to consider the benefit-harm trade-off between alternative treatments.

A systematic assessment of the epidemiologic literature regarding an association between acetaminophen exposure and cancer.

Introduced in the 1950s, acetaminophen is one of the most widely used antipyretics and analgesics worldwide. In 1999, the International Agency for Research on Cancer (IARC) reviewed the epidemiologic studies of acetaminophen and the data were judged to be "inadequate" to conclude that it is carcinogenic. In 2019 the California Office of Environmental Health Hazard Assessment initiated a review process on the carcinogenic hazard potential of acetaminophen. To inform this review process, the authors performed a comprehensive literature search and identified 136 epidemiologic studies, which for most cancer types suggest no alteration in risk associated with acetaminophen use. For 3 cancer types, renal cell, liver, and some forms of lymphohematopoietic, some studies suggest an increased risk; however, multiple factors unique to acetaminophen need to be considered to determine if these results are real and clinically meaningful. The objective of this publication is to analyze the results of these epidemiologic studies using a framework that accounts for the inherent challenge of evaluating acetaminophen, including, broad population-wide use in multiple disease states, challenges with exposure measurement, protopathic bias, channeling bias, and recall bias. When evaluated using this framework, the data do not support a causal association between acetaminophen use and cancer.

Wisdom of the CROUD: Development and validation of a patient-level prediction model for opioid use disorder using population-level claims data.

OBJECTIVE: Some patients who are given opioids for pain could develop opioid use disorder. If it was possible to identify patients who are at a higher risk of opioid use disorder, then clinicians could spend more time educating these patients about the risks. We develop and validate a model to predict a person's future risk of opioid use disorder at the point before being dispensed their first opioid. METHODS: A cohort study patient-level prediction using four US claims databases with target populations ranging between 343,552 and 384,424 patients. The outcome was recorded diagnosis of opioid abuse, dependency or unspecified drug abuse as a proxy for opioid use disorder from 1 day until 365 days after the first opioid is dispensed. We trained a regularized logistic regression using candidate predictors consisting of demographics and any conditions, drugs, procedures or visits prior to the first opioid. We then selected the top predictors and created a simple 8 variable score model. RESULTS: We estimated the percentage of new users of opioids with reported opioid use disorder within a year to range between 0.04%-0.26% across US claims data. We developed an 8 variable Calculator of Risk for Opioid Use Disorder (CROUD) score, derived from the prediction models to stratify patients into higher and lower risk groups. The 8 baseline variables were age 15-29, medical history of substance abuse, mood disorder, anxiety disorder, low back pain, renal impairment, painful neuropathy and recent ER visit. 1.8% of people were in the high risk group for opioid use disorder and had a score > = 23 with the model obtaining a sensitivity of 13%, specificity of 98% and PPV of 1.14% for predicting opioid use disorder. CONCLUSIONS: CROUD could be used by clinicians to obtain personalized risk scores. CROUD could be used to further educate those at higher risk and to personalize new opioid dispensing guidelines such as urine testing. Due to the high false positive rate, it should not be used for contraindication or to restrict utilization.

Inferring disease severity in rheumatoid arthritis using predictive modeling in administrative claims databases.

BACKGROUND: Confounding by disease severity is an issue in pharmacoepidemiology studies of rheumatoid arthritis (RA), due to channeling of sicker patients to certain therapies. To address the issue of limited clinical data for confounder adjustment, a patient-level prediction model to differentiate between patients prescribed and not prescribed advanced therapies was developed as a surrogate for disease severity, using all available data from a US claims database. METHODS: Data from adult RA patients were used to build regularized logistic regression models to predict current and future disease severity using a biologic or tofacitinib prescription claim as a surrogate for moderate-to-severe disease. Model discrimination was assessed using the area under the receiver (AUC) operating characteristic curve, tested and trained in Optum Clinformatics® Extended DataMart (Optum) and additionally validated in three external IBM MarketScan® databases. The model was further validated in the Optum database across a range of patient cohorts. RESULTS: In the Optum database (n = 68,608), the AUC for discriminating RA patients with a prescription claim for a biologic or tofacitinib versus those without in the 90 days following index diagnosis was 0.80. Model AUCs were 0.77 in IBM CCAE (n = 75,579) and IBM MDCD (n = 7,537) and 0.75 in IBM MDCR (n = 36,090). There was little change in the prediction model assessing discrimination 730 days following index diagnosis (prediction model AUC in Optum was 0.79). CONCLUSIONS: A prediction model demonstrated good discrimination across multiple claims databases to identify RA patients with a prescription claim for advanced therapies during different time-at-risk periods as proxy for current and future moderate-to-severe disease. This work provides a robust model-derived risk score that can be used as a potential covariate and proxy measure to adjust for confounding by severity in multivariable models in the RA population. An R package to develop the prediction model and risk score are available in an open source platform for researchers.

Supplementing claims data analysis using self-reported data to develop a probabilistic phenotype model for current smoking status.

OBJECTIVES: Smoking status is poorly record in US claims data. IBM MarketScan Commercial is a claims database that can be linked to an additional health risk assessment with self-reported smoking status for a subset of 1,966,174 patients. We investigate whether this subset could be used to learn a smoking status phenotype model generalizable to all US claims data that calculates the probability of being a current smoker. METHODS: 251,643 (12.8%) had self-reported their smoking status as 'current smoker'. A regularized logistic regression model, the Current Risk of Smoking Status (CROSS), was trained using the subset of patients with self-reported smoking status. CROSS considered 53,027 candidate covariates including demographics and conditions/drugs/measurements/procedures/observations recorded in the prior 365 days, The CROSS phenotype model was validated across multiple other claims data. RESULTS: The internal validation showed the CROSS model achieved an area under the receiver operating characteristic curve (AUC) of 0.76 and the calibration plots indicated it was well calibrated. The external validation across three US claims databases obtained AUCs ranging between 0.82 and 0.87 showing the model appears to be transportable across Claims data. CONCLUSION: CROSS predicts current smoking status based on the claims records in the prior year. CROSS can be readily implemented to any US insurance claims mapped to the OMOP common data model and will be a useful way to impute smoking status when conducting epidemiology studies where smoking is a known confounder but smoking status is not recorded. CROSS is available from https://github.com/OHDSI/StudyProtocolSandbox/tree/master/SmokingModel.

Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data.

Objective: To develop a conceptual prediction model framework containing standardized steps and describe the corresponding open-source software developed to consistently implement the framework across computational environments and observational healthcare databases to enable model sharing and reproducibility. Methods: Based on existing best practices we propose a 5 step standardized framework for: (1) transparently defining the problem; (2) selecting suitable datasets; (3) constructing variables from the observational data; (4) learning the predictive model; and (5) validating the model performance. We implemented this framework as open-source software utilizing the Observational Medical Outcomes Partnership Common Data Model to enable convenient sharing of models and reproduction of model evaluation across multiple observational datasets. The software implementation contains default covariates and classifiers but the framework enables customization and extension. Results: As a proof-of-concept, demonstrating the transparency and ease of model dissemination using the software, we developed prediction models for 21 different outcomes within a target population of people suffering from depression across 4 observational databases. All 84 models are available in an accessible online repository to be implemented by anyone with access to an observational database in the Common Data Model format. Conclusions: The proof-of-concept study illustrates the framework's ability to develop reproducible models that can be readily shared and offers the potential to perform extensive external validation of models, and improve their likelihood of clinical uptake. In future work the framework will be applied to perform an "all-by-all" prediction analysis to assess the observational data prediction domain across numerous target populations, outcomes and time, and risk settings.

Risk Prediction for Ischemic Stroke and Transient Ischemic Attack in Patients Without Atrial Fibrillation: A Retrospective Cohort Study.

BACKGROUND: Stroke mainly occurs in patients without atrial fibrillation (AF). This study explored risk prediction models for ischemic stroke and transient ischemic attack (TIA) in patients without AF. METHODS: Three US-based healthcare databases (Truven MarketScan Commercial Claims and Encounters [CCAE], Medicare Supplemental [MDCR], and Optum Clinformatics [Optum]) were used to establish patient cohorts without AF during the index period of 2008-2012. The performance of 2 existing models (CHADS2 and CHA2DS2-VASc) for predicting stroke and TIA was examined by fitting a logistic regression to a training dataset and evaluating predictive accuracy in a validation dataset (area under the curve, AUC) using patients with complete follow-up of 1 or 3 years, separately. RESULTS: The commercial populations were younger and had fewer comorbidities than Medicare-eligible population. The incidence proportions of ischemic stroke and TIA during 1 and 3 years of follow-up were .5% and 1.9% (CCAE), .6% and 2.2% (Optum), and 4.6% and 13.1% (MDCR), respectively. The models performed consistently across all 3 databases, with the AUC ranging from .69 to .77 and from .68 to .73 for 1- and 3-year prediction, respectively. Predictive accuracy was lower than the initial work of CHADS2 evaluation in patients with AF (AUC: .82), but consistent with a subsequent meta-analysis of CHADS2 (.60-.80) and CHA2DS2-VASc performance (.64-.79). CONCLUSION: Although the existing schemes for predicting ischemic stroke and TIA in patients with AF can be applied to patients without AF with comparable predictive accuracy, the evidence suggests that there is room for improvement in these models' performance.

A Comparison of Data Quality Assessment Checks in Six Data Sharing Networks.

OBJECTIVE: To compare rule-based data quality (DQ) assessment approaches across multiple national clinical data sharing organizations. METHODS: Six organizations with established data quality assessment (DQA) programs provided documentation or source code describing current DQ checks. DQ checks were mapped to the categories within the data verification context of the harmonized DQA terminology. To ensure all DQ checks were consistently mapped, conventions were developed and four iterations of mapping performed. Difficult-to-map DQ checks were discussed with research team members until consensus was achieved. RESULTS: Participating organizations provided 11,026 DQ checks, of which 99.97 percent were successfully mapped to a DQA category. Of the mapped DQ checks (N=11,023), 214 (1.94 percent) mapped to multiple DQA categories. The majority of DQ checks mapped to Atemporal Plausibility (49.60 percent), Value Conformance (17.84 percent), and Atemporal Completeness (12.98 percent) categories. DISCUSSION: Using the common DQA terminology, near-complete (99.97 percent) coverage across a wide range of DQA programs and specifications was reached. Comparing the distributions of mapped DQ checks revealed important differences between participating organizations. This variation may be related to the organization's stakeholder requirements, primary analytical focus, or maturity of their DQA program. Not within scope, mapping checks within the data validation context of the terminology may provide additional insights into DQA practice differences. CONCLUSION: A common DQA terminology provides a means to help organizations and researchers understand the coverage of their current DQA efforts as well as highlight potential areas for additional DQA development. Sharing DQ checks between organizations could help expand the scope of DQA across clinical data networks.

The Impact of Censoring Drug Switching in Medication Adherence Measures of Chronic Single Ingredient Oral Drugs.

We explored how drug switching impacts adherence measures for common chronic oral medications. Switching between ingredients with the same indication was detected within a 30-day grace period. The proportion of days covered (PDC) and adherent status (cutoff 0.8) for each ingredient was calculated and compared between different censoring approaches: censoring drug switching (PDCswitch), censoring the end of dispensing (PDCend), and fixed 365-day period (PDC365). Overall, 854,380 (15.9%) patients in the Optum ClinFormatics (Optum) and 150,785 (22.0%) patients in the MarketScan Multi-state Medicaid (MDCD) had at least one switch within one year. Compared with PDC365 in Optum, PDCswitch means were higher: 0.85 vs. 0.41 for antihypertensive, 0.82 vs. 0.46 for antihyperglycemics, and 0.84 vs. 0.33 for antihyerlipidemia. Further, the percentages of adherent patients were higher: 95.8% vs. 17.9% for antihypertensive, 85.5% vs. 18.9% for antihyperglycemics, and 72.1% vs. 5.3% for antihyerlipidemia. Significant and modest changes were observed between PDCswitch and PDCend.

Detecting adverse drug reactions following long-term exposure in longitudinal observational data: The exposure-adjusted self-controlled case series.

Most approaches used in postmarketing drug safety monitoring, including spontaneous reporting and statistical risk identification using electronic health care records, are primarily suited to pick up only acute adverse drug effects. With the availability of increasingly larger electronic health record and administrative claims databases comes the opportunity to monitor for potential adverse effects that occur only after prolonged exposure to a drug, but analysis methods are lacking. We propose an adaptation of the self-controlled case series design that uses the notion of accumulated exposure to capture long-term effects of drugs and evaluate extensions to correct for age and recurrent events. Several variations of the approach are tested on simulated data and two large insurance claims databases. To evaluate performance a set of positive and negative control drug-event pairs was created by medical experts based on drug product labels and review of the literature. Performance on the real data was measured using the area under the receiver operator characteristics curve. The best performing method achieved an area under the receiver operator characteristics curve of 0.86 in the largest database using a spline model, adjustment for age, and ignoring recurrent events, but it appears this performance can only be achieved with very large data sets.

The impact of standardizing the definition of visits on the consistency of multi-database observational health research.

BACKGROUND: Use of administrative claims from multiple sources for research purposes is challenged by the lack of consistency in the structure of the underlying data and definition of data across claims data providers. This paper evaluates the impact of applying a standardized revenue code-based logic for defining inpatient encounters across two different claims databases. METHODS: We selected members who had complete enrollment in 2012 from the Truven MarketScan Commercial Claims and Encounters (CCAE) and the Optum Clinformatics (Optum) databases. The overall prevalence of inpatient conditions in the raw data was compared to that in the common data model (CDM) with the standardized visit definition applied. RESULTS: In CCAE, 87.18% of claims from 2012 that were classified as part of inpatient visits in the raw data were also classified as part of inpatient visits after the data were standardized to CDM, and this overlap was consistent from 2006 to 2011. In contrast, Optum had 83.18% concordance in classification of 2012 claims from inpatient encounters before and after standardization, but the consistency varied over time. The re-classification of inpatient encounters substantially impacted the observed prevalence of medical conditions occurring in the inpatient setting and the consistency in prevalence estimates between the databases. On average, before standardization, each condition in Optum was 12% more prevalent than that same condition in CCAE; after standardization, the prevalence of conditions had a mean difference of only 1% between databases. Amongst 7,039 conditions reviewed, the difference in the prevalence of 67% of conditions in these two databases was reduced after standardization. CONCLUSIONS: In an effort to improve consistency in research results across database one should review sources of database heterogeneity, such as the way data holders process raw claims data. Our study showed that applying the Observational Medical Outcomes Partnership (OMOP) CDM with a standardized approach for defining inpatient visits during the extract, transfer, and load process can decrease the heterogeneity observed in disease prevalence estimates across two different claims data sources.

Switching from risperidone long-acting injectable to paliperidone long-acting injectable or oral antipsychotics: analysis of a Medicaid claims database.

This report examines relapse risk following a switch from risperidone long-acting injectable (RLAI) to another long-acting injectable antipsychotic [paliperidone palmitate (PP)] versus a switch to oral antipsychotics (APs). Truven Health's MarketScan Multistate Medicaid Database compared relapses following switches from RLAI. New user cohorts for these two groups were created on the basis of first incidence of exposure to the 'switched to' drug. Groups were balanced using 1:1 propensity score matching. Time-to-event analysis assessed schizophrenia-related hospital/emergency department visits. A total of 188 patients switched from RLAI to PP, and 131 patients switched from RLAI to oral AP. Propensity score-matched cohort included 109 patients who switched to PP and 109 patients who switched to an oral AP. Patients who switched from RLAI to PP had fewer events (26 vs. 32), longer time to an event (mean 70 vs. 47 days), and lower risk of relapse (hazard ratio, 0.54; 95% confidence interval, 0.32-0.92; P=0.024) compared with those who switched from RLAI to oral AP. Switching from RLAI to PP may be associated with a lower risk for relapse and longer duration of therapy compared with switching to oral AP. Given the limitations of observational studies, these results should be confirmed by other prospective evaluations.

Assessment of case definitions for identifying acute liver injury in large observational databases.

BACKGROUND: Determining the aetiology of acute liver injury (ALI) may be challenging to both clinicians and researchers. Observational research is particularly useful in studying rare medical outcomes such as ALI; however, case definitions for ALI in previous observational studies lack consistency and sensitivity. ALI is a clinically important condition with various aetiologies, including drug exposure. OBJECTIVE: The aim of this study was to evaluate four distinct case definitions for ALI across a diverse set of large observational databases, providing a better understanding of ALI prevalence and natural history. DATA SOURCES: Seven healthcare databases: GE Healthcare, MarketScan(®) Lab Database, Humana Inc., Partners HealthCare System, Regenstrief Institute, SDI Health (now IMS Health, Inc.), and the National Patient Care Database of the Veterans Health Administration. METHODS: We evaluated prevalence of ALI through the application of four distinct case definitions across seven observational healthcare databases. We described how laboratory and clinical characteristics of identified case populations varied across definitions and examined the prevalence of other hepatobiliary disorders among identified ALI cases that may decrease suspicion of drug-induced liver injury (DILI) in particular. RESULTS: This study demonstrated that increasing the restrictiveness of the case definition resulted in fewer cases, but greater prevalence of ALI clinical features. Considerable heterogeneity in the frequency of laboratory testing and results observed among cases meeting the most restrictive definition suggests that the clinical features, monitoring patterns and suspicion of ALI are highly variable among patients. CONCLUSIONS: Creation of four distinct case definitions and application across a disparate set of observational databases resulted in significant variation in the prevalence of ALI. A greater understanding of the natural history of ALI through examination of electronic healthcare data can facilitate development of reliable and valid ALI case definitions that may enhance the ability to accurately identify associations between ALI and drug exposures. Considerable heterogeneity in laboratory values and frequency of laboratory testing among individuals meeting the criteria for ALI suggests that the evaluation of ALI is highly variable.

Empirical assessment of methods for risk identification in healthcare data: results from the experiments of the Observational Medical Outcomes Partnership.

BACKGROUND: Expanded availability of observational healthcare data (both administrative claims and electronic health records) has prompted the development of statistical methods for identifying adverse events associated with medical products, but the operating characteristics of these methods when applied to the real-world data are unknown. METHODS: We studied the performance of eight analytic methods for estimating of the strength of association-relative risk (RR) and associated standard error of 53 drug-adverse event outcome pairs, both positive and negative controls. The methods were applied to a network of ten observational healthcare databases, comprising over 130 million lives. Performance measures included sensitivity, specificity, and positive predictive value of methods at RR thresholds achieving statistical significance of p < 0.05 or p < 0.001 and with absolute threshold RR > 1.5, as well as threshold-free measures such as area under receiver operating characteristic curve (AUC). RESULTS: Although no specific method demonstrated superior performance, the aggregate results provide a benchmark and baseline expectation for risk identification method performance. At traditional levels of statistical significance (RR > 1, p < 0.05), all methods have a false positive rate >18%, with positive predictive value <38%. The best predictive model, high-dimensional propensity score, achieved an AUC = 0.77. At 50% sensitivity, false positive rate ranged from 16% to 30%. At 10% false positive rate, sensitivity of the methods ranged from 9% to 33%. CONCLUSIONS: Systematic processes for risk identification can provide useful information to supplement an overall safety assessment, but assessment of methods performance suggests a substantial chance of identifying false positive associations.

Design and validation of a data simulation model for longitudinal healthcare data.

Evaluating performance characteristics of analytic methods developed to identify treatment effects in longitudinal healthcare data has been hindered by lack of an objective benchmark to measure performance. Relationships between drugs and subsequent treatment effects are not precisely quantified in real-world data, and simulated data offer potential to augment method development by providing data with known, measurable characteristics. However, the use of simulated data has been limited due to its inability to adequately reflect the complexities inherent in real-world databases that are necessary for effective method development. The goal of this study was to develop and evaluate a model for simulating longitudinal healthcare data that adequately captures these complexities. An empiric design was chosen that utilizes the characteristics of a real healthcare database as simulation input. This model demonstrates the potential for simulated data with known characteristics to adequately reflect complex relationships among diseases and treatments as recorded in healthcare databases.

Advancing the science for active surveillance: rationale and design for the Observational Medical Outcomes Partnership.

The U.S. Food and Drug Administration (FDA) Amendments Act of 2007 mandated that the FDA develop a system for using automated health care data to identify risks of marketed drugs and other medical products. The Observational Medical Outcomes Partnership is a public-private partnership among the FDA, academia, data owners, and the pharmaceutical industry that is responding to the need to advance the science of active medical product safety surveillance by using existing observational databases. The Observational Medical Outcomes Partnership's transparent, open innovation approach is designed to systematically and empirically study critical governance, data resource, and methodological issues and their interrelationships in establishing a viable national program of active drug safety surveillance by using observational data. This article describes the governance structure, data-access model, methods-testing approach, and technology development of this effort, as well as the work that has been initiated.

Patterns in nursing home medication errors: disproportionality analysis as a novel method to identify quality improvement opportunities.

PURPOSE: To explore the use of disproportionality analysis of medication error data as a novel method to identify relationships that might not be obvious through traditional analyses. This approach can supplement descriptive data and target quality improvement efforts. METHODS: Data came from the Medication Error Quality Initiative (MEQI) individual event reporting system. Participants were North Carolina nursing homes who submitted incident reports to the Web-based MEQI data repository during the 2006 and 2007 reporting years. Data from 206 nursing homes were summarized descriptively and then disproportionality analysis was applied. Associations between medication type and possible causes at the state level were explored. A single nursing home was selected to illustrate how the method might inform quality improvement at the facility level. Disproportionality analysis of drug errors in this home was compared with benchmarking. RESULTS: Statewide, 59 drug-cause pairs met the disproportionality signal and 11 occurred in 10 or more reports. Among these, warfarin was co-reported with communication errors; esomeprazole, risperidone, and nitrofurantoin were disproportionately associated with transcription error; and oxycodone and morphine were disproportionately reported with name confusion. Facility-level analyses illustrate how descriptive frequencies and disproportionality analysis are complementary, but also identify different safety targets. CONCLUSIONS: Exploratory analysis tools can help identify medication error types that occur at disproportionate rates. Candidate associations might be used to target patient safety work, although further evaluation is needed to determine the value of this information.