Development and Evaluation of the Algorithm CErtaInty Tool (ACE-IT) to Assess Electronic Medical Record and Claims-based Algorithms' Fit for Purpose for Safety Outcomes.

INTRODUCTION: Electronic health record (EHR) or medical claims-based algorithms (i.e., operational definitions) can be used to define safety outcomes using real-world data. However, existing tools do not allow researchers and decision-makers to adequately appraise whether a particular algorithm is fit for purpose (FFP) to support regulatory decisions on drug safety surveillance. Our objective was to develop a tool to enable regulatory decision-makers and other stakeholders to appraise whether a given algorithm is FFP for a specific decision context. METHODS: We drafted a set of 77 generic items informed by regulatory guidance documents, existing instruments, and publications. The outcome of ischemic stroke served as an exemplar to inform the development of draft items. The items were designed to be outcome independent. We conducted a three-round online Delphi panel to develop and refine the tool and achieve consensus on items (> 70% agreement) among panel participants composed of regulators, researchers from pharmaceutical organizations, academic clinicians, methodologists, pharmacoepidemiologists, and cardiologists. We conducted a qualitative analysis of panel responses. Five pairs of reviewers independently evaluated two ischemic stroke algorithm validation studies to test its application. We developed a user guide, with explanation and elaboration for each item, guidance on essential and additional elements for user responses, and an illustrative example of a complete assessment. Furthermore, we conducted a 2-h online stakeholder panel of 16 participants from regulatory agencies, academic institutions, and industry. We solicited input on key factors for an FFP assessment, their general reaction to the Algorithm CErtaInty Tool (ACE-IT), limitations of the tool, and its potential use. RESULTS: The expert panel reviewed and made changes to the initial list of 77 items. The panel achieved consensus on 38 items, and the final version of the ACE-IT includes 34 items after removal of duplicate items. Applying the tool to two ischemic stroke algorithms demonstrated challenges in its application and identified shared concepts addressed by more than one item. The ACE-IT was viewed positively by the majority of stakeholders. They identified that the tool could serve as an educational resource as well as an information-sharing platform. The time required to complete the assessment was identified as an important limitation. We consolidated items with shared concepts and added a preliminary screen section and a summary assessment box based on their input. The final version of the ACE-IT is a 34-item tool for assessing whether algorithm validation studies on safety outcomes are FFP. It comprises the domains of internal validity (24 items), external validity (seven items), and ethical conduct and reporting of the validation study (three items). The internal validity domain includes sections on objectives, data sources, population, outcomes, design and setting, statistical methods, reference standard, accuracy, and strengths and limitations. The external validity domain includes items that assess the generalizability to a proposed target study. The domain on ethics and transparency includes items on ethical conduct and reporting of the validation study. CONCLUSION: The ACE-IT supports a structured, transparent, and flexible approach for decision-makers to appraise whether electronic health record or medical claims-based algorithms for safety outcomes are FFP for a specific decision context. Reliability and validity testing using a larger sample of participants in other therapeutic areas and further modifications to reduce the time needed to complete the assessment are needed to fully evaluate its utility for regulatory decision-making.

A review of stakeholder recommendations for defining fit-for-purpose real-world evidence algorithms.

Aim: The credibility and value of real-world evidence (RWE) are either supported or undermined by the algorithms (i.e., operational definitions) used. Methods: We conducted a targeted evidence review of key RWE decision makers' published recommendations on RWE algorithms through April 2021. Stakeholders were regulatory bodies, other governmental agencies and payer organizations. Results: Our review identified recommended criteria: relevance, validity, reliability, responsiveness, transparency and replicability, safety, feasibility and quality process. Stakeholders routinely recommended accuracy measures, subgroups evaluation and specific considerations for assessing exposures and covariates and the underlying real-world data (RWD) quality. Conclusion: The importance of stakeholder guidance on fit-for-purpose RWE algorithms is growing. We highlight gaps that future guidance and stakeholder recommendations could address.

The utility of real-world evidence for benefit-risk assessment, communication, and evaluation of pharmaceuticals: Case studies.

PURPOSE: In recent years, novel types of real-world evidence (RWE) have played a role in various decision-making processes relating to medicinal products, including regulatory approval, patient access, health technology assessment, safety monitoring, clinical use, and post-approval lifecycle management. We therefore reviewed the potential utility of RWE in the cycle of medicinal product benefit-risk (BR) assessment, communication/risk minimization and evaluation ("BRACE"). METHODS: A convenience sample of illustrative studies was drawn from the published literature and examined. Specifically, we examined the purpose for using RWE, the type of RWE used, its novelty and how it might be integrated with other data and activities of the BRACE cycle, and how it contributed to regulatory decision-making. RESULTS: Eight studies were selected with each illustrating a different activity in the BRACE cycle ranging from BR assessment in the preapproval setting, post-approval assessment of safety or effectiveness, communicating BR information to patients and healthcare professionals, and evaluating the effectiveness of risk minimization initiatives to support a positive BR balance. CONCLUSIONS: RWE has an important role in informing regulatory decision-making regarding the BR management of medicines. With increasing digitalization, facilitating data collection and stakeholder engagement in health, this role is only expected to expand in the future. To reach the full potential of RWE, both regulators and sponsors will need to be familiar with a range of existing and emerging methods for generating and analyzing such evidence appropriately and achieve convergence regarding how different types of RWE can best be used to inform BR management and decision-making.

Electronic Health Record-Based Strategy to Promote Medication Adherence Among Patients With Diabetes: Longitudinal Observational Study.

BACKGROUND: Poor medication adherence is common; however, few mechanisms exist in clinical practice to monitor how patients take medications in outpatient settings. OBJECTIVE: This study aimed to pilot test the Electronic Medication Complete Communication (EMC2) strategy, a low-cost, sustainable approach that uses functionalities within the electronic health record to promote outpatient medication adherence and safety. METHODS: The EMC2 strategy was implemented in 2 academic practices for 14 higher-risk diabetes medications. The strategy included: (1) clinical decision support alerts to prompt provider counseling on medication risks, (2) low-literacy medication summaries for patients, (3) a portal-based questionnaire to monitor outpatient medication use, and (4) clinical outreach for identified concerns. We recruited adult patients with diabetes who were prescribed a higher-risk diabetes medication. Participants completed baseline and 2-week interviews to assess receipt of, and satisfaction with, intervention components. RESULTS: A total of 100 patients were enrolled; 90 completed the 2-week interview. Patients were racially diverse, 30.0% (30/100) had a high school education or less, and 40.0% (40/100) had limited literacy skills. About a quarter (28/100) did not have a portal account; socioeconomic disparities were noted in account ownership by income and education. Among patients with a portal account, 58% (42/72) completed the questionnaire; 21 of the 42 patients reported concerns warranting clinical follow-up. Of these, 17 were contacted by the clinic or had their issue resolved within 24 hours. Most patients (33/38, 89%) who completed the portal questionnaire and follow-up interview reported high levels of satisfaction (score of 8 or greater on a scale of 1-10). CONCLUSIONS: Findings suggest that the EMC2 strategy can be reliably implemented and delivered to patients, with high levels of satisfaction. Disparities in portal use may restrict intervention reach. Although the EMC2 strategy can be implemented with minimal impact on clinic workflow, future trials are needed to evaluate its effectiveness to promote adherence and safety.

Benefit-Risk Assessment, Communication, and Evaluation (BRACE) throughout the life cycle of therapeutic products: overall perspective and role of the pharmacoepidemiologist.

PURPOSE: Optimizing a therapeutic product's benefit-risk profile is an on-going process throughout the product's life cycle. Different, yet related, benefit-risk assessment strategies and frameworks are being developed by various regulatory agencies, industry groups, and stakeholders. This paper summarizes current best practices and discusses the role of the pharmacoepidemiologist in these activities, taking a life-cycle approach to integrated Benefit-Risk Assessment, Communication, and Evaluation (BRACE). METHODS: A review of the medical and regulatory literature was performed for the following steps involved in therapeutic benefit-risk optimization: benefit-risk evidence generation; data integration and analysis; decision making; regulatory and policy decision making; benefit-risk communication and risk minimization; and evaluation. Feedback from International Society for Pharmacoepidemiology members was solicited on the role of the pharmacoepidemiologist. The case example of natalizumab is provided to illustrate the cyclic nature of the benefit-risk optimization process. RESULTS: No single, globally adopted benefit-risk assessment process exists. The BRACE heuristic offers a way to clarify research needs and to promote best practices in a cyclic and integrated manner and highlight the critical importance of cross-disciplinary input. Its approach focuses on the integration of BRACE activities for risk minimization and optimization of the benefit-risk profile. CONCLUSION: The activities defined in the BRACE heuristic contribute to the optimization of the benefit-risk profile of therapeutic products in the clinical world at both the patient and population health level. With interdisciplinary collaboration, pharmacoepidemiologists are well suited for bringing in methodology expertise, relevant research, and public health perspectives into the BRACE process.

Risk of hepatic events in patients treated with vancomycin in clinical studies: a systematic review and meta-analysis.

BACKGROUND: Routine surveillance of spontaneous reporting data and subsequent disproportionality analyses have indicated that the use of vancomycin might be associated with an increased risk of hepatic events. OBJECTIVE: To conduct a meta-analysis of published randomized controlled clinical trials (RCTs) to better understand if the use of vancomycin is potentially associated with an increased risk of hepatic events. DATA SOURCES: A comprehensive search and review of published clinical studies indexed in MEDLINE, PubMed, International Pharmaceutical Abstracts and the Cochrane Library from 1950 to June 2010 was conducted. STUDY SELECTION: The inclusion criteria consisted of (i) published RCTs comparing vancomycin with/without other additional treatments to other comparators; and (ii) studies that reported hepatic events. DATA EXTRACTION: The data related to any hepatic events reported in RCTs were extracted and examined. The quality of selected studies was assessed based on the Jadad scale. The effect size was presented as a risk ratio (RR) with a 95% CI and number needed to harm. The pooled RRs were calculated by using both fixed-effects and random-effects models. The impact of publication bias was assessed by funnel plot and the Egger's test. DATA SYNTHESIS: A total of 20 RCTs, including 7419 patients, met the study inclusion criteria and were selected. An increased incidence of hepatic events, specifically elevated serum aminotransferase levels, was observed in patients receiving vancomycin, when compared with other comparators (pooled RR=1.95; 95% CI 1.62, 2.36; p<0.001), but the majority of the events were mild to moderate in nature. No evidence is currently available suggesting that the use of vancomcycin confers a risk of progressive or severe drug-induced liver injury. CONCLUSIONS: Continuous monitoring of hepatic events on a routine basis among patients with the use of vancomycin is suggested.

Co-possession of phosphodiesterase type-5 inhibitors (PDE5-I) with nitrates.

OBJECTIVE: Estimate the proportion of phosphodiesterase type-5 inhibitor (PDE5-I) patients who co-possess nitrates and compare the proportion of tadalafil patients dispensed nitrates to a matched control group. Secondarily, examine the percentage of co-possession of PDE5-Is and nitrates where the products were dispensed on the same day or written by the same prescriber. METHODS: Male patients aged 18+ years filling PDE5-I prescriptions between December 2003 and March 2006 were identified using a U.S. longitudinal prescription database (IMS Health LRx). Similar patients not dispensed a PDE5-I during this period were matched to the tadalafil-dispensed cohort using a propensity score approach. Co-possession, as a proxy for concurrent use, was defined as an overlap in time on therapy for a PDE5-I and nitrate and was compared for the three PDE5-Is and for tadalafil to the matched control group. RESULTS: Among 601,063 tadalafil patients, 3.31% were dispensed a nitrate during the study period, compared to 6.18% in control patients (n = 601,063). When co-possessed prescriptions were defined by overlapping exposure periods, the proportion of PDE5-I patients with co-possessed nitrates ranged from 1.44% (tadalafil) to 1.72% (vardenafil) and 2.13% (sildenafil). Co-possession percentages of PDE5-I prescriptions were 0.83% for tadalafil and 1.07% for sildenafil and vardenafil. The majority (54.29%) of co-possessed PDE5-I and nitrate prescriptions had the nitrate dispensed prior to the PDE5-I prescription identified in the study cohort. CONCLUSIONS: Keeping in mind the limitations of observational studies, these results suggest that co-dispensing of nitrates and PDE5-Is is low. Compared to control patients, the proportion of nitrate co-possession was lowest for patients filling tadalafil. Tadalafil patients also had the lowest co-possessed proportion among the three PDE5-I cohorts. While the majority of co-possessed drug pairs were prescribed by different providers, the highest percentage of co-prescribing from the same physician was among cardiologists. These results suggest that physicians adhere to contraindications and are careful about co-prescribing of nitrates with PDE-5Is.

The effect of pharmacotherapy for attention deficit hyperactivity disorder on risk of seizures in pediatric patients as assessed in an insurance claims database.

PURPOSE: To estimate the rate of new-onset seizure in ADHD patients in relation to ADHD pharmacotherapy. METHODS: A retrospective cohort study of 34,727 patients, ages 6 to 17, with at least two insurance claims bearing ADHD diagnoses during 2003 in the UnitedHealthcare database. Incidence of seizure was calculated for observation time during treatment with atomoxetine and stimulants/bupropion. RESULTS: Seizure incidence among ADHD patients was 4.5/1,000 person-years (p-y; 95% confidence interval 3.7 - 5.5). ADHD patients who received any ADHD medication had an incidence of 3.8/1,000 p-y (3.0 - 4.8) compared to 8.7 (5.8 - 12.4) for patients who did not receive any ADHD medication. The relative risk (RR) for current vs non-use of atomoxetine was 1.1 (0.6 - 2.1). For stimulants and bupropion, the RR for current vs non-use was 0.8 (0.6 - 1.3). Elevated seizure risks were found in association with central nervous system (CNS) disease (OR 3.9, 1.2 - 10.9), CNS medications (OR 2.2, 1.3 - 3.6), metabolic disease (OR 2.9, 1.1 - 6.8), and psychiatric disease risk factors (OR 1.7, 1.1 - 2.6). CONCLUSIONS: In this study, there was no statistically significant association between use of atomoxetine or stimulants and seizure risk in children ages 6 to 17 years with ADHD and without prior seizure disorder.

Diabetes mellitus and antipsychotic treatment in the United Kingdom.

OBJECTIVE: Treatment-emergent diabetes has been reported during exposure to conventional and atypical antipsychotics. This retrospective cohort study explored the UK General Practice Research Database (GPRD) to determine hazard ratios of diabetes for patients prescribed antipsychotics. METHODS: A Cox proportional hazard regression model using age, gender, and obesity (BMI > 30 kg/m2) was used to determine the hazard ratio (HR) of diabetes development in conventional antipsychotic (N = 59,089), atypical antipsychotic (N = 9053), individual antipsychotic, and general patient population cohorts (N = 1,491,548). RESULTS: Compared with the general GPRD patient population, patients exposed to conventional or atypical antipsychotics had a higher risk of developing diabetes (atypical antipsychotic cohort: HR = 2.9, CI = 2.0-4.4; and conventional antipsychotic cohort: HR = 1.9, CI = 1.6-2.3). The risk of developing diabetes during thioridazine, risperidone, or olanzapine treatment was significantly higher compared with the general GPRD patient population. CONCLUSION: Consistent with other epidemiology studies, this study supports an increased risk of developing diabetes during treatment with antipsychotics.

Retrospective cohort study of diabetes mellitus and antipsychotic treatment in a geriatric population in the United States.

OBJECTIVES: The objective of this study was to investigate risk of diabetes among elderly patients during treatment with antipsychotic medications. DESIGN: We conducted a longitudinal, retrospective study assessing the incidence of new prescription claims for antihyperglycemic agents during antipsychotic therapy. SETTING: Prescription claims from the AdvancePCS claim database were followed for 6 to 9 months. PARTICIPANTS: Study participants consisted of patients in the United States aged 60+ and receiving antipsychotic monotherapy. The following cohorts were studied: an elderly reference population (no antipsychotics: n = 1,836,799), those receiving haloperidol (n = 6481) or thioridazine (n = 1658); all patients receiving any conventional antipsychotic monotherapy (n = 11,546), clozapine (n = 117), olanzapine (n = 5382), quetiapine (n = 1664), and risperidone (n = 12,244), and all patients receiving any atypical antipsychotic monotherapy (n = 19,407). MEASUREMENTS: We used Cox proportional hazards regression to determine the risk ratio of diabetes for antipsychotic cohorts relative to the reference population. Covariates included sex and exposure duration. RESULTS: New antihyperglycemic prescription rates were higher in each antipsychotic cohort than in the reference population. Overall rates were no different between atypical and conventional antipsychotic cohorts. Among individual antipsychotic cohorts, rates were highest among patients treated with thioridazine (95% confidence interval [CI], 3.1- 5.7), lowest with quetiapine (95% CI, 1.3-2.9), and intermediate with haloperidol, olanzapine, and risperidone. Among atypical cohorts, only risperidone users had a significantly higher risk (95% CI, 1.05-1.60; P = 0.016) than for haloperidol. Conclusions about clozapine were hampered by the low number of patients. CONCLUSION: These data suggest that diabetes risk is elevated among elderly patients receiving antipsychotic treatment. However, causality remains to be demonstrated. As a group, the risk for atypical antipsychotic users was not significantly different than for users of conventional antipsychotics.

A retrospective cohort study of diabetes mellitus and antipsychotic treatment in the United States.

Treatment-emergent diabetes mellitus (DM) has been described for conventional and atypical antipsychotics. In our study, antipsychotic prescription claims from AdvancePCS's database were used to identify patients starting antipsychotic monotherapy. The relative risk of developing DM was determined using prescription claims for antidiabetic agents in the following cohorts: AdvancePCS general patient population, combined conventional antipsychotics, and combined atypical antipsychotics. Cox proportional hazards regression was used to adjust for differences in age, gender, and duration of antipsychotic exposure between cohorts in the estimation of risk of developing diabetes. Hazard ratios for developing DM in the combined conventional, combined atypical, and individual conventional and atypical antipsychotic treatment cohorts were greater than the AdvancePCS general patient population cohort. An increased risk of developing diabetes compared with the AdvancePCS general patient population was observed during treatment with conventional or atypical antipsychotics.