Signal Detection in EUROmediCAT: Identification and Evaluation of Medication-Congenital Anomaly Associations and Use of VigiBase as a Complementary Source of Reference.

INTRODUCTION: Knowledge on the safety of medication use during pregnancy is often sparse. Pregnant women are generally excluded from clinical trials, and there is a dependence on post-marketing surveillance to identify teratogenic medications. AIMS: This study aimed to identify signals of potentially teratogenic medications using EUROmediCAT registry data on medication exposure in pregnancies with a congenital anomaly, and to investigate the use of VigiBase reports of adverse events of medications in the evaluation of these signals. METHODS: Signals of medication-congenital anomaly associations were identified in EUROmediCAT (21,636 congenital anomaly cases with 32,619 medication exposures), then investigated in a subset of VigiBase (45,749 cases and 165,121 exposures), by reviewing statistical reporting patterns and VigiBase case reports. Evidence from the literature and quantitative and qualitative aspects of both datasets were considered before recommending signals as warranting further independent investigation. RESULTS: EUROmediCAT analysis identified 49 signals of medication-congenital anomaly associations. Incorporating investigation in VigiBase and the literature, these were categorised as follows: four non-specific medications; 11 likely due to maternal disease; 11 well-established teratogens; two reviewed in previous EUROmediCAT studies with limited additional evidence; and 13 with insufficient basis for recommending follow-up. Independent investigations are recommended for eight signals: pregnen (4) derivatives with limb reduction; nitrofuran derivatives with cleft palate and patent ductus arteriosus; salicylic acid and derivatives with atresia or stenosis of other parts of the small intestine and tetralogy of Fallot; carbamazepine with atrioventricular septal defect and severe congenital heart defect; and selective beta-2-adrenoreceptor agonists with posterior urethral valve and/or prune belly. CONCLUSION: EUROmediCAT data should continue to be used for signal detection, accompanied by information from VigiBase and review of the existing literature to prioritise signals for further independent evaluation.

A Feasibility Study of Drug-Drug Interaction Signal Detection in Regular Pharmacovigilance.

INTRODUCTION: Adverse drug reactions related to drug-drug interactions cause harm to patients. There is a body of research on signal detection for drug interactions in collections of individual case reports, but limited use in regular pharmacovigilance. OBJECTIVE: The aim of this study was to evaluate the feasibility of signal detection of drug-drug interactions in collections of individual case reports of suspected adverse drug reactions. METHODS: This study was conducted in VigiBase, the WHO global database of individual case safety reports. The data lock point was 31 August 2016, which provided 13.6 million reports for analysis after deduplication. Statistical signal detection was performed using a previously developed predictive model for possible drug interactions. The model accounts for an interaction disproportionality measure, expressed suspicion of an interaction by the reporter, potential for interaction through cytochrome P450 activity of drugs, and reported information indicative of unexpected therapeutic response or altered therapeutic effect. Triage filters focused the preliminary signal assessment on combinations relating to serious adverse events with case series of no more than 30 reports from at least two countries, with at least one report during the previous 2 years. Additional filters sought to eliminate already known drug interactions through text mining of standard literature sources. Preliminary signal assessment was performed by a multidisciplinary group of pharmacovigilance professionals from Uppsala Monitoring Centre and collaborating organizations, whereas in-depth signal assessment was performed by experienced pharmacovigilance assessors. RESULTS: We performed preliminary signal assessment for 407 unique drug pairs. Of these, 157 drug pairs were considered already known to interact, whereas 232 were closed after preliminary assessment for other reasons. Ten drug pairs were subjected to in-depth signal assessment and an additional eight were decided to be kept under review awaiting additional reports. The triage filters had a major impact in focusing our preliminary signal assessment on just 14% of the statistical signals generated by the predictive model for drug interactions. In-depth assessment led to three signals communicated with the broader pharmacovigilance community, six closed signals and one to be kept under review. CONCLUSION: This study shows that signals of adverse drug interactions can be detected through broad statistical screening of individual case reports. It further shows that signal assessment related to possible drug interactions requires more detailed information on the temporal relationship between different drugs and the adverse event. Future research may consider whether interaction signal detection should be performed not for individual adverse event terms but for pairs of drugs across a spectrum of adverse events.

Prospective Evaluation of Adverse Event Recognition Systems in Twitter: Results from the Web-RADR Project.

INTRODUCTION: A large number of studies on systems to detect and sometimes normalize adverse events (AEs) in social media have been published, but evidence of their practical utility is scarce. This raises the question of the transferability of such systems to new settings. OBJECTIVES: The aims of this study were to develop an AE recognition system, prospectively evaluate its performance on an external benchmark dataset and identify potential factors influencing the transferability of AE recognition systems. METHODS: A pipeline based on dictionary lookups and logistic regression classifiers was developed using a proprietary dataset of 196,533 Tweets manually annotated for AE relations and prospectively evaluated the system on the publicly available WEB-RADR reference dataset, exploring different aspects affecting transferability. RESULTS: Our system achieved 0.53 precision, 0.52 recall and 0.52 F1-score on the development test set; however, when applied to the WEB-RADR reference dataset, system performance dropped to 0.38 precision, 0.20 recall and 0.26 F1-score. Similarly, a previously published method aiming at automatically detecting adverse event posts reported 0.5 precision, 0.92 recall and 0.65 F1-score on thus another dataset, while performance on the WEB-RADR reference dataset was reduced to 0.37 precision, 0.63 recall and 0.46 F1-score. We identified four potential factors leading to poor transferability: overfitting, selection bias, label bias and prevalence. CONCLUSION: We warn the community about a potentially large discrepancy between the expected performance of automated AE recognition systems based on published results and the actual observed performance on independent data. This study highlights the difficulty of implementing an all-purpose system for automatic adverse event recognition in Twitter, which could explain the lack of such systems in practical pharmacovigilance settings. Our recommendation is to use benchmark independent datasets, such as the WEB-RADR reference, to investigate the transferability of the adverse event recognition systems and ultimately enforce rigorous comparisons across studies on the task.

Recommendations for the Use of Social Media in Pharmacovigilance: Lessons from IMI WEB-RADR.

Over a period of 3 years, the European Union's Innovative Medicines Initiative WEB-RADR project has explored the value of social media (i.e., information exchanged through the internet, typically via online social networks) for identifying adverse events as well as for safety signal detection. Many patients and clinicians have taken to social media to discuss their positive and negative experiences of medications, creating a source of publicly available information that has the potential to provide insights into medicinal product safety concerns. The WEB-RADR project has developed a collaborative English language workspace for visualising and analysing social media data for a number of medicinal products. Further, novel text and data mining methods for social media analysis have been developed and evaluated. From this original research, several recommendations are presented with supporting rationale and consideration of the limitations. Recommendations for further research that extend beyond the scope of the current project are also presented.

Correction to: vigiGrade: A Tool to Identify Well-Documented Individual Case Reports and Highlight Systematic Data Quality Issues.

The article vigiGrade: A Tool to Identify Well-Documented Individual Case Reports and Highlight Systematic Data Quality Issues, written by Tomas Bergvall. G. Niklas Norén. Marie Lindquist, was originally published Online First without open access.

Paediatric safety signals identified in VigiBase: Methods and results from Uppsala Monitoring Centre.

PURPOSE: The purpose of this study is to uncover previously unrecognised risks of medicines in paediatric pharmacovigilance reports and thereby advance a safer use of medicines in paediatrics. METHODS: Individual case safety reports (ICSRs) with ages less than 18 years were retrieved from VigiBase, the World Health Organization (WHO) global database of ICSRs, in September 2014. The reports were grouped according to the following age spans: 0 to 27 days; 28 days to 23 months; 2 to 11 years; and 12 to 17 years. vigiRank, a data-driven predictive model for emerging safety signals, was used to prioritise the list of drug events by age groups. The list was manually assessed, and potential signals were identified to undergo in-depth assessment to determine whether a signal should be communicated. RESULTS: A total of 472 drug-event pairs by paediatric age groups were the subject of an initial manual assessment. Twenty-seven drug events from the two older age groups were classified as potential signals. An in-depth assessment resulted in eight signals, of which one concerned harm in connection with off-label use of dextromethorphan and another with accidental overdose of olanzapine by young children, and the remaining signals referred to potentially new causal associations for atomoxetine (two signals), temozolamide, deferasirox, levetiracetam, and desloratadine that could be relevant also for adults. CONCLUSIONS: Clinically relevant signals were uncovered in VigiBase by using vigiRank applied to paediatric age groups. Further refinement of the methodology is needed to identify signals in reports with ages under 2 years and to capture signals specific to the paediatric population as a risk group.

vigiRank for statistical signal detection in pharmacovigilance: First results from prospective real-world use.

PURPOSE: vigiRank is a data-driven predictive model for emerging safety signals. In addition to disproportionate reporting patterns, it also accounts for the completeness, recency, and geographic spread of individual case reporting, as well as the availability of case narratives. Previous retrospective analysis suggested that vigiRank performed better than disproportionality analysis alone. The purpose of the present analysis was to evaluate its prospective performance. METHODS: The evaluation of vigiRank was based on real-world signal detection in VigiBase. In May 2014, vigiRank scores were computed for pairs of new drugs and WHO Adverse Reaction Terminology critical terms with at most 30 reports from at least 2 countries. Initial manual assessments were performed in order of descending score, selecting a subset of drug-adverse drug reaction pairs for in-depth expert assessment. The primary performance metric was the proportion of initial assessments that were decided signals during in-depth assessment. As comparator, the historical performance for disproportionality- guided signal detection in VigiBase was computed from a corresponding cohort of drug-adverse drug reaction pairs assessed between 2009 and 2013. During this period, the requirement for initial manual assessment was a positive lower endpoint of the 95% credibility interval of the Information Component measure of disproportionality, observed for the first time. RESULTS: 194 initial assessments suggested by vigiRank's ordering eventually resulted in 6 (3.1%) signals. Disproportionality analysis yielded 19 signals from 1592 initial assessments (1.2%; P < .05). CONCLUSIONS: Combining multiple strength-of-evidence aspects as in vigiRank significantly outperformed disproportionality analysis alone in real-world pharmacovigilance signal detection, for VigiBase.

An Interoperability Platform Enabling Reuse of Electronic Health Records for Signal Verification Studies.

Depending mostly on voluntarily sent spontaneous reports, pharmacovigilance studies are hampered by low quantity and quality of patient data. Our objective is to improve postmarket safety studies by enabling safety analysts to seamlessly access a wide range of EHR sources for collecting deidentified medical data sets of selected patient populations and tracing the reported incidents back to original EHRs. We have developed an ontological framework where EHR sources and target clinical research systems can continue using their own local data models, interfaces, and terminology systems, while structural interoperability and Semantic Interoperability are handled through rule-based reasoning on formal representations of different models and terminology systems maintained in the SALUS Semantic Resource Set. SALUS Common Information Model at the core of this set acts as the common mediator. We demonstrate the capabilities of our framework through one of the SALUS safety analysis tools, namely, the Case Series Characterization Tool, which have been deployed on top of regional EHR Data Warehouse of the Lombardy Region containing about 1 billion records from 16 million patients and validated by several pharmacovigilance researchers with real-life cases. The results confirm significant improvements in signal detection and evaluation compared to traditional methods with the missing background information.

Cheminformatics-aided pharmacovigilance: application to Stevens-Johnson Syndrome.

OBJECTIVE: Quantitative Structure-Activity Relationship (QSAR) models can predict adverse drug reactions (ADRs), and thus provide early warnings of potential hazards. Timely identification of potential safety concerns could protect patients and aid early diagnosis of ADRs among the exposed. Our objective was to determine whether global spontaneous reporting patterns might allow chemical substructures associated with Stevens-Johnson Syndrome (SJS) to be identified and utilized for ADR prediction by QSAR models. MATERIALS AND METHODS: Using a reference set of 364 drugs having positive or negative reporting correlations with SJS in the VigiBase global repository of individual case safety reports (Uppsala Monitoring Center, Uppsala, Sweden), chemical descriptors were computed from drug molecular structures. Random Forest and Support Vector Machines methods were used to develop QSAR models, which were validated by external 5-fold cross validation. Models were employed for virtual screening of DrugBank to predict SJS actives and inactives, which were corroborated using knowledge bases like VigiBase, ChemoText, and MicroMedex (Truven Health Analytics Inc, Ann Arbor, Michigan). RESULTS: We developed QSAR models that could accurately predict if drugs were associated with SJS (area under the curve of 75%-81%). Our 10 most active and inactive predictions were substantiated by SJS reports (or lack thereof) in the literature. DISCUSSION: Interpretation of QSAR models in terms of significant chemical descriptors suggested novel SJS structural alerts. CONCLUSIONS: We have demonstrated that QSAR models can accurately identify SJS active and inactive drugs. Requiring chemical structures only, QSAR models provide effective computational means to flag potentially harmful drugs for subsequent targeted surveillance and pharmacoepidemiologic investigations.

vigiGrade: a tool to identify well-documented individual case reports and highlight systematic data quality issues.

BACKGROUND: Individual case safety reports of suspected harm from medicines are fundamental to post-marketing surveillance. Their value is directly proportional to the amount of clinically relevant information they include. To improve the quality of the data, communication between stakeholders is essential and can be facilitated by a simple score and visualisation of the results. OBJECTIVE: The objective of this study was to propose a measure of completeness and identify predictors of well-documented reports, globally. METHODS: The Uppsala Monitoring Centre has developed the vigiGrade completeness score to measure the amount of clinically relevant information in structured format, without reflecting whether the information establishes causality between the drug and adverse event. The vigiGrade completeness score (C) starts at 1 for reports with information on time-to-onset, age, sex, indication, outcome, report type, dose, country, primary reporter and comments. For each missing dimension, a penalty is detracted which varies with clinical relevance. We classified reports with C > 0.8 as well-documented and identified all such reports in the WHO global individual case safety report database, VigiBase, from 2007 to January 2012. We utilised odds ratios with statistical shrinkage to identify subgroups with unexpectedly high proportions of well-documented reports. RESULTS: Altogether, 430,000 (13 %) of the studied reports achieved C > 0.8 in VigiBase. For VigiBase as a whole, the median completeness was 0.41 with an interquartile range of 0.26-0.63. Two out of three well-documented reports come from Europe, and two out of three from physicians. Among the countries with more than 1,000 reports in total, the highest rate of well-documented reports is 65 % in Italy. Tunisia, Spain, Portugal, Croatia and Denmark each have rates above 50 %, and another 20 countries have rates above 30 %. On the whole, 24 % of the reports from physicians are well-documented compared with only 4 % for consumers/non-health professionals. Notably, Denmark and Norway have more than 50 % well-documented reports from consumers/non-health professionals and higher rates than for physicians. The rate of well-documented reports for the E2B format is 11 % compared with 22 % for the older INTDIS (International Drug Information System) format. However, for E2B reports entered via the WHO programme's e-reporting system VigiFlow, the rate is 29 %. CONCLUSION: Overall, only one report in eight provides the desired level of information, but much higher proportions are observed for individual countries. Physicians and e-reporting tools also generate greater proportions of well-documented reports overall. Reports from consumers/non-health professionals in specific regions have excellent quality, which illustrates their potential for the future. vigiGrade has already provided valuable information by highlighting data quality issues both in Italy and the USA.

Empirical performance of the calibrated self-controlled cohort analysis within temporal pattern discovery: lessons for developing a risk identification and analysis system.

BACKGROUND: Observational healthcare data offer the potential to identify adverse drug reactions that may be missed by spontaneous reporting. The self-controlled cohort analysis within the Temporal Pattern Discovery framework compares the observed-to-expected ratio of medical outcomes during post-exposure surveillance periods with those during a set of distinct pre-exposure control periods in the same patients. It utilizes an external control group to account for systematic differences between the different time periods, thus combining within- and between-patient confounder adjustment in a single measure. OBJECTIVES: To evaluate the performance of the calibrated self-controlled cohort analysis within Temporal Pattern Discovery as a tool for risk identification in observational healthcare data. RESEARCH DESIGN: Different implementations of the calibrated self-controlled cohort analysis were applied to 399 drug-outcome pairs (165 positive and 234 negative test cases across 4 health outcomes of interest) in 5 real observational databases (four with administrative claims and one with electronic health records). MEASURES: Performance was evaluated on real data through sensitivity/specificity, the area under receiver operator characteristics curve (AUC), and bias. RESULTS: The calibrated self-controlled cohort analysis achieved good predictive accuracy across the outcomes and databases under study. The optimal design based on this reference set uses a 360 days surveillance period and a single control period 180 days prior to new prescriptions. It achieved an average AUC of 0.75 and AUC >0.70 in all but one scenario. A design with three separate control periods performed better for the electronic health records database and for acute renal failure across all data sets. The estimates for negative test cases were generally unbiased, but a minor negative bias of up to 0.2 on the RR-scale was observed with the configurations using multiple control periods, for acute liver injury and upper gastrointestinal bleeding. CONCLUSIONS: The calibrated self-controlled cohort analysis within Temporal Pattern Discovery shows promise as a tool for risk identification; it performs well at discriminating positive from negative test cases. The optimal parameter configuration may vary with the data set and medical outcome of interest.