Adverse Events in Twitter-Development of a Benchmark Reference Dataset: Results from IMI WEB-RADR.

INTRODUCTION AND OBJECTIVE: Social media has been suggested as a source for safety information, supplementing existing safety surveillance data sources. This article summarises the activities undertaken, and the associated challenges, to create a benchmark reference dataset that can be used to evaluate the performance of automated methods and systems for adverse event recognition. METHODS: A retrospective analysis of public English-language Twitter posts (Tweets) was performed. We sampled 57,473 Tweets out of 5,645,336 Tweets created between 1 March, 2012 and 1 March, 2015 that mentioned at least one of six medicinal products of interest (insulin glargine, levetiracetam, methylphenidate, sorafenib, terbinafine, zolpidem). Products, adverse events, indications, product-event combinations, and product-indication combinations were extracted and coded by two independent teams of safety reviewers. RESULTS: The benchmark reference dataset consisted of 1056 positive controls ("adverse event Tweets") and 56,417 negative controls ("non-adverse event Tweets"). The 1056 adverse event Tweets contained 1396 product-event combinations referring to personal adverse event experiences, comprising 292 different MedDRA® Preferred Terms. The 1171 product-event combinations (83.9%) were confined to four MedDRA® System Organ Classes. The 195 Tweets (18.5%) contained indication information, comprising 25 different Preferred Terms. CONCLUSIONS: A manually curated benchmark reference dataset based on Twitter data has been created and is made available to the research community to evaluate the performance of automated methods and systems for adverse event recognition in unstructured free-text information.

Investigating Overlap in Signals from EVDAS, FAERS, and VigiBase®.

INTRODUCTION: The Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) and VigiBase® are two established databases for safety monitoring of medicinal products, recently complemented with the EudraVigilance Data Analysis System (EVDAS). OBJECTIVE: Signals of disproportionate reporting (SDRs) can characterize the reporting profile of a drug, accounting for the distribution of all drugs and all events in the database. This study aims to quantify the redundancy among the three databases when characterized by two disproportionality-based analyses (DPA). METHODS: SDRs for 100 selected products were identified with two sets of thresholds (standard EudraVigilance SDR criteria for all vs Bayesian approach for FAERS and VigiBase®). Per product and database, the presence or absence of SDRs was determined and compared. Adverse events were considered at three levels: MedDRA® Preferred Term (PT), High Level Term (HLT), and HLT combined with Standardized MedDRA® Query (SMQ). Redundancy was measured in terms of recall (SDRs in EVDAS divided by SDRs from any database) and overlap (SDRs in EVDAS and at least one other database, divided by SDRs in EVDAS). Covariates with potential impact on results were explored with linear regression models. RESULTS: The median overlap between EVDAS and FAERS or VigiBase® was 85.0% at the PT level, 94.5% at the HLT level, and 97.7% at the HLT or SMQ level. The corresponding median recall of signals in EVDAS as a percentage of all signals generated in all three databases was 59.4%, 74.1%, and 87.9% at the PT, HLT, and HLT or SMQ levels, respectively. The overlap difference is partially explained by the relative number of EU cases in EudraVigilance and the ratio of EVDAS cases and FAERS cases, presumably due to differences in marketing authorizations, or market penetration in different regions. Products with few cases in EVDAS (< 1500) also display limited recall of signals relative to FAERs/VigiBase®. Time-on-market does not predict signal redundancy between the three databases. The choice of the DPA has an expected but somewhat small effect on redundancy. CONCLUSIONS: Organizations typically consider regulatory expectations, operating performance (e.g., positive predictive value), and procedural complexity when selecting databases for signal management. As SDRs can be seen as a proxy of general reporting characteristics identifiable in a systematic screening process, our results indicate that, for most products, these characteristics are largely similar in each of the databases.

Recommendations on the Use of Mobile Applications for the Collection and Communication of Pharmaceutical Product Safety Information: Lessons from IMI WEB-RADR.

Over a period of 3 years, the European Union's Innovative Medicines Initiative WEB-RADR (Recognising Adverse Drug Reactions; https://web-radr.eu/ ) project explored the value of two digital tools for pharmacovigilance (PV): mobile applications (apps) for reporting the adverse effects of drugs and social media data for its contribution to safety signalling. The ultimate intent of WEB-RADR was to provide policy, technical and ethical recommendations on how to develop and implement such digital tools to enhance patient safety. Recommendations relating to the use of mobile apps for PV are summarised in this paper. There is a presumption amongst at least some patients and healthcare professionals that information ought to be accessed and reported from any setting, including mobile apps. WEB-RADR has focused on the use of such technology for reporting suspected adverse drug reactions and for broadcasting safety information to its users, i.e. two-way risk communication. Three apps were developed and publicly launched within Europe as part of the WEB-RADR project and subsequently assessed by a range of stakeholders to determine their value as effective tools for improving patient safety; a fourth generic app was later piloted in two African countries. The recommendations from the development and evaluation of the European apps are presented here with supporting considerations, rationales and caveats as well as suggested areas for further research.

Recommendations for the Use of Social Media in Pharmacovigilance: Lessons from IMI WEB-RADR.

Over a period of 3 years, the European Union's Innovative Medicines Initiative WEB-RADR project has explored the value of social media (i.e., information exchanged through the internet, typically via online social networks) for identifying adverse events as well as for safety signal detection. Many patients and clinicians have taken to social media to discuss their positive and negative experiences of medications, creating a source of publicly available information that has the potential to provide insights into medicinal product safety concerns. The WEB-RADR project has developed a collaborative English language workspace for visualising and analysing social media data for a number of medicinal products. Further, novel text and data mining methods for social media analysis have been developed and evaluated. From this original research, several recommendations are presented with supporting rationale and consideration of the limitations. Recommendations for further research that extend beyond the scope of the current project are also presented.

Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project.

INTRODUCTION AND OBJECTIVE: Social media has been proposed as a possibly useful data source for pharmacovigilance signal detection. This study primarily aimed to evaluate the performance of established statistical signal detection algorithms in Twitter/Facebook for a broad range of drugs and adverse events. METHODS: Performance was assessed using a reference set by Harpaz et al., consisting of 62 US Food and Drug Administration labelling changes, and an internal WEB-RADR reference set consisting of 200 validated safety signals. In total, 75 drugs were studied. Twitter/Facebook posts were retrieved for the period March 2012 to March 2015, and drugs/events were extracted from the posts. We retrieved 4.3 million and 2.0 million posts for the WEB-RADR and Harpaz drugs, respectively. Individual case reports were extracted from VigiBase for the same period. Disproportionality algorithms based on the Information Component or the Proportional Reporting Ratio and crude post/report counting were applied in Twitter/Facebook and VigiBase. Receiver operating characteristic curves were generated, and the relative timing of alerting was analysed. RESULTS: Across all algorithms, the area under the receiver operating characteristic curve for Twitter/Facebook varied between 0.47 and 0.53 for the WEB-RADR reference set and between 0.48 and 0.53 for the Harpaz reference set. For VigiBase, the ranges were 0.64-0.69 and 0.55-0.67, respectively. In Twitter/Facebook, at best, 31 (16%) and four (6%) positive controls were detected prior to their index dates in the WEB-RADR and Harpaz references, respectively. In VigiBase, the corresponding numbers were 66 (33%) and 17 (27%). CONCLUSIONS: Our results clearly suggest that broad-ranging statistical signal detection in Twitter and Facebook, using currently available methods for adverse event recognition, performs poorly and cannot be recommended at the expense of other pharmacovigilance activities.

An Empirical Approach to Explore the Relationship Between Measures of Disproportionate Reporting and Relative Risks from Analytical Studies.

INTRODUCTION: Although it seems reasonable to suppose that a drug that increases the risk of an adverse event might tend to show increased disproportionality statistics in spontaneous reporting databases, that relationship is not clear. Therefore, an empirical approach was taken to investigate the relationship between proportional reporting ratios (PRRs) and relative risk (RR) estimates from formal studies in a set of known adverse drug reactions (ADRs). METHODS: Drug-event pairs that were the subject of pharmacovigilance-driven European regulatory actions from 2007 to 2010 were selected. Only pairs having RR derived from formal studies and where it was considered that there was well-established evidence supporting the actions were included. A best estimate of the RR for each ADR was chosen based on pre-specified rules. PRRs were then calculated in Eudravigilance using only those cases reported before the date of first recognition of the ADR in the medical community. An additional analysis was carried out in FEDRA, the Spanish spontaneous reports database. A descriptive analysis and an orthogonal regression model were performed. RESULTS: From an initial dataset of 78 drug-event pairs, 15 were selected. The regression model (ln RR = 0.203 + 0.463 × ln PRR) showed a significant (p < 0.001) correlation between RR and PRR in Eudravigilance. None of the ADR-related variables analysed modified the relationship. Exploratory results in FEDRA went in the same direction. CONCLUSIONS: Disproportionality measures should not replace formal studies but could provide an initial indication of the likely clinical importance of an ADR, should the signal be confirmed subsequently. Whether the same conclusions can be applied to other datasets should be further studied.

Performance of Stratified and Subgrouped Disproportionality Analyses in Spontaneous Databases.

INTRODUCTION: Disproportionality analyses are used in many organisations to identify adverse drug reactions (ADRs) from spontaneous report data. Reporting patterns vary over time, with patient demographics, and between different geographical regions, and therefore subgroup analyses or adjustment by stratification may be beneficial. OBJECTIVE: The objective of this study was to evaluate the performance of subgroup and stratified disproportionality analyses for a number of key covariates within spontaneous report databases of differing sizes and characteristics. METHODS: Using a reference set of established ADRs, signal detection performance (sensitivity and precision) was compared for stratified, subgroup and crude (unadjusted) analyses within five spontaneous report databases (two company, one national and two international databases). Analyses were repeated for a range of covariates: age, sex, country/region of origin, calendar time period, event seriousness, vaccine/non-vaccine, reporter qualification and report source. RESULTS: Subgroup analyses consistently performed better than stratified analyses in all databases. Subgroup analyses also showed benefits in both sensitivity and precision over crude analyses for the larger international databases, whilst for the smaller databases a gain in precision tended to result in some loss of sensitivity. Additionally, stratified analyses did not increase sensitivity or precision beyond that associated with analytical artefacts of the analysis. The most promising subgroup covariates were age and region/country of origin, although this varied between databases. CONCLUSIONS: Subgroup analyses perform better than stratified analyses and should be considered over the latter in routine first-pass signal detection. Subgroup analyses are also clearly beneficial over crude analyses for larger databases, but further validation is required for smaller databases.

Good Signal Detection Practices: Evidence from IMI PROTECT.

Over a period of 5 years, the Innovative Medicines Initiative PROTECT (Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium) project has addressed key research questions relevant to the science of safety signal detection. The results of studies conducted into quantitative signal detection in spontaneous reporting, clinical trial and electronic health records databases are summarised and 39 recommendations have been formulated, many based on comparative analyses across a range of databases (e.g. regulatory, pharmaceutical company). The recommendations point to pragmatic steps that those working in the pharmacovigilance community can take to improve signal detection practices, whether in a national or international agency or in a pharmaceutical company setting. PROTECT has also pointed to areas of potentially fruitful future research and some areas where further effort is likely to yield less.

Comparison of statistical signal detection methods within and across spontaneous reporting databases.

BACKGROUND: Most pharmacovigilance departments maintain a system to identify adverse drug reactions (ADRs) through analysis of spontaneous reports. The signal detection algorithms (SDAs) and the nature of the reporting databases vary between operators and it is unclear whether any algorithm can be expected to provide good performance in a wide range of environments. OBJECTIVE: The objective of this study was to compare the performance of commonly used algorithms across spontaneous reporting databases operated by pharmaceutical companies and national and international pharmacovigilance organisations. METHODS: 220 products were chosen and a reference set of ADRs was compiled. Within four company, one national and two international databases, 15 SDAs based on five disproportionality methods were tested. Signals of disproportionate reporting (SDRs) were calculated at monthly intervals and classified by comparison with the reference set. These results were summarised as sensitivity and precision for each algorithm in each database. RESULTS: Different algorithms performed differently between databases but no method dominated all others. Performance was strongly dependent on the thresholds used to define a statistical signal. However, the different disproportionality statistics did not influence the achievable performance. The relative performance of two algorithms was similar in different databases. Over the lifetime of a product there is a reduction in precision for any method. CONCLUSIONS: In designing signal detection systems, careful consideration should be given to the criteria that are used to define an SDR. The choice of disproportionality statistic does not appreciably affect the achievable range of signal detection performance and so this can primarily be based on ease of implementation, interpretation and minimisation of computing resources. The changes in sensitivity and precision obtainable by replacing one algorithm with another are predictable. However, the absolute performance of a method is specific to the database and is best assessed directly on that database. New methods may be required to gain appreciable improvements.