Monitoring Adverse Drug Events in Web Forums: Evaluation of a Pipeline and Use Case Study.

BACKGROUND: To mitigate safety concerns, regulatory agencies must make informed decisions regarding drug usage and adverse drug events (ADEs). The primary pharmacovigilance data stem from spontaneous reports by health care professionals. However, underreporting poses a notable challenge within the current system. Explorations into alternative sources, including electronic patient records and social media, have been undertaken. Nevertheless, social media's potential remains largely untapped in real-world scenarios. OBJECTIVE: The challenge faced by regulatory agencies in using social media is primarily attributed to the absence of suitable tools to support decision makers. An effective tool should enable access to information via a graphical user interface, presenting data in a user-friendly manner rather than in their raw form. This interface should offer various visualization options, empowering users to choose representations that best convey the data and facilitate informed decision-making. Thus, this study aims to assess the potential of integrating social media into pharmacovigilance and enhancing decision-making with this novel data source. To achieve this, our objective was to develop and assess a pipeline that processes data from the extraction of web forum posts to the generation of indicators and alerts within a visual and interactive environment. The goal was to create a user-friendly tool that enables regulatory authorities to make better-informed decisions effectively. METHODS: To enhance pharmacovigilance efforts, we have devised a pipeline comprising 4 distinct modules, each independently editable, aimed at efficiently analyzing health-related French web forums. These modules were (1) web forums' posts extraction, (2) web forums' posts annotation, (3) statistics and signal detection algorithm, and (4) a graphical user interface (GUI). We showcase the efficacy of the GUI through an illustrative case study involving the introduction of the new formula of Levothyrox in France. This event led to a surge in reports to the French regulatory authority. RESULTS: Between January 1, 2017, and February 28, 2021, a total of 2,081,296 posts were extracted from 23 French web forums. These posts contained 437,192 normalized drug-ADE couples, annotated with the Anatomical Therapeutic Chemical (ATC) Classification and Medical Dictionary for Regulatory Activities (MedDRA). The analysis of the Levothyrox new formula revealed a notable pattern. In August 2017, there was a sharp increase in posts related to this medication on social media platforms, which coincided with a substantial uptick in reports submitted by patients to the national regulatory authority during the same period. CONCLUSIONS: We demonstrated that conducting quantitative analysis using the GUI is straightforward and requires no coding. The results aligned with prior research and also offered potential insights into drug-related matters. Our hypothesis received partial confirmation because the final users were not involved in the evaluation process. Further studies, concentrating on ergonomics and the impact on professionals within regulatory agencies, are imperative for future research endeavors. We emphasized the versatility of our approach and the seamless interoperability between different modules over the performance of individual modules. Specifically, the annotation module was integrated early in the development process and could undergo substantial enhancement by leveraging contemporary techniques rooted in the Transformers architecture. Our pipeline holds potential applications in health surveillance by regulatory agencies or pharmaceutical companies, aiding in the identification of safety concerns. Moreover, it could be used by research teams for retrospective analysis of events.

Identification of COVID-19 Vaccines Concerns in Health-Related French Web Forums: A Topic Modelling Approach.

Since December 2019 and the first reported cases of COVID-19 in Wuhan, China, there have been 199,466,211 confirmed cases of COVID-19 in the World. The WHO defined vaccination hesitancy as one of the top ten threats to global health in 2019. Our objective was thus to identify topics and trends about COVID-19 vaccines from French web forums to understand the perception of the French population on these vaccines before the vaccination campaign started. We performed a topic model analysis on 485 web forums' posts. 10 topics were found. We reviewed 120 posts from 6 of these 10 topics. One topic was about vaccine hesitancy, refusal, and mistrust, and two topics were related to what the users think about the government, the political and economic choices made towards this epidemic.

Discrepancy Between Personal Experience and Negative Opinion with Human Papillomavirus Vaccine in Web Forums.

While vaccines are intended to protect people from infectious diseases, public confidence in vaccination has evolved as patients have reservation about vaccination, with a major concern about its safety. Social media may help regulatory authorities to better understand opposition to vaccination and make informed decisions for better promotion of vaccines' benefits towards the public. Our objective was to explore French web forums for potential pharmacovigilance signals associated with human papillomavirus infections (HPV) vaccines. Among 138 posts associated with a signal randomly chosen for manual review, 29% were actually adverse drug reactions to the vaccine described in clinical studies, and only 2 were personal experiences. Only 14% of the reviewed posts described positive opinion about the vaccine whereas 46% were neutral and 40% were negative. While few personal experiences of adverse reactions were actually reported by users, our case study showed a large proportion of negative opinions.

Qualitative and Quantitative Analysis of Web Forums for Adverse Events Detection: "Strontium Ranelate" Case Study.

Social media are proposed as a complementary data source for detection and characterisation of adverse drug reactions. While signal detection algorithms were implemented for generating signals in pharmacovigilance databases, the implementation of a graphical user interface supporting the selection and display of algorithms' results is not documented in the medical literature. Although collecting information on the chronology and the impact of adverse drug reactions is desirable to enable causality and quality assessment of potential signals detected in patients' posts, no tool has been proposed yet to consider such data. We describe here two approaches, and the corresponding tools we implemented for: (1) quantitative approach based on signal detection algorithms, and (2) qualitative approach based on expert review of patient's posts. Future work will focus on implementing other statistical methods, exploring the complementarity of both approaches on a larger scale, and prioritizing the posts to manually evaluate after applying appropriate signal detection methods.

[Informativity of French web forums for the evaluation of side effects of baclofen]. / Informativité des forums de discussion français pour l'évaluation des effets indésirables du baclofène.

OBJECTIVE: To evaluate the informativity, quality of French discussion forums for evaluation of baclofen safety. METHODS: We evaluated the quality of potential pharmacovigilance case reports associated to baclofen in 22 French discussion forums. We compared the informativity concerning the patient, treatment, seriousness and expectedness of adverse events described on these posts, with similar information coded in case reports from the French pharmacovigilance database (FPVD). RESULTS: A total of 782 potential case reports were identified among 2621 French language forums' posts. Cases in the FPVD were significantly more informative than web forums' posts for patient information (3%/6% vs. 88% for the age/class of age and 46% vs. 99% for the gender), treatment duration (9% vs. 24%) and outcome of the ADR (1% vs. 64%). But both indication and dose were more frequently retrieved in forums than in the FPVD (67% vs. 24% and 27% vs. 9%, respectively). Cases from web forums were significantly more frequently non-serious than the FPVD's ones (38% vs. 0.7%). Adverse events were significantly more often unexpected in forums than in the FPVD (43.8% vs. 11.6%). CONCLUSION: Indication and posology were more often documented in posts than in case reports which makes forums an interesting resource for monitoring use of baclofen. While posts contain more unexpected events, informativity is low which makes causality assessment difficult. Nevertheless, we consider forums as a secondary, but complementary source for pharmacovigilance about baclofen.

The Adverse Drug Reactions From Patient Reports in Social Media Project: Protocol for an Evaluation Against a Gold Standard.

BACKGROUND: Social media is a potential source of information on postmarketing drug safety surveillance that still remains unexploited nowadays. Information technology solutions aiming at extracting adverse reactions (ADRs) from posts on health forums require a rigorous evaluation methodology if their results are to be used to make decisions. First, a gold standard, consisting of manual annotations of the ADR by human experts from the corpus extracted from social media, must be implemented and its quality must be assessed. Second, as for clinical research protocols, the sample size must rely on statistical arguments. Finally, the extraction methods must target the relation between the drug and the disease (which might be either treated or caused by the drug) rather than simple co-occurrences in the posts. OBJECTIVE: We propose a standardized protocol for the evaluation of a software extracting ADRs from the messages on health forums. The study is conducted as part of the Adverse Drug Reactions from Patient Reports in Social Media project. METHODS: Messages from French health forums were extracted. Entity recognition was based on Racine Pharma lexicon for drugs and Medical Dictionary for Regulatory Activities terminology for potential adverse events (AEs). Natural language processing-based techniques automated the ADR information extraction (relation between the drug and AE entities). The corpus of evaluation was a random sample of the messages containing drugs and/or AE concepts corresponding to recent pharmacovigilance alerts. A total of 2 persons experienced in medical terminology manually annotated the corpus, thus creating the gold standard, according to an annotator guideline. We will evaluate our tool against the gold standard with recall, precision, and f-measure. Interannotator agreement, reflecting gold standard quality, will be evaluated with hierarchical kappa. Granularities in the terminologies will be further explored. RESULTS: Necessary and sufficient sample size was calculated to ensure statistical confidence in the assessed results. As we expected a global recall of 0.5, we needed at least 384 identified ADR concepts to obtain a 95% CI with a total width of 0.10 around 0.5. The automated ADR information extraction in the corpus for evaluation is already finished. The 2 annotators already completed the annotation process. The analysis of the performance of the ADR information extraction module as compared with gold standard is ongoing. CONCLUSIONS: This protocol is based on the standardized statistical methods from clinical research to create the corpus, thus ensuring the necessary statistical power of the assessed results. Such evaluation methodology is required to make the ADR information extraction software useful for postmarketing drug safety surveillance. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR1-10.2196/11448.

Mining Patients' Narratives in Social Media for Pharmacovigilance: Adverse Effects and Misuse of Methylphenidate.

Background: The Food and Drug Administration (FDA) in the United States and the European Medicines Agency (EMA) have recognized social media as a new data source to strengthen their activities regarding drug safety. Objective: Our objective in the ADR-PRISM project was to provide text mining and visualization tools to explore a corpus of posts extracted from social media. We evaluated this approach on a corpus of 21 million posts from five patient forums, and conducted a qualitative analysis of the data available on methylphenidate in this corpus. Methods: We applied text mining methods based on named entity recognition and relation extraction in the corpus, followed by signal detection using proportional reporting ratio (PRR). We also used topic modeling based on the Correlated Topic Model to obtain the list of the matics in the corpus and classify the messages based on their topics. Results: We automatically identified 3443 posts about methylphenidate published between 2007 and 2016, among which 61 adverse drug reactions (ADR) were automatically detected. Two pharmacovigilance experts evaluated manually the quality of automatic identification, and a f-measure of 0.57 was reached. Patient's reports were mainly neuro-psychiatric effects. Applying PRR, 67% of the ADRs were signals, including most of the neuro-psychiatric symptoms but also palpitations. Topic modeling showed that the most represented topics were related to Childhood and Treatment initiation, but also Side effects. Cases of misuse were also identified in this corpus, including recreational use and abuse. Conclusion: Named entity recognition combined with signal detection and topic modeling have demonstrated their complementarity in mining social media data. An in-depth analysis focused on methylphenidate showed that this approach was able to detect potential signals and to provide better understanding of patients' behaviors regarding drugs, including misuse.

Descriptions of Adverse Drug Reactions Are Less Informative in Forums Than in the French Pharmacovigilance Database but Provide More Unexpected Reactions.

Background: Social media have drawn attention for their potential use in Pharmacovigilance. Recent work showed that it is possible to extract information concerning adverse drug reactions (ADRs) from posts in social media. The main objective of the Vigi4MED project was to evaluate the relevance and quality of the information shared by patients on web forums about drug safety and its potential utility for pharmacovigilance. Methods: After selecting websites of interest, we manually evaluated the relevance of the content of posts for pharmacovigilance related to six drugs (agomelatine, baclofen, duloxetine, exenatide, strontium ranelate, and tetrazepam). We compared forums to the French Pharmacovigilance Database (FPVD) to (1) evaluate whether they contained relevant information to characterize a pharmacovigilance case report (patient's age and sex; treatment indication, dose and duration; time-to-onset (TTO) and outcome of the ADR, and drug dechallenge and rechallenge) and (2) perform impact analysis (nature, seriousness, unexpectedness, and outcome of the ADR). Results: The cases in the FPVD were significantly more informative than posts in forums for patient description (age, sex), treatment description (dose, duration, TTO), and outcome of the ADR, but the indication for the treatment was more often found in forums. Cases were more often serious in the FPVD than in forums (46% vs. 4%), but forums more often contained an unexpected ADR than the FPVD (24% vs. 17%). Moreover, 197 unexpected ADRs identified in forums were absent from the FPVD and the distribution of the MedDRA System Organ Classes (SOCs) was different between the two data sources. Discussion: This study is the first to evaluate if patients' posts may qualify as potential and informative case reports that should be stored in a pharmacovigilance database in the same way as case reports submitted by health professionals. The posts were less informative (except for the indication) and focused on less serious ADRs than the FPVD cases, but more unexpected ADRs were presented in forums than in the FPVD and their SOCs were different. Thus, web forums should be considered as a secondary, but complementary source for pharmacovigilance.

Signal Detection for Baclofen in Web Forums: A Preliminary Study.

Web forums are proposed as a new complementary source of knowledge to spontaneous reports by patients and healthcare professionals due to underreporting of adverse drug reactions (ADRs). Some authors suggest that signal detection could be a convenient method for gathering mentions of ADRs in patients' posts. Signal detection methods were proposed to mine pharmacovigilance databases, but little is known about their applicability to web forums. We describe a method implementing several traditional decision rules on signal detection with baclofen applied to a set of more than 6 million posts. We then cross-validated four unexpected signals applying a logistic regression method. Most adverse effects (AEs) described in the summary of product characteristics of baclofen were detected by signal detection methods. Some unexpected AEs were too. Therefore, web forums are confirmed as a complementary resource for improving current knowledge in pharmacovigilance by detecting unexpected adverse drug reactions.

Mining Adverse Drug Reactions in Social Media with Named Entity Recognition and Semantic Methods.

Suspected adverse drug reactions (ADR) reported by patients through social media can be a complementary source to current pharmacovigilance systems. However, the performance of text mining tools applied to social media text data to discover ADRs needs to be evaluated. In this paper, we introduce the approach developed to mine ADR from French social media. A protocol of evaluation is highlighted, which includes a detailed sample size determination and evaluation corpus constitution. Our text mining approach provided very encouraging preliminary results with F-measures of 0.94 and 0.81 for recognition of drugs and symptoms respectively, and with F-measure of 0.70 for ADR detection. Therefore, this approach is promising for downstream pharmacovigilance analysis.