Monitoring Adverse Drug Events in Web Forums: Evaluation of a Pipeline and Use Case Study.

BACKGROUND: To mitigate safety concerns, regulatory agencies must make informed decisions regarding drug usage and adverse drug events (ADEs). The primary pharmacovigilance data stem from spontaneous reports by health care professionals. However, underreporting poses a notable challenge within the current system. Explorations into alternative sources, including electronic patient records and social media, have been undertaken. Nevertheless, social media's potential remains largely untapped in real-world scenarios. OBJECTIVE: The challenge faced by regulatory agencies in using social media is primarily attributed to the absence of suitable tools to support decision makers. An effective tool should enable access to information via a graphical user interface, presenting data in a user-friendly manner rather than in their raw form. This interface should offer various visualization options, empowering users to choose representations that best convey the data and facilitate informed decision-making. Thus, this study aims to assess the potential of integrating social media into pharmacovigilance and enhancing decision-making with this novel data source. To achieve this, our objective was to develop and assess a pipeline that processes data from the extraction of web forum posts to the generation of indicators and alerts within a visual and interactive environment. The goal was to create a user-friendly tool that enables regulatory authorities to make better-informed decisions effectively. METHODS: To enhance pharmacovigilance efforts, we have devised a pipeline comprising 4 distinct modules, each independently editable, aimed at efficiently analyzing health-related French web forums. These modules were (1) web forums' posts extraction, (2) web forums' posts annotation, (3) statistics and signal detection algorithm, and (4) a graphical user interface (GUI). We showcase the efficacy of the GUI through an illustrative case study involving the introduction of the new formula of Levothyrox in France. This event led to a surge in reports to the French regulatory authority. RESULTS: Between January 1, 2017, and February 28, 2021, a total of 2,081,296 posts were extracted from 23 French web forums. These posts contained 437,192 normalized drug-ADE couples, annotated with the Anatomical Therapeutic Chemical (ATC) Classification and Medical Dictionary for Regulatory Activities (MedDRA). The analysis of the Levothyrox new formula revealed a notable pattern. In August 2017, there was a sharp increase in posts related to this medication on social media platforms, which coincided with a substantial uptick in reports submitted by patients to the national regulatory authority during the same period. CONCLUSIONS: We demonstrated that conducting quantitative analysis using the GUI is straightforward and requires no coding. The results aligned with prior research and also offered potential insights into drug-related matters. Our hypothesis received partial confirmation because the final users were not involved in the evaluation process. Further studies, concentrating on ergonomics and the impact on professionals within regulatory agencies, are imperative for future research endeavors. We emphasized the versatility of our approach and the seamless interoperability between different modules over the performance of individual modules. Specifically, the annotation module was integrated early in the development process and could undergo substantial enhancement by leveraging contemporary techniques rooted in the Transformers architecture. Our pipeline holds potential applications in health surveillance by regulatory agencies or pharmaceutical companies, aiding in the identification of safety concerns. Moreover, it could be used by research teams for retrospective analysis of events.

Users' Reactions to Announced Vaccines Against COVID-19 Before Marketing in France: Analysis of Twitter Posts.

BACKGROUND: Within a few months, the COVID-19 pandemic had spread to many countries and had been a real challenge for health systems all around the world. This unprecedented crisis has led to a surge of online discussions about potential cures for the disease. Among them, vaccines have been at the heart of the debates and have faced lack of confidence before marketing in France. OBJECTIVE: This study aims to identify and investigate the opinions of French Twitter users on the announced vaccines against COVID-19 through sentiment analysis. METHODS: This study was conducted in 2 phases. First, we filtered a collection of tweets related to COVID-19 available on Twitter from February 2020 to August 2020 with a set of keywords associated with vaccine mistrust using word embeddings. Second, we performed sentiment analysis using deep learning to identify the characteristics of vaccine mistrust. The model was trained on a hand-labeled subset of 4548 tweets. RESULTS: A set of 69 relevant keywords were identified as the semantic concept of the word "vaccin" (vaccine in French) and focused mainly on conspiracies, pharmaceutical companies, and alternative treatments. Those keywords enabled us to extract nearly 350,000 tweets in French. The sentiment analysis model achieved 0.75 accuracy. The model then predicted 16% of positive tweets, 41% of negative tweets, and 43% of neutral tweets. This allowed us to explore the semantic concepts of positive and negative tweets and to plot the trends of each sentiment. The main negative rhetoric identified from users' tweets was that vaccines are perceived as having a political purpose and that COVID-19 is a commercial argument for the pharmaceutical companies. CONCLUSIONS: Twitter might be a useful tool to investigate the arguments for vaccine mistrust because it unveils political criticism contrasting with the usual concerns on adverse drug reactions. As the opposition rhetoric is more consistent and more widely spread than the positive rhetoric, we believe that this research provides effective tools to help health authorities better characterize the risk of vaccine mistrust.

Investigating ADR mechanisms with Explainable AI: a feasibility study with knowledge graph mining.

BACKGROUND: Adverse drug reactions (ADRs) are statistically characterized within randomized clinical trials and postmarketing pharmacovigilance, but their molecular mechanism remains unknown in most cases. This is true even for hepatic or skin toxicities, which are classically monitored during drug design. Aside from clinical trials, many elements of knowledge about drug ingredients are available in open-access knowledge graphs, such as their properties, interactions, or involvements in pathways. In addition, drug classifications that label drugs as either causative or not for several ADRs, have been established. METHODS: We propose in this paper to mine knowledge graphs for identifying biomolecular features that may enable automatically reproducing expert classifications that distinguish drugs causative or not for a given type of ADR. In an Explainable AI perspective, we explore simple classification techniques such as Decision Trees and Classification Rules because they provide human-readable models, which explain the classification itself, but may also provide elements of explanation for molecular mechanisms behind ADRs. In summary, (1) we mine a knowledge graph for features; (2) we train classifiers at distinguishing, on the basis of extracted features, drugs associated or not with two commonly monitored ADRs: drug-induced liver injuries (DILI) and severe cutaneous adverse reactions (SCAR); (3) we isolate features that are both efficient in reproducing expert classifications and interpretable by experts (i.e., Gene Ontology terms, drug targets, or pathway names); and (4) we manually evaluate in a mini-study how they may be explanatory. RESULTS: Extracted features reproduce with a good fidelity classifications of drugs causative or not for DILI and SCAR (Accuracy = 0.74 and 0.81, respectively). Experts fully agreed that 73% and 38% of the most discriminative features are possibly explanatory for DILI and SCAR, respectively; and partially agreed (2/3) for 90% and 77% of them. CONCLUSION: Knowledge graphs provide sufficiently diverse features to enable simple and explainable models to distinguish between drugs that are causative or not for ADRs. In addition to explaining classifications, most discriminative features appear to be good candidates for investigating ADR mechanisms further.

How to interact with medical terminologies? Formative usability evaluations comparing three approaches for supporting the use of MedDRA by pharmacovigilance specialists.

BACKGROUND: Medical terminologies are commonly used in medicine. For instance, to answer a pharmacovigilance question, pharmacovigilance specialists (PVS) search in a pharmacovigilance database for reports in relation to a given drug. To do that, they first need to identify all MedDRA terms that might have been used to code an adverse reaction in the database, but terms may be numerous and difficult to select as they may belong to different parts of the hierarchy. In previous studies, three tools have been developed to help PVS identify and group all relevant MedDRA terms using three different approaches: forms, structured query-builder, and icons. Yet, a poor usability of the tools may increase PVS' workload and reduce their performance. This study aims to evaluate, compare and improve the three tools during two rounds of formative usability evaluation. METHODS: First, a cognitive walkthrough was performed. Based on the design recommendations obtained from this evaluation, designers made modifications to their tools to improve usability. Once this re-engineering phase completed, six PVS took part in a usability test: difficulties, errors and verbalizations during their interaction with the three tools were collected. Their satisfaction was measured through the System Usability Scale. The design recommendations issued from the tests were used to adapt the tools. RESULTS: All tools had usability problems related to the lack of guidance in the graphical user interface (e.g., unintuitive labels). In two tools, the use of the SNOMED CT to find MedDRA terms hampered their use because French PVS were not used to it. For the most obvious and common terms, the icons-based interface would appear to be more useful. For the less frequently used MedDRA terms or those distributed in different parts of the hierarchy, the structured query-builder would be preferable thanks to its great power and flexibility. The form-based tool seems to be a compromise. CONCLUSION: These evaluations made it possible to identify the strengths of each tool but also their weaknesses to address them before further evaluation. Next step is to assess the acceptability of tools and the expressiveness of their results to help identify and group MedDRA terms.

OntoADR a semantic resource describing adverse drug reactions to support searching, coding, and information retrieval.

INTRODUCTION: Efficient searching and coding in databases that use terminological resources requires that they support efficient data retrieval. The Medical Dictionary for Regulatory Activities (MedDRA) is a reference terminology for several countries and organizations to code adverse drug reactions (ADRs) for pharmacovigilance. Ontologies that are available in the medical domain provide several advantages such as reasoning to improve data retrieval. The field of pharmacovigilance does not yet benefit from a fully operational ontology to formally represent the MedDRA terms. Our objective was to build a semantic resource based on formal description logic to improve MedDRA term retrieval and aid the generation of on-demand custom groupings by appropriately and efficiently selecting terms: OntoADR. METHODS: The method consists of the following steps: (1) mapping between MedDRA terms and SNOMED-CT, (2) generation of semantic definitions using semi-automatic methods, (3) storage of the resource and (4) manual curation by pharmacovigilance experts. RESULTS: We built a semantic resource for ADRs enabling a new type of semantics-based term search. OntoADR adds new search capabilities relative to previous approaches, overcoming the usual limitations of computation using lightweight description logic, such as the intractability of unions or negation queries, bringing it closer to user needs. Our automated approach for defining MedDRA terms enabled the association of at least one defining relationship with 67% of preferred terms. The curation work performed on our sample showed an error level of 14% for this automated approach. We tested OntoADR in practice, which allowed us to build custom groupings for several medical topics of interest. DISCUSSION: The methods we describe in this article could be adapted and extended to other terminologies which do not benefit from a formal semantic representation, thus enabling better data retrieval performance. Our custom groupings of MedDRA terms were used while performing signal detection, which suggests that the graphical user interface we are currently implementing to process OntoADR could be usefully integrated into specialized pharmacovigilance software that rely on MedDRA.

[Automated grouping of terms associated to cardiac valve fibrosis in MedDRA]. / Groupement automatisé de termes liés aux valvulopathies médicamenteuses dans MedDRA.

AIM: To propose an alternative approach for building custom groupings of terms that complements the usual approach based on both hierarchical method (selection of reference groupings in medical dictionary for regulatory activities [MedDRA]) and/or textual method (string search), for case reports extraction from a pharmacovigilance database in response to a safety problem. Here we take cardiac valve fibrosis as an example. METHODS: The list of terms obtained by an automated approach, based on querying ontology of adverse drug reactions (OntoADR), a knowledge base defining MedDRA terms through relationships with systematized nomenclature of medicine-clinical terms (SNOMED CT) concepts, was compared with the reference list consisting of 53 preferred terms obtained by hierarchical and textual method. Two queries were performed on OntoADR by using a dedicated software: OntoADR query tools. Both queries excluded congenital diseases, and included a procedure or an auscultation method performed on cardiac valve structures. Query 1 also considered MedDRA terms related to fibrosis, narrowing or calcification of heart valves, and query 2 MedDRA terms described according to one of these four SNOMED CT terms: "Insufficiency", "Valvular sclerosis", "Heart valve calcification" or "Heart valve stenosis". RESULTS: The reference grouping consisted of 53 MedDRA preferred terms. Our automated method achieved recall of 79% and precision of 100% for query 1 privileging morphological abnormalities, and recall of 100% and precision of 96% for query 2 privileging functional abnormalities. CONCLUSION: An alternative approach to MedDRA reference groupings for building custom groupings is feasible for cardiac valve fibrosis. OntoADR is still in development. Its application to other adverse reactions would require significant work for a knowledge engineer to define every MedDRA term, but such definitions could then be queried as many times as necessary by pharmacovigilance professionals.

Adverse Drug Reaction Identification and Extraction in Social Media: A Scoping Review.

BACKGROUND: The underreporting of adverse drug reactions (ADRs) through traditional reporting channels is a limitation in the efficiency of the current pharmacovigilance system. Patients' experiences with drugs that they report on social media represent a new source of data that may have some value in postmarketing safety surveillance. OBJECTIVE: A scoping review was undertaken to explore the breadth of evidence about the use of social media as a new source of knowledge for pharmacovigilance. METHODS: Daubt et al's recommendations for scoping reviews were followed. The research questions were as follows: How can social media be used as a data source for postmarketing drug surveillance? What are the available methods for extracting data? What are the different ways to use these data? We queried PubMed, Embase, and Google Scholar to extract relevant articles that were published before June 2014 and with no lower date limit. Two pairs of reviewers independently screened the selected studies and proposed two themes of review: manual ADR identification (theme 1) and automated ADR extraction from social media (theme 2). Descriptive characteristics were collected from the publications to create a database for themes 1 and 2. RESULTS: Of the 1032 citations from PubMed and Embase, 11 were relevant to the research question. An additional 13 citations were added after further research on the Internet and in reference lists. Themes 1 and 2 explored 11 and 13 articles, respectively. Ways of approaching the use of social media as a pharmacovigilance data source were identified. CONCLUSIONS: This scoping review noted multiple methods for identifying target data, extracting them, and evaluating the quality of medical information from social media. It also showed some remaining gaps in the field. Studies related to the identification theme usually failed to accurately assess the completeness, quality, and reliability of the data that were analyzed from social media. Regarding extraction, no study proposed a generic approach to easily adding a new site or data source. Additional studies are required to precisely determine the role of social media in the pharmacovigilance system.

Formalizing MedDRA to support semantic reasoning on adverse drug reaction terms.

Although MedDRA has obvious advantages over previous terminologies for coding adverse drug reactions and discovering potential signals using data mining techniques, its terminological organization constrains users to search terms according to predefined categories. Adding formal definitions to MedDRA would allow retrieval of terms according to a case definition that may correspond to novel categories that are not currently available in the terminology. To achieve semantic reasoning with MedDRA, we have associated formal definitions to MedDRA terms in an OWL file named OntoADR that is the result of our first step for providing an "ontologized" version of MedDRA. MedDRA five-levels original hierarchy was converted into a subsumption tree and formal definitions of MedDRA terms were designed using several methods: mappings to SNOMED-CT, semi-automatic definition algorithms or a fully manual way. This article presents the main steps of OntoADR conception process, its structure and content, and discusses problems and limits raised by this attempt to "ontologize" MedDRA.

[Drug-related anaphylactic shocks: under-reporting and PMSI]. / Chocs anaphylactiques d'origine médicamenteuse: sous-notification et PMSI.

AIM: To evaluate the value of research in the case-mix database to identify cases of drug-related anaphylactic or anaphylactoid shock. METHODS: Hospital stays of patients discharged from the University Hospital of Saint-Étienne between July 1st 2009 and June 30th 2012. Five codes from the international classification of diseases were selected: T88.6, T88.2, J39.3, T80.5 and T78.2. RESULTS: Among 89 cases identified by the programme for medicalization of information system (programme de médicalisation des systèmes d'information, PMSI), 40 were selected (45%). Of these, 16 cases were spontaneously reported by physicians. The unspecific code "anaphylactic shock unspecified (T78.2)" was coded for 57.5% of cases. CONCLUSION: The study confirms the interest of the PMSI as a tool for health monitoring, in addition to spontaneous reporting. Nevertheless, coding with insufficient precision about the causal role of the drug, requires a return to the medical record and so an important time consuming process.

[Not Available]. / Chocs anaphylactiques d'origine médicamenteuse : sous-notification et PMSI.

AIM: To evaluate the value of research in the case-mix database to identify cases of drug-related anaphylactic or anaphylactoid shock. METHODS: Hospital stays of patients discharged from the University Hospital of Saint-Étienne between July 1st 2009 and June 30th 2012. Five codes from the international classification of diseases were selected: T88.6, T88.2, J39.3, T80.5 and T78.2. RESULTS: Among 89 cases identified by the programme for medicalization of information system (programme de médicalisation des systèmes d'information, PMSI), 40 were selected (45%). Of these, 16 cases were spontaneously reported by physicians. The unspecific code "anaphylactic shock unspecified (T78.2)" was coded for 57.5% of cases. CONCLUSION: The study confirms the interest of the PMSI as a tool for health monitoring, in addition to spontaneous reporting. Nevertheless, coding with insufficient precision about the causal role of the drug, requires a return to the medical record and so an important time consuming process.

Rapid Review on Publicly Available Datasets for Health Misinformation Detection.

The proliferation of health misinformation in recent years has prompted the development of various methods for detecting and combatting this issue. This review aims to provide an overview of the implementation strategies and characteristics of publicly available datasets that can be used for health misinformation detection. Since 2020, a large number of such datasets have emerged, half of which are focused on COVID-19. Most of the datasets are based on fact-checkable websites, while only a few are annotated by experts. Furthermore, some datasets provide additional information such as social engagement and explanations, which can be utilized to study the spread of misinformation. Overall, these datasets offer a valuable resource for researchers working to combat the spread and consequences of health misinformation.

Automatic generation of MedDRA terms groupings using an ontology.

In the context of PROTECT European project, we have developed an ontology of adverse drug reactions (OntoADR) based on the original MedDRA hierarchy and a query-based method to achieve automatic MedDRA terms groupings for improving pharmacovigilance signal detection. Those groupings were evaluated against standard handmade MedDRA groupings corresponding to first priority pharmacovigilance safety topics. Our results demonstrate that this automatic method allows catching most of the terms present in the reference groupings, and suggest that it could offer an important saving of time for the achievement of pharmacovigilance groupings. This paper describes the theoretical context of this work, the evaluation methodology, and presents the principal results.

Method for mapping the French CCAM terminology to the UMLS metathesaurus.

The French coding system of surgical procedures, the Classification Commune des Actes Médicaux (CCAM), is used in France for DRG databases and fee for services payment. Mapping between CCAM and other clinical procedures terminologies by the means of UMLS metathesaurus is essential in order to increase semantic interoperability between different healthcare terminologies and between different case mix systems. In a previous work the CISMeF team used an automatic approach to map CCAM descriptors to the French part of the UMLS metathesaurus. In another way for the French funded research project InterSTIS, we performed a mapping using MetaMap based on the top level semantic structure descriptors of anatomy and action of CCAM translated from French to English. This paper aims to present this new approach and to compare the results with the previous one. The combination of both approaches significantly improved the coverage of the mapping to 68 % for both descriptors and 95 % for at least one descriptor.

Machine Learning in Medicine: To Explain, or Not to Explain, That Is the Question.

In 2022, the Medical Informatics Europe conference created a special topic called "Challenges of trustable AI and added-value on health" which was centered around the theme of eXplainable Artificial Intelligence. Unfortunately, two opposite views remain for biomedical applications of machine learning: accepting to use reliable but opaque models, vs. enforce models to be explainable. In this contribution we discuss these two opposite approaches and illustrate with examples the differences between them.

Named Entity Recognition in Pubmed Abstracts for Pharmacovigilance Using Deep Learning.

Methods of natural language processing associated with machine learning or deep learning can support detection of adverse drug reactions in abstracts of case reports available on Pubmed. In 2012, Gurulingappa et al. proposed a training set for the recognition of named entities corresponding to drugs and adverse reactions on 3000 Pubmed abstracts. We implemented a classifier using deep learning with a Bi-LSTM and a CRF layer that achieves an F-measure of 87.8%. Perspectives consist in using BERT for improving the classifier, and applying it to a large number of Pubmed abstract to build a database of case reports available in the literature.

Implementing a Microservices Architecture for Predicting the Opinion of Twitter Users on COVID Vaccines.

A strong trend in the software industry is to merge the activities of deployment and operationalization through the DevOps approach, which in the case of artificial intelligence is called Machine Learning Operations (MLOps). We present here a microservices architecture containing the whole pipeline (frontend, backend, data predictions) hosted in Docker containers which exposes a model implemented for opinion prediction in Twitter on the COVID vaccines. This is the first description in the literature of implementing a microservice architecture using TorchServe, a library for serving Pytorch models.

Ethical Issues in the Utilization of Black Boxes for Artificial Intelligence in Medicine.

Artificial Intelligence (AI) has made major progress in recent years in many fields. With regard of medicine however, the utilization of AI raises numerous ethical questions, especially since newer and much more accurate algorithms function as black boxes. A trade-off must then be made between having algorithms being very accurate and effective, and algorithms that are explainable but less proficient. In this paper we address the ethical implications of utilizing black box algorithms in medicine.

Inference Time of a CamemBERT Deep Learning Model for Sentiment Analysis of COVID Vaccines on Twitter.

In previous work, we implemented a deep learning model with CamemBERT and PyTorch, and built a microservices architecture using the TorchServe serving library. Without TorchServe, inference time was three times faster when the model was loaded once in memory compared when the model was loaded each time. The preloaded model without TorchServe presented comparable inference time with the TorchServe instance. However, using a PyTorch preloaded model in a web application without TorchServe would necessitate to implement functionalities already present in TorchServe.

Identification of COVID-19 Vaccines Concerns in Health-Related French Web Forums: A Topic Modelling Approach.

Since December 2019 and the first reported cases of COVID-19 in Wuhan, China, there have been 199,466,211 confirmed cases of COVID-19 in the World. The WHO defined vaccination hesitancy as one of the top ten threats to global health in 2019. Our objective was thus to identify topics and trends about COVID-19 vaccines from French web forums to understand the perception of the French population on these vaccines before the vaccination campaign started. We performed a topic model analysis on 485 web forums' posts. 10 topics were found. We reviewed 120 posts from 6 of these 10 topics. One topic was about vaccine hesitancy, refusal, and mistrust, and two topics were related to what the users think about the government, the political and economic choices made towards this epidemic.