Validation of Artificial Intelligence to Support the Automatic Coding of Patient Adverse Drug Reaction Reports, Using Nationwide Pharmacovigilance Data.

INTRODUCTION: Adverse drug reaction reports are usually manually assessed by pharmacovigilance experts to detect safety signals associated with drugs. With the recent extension of reporting to patients and the emergence of mass media-related sanitary crises, adverse drug reaction reports currently frequently overwhelm pharmacovigilance networks. Artificial intelligence could help support the work of pharmacovigilance experts during such crises, by automatically coding reports, allowing them to prioritise or accelerate their manual assessment. After a previous study showing first results, we developed and compared state-of-the-art machine learning models using a larger nationwide dataset, aiming to automatically pre-code patients' adverse drug reaction reports. OBJECTIVES: We aimed to determine the best artificial intelligence model identifying adverse drug reactions and assessing seriousness in patients reports from the French national pharmacovigilance web portal. METHODS: Reports coded by 27 Pharmacovigilance Centres between March 2017 and December 2020 were selected (n = 11,633). For each report, the Portable Document Format form containing free-text information filled by the patient, and the corresponding encodings of adverse event symptoms (in Medical Dictionary for Regulatory Activities Preferred Terms) and seriousness were obtained. This encoding by experts was used as the reference to train and evaluate models, which contained input data processing and machine-learning natural language processing to learn and predict encodings. We developed and compared different approaches for data processing and classifiers. Performance was evaluated using receiver operating characteristic area under the curve (AUC), F-measure, sensitivity, specificity and positive predictive value. We used data from 26 Pharmacovigilance Centres for training and internal validation. External validation was performed using data from the remaining Pharmacovigilance Centres during the same period. RESULTS: Internal validation: for adverse drug reaction identification, Term Frequency-Inverse Document Frequency (TF-IDF) + Light Gradient Boosted Machine (LGBM) achieved an AUC of 0.97 and an F-measure of 0.80. The Cross-lingual Language Model (XLM) [transformer] obtained an AUC of 0.97 and an F-measure of 0.78. For seriousness assessment, FastText + LGBM achieved an AUC of 0.85 and an F-measure of 0.63. CamemBERT (transformer) + Light Gradient Boosted Machine obtained an AUC of 0.84 and an F-measure of 0.63. External validation for both adverse drug reaction identification and seriousness assessment tasks yielded consistent and robust results. CONCLUSIONS: Our artificial intelligence models showed promising performance to automatically code patient adverse drug reaction reports, with very similar results across approaches. Our system has been deployed by national health authorities in France since January 2021 to facilitate pharmacovigilance of COVID-19 vaccines. Further studies will be needed to validate the performance of the tool in real-life settings.

French organization for the pharmacovigilance of COVID-19 vaccines: A major challenge.

In this special issue, we present the main highlights of the first weeks of pharmacovigilance monitoring of coronavirus disease 2019 (COVID-19) vaccines in this unprecedented situation in France: the deployment of a vaccination during an epidemic period with the aim of vaccinating the entire population and the intense pharmacovigilance and surveillance of these vaccines still under conditional marketing authorizations. In this unprecedented situation, the cross approach and interaction between the French pharmacovigilance network and French National Agency for the Safety of Medicines and Health Products (ANSM) has been optimized to provide a real-time safety related to COVID-19 vaccines. Every week, pair of regional pharmacovigilance centers gathered safety data from the French pharmacovigilance network, to acutely expertise all the adverse drug reactions (ADRs) reported with each COVID-19 vaccine within a direct circuit with ANSM. Results of this expertise are presented and discussed with ANSM in order to raise safety signals and take appropriate measures if necessary. These reports are then published online. At the 25th of March 2021, more than 9 815 000 doses were injected and 20,265 ADRs were reported, mostly non-serious (76%). Several potential or confirmed signals were raised at the european level for those vaccines and others ADRs are under special attentions. This underlines the adaptiveness of the French pharmacovigilance system to both the identification of new patient profiles experiencing ADRs and the evolution of the vaccine strategy. Such an efficiency is necessary to manage a careful and acute surveillance of these new COVID-19 vaccines for and to face the pandemic at the same time.

Artificial Intelligence for Unstructured Healthcare Data: Application to Coding of Patient Reporting of Adverse Drug Reactions.

Adverse drug reaction (ADR) reporting is a major component of drug safety monitoring; its input will, however, only be optimized if systems can manage to deal with its tremendous flow of information, based primarily on unstructured text fields. The aim of this study was to develop an automated system allowing to code ADRs from patient reports. Our system was based on a knowledge base about drugs, enriched by supervised machine learning (ML) models trained on patients reporting data. To train our models, we selected all cases of ADRs reported by patients to a French Pharmacovigilance Centre through a national web-portal between March 2017 and March 2019 (n = 2,058 reports). We tested both conventional ML models and deep-learning models. We performed an external validation using a dataset constituted of a random sample of ADRs reported to the Marseille Pharmacovigilance Centre over the same period (n = 187). Here, we show that regarding area under the curve (AUC) and F-measure, the best model to identify ADRs was gradient boosting trees (LGBM), with an AUC of 0.93 (0.92-0.94) and F-measure of 0.72 (0.68-0.75). This model was run for external validation showing an AUC of 0.91 and a F-measure of 0.58. We evaluated an artificial intelligence pipeline that was found able to learn how to identify correctly ADRs from unstructured data. This result allowed us to start a new study using more data to further improve our performance and offer a tool that is useful in practice to efficiently manage drug safety information.

Contribution of Causality Assessment for an Automated Detection of Safety Signals: An Example Using the French Pharmacovigilance Database.

INTRODUCTION: Qualitative approaches based on drug causality assessment estimate the causal link between a drug and the occurrence of an adverse event from individual case safety reports. Quantitative approaches based on disproportionality analyses were developed subsequently to allow automated statistical signal detection from pharmacovigilance databases. This study assessed the potential value of causality assessment for automated safety signal detection. METHODS: All drug-serious adverse event pairs with a positive rechallenge and a semiology suggestive of drug causality were identified in the French pharmacovigilance database (BNPV) from 2011 to 2017. The results were compared with those obtained from automated disproportionality analyses of the BNPV/World Health Organization (WHO) VigiBase®, complemented by the list of signals validated by the WHO-UMC (Uppsala Monitoring Centre). Summary of Product Characteristics (SmPCs), Martindale®, Meyler's® and MedLINE® were used as other sources of information for the purpose of comparison. RESULTS: Of the 155 pairs of interest, 115 (74.2%) were also identified by another source of information. Since the individual case reporting in the BNPV, 23 (14.8%) of the adverse events (AEs) have been added to the SmPC, seven of which were not identified by disproportionality. Finally, 40 pairs were not identified by any other source of information, 13 of which were considered as potential new safety signals after analysis of case reports by pharmacovigilance experts. The signals identified by causality assessment involved antineoplastic and immunomodulatory drugs especially, in comparison with signals identified by WHO-UMC or by disproportionality within the BNPV. CONCLUSION: The approach therefore appears useful as an additional tool for safety signal detection, especially for antineoplastic and immunomodulating agents.

Benefits of combining change-point analysis with disproportionality analysis in pharmacovigilance signal detection.

BACKGROUND: Change-point analysis (CPA) is a powerful method to analyse pharmacovigilance data but it has never been used on the disproportionality metric. OBJECTIVES: To optimize signal detection investigating the interest of time-series analysis in pharmacovigilance and the benefits of combining CPA with the proportional reporting ratio (PRR). METHODS: We investigated the couple benfluorex and aortic valve incompetence (AVI) using the French National Pharmacovigilance and EudraVigilance databases: CPA was applied on monthly counts of reports and the lower bound of monthly computed PRR (PRR-). We stated a CPA hypothesis that the substance-event combination is more likely to be a signal when the 2 following criteria are fulfilled: PRR- is greater than 1 with at least 5 cases, and CPA method detects at least 2 successive change points of PRR- which made consecutively increasing segments. We tested this hypothesis by 95 test cases identified from a drug safety reference set and 2 validated signals from EudraVigilance database: CPA was applied on PRR-. RESULTS: For benfluorex and AVI, change points detected by CPA on PRR- were more meaningful compared with monthly counts of reports: More change points detected and detected earlier. In the reference set, 14 positive controls satisfied CPA hypothesis, 6 positive controls only met first requirements, 3 negative controls only met first requirement, and 2 validated signals satisfied CPA hypothesis. CONCLUSIONS: The combination of CPA and PRR represents a significant advantage in detecting earlier signals and reducing false-positive signals. This approach should be confirmed in further studies.

A pre-clinical assessment of an atlas-based automatic segmentation tool for the head and neck.

BACKGROUND AND PURPOSE: Accurate conformal radiotherapy treatment requires manual delineation of target volumes and organs at risk (OAR) that is both time-consuming and subject to large inter-user variability. One solution is atlas-based automatic segmentation (ABAS) where a priori information is used to delineate various organs of interest. The aim of the present study is to establish the accuracy of one such tool for the head and neck (H&N) using two different evaluation methods. MATERIALS AND METHODS: Two radiotherapy centres were provided with an ABAS tool that was used to outline the brainstem, parotids and mandible on several patients. The results were compared to manual delineations for the first centre (EM1) and reviewed/edited for the second centre (EM2), both of which were deemed as equally valid gold standards. The contours were compared in terms of their volume, sensitivity and specificity with the results being interpreted using the Dice similarity coefficient and a receiver operator characteristic (ROC) curve. RESULTS: Automatic segmentation took typically approximately 7min for each patient on a standard PC. The results indicated that the atlas contour volume was generally within +/-1SD of each gold standard apart from the parotids for EM1 and brainstem for EM2 that were over- and under-estimated, respectively (within +/-2SD). The similarity of the atlas contours with their respective gold standard was satisfactory with an average Dice coefficient for all OAR of 0.68+/-0.25 for EM1 and 0.82+/-0.13 for EM2. All data had satisfactory sensitivity and specificity resulting in a favourable position in ROC space. CONCLUSIONS: These tests have shown that the ABAS tool exhibits satisfactory sensitivity and specificity for the OAR investigated. There is, however, a systematic over-segmentation of the parotids (EM1) and under-segmentation of the brainstem (EM2) that require careful review and editing in the majority of cases. Such issues have been discussed with the software manufacturer and a revised version is due for release.