Validation of Artificial Intelligence to Support the Automatic Coding of Patient Adverse Drug Reaction Reports, Using Nationwide Pharmacovigilance Data.

INTRODUCTION: Adverse drug reaction reports are usually manually assessed by pharmacovigilance experts to detect safety signals associated with drugs. With the recent extension of reporting to patients and the emergence of mass media-related sanitary crises, adverse drug reaction reports currently frequently overwhelm pharmacovigilance networks. Artificial intelligence could help support the work of pharmacovigilance experts during such crises, by automatically coding reports, allowing them to prioritise or accelerate their manual assessment. After a previous study showing first results, we developed and compared state-of-the-art machine learning models using a larger nationwide dataset, aiming to automatically pre-code patients' adverse drug reaction reports. OBJECTIVES: We aimed to determine the best artificial intelligence model identifying adverse drug reactions and assessing seriousness in patients reports from the French national pharmacovigilance web portal. METHODS: Reports coded by 27 Pharmacovigilance Centres between March 2017 and December 2020 were selected (n = 11,633). For each report, the Portable Document Format form containing free-text information filled by the patient, and the corresponding encodings of adverse event symptoms (in Medical Dictionary for Regulatory Activities Preferred Terms) and seriousness were obtained. This encoding by experts was used as the reference to train and evaluate models, which contained input data processing and machine-learning natural language processing to learn and predict encodings. We developed and compared different approaches for data processing and classifiers. Performance was evaluated using receiver operating characteristic area under the curve (AUC), F-measure, sensitivity, specificity and positive predictive value. We used data from 26 Pharmacovigilance Centres for training and internal validation. External validation was performed using data from the remaining Pharmacovigilance Centres during the same period. RESULTS: Internal validation: for adverse drug reaction identification, Term Frequency-Inverse Document Frequency (TF-IDF) + Light Gradient Boosted Machine (LGBM) achieved an AUC of 0.97 and an F-measure of 0.80. The Cross-lingual Language Model (XLM) [transformer] obtained an AUC of 0.97 and an F-measure of 0.78. For seriousness assessment, FastText + LGBM achieved an AUC of 0.85 and an F-measure of 0.63. CamemBERT (transformer) + Light Gradient Boosted Machine obtained an AUC of 0.84 and an F-measure of 0.63. External validation for both adverse drug reaction identification and seriousness assessment tasks yielded consistent and robust results. CONCLUSIONS: Our artificial intelligence models showed promising performance to automatically code patient adverse drug reaction reports, with very similar results across approaches. Our system has been deployed by national health authorities in France since January 2021 to facilitate pharmacovigilance of COVID-19 vaccines. Further studies will be needed to validate the performance of the tool in real-life settings.

Artificial Intelligence for Unstructured Healthcare Data: Application to Coding of Patient Reporting of Adverse Drug Reactions.

Adverse drug reaction (ADR) reporting is a major component of drug safety monitoring; its input will, however, only be optimized if systems can manage to deal with its tremendous flow of information, based primarily on unstructured text fields. The aim of this study was to develop an automated system allowing to code ADRs from patient reports. Our system was based on a knowledge base about drugs, enriched by supervised machine learning (ML) models trained on patients reporting data. To train our models, we selected all cases of ADRs reported by patients to a French Pharmacovigilance Centre through a national web-portal between March 2017 and March 2019 (n = 2,058 reports). We tested both conventional ML models and deep-learning models. We performed an external validation using a dataset constituted of a random sample of ADRs reported to the Marseille Pharmacovigilance Centre over the same period (n = 187). Here, we show that regarding area under the curve (AUC) and F-measure, the best model to identify ADRs was gradient boosting trees (LGBM), with an AUC of 0.93 (0.92-0.94) and F-measure of 0.72 (0.68-0.75). This model was run for external validation showing an AUC of 0.91 and a F-measure of 0.58. We evaluated an artificial intelligence pipeline that was found able to learn how to identify correctly ADRs from unstructured data. This result allowed us to start a new study using more data to further improve our performance and offer a tool that is useful in practice to efficiently manage drug safety information.

Health research and innovation: Can we optimize the interface between startups/pharmaceutical companies and academic health care institutions or not?

Although France has numerous assets in the realm of health care, such as the excellence of its research teams, the reputation of its healthcare system, and the presence of many startups, all of which are necessary to become a leader in innovation, it also has combined cultural and regulatory barriers that limit the flexibility and efficiency of interactions between companies/startups and public health institutions. Therefore, the aim of the roundtable discussion was to optimize the interface between those businesses and institutions. Several institutions have successfully implemented teams and procedures which aim to facilitate this interface, with regard to assessments of technology, services provided, the transfer of biological material, R&D collaboration, and licensing agreements. However, there is still a notable absence of entrepreneurial culture among hospital and academic research practitioners; their training regarding innovation remains insufficient and business-related value-creation is non-existent in their career evolution. Pharmaceutical companies, and particularly startups, often lack knowledge about hospital environments and their constraints. As a result, the recommendations of the roundtable participants are as follows: (1) promote reciprocal acculturation between public health institutions and startups through multidisciplinary training in innovation, promoting project development and staff recognition within the institution, and improving pharmaceutical companies' understanding regarding the health care system; (2) provide those involved with means and resources dedicated to innovation by reserving time for innovation at work, securing the status of the staff involved, and aiding in the search for funding; (3) develop and use standard methodologies and tools; and (4) co-design and co-construct innovative health solutions, encouraging the emergence of participatory and interdisciplinary creative spaces. All of these recommendations should help to make the interface between startups/companies and public health institutions more fluid and attractive for those in the health sector.