Automated Classification of Quality Defect Issues Relating to Substandard Medicines Using a Hybrid Machine Learning and Rule-Based Approach.

BACKGROUND AND OBJECTIVE: Substandard medicines can lead to serious safety issues affecting public health; however, the nature of such issues can be widely heterogeneous. Health product regulators seek to prioritise critical product quality defects for review to ensure that prompt risk mitigation measures are taken. This study aims to classify the nature of issues for substandard medicines using machine learning to augment a risk-based and timely review of cases. METHODS: A combined machine learning algorithm with a keyword-based model was developed to classify quality issues using text relating to substandard medicines (CISTERM). The nature of issues for product defect cases were classified based on Medical Dictionary for Regulatory Activities-Health Sciences Authority (MedDRA-HSA) lowest-level terms. RESULTS: Product defect cases received from January 2010 to December 2021 were used for training (n = 11,082) and for testing (n = 2771). The machine learning model achieved a good recall (precision) of 92% (96%) for 'Product adulterated and/or contains prohibited substance', 86% (90%) for 'Out of specification or out of trend test result' and 90% (91%) for 'Manufacturing non-compliance'. CONCLUSION: Post-market surveillance of substandard medicines remains a key activity for drug regulatory authorities. A combined machine learning algorithm with keyword-based model can help to prioritise the review of product quality defect issues in a timely manner.

Applying the OMOP Common Data Model to Facilitate Benefit-Risk Assessments of Medicinal Products Using Real-World Data from Singapore and South Korea.

OBJECTIVES: The aim of this study was to characterize the benefits of converting Electronic Medical Records (EMRs) to a common data model (CDM) and to assess the potential of CDM-converted data to rapidly generate insights for benefit-risk assessments in post-market regulatory evaluation and decisions. METHODS: EMRs from January 2013 to December 2016 were mapped onto the Observational Medical Outcomes Partnership-CDM (OMOP-CDM) schema. Vocabulary mappings were applied to convert source data values into OMOP-CDM-endorsed terminologies. Existing analytic codes used in a prior OMOP-CDM drug utilization study were modified to conduct an illustrative analysis of oral anticoagulants used for atrial fibrillation in Singapore and South Korea, resembling a typical benefit-risk assessment. A novel visualization is proposed to represent the comparative effectiveness, safety and utilization of the drugs. RESULTS: Over 90% of records were mapped onto the OMOP-CDM. The CDM data structures and analytic code templates simplified the querying of data for the analysis. In total, 2,419 patients from Singapore and South Korea fulfilled the study criteria, the majority of whom were warfarin users. After 3 months of follow-up, differences in cumulative incidence of bleeding and thromboembolic events were observable via the proposed visualization, surfacing insights as to the agent of preference in a given clinical setting, which may meaningfully inform regulatory decision-making. CONCLUSIONS: While the structure of the OMOP-CDM and its accessory tools facilitate real-world data analysis, extending them to fulfil regulatory analytic purposes in the post-market setting, such as benefit-risk assessments, may require layering on additional analytic tools and visualization techniques.

A risk classification model for prioritising the management of quality issues relating to substandard medicines in Singapore.

BACKGROUND: Monitoring for substandard medicines by regulatory agencies is a key post-market surveillance activity. It is important to prioritise critical product defects for review to ensure that prompt risk mitigation actions are taken. METHODS: A regulatory risk impact prioritisation model for product defects (RISMED) with 11 factors considering the seriousness and extent of impact of a defect was developed. The model generated an overall score that categorised cases into high, medium or low impact. The model was further developed into a statistical risk scoring model (stat-RISMED) using multivariate logistic regression that classified cases into high and non-high impact. Both models were evaluated against an expert-derived gold standard annotation corpus and tested on an independent dataset. RESULTS: Product defect cases received from January 2011 to June 2020 (n = 660) were used to train stat-RISMED and cases from July 2020 to June 2021 (n = 220) for validation. The stat-RISMED identified four factors associated with high impact cases, namely defect classification based on MedDRA-HSA terms, therapeutic indication of product, detectability of defect and whether any overseas regulatory actions were performed. Compared to RISMED, stat-RISMED achieved an improved sensitivity (94% vs 42%) and positive predictive value (47% vs 43%) for the identification of high impact cases, against the gold standard labels. CONCLUSIONS: This study reported characteristics that predicts cases with high impact, and the use of a statistical model to identify such cases. The model may potentially be applied to prioritise product defect issues and strengthen overall surveillance efforts of substandard medicines.

Augmenting Product Defect Surveillance Through Web Crawling and Machine Learning in Singapore.

INTRODUCTION: Substandard medicines are medicines that fail to meet their quality standards and/or specifications. Substandard medicines can lead to serious safety issues affecting public health. With the increasing number of pharmaceuticals and the complexity of the pharmaceutical manufacturing supply chain, monitoring for substandard medicines via manual environmental scanning can be laborious and time consuming. METHODS: A web crawler was developed to automatically detect and extract alerts on substandard medicines published on the Internet by regulatory agencies. The crawled data were labelled as related to substandard medicines or not. An expert-derived keyword-based classification algorithm was compared against machine learning algorithms to identify substandard medicine alerts on two validation datasets (n = 4920 and n = 2458) from a later time period than training data. Models were comparatively assessed for recall, precision and their F1 scores (harmonic mean of precision and recall). RESULTS: The web crawler routinely extracted alerts from the 46 web pages belonging to nine regulatory agencies. From October 2019 to May 2020, 12,156 unique alerts were crawled of which 7378 (60.7%) alerts were set aside for validation and contained 1160 substandard medicine alerts (15.7%). An ensemble approach of combining machine learning and keywords achieved the best recall (94% and 97%), precision (85% and 80%) and F1 scores (89% and 88%) on temporal validation. CONCLUSIONS: Combining robust web crawler programmes with rigorously tested filtering algorithms based on machine learning and keyword models can automate and expand horizon scanning capabilities for issues relating to substandard medicines.

Drug-induced liver injury associated with Complementary and Alternative Medicine: a review of adverse event reports in an Asian community from 2009 to 2014.

BACKGROUND: The use of Complementary and Alternative Medicine (CAM) has been increasing over the years. A recent review of adverse event reports (AERs) associated with CAM in Singapore found a notable number of AERs submitted. The objectives of this study are to analyse hepatotoxicity cases associated with CAM in Singapore based on spontaneous adverse event reporting to the Health Sciences Authority (HSA), and to highlight safety signals for specific herbal ingredients. METHODS: AERs associated with CAM and hepatotoxicity submitted to the Vigilance and Compliance Branch (VCB) of the HSA from 2009 to 2014 were compiled. The following information was extracted and analysed: Demographic information; time to onset; hospitalisation status; outcome; type of hepatotoxicity; ingredients of CAM, and the total daily doses (TDD); concurrent western medicines and health supplements; and reporter details. RESULTS: Fifty-seven reports were eligible for analysis. Thirty-five (61.4 %) cases involved Traditional Chinese Medicine (TCM). The Roussel Uclaf Causality Assessment Method was applied in 29 (82.9 %) of these cases, and the median score was 4 (range: 1-8). Chai Hu (Radix bupleuri) was suspected in 11 (31.4 %) cases. TDDs of most ingredients were within recommended doses of the Chinese Pharmacopoeia. CONCLUSIONS: Drug-induced liver injury is still poorly understood and more objective assessments are warranted. Reporting of adverse events should be strongly advocated to facilitate future analyses and the understanding of risk-benefit profiles of CAM.