An updated analysis on myocarditis and pericarditis cases reported following mRNA SARS-CoV-2 vaccination in Singapore.

INTRODUCTION: Messenger ribonucleic acid (mRNA) severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) vaccines have been associated with myocarditis/pericarditis, especially in young males. We evaluated the risk of myocarditis/pericarditis following mRNA vaccines by brand, age, sex and dose number in Singapore. METHODS: Adverse event reports of myocarditis/pericarditis following mRNA vaccines received by the Health Sciences Authority from 30 December 2020 to 25 July 2022 were included, with a data lock on 30 September 2022. Case adjudication was done by an independent panel of cardiologists using the US Centers for Disease Control and Prevention case definition. Reporting rates were compared with expected rates using historical data from 2018 to 2020. RESULTS: Of the 152 adjudicated cases, males comprised 75.0%. The median age was 30 years. Most cases occurred after Dose 2 (49.3%). The median time to onset was 2 days. Reporting rates were highest in males aged 12-17 years for both primary series (11.5 [95% confidence interval [CI] 6.7-18.4] per 100,000 doses, post-Dose 2) and following booster doses (7.1 [95% CI 3.0-13.9] per 100,000 doses). In children aged 5-11 years, myocarditis remained very rare (0.2 per 100,000 doses). The reporting rates for Booster 1 were generally similar or lower than those for Dose 2. CONCLUSIONS: The risk of myocarditis/pericarditis with mRNA vaccines was highest in adolescent males following Dose 2, and this was higher than historically observed background rates. Most cases were clinically mild. The risk of myocarditis should be weighed against the benefits of receiving an mRNA vaccine, keeping in mind that SARS-CoV-2 infections carry substantial risks of myocarditis/pericarditis, as well as the evolving landscape of the disease.

Nationwide safety surveillance of COVID-19 mRNA vaccines following primary series and first booster vaccination in Singapore.

Background: The real-world safety profile of COVID-19 mRNA vaccines remains incompletely elucidated. Methods: We performed a nationwide post-market safety surveillance analysis in Singapore, on vacinees aged 5 years and older, through mid-September 2022. Observed-over-expected (O/E) analyses were performed to identify potential safety signals among eight shortlisted adverse events of special interest (AESIs): strokes, cerebral venous thrombosis (CVT), acute myocardial infarction, myocarditis/pericarditis, pulmonary embolism, immune thrombocytopenia, convulsions and appendicitis. Self-controlled case series analyses (SCCS) were performed to validate signals of concern, occurring within 42 days of vaccination. Findings: Elevated risks were observed on O/E analyses for the following AESIs: myocarditis/pericarditis, [rate ratio (RR): 3.66, 95 % confidence interval (95 % CI): 2.71 to 4.94], appendicitis [RR: 1.14 (1.02 to 1.27)] and CVT [RR: 2.11 (1.18 to 3.77)]. SCCS analyses generated corroborative findings: myocarditis/pericarditis, [relative incidence (RI): 6.96 (3.95 to 12.27) at 1 to 7 days post-dose 2], CVT [RI: 4.30 (1.30 to 14.20) at 22 to 42 days post-dose 1] and appendicitis [RI: 1.31 (1.03 to 1.67) at 1 to 7 days post-dose 1]. Booster dose 1 continued to be associated with higher rates of myocarditis/pericarditis on O/E analysis [RR: 2.30, (1.39 to 3.80) and 1.69, (1.11 to 2.59)] at 21- and 42-days post-booster dose 1, respectively. Males aged 12 to 17 exhibited highest risks of both myocarditis/pericarditis [RI: 6.31 (1.36 to 29.3)] and appendicitis [RI: 2.01 (1.12 to 3.64)] after primary vaccination. Similarly, CVT was also predominantly observed in males aged above 50 (11 out of 16 cases), within 42-days of vaccination. Interpretation: Our data suggest that myocarditis/pericarditis, appendicitis and CVT are associated with primary vaccination using COVID-19 mRNA vaccines. Males at specific ages exhibit higher risks for all three AEs identified. The risk of myocarditis/pericarditis continues to be elevated after booster dose 1.

Automated Classification of Quality Defect Issues Relating to Substandard Medicines Using a Hybrid Machine Learning and Rule-Based Approach.

BACKGROUND AND OBJECTIVE: Substandard medicines can lead to serious safety issues affecting public health; however, the nature of such issues can be widely heterogeneous. Health product regulators seek to prioritise critical product quality defects for review to ensure that prompt risk mitigation measures are taken. This study aims to classify the nature of issues for substandard medicines using machine learning to augment a risk-based and timely review of cases. METHODS: A combined machine learning algorithm with a keyword-based model was developed to classify quality issues using text relating to substandard medicines (CISTERM). The nature of issues for product defect cases were classified based on Medical Dictionary for Regulatory Activities-Health Sciences Authority (MedDRA-HSA) lowest-level terms. RESULTS: Product defect cases received from January 2010 to December 2021 were used for training (n = 11,082) and for testing (n = 2771). The machine learning model achieved a good recall (precision) of 92% (96%) for 'Product adulterated and/or contains prohibited substance', 86% (90%) for 'Out of specification or out of trend test result' and 90% (91%) for 'Manufacturing non-compliance'. CONCLUSION: Post-market surveillance of substandard medicines remains a key activity for drug regulatory authorities. A combined machine learning algorithm with keyword-based model can help to prioritise the review of product quality defect issues in a timely manner.

Combining Machine Learning with a Rule-Based Algorithm to Detect and Identify Related Entities of Documented Adverse Drug Reactions on Hospital Discharge Summaries.

INTRODUCTION: Discharge summaries contain valuable information about adverse drug reactions, but their unstructured nature makes them challenging to analyse and use as a signal source for pharmacovigilance. Machine learning has shown promise in identifying discharge summaries that contain related drug-adverse event pairs but has fared relatively poorer in entity extraction. METHODS: A hybrid model is developed combining rule-based and machine learning algorithms using discharge summaries with the aim of maximising capture of related drug-adverse event pairs. The rule first identifies segments containing adverse event entities within a 100-character distance from a drug term; machine learning subsequently estimates the relatedness of the drug and adverse event entities contained. The approach is validated on four independent datasets that are temporally and geographically separated from model development data. The impact of restricted drug-adverse event pair detection on recall is evaluated by using two of the four validation datasets that do not impose rule-based restrictions to annotations. RESULTS: The hybrid model achieves a recall of 0.80 (fivefold cross validation), 0.80 (temporal) and 0.76 (geographical) on validation using datasets containing only pre-identified target text segments that fulfil the rule-based algorithm criteria. When tested on datasets that additionally contained drug-adverse event pairs not restricted by the rule-based criteria, recall of the model declines to 0.68 and 0.62 on temporally and geographically separated datasets, respectively. CONCLUSIONS: The proposed hybrid model demonstrates reasonable generalisability on external validation. Rule-based restriction of the detection space results in an approximately 12-14% reduction in recall but improves identification of the related drug and adverse event terms.

Applying the OMOP Common Data Model to Facilitate Benefit-Risk Assessments of Medicinal Products Using Real-World Data from Singapore and South Korea.

OBJECTIVES: The aim of this study was to characterize the benefits of converting Electronic Medical Records (EMRs) to a common data model (CDM) and to assess the potential of CDM-converted data to rapidly generate insights for benefit-risk assessments in post-market regulatory evaluation and decisions. METHODS: EMRs from January 2013 to December 2016 were mapped onto the Observational Medical Outcomes Partnership-CDM (OMOP-CDM) schema. Vocabulary mappings were applied to convert source data values into OMOP-CDM-endorsed terminologies. Existing analytic codes used in a prior OMOP-CDM drug utilization study were modified to conduct an illustrative analysis of oral anticoagulants used for atrial fibrillation in Singapore and South Korea, resembling a typical benefit-risk assessment. A novel visualization is proposed to represent the comparative effectiveness, safety and utilization of the drugs. RESULTS: Over 90% of records were mapped onto the OMOP-CDM. The CDM data structures and analytic code templates simplified the querying of data for the analysis. In total, 2,419 patients from Singapore and South Korea fulfilled the study criteria, the majority of whom were warfarin users. After 3 months of follow-up, differences in cumulative incidence of bleeding and thromboembolic events were observable via the proposed visualization, surfacing insights as to the agent of preference in a given clinical setting, which may meaningfully inform regulatory decision-making. CONCLUSIONS: While the structure of the OMOP-CDM and its accessory tools facilitate real-world data analysis, extending them to fulfil regulatory analytic purposes in the post-market setting, such as benefit-risk assessments, may require layering on additional analytic tools and visualization techniques.

A risk classification model for prioritising the management of quality issues relating to substandard medicines in Singapore.

BACKGROUND: Monitoring for substandard medicines by regulatory agencies is a key post-market surveillance activity. It is important to prioritise critical product defects for review to ensure that prompt risk mitigation actions are taken. METHODS: A regulatory risk impact prioritisation model for product defects (RISMED) with 11 factors considering the seriousness and extent of impact of a defect was developed. The model generated an overall score that categorised cases into high, medium or low impact. The model was further developed into a statistical risk scoring model (stat-RISMED) using multivariate logistic regression that classified cases into high and non-high impact. Both models were evaluated against an expert-derived gold standard annotation corpus and tested on an independent dataset. RESULTS: Product defect cases received from January 2011 to June 2020 (n = 660) were used to train stat-RISMED and cases from July 2020 to June 2021 (n = 220) for validation. The stat-RISMED identified four factors associated with high impact cases, namely defect classification based on MedDRA-HSA terms, therapeutic indication of product, detectability of defect and whether any overseas regulatory actions were performed. Compared to RISMED, stat-RISMED achieved an improved sensitivity (94% vs 42%) and positive predictive value (47% vs 43%) for the identification of high impact cases, against the gold standard labels. CONCLUSIONS: This study reported characteristics that predicts cases with high impact, and the use of a statistical model to identify such cases. The model may potentially be applied to prioritise product defect issues and strengthen overall surveillance efforts of substandard medicines.

Augmenting Product Defect Surveillance Through Web Crawling and Machine Learning in Singapore.

INTRODUCTION: Substandard medicines are medicines that fail to meet their quality standards and/or specifications. Substandard medicines can lead to serious safety issues affecting public health. With the increasing number of pharmaceuticals and the complexity of the pharmaceutical manufacturing supply chain, monitoring for substandard medicines via manual environmental scanning can be laborious and time consuming. METHODS: A web crawler was developed to automatically detect and extract alerts on substandard medicines published on the Internet by regulatory agencies. The crawled data were labelled as related to substandard medicines or not. An expert-derived keyword-based classification algorithm was compared against machine learning algorithms to identify substandard medicine alerts on two validation datasets (n = 4920 and n = 2458) from a later time period than training data. Models were comparatively assessed for recall, precision and their F1 scores (harmonic mean of precision and recall). RESULTS: The web crawler routinely extracted alerts from the 46 web pages belonging to nine regulatory agencies. From October 2019 to May 2020, 12,156 unique alerts were crawled of which 7378 (60.7%) alerts were set aside for validation and contained 1160 substandard medicine alerts (15.7%). An ensemble approach of combining machine learning and keywords achieved the best recall (94% and 97%), precision (85% and 80%) and F1 scores (89% and 88%) on temporal validation. CONCLUSIONS: Combining robust web crawler programmes with rigorously tested filtering algorithms based on machine learning and keyword models can automate and expand horizon scanning capabilities for issues relating to substandard medicines.

Comparison of laboratory threshold criteria in drug-induced liver injury detection algorithms for use in pharmacovigilance.

PURPOSE: For the purpose of pharmacovigilance, we sought to determine the best performing laboratory threshold criteria to detect drug-induced liver injury (DILI) in the electronic medical records (EMR). METHODS: We compared three commonly used liver chemistry criteria from the DILI expert working group (DEWG), DILI network (DILIN), and Council for International Organizations of Medical Sciences (CIOMS), based on hospital EMR for years 2010 and 2011 (42 176 admissions), using independent medical record review. The performance characteristics were compared in terms of sensitivity, specificity, positive predictive value (PPV), negative predictive value, accuracy, F-measure, and area under the receiver operating characteristic curve (AUROC). RESULTS: DEWG had the highest PPV (5.5%, 95% CI: 4.1%-7.2%), specificity (97.0%, 95% CI: 96.8%-97.2%), accuracy (96.8%, 95% CI: 96.6%-97.0%) and F-measure (0.099). CIOMS had the highest sensitivity (74.0%, 95% CI: 64.3%-82.3%) and AUROC (85.2%, 95% CI: 80.8%-89.7%). Besides the laboratory criteria, including additional keywords in the classification algorithm improved the PPV and F-measure to a maximum of 29.0% (95% CI: 22.3%-36.5%) and 0.379, respectively. CONCLUSIONS: More stringent criteria (DEWG and DILIN) performed better in terms of PPV, specificity, accuracy and F-measure. CIOMS performed better in terms of sensitivity. An algorithm with high sensitivity is useful in pharmacovigilance for detecting rare events and to avoid missing cases. Requiring at least two abnormal liver chemistries during hospitalization and text-word searching in the discharge summaries decreased false positives without loss in sensitivity.

Quantitative data mining in signal detection: the Singapore experience.

Background: In Singapore, the Health Sciences Authority (HSA) reviews an average of 20,000 spontaneous adverse event (AE) reports yearly. Potential safety signals are identified manually and discussed on a weekly basis. In this study, we compared the use of four quantitative data mining (QDM) methods with weekly manual review to determine if signals of disproportionate reporting (SDRs) can improve the efficiency of manual reviews and thereby enhance drug safety signal detection.Methods: We formulated a QDM triage strategy to reduce the number of SDRs for weekly review and compared the results against those derived from manual reviews alone for the same 6-month period. We then incorporated QDM triage into the manual review workflow for the subsequent two 6-month periods and made further comparisons against QDM triage alone.Results: The incorporation of QDM triage into routine manual reviews resulted in a reduction of 20% to 30% in the number of drug-AE pairs identified for further evaluation. Sequential Probability Ratio Test (SPRT) detected more signals that mirror human manual signal detection than the other three methods.Conclusions: The adoption of QDM triage into our manual reviews is a more efficient way forward in signal detection, avoiding missing important drug safety signals.

Detecting adverse drug reactions in discharge summaries of electronic medical records using Readpeer.

BACKGROUND: Hospital discharge summaries offer a potentially rich resource to enhance pharmacovigilance efforts to evaluate drug safety in real-world clinical practice. However, it is infeasible for experts to read through all discharge summaries to find cases of drug-adverse event (AE) relations. PURPOSE: The objective of this paper is to develop a natural language processing (NLP) framework to detect drug-AE relations from unstructured hospital discharge summaries. BASIC PROCEDURES: An NLP algorithm was designed using customized dictionaries of drugs, adverse event (AE) terms, and rules based on trigger phrases, negations, fuzzy logic and word distances to recognize drug, AE terms and to detect drug-AE relations. Furthermore, a customized annotation tool was developed to facilitate expert review of discharge summaries from a tertiary hospital in Singapore in 2011. MAIN FINDINGS: A total of 33 trial sets with 50 to 100 records per set were evaluated (1620 discharge summaries) by our algorithm and reviewed by pharmacovigilance experts. After every 6 trial sets, drug and AE dictionaries were updated, and rules were modified to improve the system. Excellent performance was achieved for drug and AE entity recognition with over 92% precision and recall. On the final 6 sets of discharge summaries (600 records), our algorithm achieved 75% precision and 59% recall for identification of valid drug-AE relations. PRINCIPAL CONCLUSIONS: Adverse drug reactions are a significant contributor to health care costs and utilization. Our algorithm is not restricted to particular drugs, drug classes or specific medical specialties, which is an important attribute for a national regulatory authority to carry out comprehensive safety monitoring of drug products. Drug and AE dictionaries may be updated periodically to ensure that the tool remains relevant for performing surveillance activities. The development of the algorithm, and the ease of reviewing and correcting the results of the algorithm as part of an iterative machine learning process, is an important step towards use of hospital discharge summaries for an active pharmacovigilance program.

Application and optimisation of the Comparison on Extreme Laboratory Tests (CERT) algorithm for detection of adverse drug reactions: Transferability across national boundaries.

PURPOSE: The Singapore regulatory agency for health products (Health Sciences Authority), in performing active surveillance of medicines and their potential harms, is open to new methods to achieve this goal. Laboratory tests are a potential source of data for this purpose. We have examined the performance of the Comparison on Extreme Laboratory Tests (CERT) algorithm, developed by Ajou University, Korea, as a potential tool for adverse drug reaction detection based on the electronic medical records of the Singapore health care system. METHODS: We implemented the original CERT algorithm, comparing extreme laboratory results pre- and post-drug exposure, and 5 variations thereof using 4.5 years of National University Hospital (NUH) electronic medical record data (31 869 588 laboratory tests, 6 699 591 drug dispensings from 272 328 hospitalizations). We investigated 6 drugs from the original CERT paper and an additional 47 drugs. We benchmarked results against a reference standard that we created from UpToDate 2015. RESULTS: The original CERT algorithm applied to all 53 drugs and 44 laboratory abnormalities yielded a positive predictive value (PPV) and sensitivity of 50.3% and 54.1%, respectively. By raising the minimum number of cases for each drug-laboratory abnormality pair from 2 to 400, the PPV and sensitivity increased to 53.9% and 67.2%, respectively. This post hoc variation, named CERT400, performed particularly well for drug-induced hepatic and renal toxicities. DISCUSSION: We have demonstrated that the CERT algorithm can be applied across national boundaries. One modification (CERT400) was able to identify adverse drug reaction signals from laboratory data with reasonable PPV and sensitivity, which indicates potential utility as a supplementary pharmacovigilance tool.

Detecting Signals of Disproportionate Reporting from Singapore's Spontaneous Adverse Event Reporting System: An Application of the Sequential Probability Ratio Test.

INTRODUCTION: The ability to detect safety concerns from spontaneous adverse drug reaction reports in a timely and efficient manner remains important in public health. OBJECTIVE: This paper explores the behaviour of the Sequential Probability Ratio Test (SPRT) and ability to detect signals of disproportionate reporting (SDRs) in the Singapore context. METHODS: We used SPRT with a combination of two hypothesised relative risks (hRRs) of 2 and 4.1 to detect signals of both common and rare adverse events in our small database. We compared SPRT with other methods in terms of number of signals detected and whether labelled adverse drug reactions were detected or the reaction terms were considered serious. The other methods used were reporting odds ratio (ROR), Bayesian Confidence Propagation Neural Network (BCPNN) and Gamma Poisson Shrinker (GPS). RESULTS: The SPRT produced 2187 signals in common with all methods, 268 unique signals, and 70 signals in common with at least one other method, and did not produce signals in 178 cases where two other methods detected them, and there were 403 signals unique to one of the other methods. In terms of sensitivity, ROR performed better than other methods, but the SPRT method found more new signals. The performances of the methods were similar for negative predictive value and specificity. CONCLUSIONS: Using a combination of hRRs for SPRT could be a useful screening tool for regulatory agencies, and more detailed investigation of the medical utility of the system is merited.

A Survey on Pharmacovigilance Activities in ASEAN and Selected Non-ASEAN Countries, and the Use of Quantitative Signal Detection Algorithms.

INTRODUCTION: Most Countries have pharmacovigilance (PV) systems in place to monitor the safe use of health products. The process involves the detection and assessment of safety issues from various sources of information, communicating the risk to stakeholders and taking other relevant risk minimization measures. OBJECTIVES: This study aimed to assess the PV status in Association of Southeast Asian Nation (ASEAN) countries, sources for postmarket safety monitoring, methods used for signal detection and the need for a quantitative signal detection algorithm (QSDA). Comparisons were conducted with centres outside ASEAN. METHODS: A questionnaire was sent to all PV centres in ASEAN countries, as well as seven other countries, from November 2015 to June 2016. The questionnaire was designed to collect information on the status of PV, with a focus on the use of a QSDA. RESULTS: Data were collected from nine ASEAN countries and seven other countries. PV activities were conducted in all these countries, which were at different stages of development. In terms of adverse drug reaction (ADR) reports, the average number received per year ranged from 3 to 50,000 reports for ASEAN countries and from 7000 to 1,103,200 for non-ASEAN countries. Thirty-three percent of ASEAN countries utilized statistical methods to help detect signals from ADR reports compared with 100% in the other non-ASEAN countries. Eighty percent agreed that the development of a QSDA would help in drug signal detection. The main limitation identified was the lack of knowledge and/or lack of resources. CONCLUSION: Spontaneous ADR reports from healthcare professionals remains the most frequently used source for safety monitoring. The traditional method of case-by-case review of ADR reports prevailed for signal detection in ASEAN countries. As the reports continue to grow, the development of a QSDA would be useful in helping detect safety signals.

Data mining spontaneous adverse drug event reports for safety signals in Singapore - a comparison of three different disproportionality measures.

OBJECTIVES: Quantitative data mining methods can be used to identify potential signals of unexpected relationships between drug and adverse event (AE). This study aims to compare and explore the use of three data mining methods in our small spontaneous AE database. METHODS: We consider reporting odds ratio (ROR), Bayesian Confidence Propagation Neural Network (BCPNN) and Gamma Poisson Shrinker (GPS) assuming two different sets of criteria: (1) ROR-1.96SE>1, IC-1.96SD>0, EB05>1 (2) ROR-1.96SE>2, IC-1.96SD>1, EB05 >2. Count of drug-AE pairs ≥3 was considered for ROR and GPS. RESULTS: The Health Sciences Authority, Singapore received 151,180 AE reports between 1993 and 2013. ROR, BCPNN and GPS identified 2,835, 2,311 and 2,374 significant drug-AE pairs using Criterion 1, and 1,899, 1,101 and 1,358 respectively using Criterion 2. The performance of the three methods with respect to specificity, positive predictive value and negative predictive value were similar, although ROR yielded a higher sensitivity and larger area under the receiver operating characteristic curve. ROR and GPS picked up some potential signals which BCPNN missed. CONCLUSIONS: The defined threshold used for ROR (Criterion 1) is a useful screening tool for our small database. It may be used in conjunction with GPS to avoid missed signals.