Leveraging Real-World Data in Safety Signal Assessment.

PURPOSE: TransCelerate BioPharma surveyed its member biopharmaceutical companies to understand current practices and identify opportunities to complement safety signal assessment with rapid real-world data (RWD) analysis. METHODS: A voluntary 30-question questionnaire regarding the use of RWD in safety signal assessment was disseminated to subject matter experts at all TransCelerate member companies in July 2022. Responses were blinded, aggregated, summarized, and presented. RESULTS: Eighteen of 20 member companies provided responses to the questionnaire. Sixteen (89%) companies reported actively leveraging RWD in their signal assessment processes. Of 18 respondent companies, 8 (44%) routinely use rapid approaches to RWD analysis, 7 (39%) utilize rapid RWD analysis non-routinely or in a pilot setting, 2 (11%) are considering using rapid RWD analysis, and 1 (6%) has no plans to use rapid RWD analysis for their signal assessment. Most companies reported that RWD adds context to and improves quality of signal assessments. To conduct RWD analysis for signal assessment, 16 of 17 (94%) respondent companies utilize or plan to utilize internally available data, 8 (47%) utilize both internal and external data, and 3 (18%) utilize data networks. Respondents identified key challenges to rapidly performing RWD analyses, including data access/availability, time for analysis execution, and uncertainties regarding acceptance of minimal or non-protocolized approaches by health authorities. CONCLUSION: Biopharmaceutical companies reported that they see value in the use of rapid RWD analyses for complementing signal assessments. Future work is recommended to offer a framework and process for use of rapid use of RWD analyses in signal assessment.

The Reporting of a Disproportionality Analysis for Drug Safety Signal Detection Using Individual Case Safety Reports in PharmacoVigilance (READUS-PV): Development and Statement.

BACKGROUND AND AIM: Disproportionality analyses using reports of suspected adverse drug reactions are the most commonly used quantitative methods for detecting safety signals in pharmacovigilance. However, their methods and results are generally poorly reported in published articles and existing guidelines do not capture the specific features of disproportionality analyses. We here describe the development of a guideline (REporting of A Disproportionality analysis for drUg Safety signal detection using individual case safety reports in PharmacoVigilance [READUS-PV]) for reporting the results of disproportionality analyses in articles and abstracts. METHODS: We established a group of 34 international experts from universities, the pharmaceutical industry, and regulatory agencies, with expertise in pharmacovigilance, disproportionality analyses, and assessment of safety signals. We followed a three-step process to develop the checklist: (1) an open-text survey to generate a first list of items; (2) an online Delphi method to select and rephrase the most important items; (3) a final online consensus meeting. RESULTS: Among the panel members, 33 experts responded to round 1 and 30 to round 2 of the Delphi and 25 participated to the consensus meeting. Overall, 60 recommendations for the main body of the manuscript and 13 recommendations for the abstracts were retained by participants after the Delphi method. After merging of some items together and the online consensus meeting, the READUS-PV guidelines comprise a checklist of 32 recommendations, in 14 items, for the reporting of disproportionality analyses in the main body text and four items, comprising 12 recommendations, for abstracts. CONCLUSIONS: The READUS-PV guidelines will support authors, editors, peer-reviewers, and users of disproportionality analyses using individual case safety report databases. Adopting these guidelines will lead to more transparent, comprehensive, and accurate reporting and interpretation of disproportionality analyses, facilitating the integration with other sources of evidence.

The REporting of A Disproportionality Analysis for DrUg Safety Signal Detection Using Individual Case Safety Reports in PharmacoVigilance (READUS-PV): Explanation and Elaboration.

In pharmacovigilance, disproportionality analyses based on individual case safety reports are widely used to detect safety signals. Unfortunately, publishing disproportionality analyses lacks specific guidelines, often leading to incomplete and ambiguous reporting, and carries the risk of incorrect conclusions when data are not placed in the correct context. The REporting of A Disproportionality analysis for drUg Safety signal detection using individual case safety reports in PharmacoVigilance (READUS-PV) statement was developed to address this issue by promoting transparent and comprehensive reporting of disproportionality studies. While the statement paper explains in greater detail the procedure followed to develop these guidelines, with this explanation paper we present the 14 items retained for READUS-PV guidelines, together with an in-depth explanation of their rationale and bullet points to illustrate their practical implementation. Our primary objective is to foster the adoption of the READUS-PV guidelines among authors, editors, peer reviewers, and readers of disproportionality analyses. Enhancing transparency, completeness, and accuracy of reporting, as well as proper interpretation of their results, READUS-PV guidelines will ultimately facilitate evidence-based decision making in pharmacovigilance.

Supervised Machine Learning-Based Decision Support for Signal Validation Classification.

INTRODUCTION: Signal validation in pharmacovigilance is the process of evaluating data to decide whether evidence is sufficient to justify further assessment of a detected signal. During the signal validation process, safety experts in our organization are required to review signals of disproportionate reporting (SDRs) and classify them into one of six predefined categories. OBJECTIVE: This experiment explored the extent to which predictive machine learning (ML) models can support the decision making of safety experts by accurately identifying the most appropriate predefined signal validation category. METHODS: We extracted cumulative data for six medicinal products, consisting of historic SDR validations and Individual Case Safety Reports, from the company's safety database for training and testing of the ML model. We implemented a decision tree-based supervised multiclass classifier model termed Gradient Boosted Trees followed by a SHapley Additive exPlanations (SHAP) analysis to mitigate the "black box" effect of the ensemble model by identifying the key predicting features in the model. Following a retrospective analysis, a prospective experiment was conducted to test the model accuracy and user acceptance in a real-life setting. RESULTS: The prediction accuracy of our ML model ranged from 83 to 86% over 3 months for the six medicinal products. The applicability of the model was confirmed by the company's safety experts. Additionally, the systematic predictions provided valuable information to the safety experts and assisted them in reviewing the SDRs efficiently and consistently. CONCLUSIONS: This experiment demonstrated that it is possible to train a multiclass classification model to accurately predict signal validation categories for SDRs. More importantly, the transparency of the predictions provided by the SHAP analysis led to high acceptance by the safety experts.

Good Signal Detection Practices: Evidence from IMI PROTECT.

Over a period of 5 years, the Innovative Medicines Initiative PROTECT (Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium) project has addressed key research questions relevant to the science of safety signal detection. The results of studies conducted into quantitative signal detection in spontaneous reporting, clinical trial and electronic health records databases are summarised and 39 recommendations have been formulated, many based on comparative analyses across a range of databases (e.g. regulatory, pharmaceutical company). The recommendations point to pragmatic steps that those working in the pharmacovigilance community can take to improve signal detection practices, whether in a national or international agency or in a pharmaceutical company setting. PROTECT has also pointed to areas of potentially fruitful future research and some areas where further effort is likely to yield less.

Statistical Signal Detection as a Routine Pharmacovigilance Practice: Effects of Periodicity and Resignalling Criteria on Quality and Workload.

INTRODUCTION: The goal of signal detection in pharmacovigilance (PV) is to detect unknown causal associations between medicines and unexpected events. Statistical methods serve to detect signals and supplement traditional PV methods. Statistical signal detection (SSD) requires decisions about various settings that influence the quality and efficiency of SSD, as shown in several studies. To our knowledge, the effects of SSD periodicity and resignalling criteria on the quality and workload of routine SSD have not been published before. OBJECTIVE: To analyse the effects of different periodicities and resignalling criteria on signal detection quality and signal validation workload, and to test the impact of changing the signal threshold for number of cases. METHODS: We calculated signals of disproportionate reporting (SDRs) using thresholds of number of cases (N) ≥3, proportional reporting ratio ≥2 and Chi(2) ≥ 4. We retrospectively simulated recurrent SDR calculation and validation with varying periodicity (quarterly vs. monthly), resignalling criteria, and N ≥ 3 vs. N ≥ 5. RESULTS: Changing the periodicity from quarterly to monthly increased the workload by 46.6 % (0 % signal loss). More restrictive resignalling criteria reduced the workload between 36.3 % (0 % signal loss) and 74.1 % (50 % signal loss). For N ≥ 3, the most efficient monthly SSD resignalling criterion reduced the workload by 36.3 % and detected all true signals earlier than quarterly SSD. N ≥ 5 reduced the workload between 13.8 and 21.4 % (0 % signal loss). CONCLUSIONS: In real-life PV practice, signal detection and validation are recurrent periodic activities. Some true signals are only discovered upon resignalling. Our results demonstrate resignalling criteria with high signal detection quality and high efficiency. We found potential earlier detection of true signals using monthly SSD. Additional studies about resignalling should be performed to complement our findings.

Comparison of statistical signal detection methods within and across spontaneous reporting databases.

BACKGROUND: Most pharmacovigilance departments maintain a system to identify adverse drug reactions (ADRs) through analysis of spontaneous reports. The signal detection algorithms (SDAs) and the nature of the reporting databases vary between operators and it is unclear whether any algorithm can be expected to provide good performance in a wide range of environments. OBJECTIVE: The objective of this study was to compare the performance of commonly used algorithms across spontaneous reporting databases operated by pharmaceutical companies and national and international pharmacovigilance organisations. METHODS: 220 products were chosen and a reference set of ADRs was compiled. Within four company, one national and two international databases, 15 SDAs based on five disproportionality methods were tested. Signals of disproportionate reporting (SDRs) were calculated at monthly intervals and classified by comparison with the reference set. These results were summarised as sensitivity and precision for each algorithm in each database. RESULTS: Different algorithms performed differently between databases but no method dominated all others. Performance was strongly dependent on the thresholds used to define a statistical signal. However, the different disproportionality statistics did not influence the achievable performance. The relative performance of two algorithms was similar in different databases. Over the lifetime of a product there is a reduction in precision for any method. CONCLUSIONS: In designing signal detection systems, careful consideration should be given to the criteria that are used to define an SDR. The choice of disproportionality statistic does not appreciably affect the achievable range of signal detection performance and so this can primarily be based on ease of implementation, interpretation and minimisation of computing resources. The changes in sensitivity and precision obtainable by replacing one algorithm with another are predictable. However, the absolute performance of a method is specific to the database and is best assessed directly on that database. New methods may be required to gain appreciable improvements.