Exploring the impact of design criteria for reference sets on performance evaluation of signal detection algorithms: The case of drug-drug interactions.

PURPOSE: To evaluate the impact of multiple design criteria for reference sets that are used to quantitatively assess the performance of pharmacovigilance signal detection algorithms (SDAs) for drug-drug interactions (DDIs). METHODS: Starting from a large and diversified reference set for two-way DDIs, we generated custom-made reference sets of various sizes considering multiple design criteria (e.g., adverse event background prevalence). We assessed differences observed in the performance metrics of three SDAs when applied to FDA Adverse Event Reporting System (FAERS) data. RESULTS: For some design criteria, the impact on the performance metrics was neglectable for the different SDAs (e.g., theoretical evidence associated with positive controls), while others (e.g., restriction to designated medical events, event background prevalence) seemed to have opposing and effects of different sizes on the Area Under the Curve (AUC) and positive predictive value (PPV) estimates. CONCLUSIONS: The relative composition of reference sets can significantly impact the evaluation metrics, potentially altering the conclusions regarding which methodologies are perceived to perform best. We therefore need to carefully consider the selection of controls to avoid misinterpretation of signals triggered by confounding factors rather than true associations as well as adding biases to our evaluation by "favoring" some algorithms while penalizing others.

Similarity and consistency assessment of three major online drug-drug interaction resources.

AIMS: The aim of this study was to explore the level of agreement on drug-drug interaction (DDI) information listed in three major online drug information resources (DIRs) in terms of: (1) interacting drug pairs; (2) severity rating; (3) evidence rating; and (4) clinical management recommendations. METHODS: We extracted information from the British National Formulary (BNF), Thesaurus and Micromedex. Following drug name normalisation, we estimated the overlap of the DIRs in terms of DDI. We annotated clinical management recommendations either manually, where possible, or through application of a machine learning algorithm. RESULTS: The DIRs contained 51 481 (BNF), 38 037 (Thesaurus) and 65 446 (Micromedex) drug pairs involved in DDIs. The number of common DDIs across the three DIRs was 6970 (13.54% of BNF, 18.32% of Thesaurus and 10.65% of Micromedex). Micromedex and Thesaurus overall showed higher levels of similarity in their severity ratings, while the BNF agreed more with Micromedex on the critical severity ratings and with Thesaurus on the least significant ones. Evidence rating agreement between BNF and Micromedex was generally poor. Variation in clinical management recommendations was also identified, with some categories (i.e., Monitor and Adjust dose) showing higher levels of agreement compared to others (i.e., Use with caution, Wash-out, Modify administration). CONCLUSIONS: There is considerable variation in the DDIs included in the examined DIRs, together with variability in categorisation of severity and clinical advice given. DDIs labelled as critical were more likely to appear in multiple DIRs. Such variability in information could have deleterious consequences for patient safety, and there is a need for harmonisation and standardisation.

Identifying Drug-Drug Interactions in Spontaneous Reports Utilizing Signal Detection and Biological Plausibility Aspects.

Translational approaches can benefit post-marketing drug safety surveillance through the growing availability of systems pharmacology data. Here, we propose a novel Bayesian framework for identifying drug-drug interaction (DDI) signals and differentiating between individual drug and drug combination signals. This framework is coupled with a systems pharmacology approach for automated biological plausibility assessment. Integrating statistical and biological evidence, our method achieves a 16.5% improvement (AUC: from 0.620 to 0.722) with drug-target-adverse event associations, 16.0% (AUC: from 0.580 to 0.673) with drug enzyme, and 15.0% (AUC: from 0.568 to 0.653) with drug transporter information. Applying this approach to detect potential DDI signals of QT prolongation and rhabdomyolysis within the FDA Adverse Event Reporting System (FAERS), we emphasize the significance of systems pharmacology in enhancing statistical signal detection in pharmacovigilance. Our study showcases the promise of data-driven biological plausibility assessment in the context of challenging post-marketing DDI surveillance.

A reference set of clinically relevant adverse drug-drug interactions.

The accurate and timely detection of adverse drug-drug interactions (DDIs) during the postmarketing phase is an important yet complex task with potentially major clinical implications. The development of data mining methodologies that scan healthcare databases for drug safety signals requires appropriate reference sets for performance evaluation. Methodologies for establishing DDI reference sets are limited in the literature, while there is no publicly available resource simultaneously focusing on clinical relevance of DDIs and individual behaviour of interacting drugs. By automatically extracting and aggregating information from multiple clinical resources, we provide a scalable approach for generating a reference set for DDIs that could support research in postmarketing safety surveillance. CRESCENDDI contains 10,286 positive and 4,544 negative controls, covering 454 drugs and 179 adverse events mapped to RxNorm and MedDRA concepts, respectively. It also includes single drug information for the included drugs (i.e., adverse drug reactions, indications, and negative drug-event associations). We demonstrate usability of the resource by scanning a spontaneous reporting system database for signals of DDIs using traditional signal detection algorithms.