A framework enabling LLMs into regulatory environment for transparency and trustworthiness and its application to drug labeling document.

Regulatory agencies consistently deal with extensive document reviews, ranging from product submissions to both internal and external communications. Large Language Models (LLMs) like ChatGPT can be invaluable tools for these tasks, however present several challenges, particularly the proprietary information, combining customized function with specific review needs, and transparency and explainability of the model's output. Hence, a localized and customized solution is imperative. To tackle these challenges, we formulated a framework named askFDALabel on FDA drug labeling documents that is a crucial resource in the FDA drug review process. AskFDALabel operates within a secure IT environment and comprises two key modules: a semantic search and a Q&A/text-generation module. The Module S built on word embeddings to enable comprehensive semantic queries within labeling documents. The Module T utilizes a tuned LLM to generate responses based on references from Module S. As the result, our framework enabled small LLMs to perform comparably to ChatGPT with as a computationally inexpensive solution for regulatory application. To conclude, through AskFDALabel, we have showcased a pathway that harnesses LLMs to support agency operations within a secure environment, offering tailored functions for the needs of regulatory research.

Classifying Free Texts Into Predefined Sections Using AI in Regulatory Documents: A Case Study with Drug Labeling Documents.

The US Food and Drug Administration (FDA) regulatory process often involves several reviewers who focus on sets of information related to their respective areas of review. Accordingly, manufacturers that provide submission packages to regulatory agencies are instructed to organize the contents using a structure that enables the information to be easily allocated, retrieved, and reviewed. However, this practice is not always followed correctly; as such, some documents are not well structured, with similar information spreading across different sections, hindering the efficient access and review of all of the relevant data as a whole. To improve this common situation, we evaluated an artificial intelligence (AI)-based natural language processing (NLP) methodology, called Bidirectional Encoder Representations from Transformers (BERT), to automatically classify free-text information into standardized sections, supporting a holistic review of drug safety and efficacy. Specifically, FDA labeling documents were used in this study as a proof of concept, where the labeling section structure defined by the Physician Label Rule (PLR) was used to classify labels in the development of the model. The model was subsequently evaluated on texts from both well-structured labeling documents (i.e., PLR-based labeling) and less- or differently structured documents (i.e., non-PLR and Summary of Product Characteristic [SmPC] labeling.) In the training process, the model yielded 96% and 88% accuracy for binary and multiclass tasks, respectively. The testing accuracies observed for the PLR, non-PLR, and SmPC testing data sets for the binary model were 95%, 88%, and 88%, and for the multiclass model were 82%, 73%, and 68%, respectively. Our study demonstrated that automatically classifying free texts into standardized sections with AI language models could be an advanced regulatory science approach for supporting the review process by effectively processing unformatted documents.

A Weakly Supervised Deep Learning Framework for Whole Slide Classification to Facilitate Digital Pathology in Animal Study.

The pathology of animal studies is crucial for toxicity evaluations and regulatory assessments, but the manual examination of slides by pathologists remains time-consuming and requires extensive training. One inherent challenge in this process is the interobserver variability, which can compromise the consistency and accuracy of a study. Artificial intelligence (AI) has demonstrated its ability to automate similar examinations in clinical applications with enhanced efficiency, consistency, and accuracy. However, training AI models typically relies on costly pixel-level annotation of injured regions and is often not available for animal pathology. To address this, we developed the PathologAI system, a "weakly" supervised approach for WSI classification in rat images without explicit lesion annotation at the pixel level. Using rat liver imaging data from the Open TG-GATEs system, PathologAI was applied to predict necrosis of n = 816 WSIs (377 controls). TG-GATEs studied 170 compounds at three dose levels (low, middle, and high) for four time points (3, 7, 14, and 28 days). PathologAI first preprocessed WSIs at the tile level to generate a high-level representation with a Generative Adversarial Network architecture. The prediction of liver necrosis relied on an ensemble model of 5 CNN classifiers trained on 335 WSIs. The ensemble model achieved notable classification accuracy on the holdout test set: 87% among 87 control slides free of findings, 83% among 120 controls with spontaneous necrosis, 67% among 147 treated animals with spontaneous minimal or slight necrosis, and 59% among 127 treated animals with minimal or slight necrosis caused by the treatment. Importantly, PathologAI was able to discriminate WSIs with spontaneous necrosis from those with treatment related necrosis and discriminated mild lesion level findings (slight vs minimal) and between treatment dose levels. PathologAI could provide an inexpensive and rapid screening tool to assist the digital pathology analysis in preclinical applications and general toxicological studies.

Development of benchmark datasets for text mining and sentiment analysis to accelerate regulatory literature review.

In the field of regulatory science, reviewing literature is an essential and important step, which most of the time is conducted by manually reading hundreds of articles. Although this process is highly time-consuming and labor-intensive, most output of this process is not well transformed into machine-readable format. The limited availability of data has largely constrained the artificial intelligence (AI) system development to facilitate this literature reviewing in the regulatory process. In the past decade, AI has revolutionized the area of text mining as many deep learning approaches have been developed to search, annotate, and classify relevant documents. After the great advancement of AI algorithms, a lack of high-quality data instead of the algorithms has recently become the bottleneck of AI system development. Herein, we constructed two large benchmark datasets, Chlorine Efficacy dataset (CHE) and Chlorine Safety dataset (CHS), under a regulatory scenario that sought to assess the antiseptic efficacy and toxicity of chlorine. For each dataset, ∼10,000 scientific articles were initially collected, manually reviewed, and their relevance to the review task were labeled. To ensure high data quality, each paper was labeled by a consensus among multiple experienced reviewers. The overall relevance rate was 27.21% (2,663 of 9,788) for CHE and 7.50% (761 of 10,153) for CHS, respectively. Furthermore, the relevant articles were categorized into five subgroups based on the focus of their content. Next, we developed an attention-based classification language model using these two datasets. The proposed classification model yielded 0.857 and 0.908 of Area Under the Curve (AUC) for CHE and CHS dataset, respectively. This performance was significantly better than permutation test (p < 10E-9), demonstrating that the labeling processes were valid. To conclude, our datasets can be used as benchmark to develop AI systems, which can further facilitate the literature review process in regulatory science.

Systematic Identification of Molecular Targets and Pathways Related to Human Organ Level Toxicity.

The mechanisms leading to organ level toxicities are poorly understood. In this study, we applied an integrated approach to deduce the molecular targets and biological pathways involved in chemically induced toxicity for eight common human organ level toxicity end points (carcinogenicity, cardiotoxicity, developmental toxicity, hepatotoxicity, nephrotoxicity, neurotoxicity, reproductive toxicity, and skin toxicity). Integrated analysis of in vitro assay data, molecular targets and pathway annotations from the literature, and toxicity-molecular target associations derived from text mining, combined with machine learning techniques, were used to generate molecular targets for each of the organ level toxicity end points. A total of 1516 toxicity-related genes were identified and subsequently analyzed for biological pathway coverage, resulting in 206 significant pathways (p-value <0.05), ranging from 3 (e.g., developmental toxicity) to 101 (e.g., skin toxicity) for each toxicity end point. This study presents a systematic and comprehensive analysis of molecular targets and pathways related to various in vivo toxicity end points. These molecular targets and pathways could aid in understanding the biological mechanisms of toxicity and serve as a guide for the design of suitable in vitro assays for more efficient toxicity testing. In addition, these results are complementary to the existing adverse outcome pathway (AOP) framework and can be used to aid in the development of novel AOPs. Our results provide abundant testable hypotheses for further experimental validation.

Trade-off Predictivity and Explainability for Machine-Learning Powered Predictive Toxicology: An in-Depth Investigation with Tox21 Data Sets.

Selecting a model in predictive toxicology often involves a trade-off between prediction performance and explainability: should we sacrifice the model performance to gain explainability or vice versa. Here we present a comprehensive study to assess algorithm and feature influences on model performance in chemical toxicity research. We conducted over 5000 models for a Tox21 bioassay data set of 65 assays and ∼7600 compounds. Seven molecular representations as features and 12 modeling approaches varying in complexity and explainability were employed to systematically investigate the impact of various factors on model performance and explainability. We demonstrated that end points dictated a model's performance, regardless of the chosen modeling approach including deep learning and chemical features. Overall, more complex models such as (LS-)SVM and Random Forest performed marginally better than simpler models such as linear regression and KNN in the presented Tox21 data analysis. Since a simpler model with acceptable performance often also is easy to interpret for the Tox21 data set, it clearly was the preferred choice due to its better explainability. Given that each data set had its own error structure both for dependent and independent variables, we strongly recommend that it is important to conduct a systematic study with a broad range of model complexity and feature explainability to identify model balancing its predictivity and explainability.

Transcriptome analysis reveals lung-specific miRNAs associated with impaired mucociliary clearance induced by cigarette smoke in an in vitro human airway tissue model.

Exposure to cigarette smoke (CS) is strongly associated with impaired mucociliary clearance (MCC), which has been implicated in the pathogenesis of CS-induced respiratory diseases, such as chronic obstructive pulmonary diseases (COPD). In this study, we aimed to identify microRNAs (miRNAs) that are associated with impaired MCC caused by CS in an in vitro human air-liquid-interface (ALI) airway tissue model. ALI cultures were exposed to CS (diluted with 0.5 L/min, 1.0 L/min, and 4.0 L/min of clean air) from smoking five 3R4F University of Kentucky reference cigarettes under the International Organization for Standardization (ISO) machine smoking regimen, every other day for 1 week (a total of 3 days, 40 min/day). Transcriptome analyses of ALI cultures exposed to the high concentration of CS identified 5090 differentially expressed genes and 551 differentially expressed miRNAs after the third exposure. Genes involved in ciliary function and ciliogenesis were significantly perturbed by repeated CS exposures, leading to changes in cilia beating frequency and ciliary protein expression. In particular, a time-dependent decrease in the expression of miR-449a, a conserved miRNA highly enriched in ciliated airway epithelia and implicated in motile ciliogenesis, was observed in CS-exposed cultures. Similar alterations in miR-449a have been reported in smokers with COPD. Network analysis further indicates that downregulation of miR-449a by CS may derepress cell-cycle proteins, which, in turn, interferes with ciliogenesis. Investigating the effects of CS on transcriptome profile in human ALI cultures may provide not only mechanistic insights, but potential early biomarkers for CS exposure and harm.

Coordinated Regulation of UGT2B15 Expression by Long Noncoding RNA LINC00574 and hsa-miR-129-5p in HepaRG Cells.

Recent studies have shown that microRNAs and long noncoding RNAs (lncRNAs) regulate the expression of drug metabolizing enzymes (DMEs) in human hepatic cells and that a set of DMEs, including UDP glucuronosyltransferase (UGT) 2B15, is down-regulated dramatically in liver cells by toxic acetaminophen (APAP) concentrations. In this study we analyzed mRNA, microRNA, and lncRNA expression profiles in APAP-treated HepaRG cells to explore noncoding RNA-dependent regulation of DME expression. The expression of UGT2B15 and lncRNA LINC00574 was decreased in APAP-treated HepaRG cells. UGT2B15 levels were diminished by LINC00574 suppression using antisense oligonucleotides or small interfering RNA. Furthermore, we found that hsa-miR-129-5p suppressed LINC00574 and decreased UGT2B15 expression via LINC00574 in HepaRG cells. In conclusion, our results indicate that LINC00574 acts as an important regulator of UGT2B15 expression in human hepatic cells, providing experimental evidence and new clues to understand the role of cross-talk between noncoding RNAs. SIGNIFICANCE STATEMENT: We showed a molecular network that displays the cross-talk and consequences among mRNA, micro RNA, long noncoding RNA, and proteins in acetaminophen (APAP)-treated HepaRG cells. APAP treatment increased the level of hsa-miR-129-5p and decreased that of LINC00574, ultimately decreasing the production of UDP glucuronosyltransferase (UGT) 2B15. The proposed regulatory network suppresses UGT2B15 expression through interaction of hsa-miR-129-5p and LINC00574, which may be achieved potentially by recruiting RNA binding proteins.

Long noncoding RNA LINC00844-mediated molecular network regulates expression of drug metabolizing enzymes and nuclear receptors in human liver cells.

Noncoding RNAs, such as long noncoding RNAs (lncRNAs) and microRNAs (miRNAs), regulate gene expression in many physiological and pathological processes, including drug metabolism. Drug metabolizing enzymes (DMEs) are critical components in drug-induced liver toxicity. In this study, we used human hepatic HepaRG cells treated with 5 or 10 mM acetaminophen (APAP) as a model system and identified LINC00844 as a toxicity-responsive lncRNA. We analyzed the expression profiles of LINC00844 in different human tissues. In addition, we examined the correlations between the levels of LINC00844 and those of key DMEs and nuclear receptors (NRs) for APAP metabolism in humans. Our results showed that lncRNA LINC00844 is enriched in the liver and its expression correlates positively with mRNA levels of CYP3A4, CYP2E1, SULT2A1, pregnane X receptor (PXR), and hepatocyte nuclear factor (HNF) 4α. We demonstrated that LINC00844 regulates the expression of these five genes in HepaRG cells using gain- and loss-of-function assays. Further, we discovered that LINC00844 is localized predominantly in the cytoplasm and acts as an hsa-miR-486-5p sponge, via direct binding, to protect SULT2A1 from miRNA-mediated gene silencing. Our data also demonstrated a functional interaction between LINC00844 and hsa-miR-486-5p in regulating DME and NR expression in HepaRG cells and primary human hepatocytes. We depicted a LINC00844-mediated regulatory network that involves miRNA and NRs and influences DME expression in response to APAP toxicity.

DLI-IT: a deep learning approach to drug label identification through image and text embedding.

BACKGROUND: Drug label, or packaging insert play a significant role in all the operations from production through drug distribution channels to the end consumer. Image of the label also called Display Panel or label could be used to identify illegal, illicit, unapproved and potentially dangerous drugs. Due to the time-consuming process and high labor cost of investigation, an artificial intelligence-based deep learning model is necessary for fast and accurate identification of the drugs. METHODS: In addition to image-based identification technology, we take advantages of rich text information on the pharmaceutical package insert of drug label images. In this study, we developed the Drug Label Identification through Image and Text embedding model (DLI-IT) to model text-based patterns of historical data for detection of suspicious drugs. In DLI-IT, we first trained a Connectionist Text Proposal Network (CTPN) to crop the raw image into sub-images based on the text. The texts from the cropped sub-images are recognized independently through the Tesseract OCR Engine and combined as one document for each raw image. Finally, we applied universal sentence embedding to transform these documents into vectors and find the most similar reference images to the test image through the cosine similarity. RESULTS: We trained the DLI-IT model on 1749 opioid and 2365 non-opioid drug label images. The model was then tested on 300 external opioid drug label images, the result demonstrated our model achieves up-to 88% of the precision in drug label identification, which outperforms previous image-based or text-based identification method by up-to 35% improvement. CONCLUSION: To conclude, by combining Image and Text embedding analysis under deep learning framework, our DLI-IT approach achieved a competitive performance in advancing drug label identification.

Study of serious adverse drug reactions using FDA-approved drug labeling and MedDRA.

BACKGROUND: Adverse Drug Reactions (ADRs) are of great public health concern. FDA-approved drug labeling summarizes ADRs of a drug product mainly in three sections, i.e., Boxed Warning (BW), Warnings and Precautions (WP), and Adverse Reactions (AR), where the severity of ADRs are intended to decrease in the order of BW > WP > AR. Several reported studies have extracted ADRs from labeling documents, but most, if not all, did not discriminate the severity of the ADRs by the different labeling sections. Such a practice could overstate or underestimate the impact of certain ADRs to the public health. In this study, we applied the Medical Dictionary for Regulatory Activities (MedDRA) to drug labeling and systematically analyzed and compared the ADRs from the three labeling sections with a specific emphasis on analyzing serious ADRs presented in BW, which is of most drug safety concern. RESULTS: This study investigated New Drug Application (NDA) labeling documents for 1164 single-ingredient drugs using Oracle Text search to extract MedDRA terms. We found that only a small portion of MedDRA Preferred Terms (PTs), 3819 out of 21,920 or 17.42%, were observed in a whole set of documents. In detail, 466/3819 (12.0%) PTs were in BW, 2023/3819 (53.0%) were in WP, and 2961/3819 (77.5%) were in AR sections. We also found a higher overlap of top 20 occurring BW PTs with WP sections compared to AR sections. Within the MedDRA System Organ Class levels, serious ADRs (sADRs) from BW were prevalent in Nervous System disorders and Vascular disorders. A Hierarchical Cluster Analysis (HCA) revealed that drugs within the same therapeutic category shared the same ADR patterns in BW (e.g., nervous system drug class is highly associated with drug abuse terms such as dependence, substance abuse, and respiratory depression). CONCLUSIONS: This study demonstrated that combining MedDRA standard terminologies with data mining techniques facilitated computer-aided ADR analysis of drug labeling. We also highlighted the importance of labeling sections that differ in seriousness and application in drug safety. Using sADRs primarily related to BW sections, we illustrated a prototype approach for computer-aided ADR monitoring and studies which can be applied to other public health documents.

HetEnc: a deep learning predictive model for multi-type biological dataset.

BACKGROUND: Researchers today are generating unprecedented amounts of biological data. One trend in current biological research is integrated analysis with multi-platform data. Effective integration of multi-platform data into the solution of a single or multi-task classification problem; however, is critical and challenging. In this study, we proposed HetEnc, a novel deep learning-based approach, for information domain separation. RESULTS: HetEnc includes both an unsupervised feature representation module and a supervised neural network module to handle multi-platform gene expression datasets. It first constructs three different encoding networks to represent the original gene expression data using high-level abstracted features. A six-layer fully-connected feed-forward neural network is then trained using these abstracted features for each targeted endpoint. We applied HetEnc to the SEQC neuroblastoma dataset to demonstrate that it outperforms other machine learning approaches. Although we used multi-platform data in feature abstraction and model training, HetEnc does not need multi-platform data for prediction, enabling a broader application of the trained model by reducing the cost of gene expression profiling for new samples to a single platform. Thus, HetEnc provides a new solution to integrated gene expression analysis, accelerating modern biological research.

Novel reference genes in colorectal cancer identify a distinct subset of high stage tumors and their associated histologically normal colonic tissues.

BACKGROUND: Reference genes are often interchangeably called housekeeping genes due to 1) the essential cellular functions their proteins provide and 2) their constitutive expression across a range of normal and pathophysiological conditions. However, given the proliferative drive of malignant cells, many reference genes such as beta-actin (ACTB) and glyceraldehyde-3-phosphate-dehydrogenase (GAPDH) which play critical roles in cell membrane organization and glycolysis, may be dysregulated in tumors versus their corresponding normal controls METHODS: Because Next Generation Sequencing (NGS) technology has several advantages over hybridization-based technologies, such as independent detection and quantitation of transcription levels, greater sensitivity, and increased dynamic range, we evaluated colorectal cancers (CRC) and their histologically normal tissue counterparts by NGS to evaluate the expression of 21 "classical" reference genes used as normalization standards for PCR based methods. Seventy-nine paired tissue samples of CRC and their patient matched healthy colonic tissues were subjected to NGS analysis of their mRNAs. RESULTS: We affirmed that 17 out of 21 classical reference genes had upregulated expression in tumors compared to normal colonic epithelial tissue and dramatically so in some cases. Indeed, tumors were distinguished from normal controls in both unsupervised hierarchical clustering analyses (HCA) and principal component analyses (PCA). We then identified 42 novel potential reference genes with minimal coefficients of variation (CV) across 79 CRC tumor pairs. Though largely consistently expressed across tumors and normal control tissues, a subset of high stage tumors (HSTs) as well as some normal tissue samples (HSNs) located adjacent to these HSTs demonstrated dysregulated expression, thus identifying a subset of tumors with a potentially distinct and aggressive biological profile. CONCLUSION: While classical CRC reference genes were found to be differentially expressed between tumors and normal controls, novel reference genes, identified via NGS, were more consistently expressed across malignant and normal colonic tissues. Nonetheless, a subset of HST had profound dysregulation of such genes as did many of the histologically normal tissues adjacent to such HSTs, indicating that the HSTs so distinguished may have unique biological properties and that their histologically normal tissues likely harbor a small population of microscopically undetected but metabolically active tumors.

Technical advance in targeted NGS analysis enables identification of lung cancer risk-associated low frequency TP53, PIK3CA, and BRAF mutations in airway epithelial cells.

BACKGROUND: Standardized Nucleic Acid Quantification for SEQuencing (SNAQ-SEQ) is a novel method that utilizes synthetic DNA internal standards spiked into each sample prior to next generation sequencing (NGS) library preparation. This method was applied to analysis of normal appearing airway epithelial cells (AEC) obtained by bronchoscopy in an effort to define a somatic mutation field effect associated with lung cancer risk. There is a need for biomarkers that reliably detect those at highest lung cancer risk, thereby enabling more effective screening by annual low dose CT. The purpose of this study was to test the hypothesis that lung cancer risk is characterized by increased prevalence of low variant allele frequency (VAF) somatic mutations in lung cancer driver genes in AEC. METHODS: Synthetic DNA internal standards (IS) were prepared for 11 lung cancer driver genes and mixed with each AEC genomic (g) DNA specimen prior to competitive multiplex PCR amplicon NGS library preparation. A custom Perl script was developed to separate IS reads and respective specimen gDNA reads from each target into separate files for parallel variant frequency analysis. This approach identified nucleotide-specific sequencing error and enabled reliable detection of specimen mutations with VAF as low as 5 × 10- 4 (0.05%). This method was applied in a retrospective case-control study of AEC specimens collected by bronchoscopic brush biopsy from the normal airways of 19 subjects, including eleven lung cancer cases and eight non-cancer controls, and the association of lung cancer risk with AEC driver gene mutations was tested. RESULTS: TP53 mutations with 0.05-1.0% VAF were more prevalent (p < 0.05) and also enriched for tobacco smoke and age-associated mutation signatures in normal AEC from lung cancer cases compared to non-cancer controls matched for smoking and age. Further, PIK3CA and BRAF mutations in this VAF range were identified in AEC from cases but not controls. CONCLUSIONS: Application of SNAQ-SEQ to measure mutations in the 0.05-1.0% VAF range enabled identification of an AEC somatic mutation field of injury associated with lung cancer risk. A biomarker comprising TP53, PIK3CA, and BRAF somatic mutations may better stratify individuals for optimal lung cancer screening and prevention outcomes.

Multiple microRNAs function as self-protective modules in acetaminophen-induced hepatotoxicity in humans.

Acetaminophen (APAP) overdose is the leading cause of acute liver failure. Yet the mechanisms underlying adaptive tolerance toward APAP-induced liver injury are not fully understood. To better understand molecular mechanisms contributing to adaptive tolerance to APAP is an underpinning foundation for APAP-related precision medicine. In the current study, the mRNA and microRNA (miRNA) expression profiles derived from next generation sequencing data for APAP-treated (5 and 10 mM) HepaRG cells and controls were analyzed systematically. Putative miRNAs targeting key dysregulated genes involved in APAP hepatotoxicity were selected using in silico prediction algorithms, un-biased gene ontology, and network analyses. Luciferase reporter assays, RNA electrophoresis mobility shift assays, and miRNA pull-down assays were performed to investigate the role of miRNAs affecting the expression of dysregulated genes. Levels of selected miRNAs were measured in serum samples obtained from children with APAP overdose (58.6-559.4 mg/kg) and from healthy controls. As results, 2758 differentially expressed genes and 47 miRNAs were identified. Four of these miRNAs (hsa-miR-224-5p, hsa-miR-320a, hsa-miR-449a, and hsa-miR-877-5p) suppressed drug metabolizing enzyme (DME) levels involved in APAP-induced liver injury by downregulating HNF1A, HNF4A and NR1I2 expression. Exogenous transfection of these miRNAs into HepaRG cells effectively rescued them from APAP toxicity, as indicated by decreased alanine aminotransferase levels. Importantly, hsa-miR-320a and hsa-miR-877-5p levels were significantly elevated in serum samples obtained from children with APAP overdose compared to health controls. Collectively, these data indicate that hsa-miR-224-5p, hsa-miR-320a, hsa-miR-449a, and hsa-miR-877-5p suppress DME expression involved in APAP-induced hepatotoxicity and they contribute to an adaptive response in hepatocytes.

Integrating Drug's Mode of Action into Quantitative Structure-Activity Relationships for Improved Prediction of Drug-Induced Liver Injury.

Drug-induced liver injury (DILI) is complex in mechanism. Different drugs could undergo different mechanisms but result in the same DILI type, while the same drug could lead to different DILI types via different mechanisms. Therefore, predicting a drug's potential for DILI should take its underlying mechanisms into consideration. To achieve that, we constructed a novel approach by incorporating the drug's Mode of Action (MOA) into Quantitative Structure-Activity Relationship (QSAR) modeling. This MOA-DILI approach was examined using a data set of 333 drugs. The drugs were first grouped according to their MOA profiles (positive or negative in each MOA) based on the Tox21 qHTS assays. QSAR models for individual MOA assays were developed and subsequently combined to obtain the MOA-DILI model. A hold-out testing strategy (222 drugs for training and 111 drugs as a test set) was employed, which yielded a predictive accuracy of 0.711. The MOA-DILI model was directly compared with the standard QSAR approach using the same hold-out strategy, and the QSAR model yielded an accuracy of 0.662. To minimize the random chance in splitting training/test sets, the hold-out testing process was repeated 1000 times, and the observed difference in prediction accuracy between MOA-DILI and QSARs was statistically significant (P value <0.0001). Out of 17 MOAs used, four assays (i.e., antioxidant response elements, PPAR-gamma, estrogen receptor, and thyroid receptor assays) contributed most to the improved prediction of the MOA-DILI model over QSARs. In conclusion, the MOA-DILI approach has the potential to significantly improve predictive outcomes and to reveal complex relationships between MOAs and DILI, all of which would be helpful in developing DILI predictive models in drug screening and for risk assessment of industrial chemicals.

LTMap: a web server for assessing the potential liver toxicity by genome-wide transcriptional expression data.

Toxicogenomics (TGx) has played a significant role in mechanistic research related with hepatotoxicity as well as liver toxicity prediction. Currently, several large-scale preclinical TGx data sets were made freely accessible to the public, such as Open TG-GATEs. With the availability of a sufficient amount of microarray data, it is important to integrate this information to provide new insights into the risk assessment of potential drug-induced liver toxicity. Here we developed a web server for evaluating the potential liver toxicity based on genome-wide transcriptomics data, namely LTMap. In LTMap, researchers could compare signatures of query compounds against a pregenerated signature database of 20 123 Affymetrix arrays associated with about 170 compounds retrieved from the largest public toxicogenomics data set Open TG-GATEs. Results from this comparison may lead to the unexpected discovery of similar toxicological responses between chemicals. We validated our computational approach for similarity comparison using three example drugs. Our successful applications of LTMap in these case studies demonstrated its utility in revealing the connection of chemicals according to similar toxicological behaviors. Furthermore, a user-friendly web interface is provided by LTMap to browse and search toxicogenomics data (http://tcm.zju.edu.cn/ltmap).

PERform: assessing model performance with predictivity and explainability readiness formula.

In the rapidly evolving field of artificial intelligence (AI), explainability has been traditionally assessed in a post-modeling process and is often subjective. In contrary, many quantitative metrics have been routinely used to assess a model's performance. We proposed a unified formular named PERForm, by incorporating explainability as a weight into the existing statistical metrics to provide an integrated and quantitative measure of both predictivity and explainability to guide model selection, application, and evaluation. PERForm was designed as a generic formula and can be applied to any data types. We applied PERForm on a range of diverse datasets, including DILIst, Tox21, and three MAQC-II benchmark datasets, using various modeling algorithms to predict a total of 73 distinct endpoints. For example, AdaBoost algorithms exhibited superior performance (PERForm AUC for AdaBoost is 0.129 where Linear regression is 0) in DILIst prediction, where linear regression outperformed other models in the majority of Tox21 endpoints (PERForm AUC for linear regression is 0.301 where AdaBoost is 0.283 in average). This research marks a significant step toward comprehensively evaluating the utility of an AI model to advance transparency and interpretability, where the tradeoff between a model's performance and its interpretability can have profound implications.

Measurement and Mitigation of Bias in Artificial Intelligence: A Narrative Literature Review for Regulatory Science.

Artificial intelligence (AI) is increasingly being used in decision making across various industries, including the public health arena. Bias in any decision-making process can significantly skew outcomes, and AI systems have been shown to exhibit biases at times. The potential for AI systems to perpetuate and even amplify biases is a growing concern. Bias, as used in this paper, refers to the tendency toward a particular characteristic or behavior, and thus, a biased AI system is one that shows biased associations entities. In this literature review, we examine the current state of research on AI bias, including its sources, as well as the methods for measuring, benchmarking, and mitigating it. We also examine the biases and methods of mitigation specifically relevant to the healthcare field and offer a perspective on bias measurement and mitigation in regulatory science decision making.

Text summarization with ChatGPT for drug labeling documents.

Text summarization is crucial in scientific research, drug discovery and development, regulatory review, and more. This task demands domain expertise, language proficiency, semantic prowess, and conceptual skill. The recent advent of large language models (LLMs), such as ChatGPT, offers unprecedented opportunities to automate this process. We compared ChatGPT-generated summaries with those produced by human experts using FDA drug labeling documents. The labeling contains summaries of key labeling sections, making them an ideal human benchmark to evaluate ChatGPT's summarization capabilities. Analyzing >14000 summaries, we observed that ChatGPT-generated summaries closely resembled those generated by human experts. Importantly, ChatGPT exhibited even greater similarity when summarizing drug safety information. These findings highlight ChatGPT's potential to accelerate work in critical areas, including drug safety.