Reproducibility of organ-level effects in repeat dose animal studies.

This work estimates benchmarks for new approach method (NAM) performance in predicting organ-level effects in repeat dose studies of adult animals based on variability in replicate animal studies. Treatment-related effect values from the Toxicity Reference database (v2.1) for weight, gross, or histopathological changes in the adrenal gland, liver, kidney, spleen, stomach, and thyroid were used. Rates of chemical concordance among organ-level findings in replicate studies, defined by repeated chemical only, chemical and species, or chemical and study type, were calculated. Concordance was 39 - 88%, depending on organ, and was highest within species. Variance in treatment-related effect values, including lowest effect level (LEL) values and benchmark dose (BMD) values when available, was calculated by organ. Multilinear regression modeling, using study descriptors of organ-level effect values as covariates, was used to estimate total variance, mean square error (MSE), and root residual mean square error (RMSE). MSE values, interpreted as estimates of unexplained variance, suggest study descriptors accounted for 52-69% of total variance in organ-level LELs. RMSE ranged from 0.41 - 0.68 log10-mg/kg/day. Differences between organ-level effects from chronic (CHR) and subchronic (SUB) dosing regimens were also quantified. Odds ratios indicated CHR organ effects were unlikely if the SUB study was negative. Mean differences of CHR - SUB organ-level LELs ranged from -0.38 to -0.19 log10 mg/kg/day; the magnitudes of these mean differences were less than RMSE for replicate studies. Finally, in vitro to in vivo extrapolation (IVIVE) was employed to compare bioactive concentrations from in vitro NAMs for kidney and liver to LELs. The observed mean difference between LELs and mean IVIVE dose predictions approached 0.5 log10-mg/kg/day, but differences by chemical ranged widely. Overall, variability in repeat dose organ-level effects suggests expectations for quantitative accuracy of NAM prediction of LELs should be at least ± 1 log10-mg/kg/day, with qualitative accuracy not exceeding 70%.

ToxRefDB v2.1: update to curated in vivo study data in the Toxicity Reference Database.

The Toxicity Reference Database (ToxRefDB) contains in vivo study data from over 5,900 guideline or guideline-like studies for over 1,100 chemicals. The database includes information regarding study design, chemical treatment, dosing, treatment group parameters, treatment-related (significantly different from control) and critical (adverse) effects, guided by a controlled effect vocabulary, as well as endpoint testing status according to health effects guideline requirements. ToxRefDB v2.1 is an update to address a compilation error found in ToxRefDB v2.0 that resulted in some effects being inadvertently omitted from the database. Though effect data has been recovered, no new studies were added. The recovered data improves the utility of ToxRefDB as a resource for curated legacy in vivo information, which enhances scientific confidence in vitro high-throughput screening of chemicals and supports retrospective and predictive toxicology applications for which outcomes in traditional regulatory toxicology studies serve as reference information.

Synthetic Health Data Can Augment Community Research Efforts to Better Inform the Public During Emerging Pandemics.

The COVID-19 pandemic had disproportionate effects on the Veteran population due to the increased prevalence of medical and environmental risk factors. Synthetic electronic health record (EHR) data can help meet the acute need for Veteran population-specific predictive modeling efforts by avoiding the strict barriers to access, currently present within Veteran Health Administration (VHA) datasets. The U.S. Food and Drug Administration (FDA) and the VHA launched the precisionFDA COVID-19 Risk Factor Modeling Challenge to develop COVID-19 diagnostic and prognostic models; identify Veteran population-specific risk factors; and test the usefulness of synthetic data as a substitute for real data. The use of synthetic data boosted challenge participation by providing a dataset that was accessible to all competitors. Models trained on synthetic data showed similar but systematically inflated model performance metrics to those trained on real data. The important risk factors identified in the synthetic data largely overlapped with those identified from the real data, and both sets of risk factors were validated in the literature. Tradeoffs exist between synthetic data generation approaches based on whether a real EHR dataset is required as input. Synthetic data generated directly from real EHR input will more closely align with the characteristics of the relevant cohort. This work shows that synthetic EHR data will have practical value to the Veterans' health research community for the foreseeable future.

Application of evidence-based methods to construct mechanism-driven chemical assessment frameworks.

The workshop titled "Application of evidence-based methods to construct mechanism-driven chemical assessment frameworks" was co-organized by the Evidence-based Toxicology Collaboration and the European Food Safety Authority (EFSA) and hosted by EFSA at its headquarters in Parma, Italy on October 2 and 3, 2019. The goal was to explore integration of systematic review with mechanistic evidence evaluation. Participants were invited to work on concrete products to advance the exploration of how evidence-based approaches can support the development and application of adverse outcome pathways (AOP) in chemical risk assessment. The workshop discussions were centered around three related themes: 1) assessing certainty in AOPs, 2) literature-based AOP development, and 3) integrating certainty in AOPs and non-animal evidence into decision frameworks. Several challenges, mostly related to methodology, were identified and largely determined the workshop recommendations. The workshop recommendations included the comparison and potential alignment of processes used to develop AOP and systematic review methodology, including the translation of vocabulary of evidence-based methods to AOP and vice versa, the development and improvement of evidence mapping and text mining methods and tools, as well as a call for a fundamental change in chemical risk and uncertainty assessment methodology if to be conducted based on AOPs and new approach methodologies (NAM). The usefulness of evidence-based approaches for mechanism-based chemical risk assessments was stressed, particularly the potential contribution of the rigor and transparency inherent to such approaches in building stakeholders' trust for implementation of NAM evidence and AOPs into chemical risk assessment.

Knowledge Organization Systems for Systematic Chemical Assessments.

BACKGROUND: Although the implementation of systematic review and evidence mapping methods stands to improve the transparency and accuracy of chemical assessments, they also accentuate the challenges that assessors face in ensuring they have located and included all the evidence that is relevant to evaluating the potential health effects an exposure might be causing. This challenge of information retrieval can be characterized in terms of "semantic" and "conceptual" factors that render chemical assessments vulnerable to the streetlight effect. OBJECTIVES: This commentary presents how controlled vocabularies, thesauruses, and ontologies contribute to overcoming the streetlight effect in information retrieval, making up the key components of Knowledge Organization Systems (KOSs) that enable more systematic access to assessment-relevant information than is currently achievable. The concept of Adverse Outcome Pathways is used to illustrate what a general KOS for use in chemical assessment could look like. DISCUSSION: Ontologies are an underexploited element of effective knowledge organization in the environmental health sciences. Agreeing on and implementing ontologies in chemical assessment is a complex but tractable process with four fundamental steps. Successful implementation of ontologies would not only make currently fragmented information about health risks from chemical exposures vastly more accessible, it could ultimately enable computational methods for chemical assessment that can take advantage of the full richness of data described in natural language in primary studies. https://doi.org/10.1289/EHP6994.

Variability in in vivo studies: Defining the upper limit of performance for predictions of systemic effect levels.

New approach methodologies (NAMs) for chemical hazard assessment are often evaluated via comparison to animal studies; however, variability in animal study data limits NAM accuracy. The US EPA Toxicity Reference Database (ToxRefDB) enables consideration of variability in effect levels, including the lowest effect level (LEL) for a treatment-related effect and the lowest observable adverse effect level (LOAEL) defined by expert review, from subacute, subchronic, chronic, multi-generation reproductive, and developmental toxicity studies. The objectives of this work were to quantify the variance within systemic LEL and LOAEL values, defined as potency values for effects in adult or parental animals only, and to estimate the upper limit of NAM prediction accuracy. Multiple linear regression (MLR) and augmented cell means (ACM) models were used to quantify the total variance, and the fraction of variance in systemic LEL and LOAEL values explained by available study descriptors (e.g., administration route, study type). The MLR approach considered each study descriptor as an independent contributor to variance, whereas the ACM approach combined categorical descriptors into cells to define replicates. Using these approaches, total variance in systemic LEL and LOAEL values (in log10-mg/kg/day units) ranged from 0.74 to 0.92. Unexplained variance in LEL and LOAEL values, approximated by the residual mean square error (MSE), ranged from 0.20-0.39. Considering subchronic, chronic, or developmental study designs separately resulted in similar values. Based on the relationship between MSE and R-squared for goodness-of-fit, the maximal R-squared may approach 55 to 73% for a NAM-based predictive model of systemic toxicity using these data as reference. The root mean square error (RMSE) ranged from 0.47 to 0.63 log10-mg/kg/day, depending on dataset and regression approach, suggesting that a two-sided minimum prediction interval for systemic effect levels may have a width of 58 to 284-fold. These findings suggest quantitative considerations for building scientific confidence in NAM-based systemic toxicity predictions.

Python BMDS: A Python interface library and web application for the canonical EPA dose-response modeling software.

Several primary sources of publicly available, quantitative dose-response data from traditional toxicology study designs relevant to predictive toxicology applications are now available, including the redeveloped U.S. Environmental Protection Agency's Toxicity Reference Database (ToxRefDB v2.0), the Health Assessment Workspace Collaborative (HAWC), and the National Toxicology Program's Chemical Program's Chemical Effects in Biological Systems (CEBS). These resources provide effect level information but modeling these data to a curve may be more informative for predictive toxicology applications. Benchmark Dose Software (BMDS) has been recognized broadly and used for regulatory applications at multiple agencies. However, the current BMDS software was not amenable to modeling large datasets. Herein we describe development and use of a Python package that implements a wrapper around BMDS, a software that requires manual input in the dose-response modeling process (i.e., best-fitting model-selection, reporting, and dose-dropping). In the Python BMDS, users can select the BMDS version, customize model recommendation logic, and export summaries of the resultant BMDS output. Further, using the Python interface, a web-based application programming interface (API) has been developed for easy integration into other software systems, pipelines, or databases. Software utility was demonstrated via modeling nearly 28,000 datasets in ToxRefDB v2.0, re-creation of an existing, published large-scale analysis, and demonstration of usage in software such as CEBS and HAWC. Python BMDS enables rapid-batch processing of dose-response datasets using a modeling software with broad acceptance in the toxicology community, thereby providing an important tool for leveraging the publicly available quantitative toxicology data in a reproducible manner.

Progress in data interoperability to support computational toxicology and chemical safety evaluation.

New approach methodologies (NAMs) in chemical safety evaluation are being explored to address the current public health implications of human environmental exposures to chemicals with limited or no data for assessment. For over a decade since a push toward "Toxicity Testing in the 21st Century," the field has focused on massive data generation efforts to inform computational approaches for preliminary hazard identification, adverse outcome pathways that link molecular initiating events and key events to apical outcomes, and high-throughput approaches to risk-based ratios of bioactivity and exposure to inform relative priority and safety assessment. Projects like the interagency Tox21 program and the US EPA ToxCast program have generated dose-response information on thousands of chemicals, identified and aggregated information from legacy systems, and created tools for access and analysis. The resulting information has been used to develop computational models as viable options for regulatory applications. This progress has introduced challenges in data management that are new, but not unique, to toxicology. Some of the key questions require critical thinking and solutions to promote semantic interoperability, including: (1) identification of bioactivity information from NAMs that might be related to a biological process; (2) identification of legacy hazard information that might be related to a key event or apical outcomes of interest; and, (3) integration of these NAM and traditional data for computational modeling and prediction of complex apical outcomes such as carcinogenesis. This work reviews a number of toxicology-related efforts specifically related to bioactivity and toxicological data interoperability based on the goals established by Findable, Accessible, Interoperable, and Reusable (FAIR) Data Principles. These efforts are essential to enable better integration of NAM and traditional toxicology information to support data-driven toxicology applications.

ToxRefDB version 2.0: Improved utility for predictive and retrospective toxicology analyses.

The Toxicity Reference Database (ToxRefDB) structures information from over 5000 in vivo toxicity studies, conducted largely to guidelines or specifications from the US Environmental Protection Agency and the National Toxicology Program, into a public resource for training and validation of predictive models. Herein, ToxRefDB version 2.0 (ToxRefDBv2) development is described. Endpoints were annotated (e.g. required, not required) according to guidelines for subacute, subchronic, chronic, developmental, and multigenerational reproductive designs, distinguishing negative responses from untested. Quantitative data were extracted, and dose-response modeling for nearly 28,000 datasets from nearly 400 endpoints using Benchmark Dose (BMD) Modeling Software were generated and stored. Implementation of controlled vocabulary improved data quality; standardization to guideline requirements and cross-referencing with United Medical Language System (UMLS) connects ToxRefDBv2 observations to vocabularies linked to UMLS, including PubMed medical subject headings. ToxRefDBv2 allows for increased connections to other resources and has greatly enhanced quantitative and qualitative utility for predictive toxicology.

Novel application of normalized pointwise mutual information (NPMI) to mine biomedical literature for gene sets associated with disease: use case in breast carcinogenesis.

Advances in technology within biomedical sciences have led to an inundation of data across many fields, raising new challenges in how best to integrate and analyze these resources. For example, rapid chemical screening programs like the US Environmental Protection Agency's ToxCast and the collaborative effort, Tox21, have produced massive amounts of information on putative chemical mechanisms where assay targets are identified as genes; however, systematically linking these hypothesized mechanisms with in vivo toxicity endpoints like disease outcomes remains problematic. Herein we present a novel use of normalized pointwise mutual information (NPMI) to mine biomedical literature for gene associations with biological concepts as represented by Medical Subject Headings (MeSH terms) in PubMed. Resources that tag genes to articles were integrated, then cross-species orthologs were identified using UniRef50 clusters. MeSH term frequency was normalized to reflect the MeSH tree structure, and then the resulting GeneID-MeSH associations were ranked using NPMI. The resulting network, called Entity MeSH Co-occurrence Network (EMCON), is a scalable resource for the identification and ranking of genes for a given topic of interest. The utility of EMCON was evaluated with the use case of breast carcinogenesis. Topics relevant to breast carcinogenesis were used to query EMCON and retrieve genes important to each topic. A breast cancer gene set was compiled through expert literature review (ELR) to assess performance of the search results. We found that the results from EMCON ranked the breast cancer genes from ELR higher than randomly selected genes with a recall of 0.98. Precision of the top five genes for selected topics was calculated as 0.87. This work demonstrates that EMCON can be used to link in vitro results to possible biological outcomes, thus aiding in generation of testable hypotheses for furthering understanding of biological function and the contribution of chemical exposures to disease.

BCScreen: A gene panel to test for breast carcinogenesis in chemical safety screening.

Targeted gene lists have been used in clinical settings to specify breast tumor type, and to predict breast cancer prognosis and response to treatment. Separately, panels have been curated to predict systemic toxicity and xenoestrogen activity as a part of chemical screening strategies. However, currently available panels do not specifically target biological processes relevant to breast development and carcinogenesis. We have developed a gene panel called the Breast Carcinogen Screen (BCScreen) as a tool to identify potential breast carcinogens and characterize mechanisms of toxicity. First, we used four seminal reviews to identify 14 key characteristics of breast carcinogenesis, such as apoptosis, immunomodulation, and genotoxicity. Then, using a hybrid data and knowledge-driven framework, we systematically combined information from whole transcriptome data from genomic databases, biomedical literature, the CTD chemical-gene interaction database, and primary literature review to generate a panel of 500 genes relevant to breast carcinogenesis. We used normalized pointwise mutual information (NPMI) to rank genes that frequently co-occurred with key characteristics in biomedical literature. We found that many genes identified for BCScreen were not included in prognostic breast cancer or systemic toxicity panels. For example, more than half of BCScreen genes were not included in the Tox21 S1500+ general toxicity gene list. Of the 230 that did overlap between the two panels, representation varied across characteristics of carcinogenesis ranging from 21% for genes associated with epigenetics to 82% for genes associated with xenobiotic metabolism. Enrichment analysis of BCScreen identified pathways and processes including response to steroid hormones, cancer, cell cycle, apoptosis, DNA damage and breast cancer. The biologically-based systematic approach to gene prioritization demonstrated here provides a flexible framework for creating disease-focused gene panels to support discovery related to etiology. With validation, BCScreen may also be useful for toxicological screening relevant to breast carcinogenesis.