Evaluation of polygenic risk models using multiple performance measures: a critical assessment of discordant results.

PURPOSE: The area under the receiver operating characteristic curve (AUC) is commonly used for evaluating the improvement of polygenic risk models and increasingly assessed together with the net reclassification improvement (NRI) and integrated discrimination improvement (IDI). We evaluated how researchers described and interpreted AUC, NRI, and IDI when simultaneously assessed. METHODS: We reviewed how researchers described definitions of AUC, NRI, and IDI and how they computed each metric. Next, we reviewed how the increment in AUC, NRI, and IDI were interpreted, and how the overall conclusion about the improvement of the risk model was reached. RESULTS: AUC, NRI, and IDI were correctly defined in 63, 70, and 0% of the articles. All statistically significant values and almost half of the nonsignificant were interpreted as indicative of improvement, irrespective of the values of the metrics. Also, small, nonsignificant changes in the AUC were interpreted as indication of improvement when NRI and IDI were statistically significant. CONCLUSION: Researchers have insufficient knowledge about how to interpret the various metrics for the assessment of the predictive performance of polygenic risk models and rely on the statistical significance for their interpretation. A better understanding is needed to achieve more meaningful interpretation of polygenic prediction studies.

Developmental immunotoxicity of chemicals in rodents and its possible regulatory impact.

Around 25% of the children in developed countries are affected with immune-based diseases. Juvenile onset diseases such as allergic, inflammatory and autoimmune diseases have shown increasing prevalences in the last decades. The role of chemical exposures in these phenomena is unclear. It is thought that the developmental immune system is more susceptible to toxicants than the mature situation. Developmental immunotoxicity (DIT) testing is nowadays not or minimally included in regulatory toxicology requirements. We reviewed whether developmental immune parameters in rodents would provide relatively sensitive endpoints of toxicity, whose inclusion in regulatory toxicity testing might improve hazard identification and risk assessment of chemicals. For each of the nine reviewed toxicants, the developing immune system was found to be at least as sensitive or more sensitive than the general (developmental) toxicity parameters. Functional immune (antigen-challenged) parameters appear more affected than structural (non-challenged) immune parameters. Especially, antibody responses to immune challenges with keyhole limpet hemocyanine or sheep red blood cells and delayed-type hypersensitivity responses appear to provide sensitive parameters of developmental immune toxicity. Comparison with current tolerable daily intakes (TDI) and their underlying overall no observed adverse effect levels showed that for some of the compounds reviewed, the TDI may need reconsideration based on developmental immune parameters. From these data, it can be concluded that the developing immune system is very sensitive to the disruption of toxicants independent of study design. Consideration of including functional DIT parameters in current hazard identification guidelines and wider application of relevant study protocols is warranted.

Developmental immunotoxicity testing of 4-methyl anisole.

The developmental immunotoxicity of 4-methyl anisole (4MA) was investigated in the rat. Four study designs were used, with either premating or post-weaning onset of exposure, continued to postnatal day 50, and with or without additional oral gavage of pups from postnatal day 10 onward. Reduced litter size (benchmark dose lower confidence limit (BMDL) 80mg/kg bw/day) was the most sensitive developmental parameter, with pup relative organ weight effects observed at similar BMDLs, in the absence of maternal toxicity. Eosinophil numbers were reduced at lower doses (BMDL 16mg/kg bw/day). KLH challenge resulted in increased IL-13 and TNF-α responses, and variably reduced IgG production (BMDL 27mg/kg bw/day). T4 levels were reduced by 11% at maximum with a BMDL of 73mg/kg bw/day. Differences between exposure cohorts were limited and were considered to be without biological significance. This study shows that 4MA induces developmental immunotoxicity at doses below those inducing developmental and general toxicity. These observations being independent of the study designs applied suggest that the post-weaning period, included in all designs, is the most relevant sensitive period for inducing 4MA mediated developmental immunotoxicity. Moreover, this study stresses the importance of including developmental immunotoxicity testing by default in regulatory toxicology.

Implementation of toxicokinetics in toxicity studies--Toxicokinetics of 4-methylanisole and its metabolites in juvenile and adult rats.

The current risk assessment of compounds is generally based on external exposure and effect relationships. External doses are often not representative for internal exposure concentrations. The aim of this study was to show how the implementation of toxicokinetics in a scheduled toxicity study contributes to improved data interpretation without additional use of animals and to the three goals of the 3R principles for animal testing. Toxicokinetic analyses were implemented in a rat developmental immunotoxicity study with 4-methylanisole without interfering with the outcome of the study and without the use of additional animals. 4-Methylanisole and its metabolites were analysed in plasma of adult rats and in pups at postnatal day 10. 4-Methylanisole has a short half-life in adult animals and the plasma concentrations increased more than proportional with increasing dose. The metabolic profile appeared to be different at low dose as compared to high dose. This information on the dose-proportionality of the internal exposure is crucial for the interpretation of the toxicity data and helps to identify the toxic agent and the appropriate dose metric. The metabolism was similar in adult and juvenile animals. Large inter-individual variability in adult animals, as observed for 4-methylanisole, may hamper dose-response analyses of the results. In addition, 4-metylanisole was excreted via milk, but concentrations in the juvenile animals appeared to be 20- to 100-fold lower than via direct gavage exposure. The toxicokinetic parameters support the data interpretation, among others by providing better insight into internal exposures. Subsequently, it will help to prevent testing of irrelevant exposure scenarios and exposure concentrations. Overall, implementation of kinetics with limited effort provides useful information to support the interpretation of toxicological data and can contribute to reduction and refinement of animal testing.

Developmental immunotoxicity of ethanol in an extended one-generation reproductive toxicity study.

The susceptibility of developing immune system to chemical disruption warrants the assessment of immune parameters in reproductive and developmental testing protocols. In this study, a wide range of immune endpoints was included in an extended one-generation reproduction toxicity study (EOGRTS) design to determine the relative sensitivity of immune and developmental parameters to ethanol (EtOH), a well-known developmental toxicant with immunomodulatory properties. Adult Wistar rats were exposed to EtOH via drinking water (0, 1.5, 4, 6.5, 9, 11.5 and 14 % (w/v EtOH)) during premating, mating, gestation and lactation and continuation of exposure of the F(1) from weaning until killed. Immune assessments were performed at postnatal days (PNDs) 21, 42 and 70. Keyhole limpet hemocyanin (KLH)-specific immune responses were evaluated following subcutaneous immunizations on PNDs 21 and 35. EtOH exposure affected innate as well as adaptive immune responses. The most sensitive immune parameters included white blood cell subpopulations, ConA-stimulated splenocyte proliferation, LPS-induced NO and TNF-α production by adherent splenocytes and KLH-specific immune responses. Most parameters showed recovery after cessation of EtOH exposure after weaning in the 14 % exposure group. However, effects on LPS-induced NO and TNF-α production by adherent splenocytes and KLH-specific parameters persisted until PND 70. The results demonstrate the relative sensitivity to EtOH of especially functional immune parameters and confirm the added value of immune parameters in the EOGRTS. Furthermore, this study identified an expanded KLH-specific parameter set and LPS-induced NO and TNF-α production by adherent splenocytes as valuable parameters that can provide additional information on functional immune effects.

Relative sensitivity of developmental and immune parameters in juvenile versus adult male rats after exposure to di(2-ethylhexyl) phthalate.

The developing immune system displays a relatively high sensitivity as compared to both general toxicity parameters and to the adult immune system. In this study we have performed such comparisons using di(2-ethylhexyl) phthalate (DEHP) as a model compound. DEHP is the most abundant phthalate in the environment and perinatal exposure to DEHP has been shown to disrupt male sexual differentiation. In addition, phthalate exposure has been associated with immune dysfunction as evidenced by effects on the expression of allergy. Male wistar rats were dosed with corn oil or DEHP by gavage from postnatal day (PND) 10-50 or PND 50-90 at doses between 1 and 1000 mg/kg/day. Androgen-dependent organ weights showed effects at lower dose levels in juvenile versus adult animals. Immune parameters affected included TDAR parameters in both age groups, NK activity in juvenile animals and TNF-α production by adherent splenocytes in adult animals. Immune parameters were affected at lower dose levels compared to developmental parameters. Overall, more immune parameters were affected in juvenile animals compared to adult animals and effects were observed at lower dose levels. The results of this study show a relatively higher sensitivity of juvenile versus adult rats. Furthermore, they illustrate the relative sensitivity of the developing immune system in juvenile animals as compared to general toxicity and developmental parameters. This study therefore provides further argumentation for performing dedicated developmental immune toxicity testing as a default in regulatory toxicology.

Small improvement in the area under the receiver operating characteristic curve indicated small changes in predicted risks.

OBJECTIVE: Adding risk factors to a prediction model often increases the area under the receiver operating characteristic curve (AUC) only slightly, particularly when the AUC of the model was already high. We investigated whether a risk factor that minimally improves the AUC may nevertheless improve the predictive ability of the model, assessed by integrated discrimination improvement (IDI). STUDY DESIGN AND SETTING: We simulated data sets with risk factors and event status for 100,000 hypothetical individuals and created prediction models with AUCs between 0.50 and 0.95. We added a single risk factor for which the effect was modeled as a certain odds ratio (OR 2, 4, 8) or AUC increment (ΔAUC 0.01, 0.02, 0.03). RESULTS: Across all AUC values of the baseline model, for a risk factor with the same OR, both ΔAUC and IDI were lower when the AUC of the baseline model was higher. When the increment in AUC was small (ΔAUC 0.01), the IDI was also small, except when the AUC of the baseline model was >0.90. CONCLUSION: When the addition of a risk factor shows minimal improvement in AUC, predicted risks generally show minimal changes too. Updating risk models with strong risk factors may be informative for a subgroup of individuals, but not at the population level. The AUC may not be as insensitive as is frequently argued.

An adverse outcome pathway framework for neural tube and axial defects mediated by modulation of retinoic acid homeostasis.

Developmental toxicity can be caused through a multitude of mechanisms and can therefore not be captured through a single simple mechanistic paradigm. However, it may be possible to define a selected group of overarching mechanisms that might allow detection of the vast majority of developmental toxicants. Against this background, we have explored the usefulness of retinoic acid mediated regulation of neural tube and axial patterning as a general mechanism that, when perturbed, may result in manifestations of developmental toxicity that may cover a large part of malformations known to occur in experimental animals and in man. Through a literature survey, we have identified key genes in the regulation of retinoic acid homeostasis, as well as marker genes of neural tube and axial patterning, that may be used to detect developmental toxicants in in vitro systems. A retinoic acid-neural tube/axial patterning adverse outcome pathway (RA-NTA AOP) framework was designed. The framework was tested against existing data of flusilazole exposure in the rat whole embryo culture, the zebrafish embryotoxicity test, and the embryonic stem cell test. Flusilazole is known to interact with retinoic acid homeostasis, and induced common and unique NTA marker gene changes in the three test systems. Flusilazole-induced changes were similar in directionality to gene expression responses after retinoic acid exposure. It is suggested that the RA-NTA framework may provide a general tool to define mechanistic pathways and biomarkers of developmental toxicity that may be used in alternative in vitro assays for the detection of embryotoxic compounds.

A statistical approach towards the derivation of predictive gene sets for potency ranking of chemicals in the mouse embryonic stem cell test.

The embryonic stem cell test (EST) is applied as a model system for detection of embryotoxicants. The application of transcriptomics allows a more detailed effect assessment compared to the morphological endpoint. Genes involved in cell differentiation, modulated by chemical exposures, may be useful as biomarkers of developmental toxicity. We describe a statistical approach to obtain a predictive gene set for toxicity potency ranking of compounds within one class. This resulted in a gene set based on differential gene expression across concentration-response series of phthalatic monoesters. We determined the concentration at which gene expression was changed at least 1.5-fold. Genes responding with the same potency ranking in vitro and in vivo embryotoxicity were selected. A leave-one-out cross-validation showed that the relative potency of each phthalate was always predicted correctly. The classical morphological 50% effect level (ID50) in EST was similar to the predicted concentration using gene set expression responses. A general down-regulation of development-related genes and up-regulation of cell-cycle related genes was observed, reminiscent of the differentiation inhibition in EST. This study illustrates the feasibility of applying dedicated gene set selections as biomarkers for developmental toxicity potency ranking on the basis of in vitro testing in the EST.

Developmental immunotoxicity in male rats after juvenile exposure to ethanol.

The aim of the present study was to determine the sensitivity of the developing immune system to ethanol (EtOH) after exposure from postnatal day (PND) 10 onward. Adult Wistar dams and litters were exposed to EtOH via drinking water (0, 0.25, 1.5, 2.75, 4, 5.25, or 6.5% (w/v) EtOH ad libitum) and drinking water exposure of the F1 was continued from weaning until sacrifice. Immune assessments were performed at postnatal days (PNDs) 21, 42, and 70. Furthermore, Keyhole Limpet Hemocyanin (KLH) specific immune responses were evaluated following subcutaneous immunizations on PNDs 21 and 35. EtOH exposure affected innate immune responses, such as LPS-induced NO-production by adherent splenocytes, as well as adaptive immune responses as represented by KLH-specific parameters. The most sensitive developmental parameters included effects on maternal and pup bodyweight with calculated BMDs of 4.0% and 4.3% EtOH, respectively. The most sensitive immune parameters were affected at dose levels lower than those affecting developmental parameters and included KLH-specific immune responses, LPS-induced NO production by adherent splenocytes, and IL-10 production by ConA stimulated splenocytes. Calculated BMDs for these parameters were between 0.01% and 0.1% EtOH. A comparison of the results of this juvenile study with an extended one-generation reproductive toxicity study revealed that the juvenile study design may result in a higher sensitivity related to differences in the exposure design. These findings demonstrate the relative sensitivity of the developing immune system for EtOH exposure, the additional value of assessing functional immune parameters, and the importance of the juvenile window in developmental immunotoxicity testing.

Valproic acid-induced gene expression responses in rat whole embryo culture and comparison across in vitro developmental and non-developmental models.

Transcriptomic evaluations may improve toxicity prediction of in vitro-based developmental models. In this study, transcriptomics was used to identify VPA-induced gene expression changes in rat whole embryo culture (WEC). Furthermore, VPA-induced responses were compared across in vitro-based developmental models, such as the cardiac and neural embryonic stem cells (ESTc and ESTn, respectively) and the zebrafish embryotoxicity model. VPA-induced gene regulation in WEC corresponded with observed morphological effects and previously suggested mechanisms of toxicity. Gene Ontology term-directed analysis showed conservation of VPA-induced gene expression changes across in vitro-based developmental models, with ESTc and ESTn exhibiting complementary responses. Furthermore, comparison of in vitro-based developmental and non-developmental models revealed that more generalized VPA-induced effects can be detected using non-developmental models whereas developmental models provide added value when assessing developmental-specific effects. These analyses can be used to optimize test batteries for the detection of developmental toxicants in vitro.

Triazole induced concentration-related gene signatures in rat whole embryo culture.

Commonly used as antifungal agents in agriculture and medicine, triazoles have been shown to cause teratogenicity in a diverse set of animal models. Here, we evaluated the dose-dependent impacts of flusilazole, cyproconazole and triadimefon, on global gene expression in relation to effects on embryonic development using the rat whole embryo culture (WEC) model. After 4 h exposure, we identified changes in gene expression due to triazole exposure which preceded morphological alterations observed at 48 h. In general, across the three triazoles, we observed similar directionality of regulation in gene expression and the magnitude of effects on gene expression correlated with the degree of induced developmental toxicity. Significantly regulated genes included key members of steroid/cholesterol and retinoic acid metabolism and hindbrain developmental pathways. Direct comparisons with previous studies suggest that triazole-gene signatures identified in the WEC overlap with zebrafish and mouse, and furthermore, triazoles impact gene expression in a similar manner as retinoic acid exposures in rat embryos. In summary, we further differentiate pathways underlying triazole-developmental toxicity using WEC and demonstrate the conservation of these response-pathways across model systems.

Juvenile toxicity testing protocols for chemicals.

There is increased awareness of the specific position of children when it comes to hazards of xenobiotic exposures. Children are not small adults, since their exposure patterns, compound kinetics and metabolism, and sensitivity of their developing organs may differ extensively from adults. Current international hazard assessment test guidelines do not specifically address juvenile exposures and effects. In conjunction with the Annual Meeting of the European Teratology Society, a satellite meeting was organized to specifically address juvenile toxicity testing issues for chemicals. The workshop focused on developmental neurotoxicity and developmental immune toxicity testing in juvenile animals. A clear case was made for the importance of juvenile toxicity testing, showing that in animal studies developmental neurotoxicity and immunotoxicity parameters express specifically high sensitivities after exposure during the juvenile period. Additional data will be generated in the coming years, and OECD initiatives will need to further the issue at the global regulatory level.

Developmental immunotoxicity of di-n-octyltin dichloride (DOTC) in an extended one-generation reproductive toxicity study.

Developmental immunotoxicity assessment is considered ready for inclusion in developmental toxicity studies. Further evaluation of proposed and additional assays is needed to determine their utility in assessing developmental immunotoxicity. In this study, a wide range of immunological parameters was included in an extended one-generation reproductive toxicity protocol. F(0) Wistar rats were exposed to DOTC via the feed (0, 3, 10, and 30mg/kg) during pre-mating, mating, gestation and lactation and subsequently F(1) were exposed from weaning until sacrifice. Immune assessments by several immune parameters were performed at PNDs 21, 42 and 70. The T cell-dependent antibody response to Keyhole Limpet hemocyanin (KLH) was assessed following subcutaneous immunizations with KLH on PNDs 21 and 35 and the delayed-type hypersensitivity response (DTH) against KLH was evaluated at PND 49. No effects were found on PND 21. While effects on lymphocyte subpopulations in the thymus were only observed in the 30mg/kg group on PND 42, effects on lymphocyte subpopulations in the spleen were found in the 30mg/kg group on both PNDs 42 and 70. The DTH response already showed an effect at 3mg/kg and was the overall critical endpoint. The results from this study support the inclusion of splenocyte subpopulation parameters in developmental toxicity studies and identified the DTH response as an important functional parameter.

Developmental immunotoxicity in male rats after juvenile exposure to di-n-octyltin dichloride (DOTC).

To determine relevant endpoints for evaluating developmental immunotoxicity due to juvenile exposure and optimal age of the animals at assessment, a wide range of immunological parameters were assessed in a juvenile toxicity study. Rats were exposed to di-n-octyltin dichloride (DOTC) by gavage from postnatal day (PND) 10 through PND 21 and via the diet after weaning using a benchmark dose (BMD) approach. Immune assessments were performed in male rats on PNDs 21, 42, and 70 and a subset of animals was used to evaluate the T-cell dependent antibody response (TDAR) to Keyhole limpet hemocyanin. Immune effects were more pronounced on PND 21 and 42 and observed at lower doses than developmental effects. The most sensitive immune parameters affected included TDAR parameters and thymocyte subpopulations with lower confidence limits of the benchmark doses (BMDLs) below the overall no-observed-adverse-effect-level (NOAEL) for DOTC reported so far in literature. These findings illustrate the relative sensitivity of the developing immune system for DOTC, the additional value of assessing functional immune parameters, and underscore the relevance of juvenile immunotoxicity testing in view of the risk assessment of chemicals.

Developmental immunotoxicity of methylmercury: the relative sensitivity of developmental and immune parameters.

Current developmental and reproductive toxicity protocols include only a limited set of parameters for effects on the developing immune system. In this study, a wide range of immunological parameters were included in a pre- and postnatal developmental toxicity study. Dose-response data were compared to determine the relative sensitivity of different immune and developmental parameters. Mated female Wistar rats were dosed daily by gavage with methylmercury (0, 0.1, 0.4, 0.7, 1.0, 1.5, and 2.0 mg/kg BW/day) from gestational day 6 to postnatal day (PND) 10. In addition to general, reproductive, and developmental parameters, a wide range of immunological parameters were assessed in male offspring at PNDs 21, 42, and 70. The T cell-dependent antibody response to keyhole limpet hemocyanin (KLH) was assessed following sc immunizations on PNDs 21 and 35. Dose-response data were analyzed using the benchmark dose (BMD) approach by fitting dose-response models to the various endpoints. Methylmercury induced effects on developmental parameters, such as growth parameters and pup mortality. Effects on the immune system were found at doses without observed developmental toxicity. Immune effects differed at the three time points and consisted mainly of effects on functional parameters. The parameter with the lowest 5% lower confidence bound of the BMD (BMDL) was the primary KLH-specific IgG antibody response, which showed a dose-dependent decrease with a BMD of 0.039 mg/kg BW/day (CI 0.010-0.12). These data show the relatively high sensitivity of the developing immune system and thereby illustrate the relevance of testing immune parameters in reproductive and developmental toxicity testing protocols.

Conversion of the low affinity ouabain-binding site of non-gastric H,K-ATPase into a high affinity binding site by substitution of only five amino acids.

P-type ATPases of the IIC subfamily exhibit large differences in sensitivity toward ouabain. This allows a strategy in which ouabain-insensitive members of this subfamily are used as template for mutational elucidation of the ouabain-binding site. With this strategy, we recently identified seven amino acids in Na,K-ATPase that conferred high affinity ouabain binding to gastric H,K-ATPase (Qiu, L. Y., Krieger, E., Schaftenaar, G., Swarts, H. G. P., Willems, P. H. G. M., De Pont, J. J. H. H. M., and Koenderink, J. B. (2005) J. Biol. Chem. 280, 32349-32355). Because important, but identical, amino acids were not recognized in that study, here we used the non-gastric H,K-ATPase, which is rather ouabain-insensitive, as template. The catalytic subunit of this enzyme, in which several amino acids from Na,K-ATPase were incorporated, was expressed with the Na,K-ATPase beta1 subunit in Xenopus laevis oocytes. A chimera containing 14 amino acids, located in M4, M5, and M6, which are unique to Na,K-ATPase, displayed high affinity ouabain binding. Four of these residues, all located in M5, appeared dispensable for high affinity binding. Individual mutation of the remaining 10 residues to their non-gastric H,K-ATPase counterparts yielded five amino acids (Glu312,Gly319, Pro778, Leu795, and Cys802) whose mutation resulted in a loss of ouabain binding. In a final gain-of-function experiment, we introduced these five amino acids in different combinations in non-gastric H,K-ATPase and demonstrated that all five were essential for high affinity ouabain binding. The non-gastric H,K-ATPase with these five mutations had a similar apparent affinity for ouabain as the wild type Na,K-ATPase and showed a 2000 times increased affinity for ouabain in the NH4+-stimulated ATPase activity in membranes of transfected Sf9 cells.