Kernel density weighted loess normalization improves the performance of detection within asymmetrical data.

BACKGROUND: Normalization of gene expression data has been studied for many years and various strategies have been formulated to deal with various types of data. Most normalization algorithms rely on the assumption that the number of up-regulated genes and the number of down-regulated genes are roughly the same. However, the well-known Golden Spike experiment presents a unique situation in which differentially regulated genes are biased toward one direction, thereby challenging the conclusions of previous bench mark studies. RESULTS: This study proposes two novel approaches, KDL and KDQ, based on kernel density estimation to improve upon the basic idea of invariant set selection. The key concept is to provide various importance scores to data points on the MA plot according to their proximity to the cluster of the null genes under the assumption that null genes are more densely distributed than those that are differentially regulated. The comparison is demonstrated in the Golden Spike experiment as well as with simulation data using the ROC curves and compression rates. KDL and KDQ in combination with GCRMA provided the best performance among all approaches. CONCLUSIONS: This study determined that methods based on invariant sets are better able to resolve the problem of asymmetry. Normalization, either before or after expression summary for probesets, improves performance to a similar degree.

Evaluation of external RNA controls for the assessment of microarray performance.

External RNA controls (ERCs), although important for microarray assay performance assessment, have yet to be fully implemented in the research community. As part of the MicroArray Quality Control (MAQC) study, two types of ERCs were implemented and evaluated; one was added to the total RNA in the samples before amplification and labeling; the other was added to the copyRNAs (cRNAs) before hybridization. ERC concentration-response curves were used across multiple commercial microarray platforms to identify problematic assays and potential sources of variation in the analytical process. In addition, the behavior of different ERC types was investigated, resulting in several important observations, such as the sample-dependent attributes of performance and the potential of using these control RNAs in a combinatorial fashion. This multiplatform investigation of the behavior and utility of ERCs provides a basis for articulating specific recommendations for their future use in evaluating assay performance across multiple platforms.

Performance comparison of one-color and two-color platforms within the MicroArray Quality Control (MAQC) project.

Microarray-based expression profiling experiments typically use either a one-color or a two-color design to measure mRNA abundance. The validity of each approach has been amply demonstrated. Here we provide a simultaneous comparison of results from one- and two-color labeling designs, using two independent RNA samples from the Microarray Quality Control (MAQC) project, tested on each of three different microarray platforms. The data were evaluated in terms of reproducibility, specificity, sensitivity and accuracy to determine if the two approaches provide comparable results. For each of the three microarray platforms tested, the results show good agreement with high correlation coefficients and high concordance of differentially expressed gene lists within each platform. Cumulatively, these comparisons indicate that data quality is essentially equivalent between the one- and two-color approaches and strongly suggest that this variable need not be a primary factor in decisions regarding experimental microarray design.

The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements.

Over the last decade, the introduction of microarray technology has had a profound impact on gene expression research. The publication of studies with dissimilar or altogether contradictory results, obtained using different microarray platforms to analyze identical RNA samples, has raised concerns about the reliability of this technology. The MicroArray Quality Control (MAQC) project was initiated to address these concerns, as well as other performance and data analysis issues. Expression data on four titration pools from two distinct reference RNA samples were generated at multiple test sites using a variety of microarray-based and alternative technology platforms. Here we describe the experimental design and probe mapping efforts behind the MAQC project. We show intraplatform consistency across test sites as well as a high level of interplatform concordance in terms of genes identified as differentially expressed. This study provides a resource that represents an important first step toward establishing a framework for the use of microarrays in clinical and regulatory settings.

The balance of reproducibility, sensitivity, and specificity of lists of differentially expressed genes in microarray studies.

BACKGROUND: Reproducibility is a fundamental requirement in scientific experiments. Some recent publications have claimed that microarrays are unreliable because lists of differentially expressed genes (DEGs) are not reproducible in similar experiments. Meanwhile, new statistical methods for identifying DEGs continue to appear in the scientific literature. The resultant variety of existing and emerging methods exacerbates confusion and continuing debate in the microarray community on the appropriate choice of methods for identifying reliable DEG lists. RESULTS: Using the data sets generated by the MicroArray Quality Control (MAQC) project, we investigated the impact on the reproducibility of DEG lists of a few widely used gene selection procedures. We present comprehensive results from inter-site comparisons using the same microarray platform, cross-platform comparisons using multiple microarray platforms, and comparisons between microarray results and those from TaqMan - the widely regarded "standard" gene expression platform. Our results demonstrate that (1) previously reported discordance between DEG lists could simply result from ranking and selecting DEGs solely by statistical significance (P) derived from widely used simple t-tests; (2) when fold change (FC) is used as the ranking criterion with a non-stringent P-value cutoff filtering, the DEG lists become much more reproducible, especially when fewer genes are selected as differentially expressed, as is the case in most microarray studies; and (3) the instability of short DEG lists solely based on P-value ranking is an expected mathematical consequence of the high variability of the t-values; the more stringent the P-value threshold, the less reproducible the DEG list is. These observations are also consistent with results from extensive simulation calculations. CONCLUSION: We recommend the use of FC-ranking plus a non-stringent P cutoff as a straightforward and baseline practice in order to generate more reproducible DEG lists. Specifically, the P-value cutoff should not be stringent (too small) and FC should be as large as possible. Our results provide practical guidance to choose the appropriate FC and P-value cutoffs when selecting a given number of DEGs. The FC criterion enhances reproducibility, whereas the P criterion balances sensitivity and specificity.

Linking ligand-induced alterations in androgen receptor structure to differential gene expression: a first step in the rational design of selective androgen receptor modulators.

We have previously identified a family of novel androgen receptor (AR) ligands that, upon binding, enable AR to adopt structures distinct from that observed in the presence of canonical agonists. In this report, we describe the use of these compounds to establish a relationship between AR structure and biological activity with a view to defining a rational approach with which to identify useful selective AR modulators. To this end, we used combinatorial peptide phage display coupled with molecular dynamic structure analysis to identify the surfaces on AR that are exposed specifically in the presence of selected AR ligands. Subsequently, we used a DNA microarray analysis to demonstrate that differently conformed receptors facilitate distinct patterns of gene expression in LNCaP cells. Interestingly, we observed a complete overlap in the identity of genes expressed after treatment with mechanistically distinct AR ligands. However, it was differences in the kinetics of gene regulation that distinguished these compounds. Follow-up studies, in cell-based assays of AR action, confirmed the importance of these alterations in gene expression. Together, these studies demonstrate an important link between AR structure, gene expression, and biological outcome. This relationship provides a firm underpinning for mechanism-based screens aimed at identifying SARMs with useful clinical profiles.

Transcript profiling of a conifer pathosystem: response of Pinus sylvestris root tissues to pathogen (Heterobasidion annosum) invasion.

The mechanisms underlying defence reactions to a pathogen attack, though well studied in crop plants, are poorly understood in conifers. To analyze changes in gene transcript abundance in Pinus sylvestris L. root tissues infected by Heterobasidion annosum (Fr.) Bref. s.l., a cDNA microarray containing 2109 ESTs from P. taeda L. was used. Mixed model statistical analysis identified 179 expressed sequence tags differentially expressed at 1, 5 or 15 days post inoculation. In general, the total number of genes differentially expressed during the infection increased over time. The most abundant group of genes up-regulated upon infection coded for enzymes involved in metabolism (phenylpropanoid pathway) and defence-related proteins with antimicrobial properties. A class III peroxidase responsible for lignin biosynthesis and cell wall thickening had increased transcript abundance at all measurement times. Real-time RT-PCR verified the microarray results with high reproducibility. The similarity of the expression profiling pattern observed in this pathosystem to those documented in crop pathology suggests that angiosperms and gymnosperms use similar genetic programs in responding to invasive growth by microbial pathogens.

Comparison of transcriptional responses in liver tissue and primary hepatocyte cell cultures after exposure to hexahydro-1, 3, 5-trinitro-1, 3, 5-triazine.

BACKGROUND: Cell culture systems are useful in studying toxicological effects of chemicals such as Hexahydro-1,3,5-trinitro-1,3,5-triazine (RDX), however little is known as to how accurately isolated cells reflect responses of intact organs. In this work, we compare transcriptional responses in livers of Sprague-Dawley rats and primary hepatocyte cells after exposure to RDX to determine how faithfully the in vitro model system reflects in vivo responses. RESULTS: Expression patterns were found to be markedly different between liver tissue and primary cell cultures before exposure to RDX. Liver gene expression was enriched in processes important in toxicology such as metabolism of amino acids, lipids, aromatic compounds, and drugs when compared to cells. Transcriptional responses in cells exposed to 7.5, 15, or 30 mg/L RDX for 24 and 48 hours were different from those of livers isolated from rats 24 hours after exposure to 12, 24, or 48 mg/Kg RDX. Most of the differentially expressed genes identified across conditions and treatments could be attributed to differences between cells and tissue. Some similarity was observed in RDX effects on gene expression between tissue and cells, but also significant differences that appear to reflect the state of the cell or tissue examined. CONCLUSION: Liver tissue and primary cells express different suites of genes that suggest they have fundamental differences in their cell physiology. Expression effects related to RDX exposure in cells reflected a fraction of liver responses indicating that care must be taken in extrapolating from primary cells to whole animal organ toxicity effects.

Mixed-model reanalysis of primate data suggests tissue and species biases in oligonucleotide-based gene expression profiles.

An emerging issue in evolutionary genetics is whether it is possible to use gene expression profiling to identify genes that are associated with morphological, physiological, or behavioral divergence between species and whether these genes have undergone positive selection. Some of these questions were addressed in a recent study (Enard et al. 2002) of the difference in gene expression among human, chimp, and orangutan, which suggested an accelerated rate of divergence in gene expression in the human brain relative to liver. Reanalysis of the Affymetrix data set using analysis of variance methods to quantify the contributions of individuals and species to variation in expression of 12,600 genes indicates that as much as one-quarter of the genome shows divergent expression between primate species at the 5% level. The magnitude of fold change ranges from 1.2-fold up to 8-fold. Similar conclusions apply to reanalysis of Enard et al. 2002 parallel murine data set. However, biases inherent to short oligonucleotide microarray technology may account for some of the tissue and species effects. At high significance levels, more differences were observed in the liver than in the brain in each of the pairwise species comparisons, so it is not clear that expression divergence is accelerated in the human brain. Further, there is an apparent bias toward upregulation of gene expression in the brain in both primates and mice, whereas genes are equally likely to be up- or downregulated in the liver when these species diverge. A small subset of genes that are candidates for adaptive divergence may be identified on the basis of a high ratio of interspecific to intraspecific divergence.

A mixed model approach to identify yeast transcriptional regulatory motifs via microarray experiments.

A genome-wide location analysis method has been introduced as a means to simultaneously study protein-DNA binding interactions for a large number of genes on a microarray platform. Identification of interactions between transcription factors (TF) and genes provide insight into the mechanisms that regulate a variety of cellular responses. Drawing proper inferences from the experimental data is key to finding statistically significant TF-gene binding interactions. We describe how the analysis and interpretation of genome-wide location data can be fit into a traditional statistical modeling framework that considers the data across all arrays and formulizes appropriate hypothesis tests. The approach is illustrated with data from a yeast transcription factor binding experiment that illustrates how identified TF-gene interactions can enhance initial exploration of transcriptional regulatory networks. Examples of five kinds of transcriptional regulatory structure are also demonstrated. Some stark differences with previously published results are explored.

Comparison of RNA-seq and microarray-based models for clinical endpoint prediction.

BACKGROUND: Gene expression profiling is being widely applied in cancer research to identify biomarkers for clinical endpoint prediction. Since RNA-seq provides a powerful tool for transcriptome-based applications beyond the limitations of microarrays, we sought to systematically evaluate the performance of RNA-seq-based and microarray-based classifiers in this MAQC-III/SEQC study for clinical endpoint prediction using neuroblastoma as a model. RESULTS: We generate gene expression profiles from 498 primary neuroblastomas using both RNA-seq and 44 k microarrays. Characterization of the neuroblastoma transcriptome by RNA-seq reveals that more than 48,000 genes and 200,000 transcripts are being expressed in this malignancy. We also find that RNA-seq provides much more detailed information on specific transcript expression patterns in clinico-genetic neuroblastoma subgroups than microarrays. To systematically compare the power of RNA-seq and microarray-based models in predicting clinical endpoints, we divide the cohort randomly into training and validation sets and develop 360 predictive models on six clinical endpoints of varying predictability. Evaluation of factors potentially affecting model performances reveals that prediction accuracies are most strongly influenced by the nature of the clinical endpoint, whereas technological platforms (RNA-seq vs. microarrays), RNA-seq data analysis pipelines, and feature levels (gene vs. transcript vs. exon-junction level) do not significantly affect performances of the models. CONCLUSIONS: We demonstrate that RNA-seq outperforms microarrays in determining the transcriptomic characteristics of cancer, while RNA-seq and microarray-based models perform similarly in clinical endpoint prediction. Our findings may be valuable to guide future studies on the development of gene expression-based predictive models and their implementation in clinical practice.

Cross-site comparison of gene expression data reveals high similarity.

Consistency and coherence of gene expression data across multiple sites depends on several factors such as platform (oligo, cDNA, etc.), environmental conditions at each laboratory, and data quality. The Hepatotoxicity Working Group of the International Life Sciences Institute Health and Environmental Sciences Institute consortium on the application of genomics to mechanism-based risk assessment is investigating these factors by comparing high-density gene expression data sets generated on two sets of RNA from methapyrilene (MP) experiments conducted at Abbott Laboratories and Boehringer-Ingelheim Pharmaceuticals, Inc. using a single platform (Affymetrix Rat Genome U34A GeneChip) at seven different sites. This article focuses on the evaluation of data quality and statistical models that facilitate the comparison of such data sets at the probe level. We present methods for exploring and quantitatively assessing differences in the data, with the principal goal being the generation of lists of site-insensitive genes responsive to low and high doses of MP. A combination of numerical and graphical techniques reveals important patterns and partitions of variability in the data, including the magnitude of the site effects. Although the site effects are significantly large in the analysis results, they appear to be primarily additive and therefore can be adjusted in the statistical calculations in a way that does not bias conclusions regarding treatment differences.

Variation in transcript abundance during somatic embryogenesis in gymnosperms.

Somatic embryogenesis of Norway spruce (Picea abies L.) is a versatile model system to study molecular mechanisms regulating embryo development because it proceeds through defined developmental stages corresponding to specific culture treatments. Normal embryonic development involves early differentiation of proembryogenic masses (PEMs) into somatic embryos, followed by early and late embryogeny leading to the formation of mature cotyledonary embryos. In some cell lines there is a developmental arrest at the PEM-somatic embryo transition. To learn more about the molecular mechanisms regulating embryogenesis, we compared the transcript profiles of two normal lines and one developmentally arrested line. Ribonucleic acid, extracted from these cell lines at successive developmental stages, was analyzed on DNA microarrays containing 2178 expressed sequence tags (ESTs) (corresponding to 2110 unique cDNAs) from loblolly pine (Pinus taeda L.). Hybridization between spruce and pine species on microarrays has been shown to be effective (van Zyl et al. 2002, Stasolla et al. 2003). In contrast to the developmentally arrested line, the early phases of normal embryo development are characterized by a precise pattern of gene expression, i.e., repression followed by induction. Comparison of transcript levels between successive stages of embryogenesis allowed us to identify several genes that showed unique expression responses during normal development. Several of these genes encode proteins involved in detoxification processes, methionine synthesis and utilization, and carbohydrate metabolism. The potential role of these genes in embryo development is discussed.

A systematic statistical linear modeling approach to oligonucleotide array experiments.

We outline and describe steps for a statistically rigorous approach to analyzing probe-level Affymetrix GeneChip data. The approach employs classical linear mixed models and operates on a gene-by-gene basis. Forgoing any attempts at gene presence or absence calls, the method simultaneously considers the data across all chips in an experiment. Primary output includes precise estimates of fold change (some as low as 1.1), their statistical significance, and measures of array and probe variability. The method can accommodate complex experiments involving many kinds of treatments and can test for their effects at the probe level. Furthermore, mismatch probe data can be incorporated in different ways or ignored altogether. Data from an ionizing radiation experiment on human cell lines illustrate the key concepts.

Comparison of microarrays and RNA-seq for gene expression analyses of dose-response experiments.

Relative to microarrays, RNA-seq has been reported to offer higher precision estimates of transcript abundance, a greater dynamic range, and detection of novel transcripts. However, previous comparisons of the 2 technologies have not covered dose-response experiments that are relevant to toxicology. Male F344 rats were exposed for 13 weeks to 5 doses of bromobenzene, and liver gene expression was measured using both microarrays and RNA-seq. Multiple normalization methods were evaluated for each technology, and gene expression changes were statistically analyzed using both analysis of variance and benchmark dose (BMD). Fold-change values were highly correlated between the 2 technologies, whereas the -log p values showed lower correlation. RNA-seq detected fewer statistically significant genes at lower doses, but more significant genes based on fold change except when a negative binomial transformation was applied. Overlap in genes significant by both p value and fold change was approximately 30%-40%. Random sampling of the RNA-seq data showed an equivalent number of differentially expressed genes compared with microarrays at ~5 million reads. Quantitative RT-PCR of differentially expressed genes uniquely identified by each technology showed a high degree of confirmation when both fold change and p value were considered. The mean dose-response expression of each gene was highly correlated between technologies, whereas estimates of sample variability and gene-based BMD values showed lower correlation. Differences in BMD estimates and statistical significance may be due, in part, to differences in the dynamic range of each technology and the degree to which normalization corrects genes at either end of the scale.

Global transcriptome analysis of Clostridium thermocellum ATCC 27405 during growth on dilute acid pretreated Populus and switchgrass.

BACKGROUND: The thermophilic anaerobe Clostridium thermocellum is a candidate consolidated bioprocessing (CBP) biocatalyst for cellulosic ethanol production. The aim of this study was to investigate C. thermocellum genes required to ferment biomass substrates and to conduct a robust comparison of DNA microarray and RNA sequencing (RNA-seq) analytical platforms. RESULTS: C. thermocellum ATCC 27405 fermentations were conducted with a 5 g/L solid substrate loading of either pretreated switchgrass or Populus. Quantitative saccharification and inductively coupled plasma emission spectroscopy (ICP-ES) for elemental analysis revealed composition differences between biomass substrates, which may have influenced growth and transcriptomic profiles. High quality RNA was prepared for C. thermocellum grown on solid substrates and transcriptome profiles were obtained for two time points during active growth (12 hours and 37 hours postinoculation). A comparison of two transcriptomic analytical techniques, microarray and RNA-seq, was performed and the data analyzed for statistical significance. Large expression differences for cellulosomal genes were not observed. We updated gene predictions for the strain and a small novel gene, Cthe_3383, with a putative AgrD peptide quorum sensing function was among the most highly expressed genes. RNA-seq data also supported different small regulatory RNA predictions over others. The DNA microarray gave a greater number (2,351) of significant genes relative to RNA-seq (280 genes when normalized by the kernel density mean of M component (KDMM) method) in an analysis of variance (ANOVA) testing method with a 5% false discovery rate (FDR). When a 2-fold difference in expression threshold was applied, 73 genes were significantly differentially expressed in common between the two techniques. Sulfate and phosphate uptake/utilization genes, along with genes for a putative efflux pump system were some of the most differentially regulated transcripts when profiles for C. thermocellum grown on either pretreated switchgrass or Populus were compared. CONCLUSIONS: Our results suggest that a high degree of agreement in differential gene expression measurements between transcriptomic platforms is possible, but choosing an appropriate normalization regime is essential.

Relative impact of incorporating pharmacokinetics on predicting in vivo hazard and mode of action from high-throughput in vitro toxicity assays.

The use of high-throughput in vitro assays has been proposed to play a significant role in the future of toxicity testing. In this study, rat hepatic metabolic clearance and plasma protein binding were measured for 59 ToxCast phase I chemicals. Computational in vitro-to-in vivo extrapolation was used to estimate the daily dose in a rat, called the oral equivalent dose, which would result in steady-state in vivo blood concentrations equivalent to the AC 50 or lowest effective concentration (LEC) across more than 600 ToxCast phase I in vitro assays. Statistical classification analysis was performed using either oral equivalent doses or unadjusted AC 50 /LEC values for the in vitro assays to predict the in vivo effects of the 59 chemicals. Adjusting the in vitro assays for pharmacokinetics did not improve the ability to predict in vivo effects as either a discrete (yes or no) response or a low effect level (LEL) on a continuous dose scale. Interestingly, a comparison of the in vitro assay with the lowest oral equivalent dose with the in vivo endpoint with the lowest LEL suggested that the lowest oral equivalent dose may provide a conservative estimate of the point of departure for a chemical in a dose-response assessment. Furthermore, comparing the oral equivalent doses for the in vitro assays with the in vivo dose range that resulted in adverse effects identified more coincident in vitro assays across chemicals than expected by chance, suggesting that the approach may also be used to identify potential molecular initiating events leading to adversity.

A comprehensive statistical analysis of predicting in vivo hazard using high-throughput in vitro screening.

Over the past 5 years, increased attention has been focused on using high-throughput in vitro screening for identifying chemical hazards and prioritizing chemicals for additional in vivo testing. The U.S. Environmental Protection Agency's ToxCast program has generated a significant amount of high-throughput screening data allowing a broad-based assessment of the utility of these assays for predicting in vivo responses. In this study, a comprehensive cross-validation model comparison was performed to evaluate the predictive performance of the more than 600 in vitro assays from the ToxCast phase I screening effort across 60 in vivo endpoints using 84 different statistical classification methods. The predictive performance of the in vitro assays was compared and combined with that from chemical structure descriptors. With the exception of chronic in vivo cholinesterase inhibition, the overall predictive power of both the in vitro assays and the chemical descriptors was relatively low. The predictive power of the in vitro assays was not significantly different from that of the chemical descriptors and aggregating the assays based on genes reduced predictive performance. Prefiltering the in vitro assay data outside the cross-validation loop, as done in some previous studies, significantly biased estimates of model performance. The results suggest that the current ToxCast phase I assays and chemicals have limited applicability for predicting in vivo chemical hazards using standard statistical classification methods. However, if viewed as a survey of potential molecular initiating events and interpreted as risk factors for toxicity, the assays may still be useful for chemical prioritization.

The MicroArray Quality Control (MAQC)-II study of common practices for the development and validation of microarray-based predictive models.

Gene expression data from microarrays are being applied to predict preclinical and clinical endpoints, but the reliability of these predictions has not been established. In the MAQC-II project, 36 independent teams analyzed six microarray data sets to generate predictive models for classifying a sample with respect to one of 13 endpoints indicative of lung or liver toxicity in rodents, or of breast cancer, multiple myeloma or neuroblastoma in humans. In total, >30,000 models were built using many combinations of analytical methods. The teams generated predictive models without knowing the biological meaning of some of the endpoints and, to mimic clinical reality, tested the models on data that had not been used for training. We found that model performance depended largely on the endpoint and team proficiency and that different approaches generated models of similar performance. The conclusions and recommendations from MAQC-II should be useful for regulatory agencies, study committees and independent investigators that evaluate methods for global gene expression analysis.

Use of short-term transcriptional profiles to assess the long-term cancer-related safety of environmental and industrial chemicals.

The process for evaluating chemical safety is inefficient, costly, and animal intensive. There is growing consensus that the current process of safety testing needs to be significantly altered to improve efficiency and reduce the number of untested chemicals. In this study, the use of short-term gene expression profiles was evaluated for predicting the increased incidence of mouse lung tumors. Animals were exposed to a total of 26 diverse chemicals with matched vehicle controls over a period of 3 years. Upon completion, significant batch-related effects were observed. Adjustment for batch effects significantly improved the ability to predict increased lung tumor incidence. For the best statistical model, the estimated predictive accuracy under honest fivefold cross-validation was 79.3% with a sensitivity and specificity of 71.4 and 86.3%, respectively. A learning curve analysis demonstrated that gains in model performance reached a plateau at 25 chemicals, indicating that the size of current data set was sufficient to provide a robust classifier. The classification results showed that a small subset of chemicals contributed disproportionately to the misclassification rate. For these chemicals, the misclassification was more closely associated with genotoxicity status than with efficacy in the original bioassay. Statistical models were also used to predict dose-response increases in tumor incidence for methylene chloride and naphthalene. The average posterior probabilities for the top models matched the results from the bioassay for methylene chloride. For naphthalene, the average posterior probabilities for the top models overpredicted the tumor response, but the variability in predictions was significantly higher. The study provides both a set of gene expression biomarkers for predicting chemically induced mouse lung tumors and a broad assessment of important experimental and analysis criteria for developing microarray-based predictors of safety-related end points.