A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction.

The utility of mouse and rat studies critically depends on their replicability in other laboratories. A widely advocated approach to improving replicability is through the rigorous control of predefined animal or experimental conditions, known as standardization. However, this approach limits the generalizability of the findings to only to the standardized conditions and is a potential cause rather than solution to what has been called a replicability crisis. Alternative strategies include estimating the heterogeneity of effects across laboratories, either through designs that vary testing conditions, or by direct statistical analysis of laboratory variation. We previously evaluated our statistical approach for estimating the interlaboratory replicability of a single laboratory discovery. Those results, however, were from a well-coordinated, multi-lab phenotyping study and did not extend to the more realistic setting in which laboratories are operating independently of each other. Here, we sought to test our statistical approach as a realistic prospective experiment, in mice, using 152 results from 5 independent published studies deposited in the Mouse Phenome Database (MPD). In independent replication experiments at 3 laboratories, we found that 53 of the results were replicable, so the other 99 were considered non-replicable. Of the 99 non-replicable results, 59 were statistically significant (at 0.05) in their original single-lab analysis, putting the probability that a single-lab statistical discovery was made even though it is non-replicable, at 59.6%. We then introduced the dimensionless "Genotype-by-Laboratory" (GxL) factor-the ratio between the standard deviations of the GxL interaction and the standard deviation within groups. Using the GxL factor reduced the number of single-lab statistical discoveries and alongside reduced the probability of a non-replicable result to be discovered in the single lab to 12.1%. Such reduction naturally leads to reduced power to make replicable discoveries, but this reduction was small (from 87% to 66%), indicating the small price paid for the large improvement in replicability. Tools and data needed for the above GxL adjustment are publicly available at the MPD and will become increasingly useful as the range of assays and testing conditions in this resource increases.

Discussion on "Optimal test procedures for multiple hypotheses controlling the familywise expected loss" by Willi Maurer, Frank Bretz, and Xiaolei Xun.

We discuss three issues. In the first part, we discuss the criteria emphasized by Maurer, Bretz, and Xun, warning that it modifies the per comparison error rate that does not address the concerns raised by multiple testing. In the second part, we strengthen the optimality results developed in the paper, based on our recent results. In the third part, we highlight the potentially important role that the use of weights may have in practice and discuss the difficulties in assigning weights that convey the importance in the gain and loss functions, especially as it pertains to multiple endpoints.

Reliability Validity and Responsiveness of the Spinal Cord Independence Measure 4th Version in a Multicultural Setup.

OBJECTIVE: To examine the fourth version of the Spinal Cord Independence Measure for reliability and validity. DESIGN: Partly blinded comparison with the criterion standard Spinal Cord Independence Measure III, and between examiners and examinations. SETTING: A multicultural cohort from 19 spinal cord injury units in 11 countries. PARTICIPANTS: A total of 648 patients with spinal cord injury. INTERVENTION: Assessment with Spinal Cord Independence Measure (SCIM IV) and Spinal Cord Independence Measure (SCIM III) on admission to inpatient rehabilitation and before discharge. MAIN OUTCOME MEASURES: SCIM IV interrater reliability, internal consistency, correlation with and difference from SCIM III, and responsiveness. RESULTS: Total agreement between examiners was above 80% on most SCIM IV tasks. All Kappa coefficients were above 0.70 and statistically significant (P<.001). Pearson's coefficients of the correlation between the examiners were above 0.90, and intraclass correlation coefficients were above 0.90. Cronbach's alpha was above 0.96 for the entire SCIM IV, above 0.66 for the subscales, and usually decreased when an item was eliminated. Reliability values were lower for the subscale of respiration and sphincter management, and on admission than at discharge. SCIM IV and SCIM III mean values were very close, and the coefficients of Pearson correlation between them were 0.91-0.96 (P<.001). The responsiveness of SCIM IV was not significantly different from that of SCIM III in most of the comparisons. CONCLUSIONS: The validity, reliability, and responsiveness of SCIM IV, which was adjusted to assess specific patient conditions or situations that SCIM III does not address, and which includes more accurate definitions of certain scoring criteria, are very good and quite similar to those of SCIM III. SCIM IV can be used for clinical and research trials, including international multi-center studies, and its group scores can be compared with those of SCIM III.

Characterization of Expression Quantitative Trait Loci in Pedigrees from Colombia and Costa Rica Ascertained for Bipolar Disorder.

The observation that variants regulating gene expression (expression quantitative trait loci, eQTL) are at a high frequency among SNPs associated with complex traits has made the genome-wide characterization of gene expression an important tool in genetic mapping studies of such traits. As part of a study to identify genetic loci contributing to bipolar disorder and other quantitative traits in members of 26 pedigrees from Costa Rica and Colombia, we measured gene expression in lymphoblastoid cell lines derived from 786 pedigree members. The study design enabled us to comprehensively reconstruct the genetic regulatory network in these families, provide estimates of heritability, identify eQTL, evaluate missing heritability for the eQTL, and quantify the number of different alleles contributing to any given locus. In the eQTL analysis, we utilize a recently proposed hierarchical multiple testing strategy which controls error rates regarding the discovery of functional variants. Our results elucidate the heritability and regulation of gene expression in this unique Latin American study population and identify a set of regulatory SNPs which may be relevant in future investigations of complex disease in this population. Since our subjects belong to extended families, we are able to compare traditional kinship-based estimates with those from more recent methods that depend only on genotype information.

A powerful statistical framework for generalization testing in GWAS, with application to the HCHS/SOL.

In genome-wide association studies (GWAS), "generalization" is the replication of genotype-phenotype association in a population with different ancestry than the population in which it was first identified. Current practices for declaring generalizations rely on testing associations while controlling the family-wise error rate (FWER) in the discovery study, then separately controlling error measures in the follow-up study. This approach does not guarantee control over the FWER or false discovery rate (FDR) of the generalization null hypotheses. It also fails to leverage the two-stage design to increase power for detecting generalized associations. We provide a formal statistical framework for quantifying the evidence of generalization that accounts for the (in)consistency between the directions of associations in the discovery and follow-up studies. We develop the directional generalization FWER (FWERg ) and FDR (FDRg ) controlling r-values, which are used to declare associations as generalized. This framework extends to generalization testing when applied to a published list of Single Nucleotide Polymorphism-(SNP)-trait associations. Our methods control FWERg or FDRg under various SNP selection rules based on P-values in the discovery study. We find that it is often beneficial to use a more lenient P-value threshold than the genome-wide significance threshold. In a GWAS of total cholesterol in the Hispanic Community Health Study/Study of Latinos (HCHS/SOL), when testing all SNPs with P-values <5×10-8 (15 genomic regions) for generalization in a large GWAS of whites, we generalized SNPs from 15 regions. But when testing all SNPs with P-values <6.6×10-5 (89 regions), we generalized SNPs from 27 regions.

Weighted false discovery rate controlling procedures for clinical trials.

Having identified that the lack of replicability of results in earlier phases of clinical medical research stems largely from unattended selective inference, we offer a new hierarchical weighted false discovery rate controlling testing procedure alongside the single-level weighted procedure. These address the special structure of clinical research, where the comparisons of treatments involve both primary and secondary endpoints, by assigning weights that reflect the relative importance of the endpoints in the error being controlled. In the hierarchical method, the primary endpoints and a properly weighted intersection hypothesis that represents all secondary endpoints are tested. Should the intersection hypothesis be among the rejected, individual secondary endpoints are tested. We identify configurations where each of the two procedures has the advantage. Both offer higher power than competing hierarchical (gatekeeper) familywise error-rate controlling procedures being used for drug approval. By their design, the advantage of the proposed methods is the increased power to discover effects on secondary endpoints, without giving up the rigor of addressing their multiplicity.

Identifying the ligated amino acid of archaeal tRNAs based on positions outside the anticodon.

Proper recognition of tRNAs by their aminoacyl-tRNA synthetase is essential for translation accuracy. Following evidence that the enzymes can recognize the correct tRNA even when anticodon information is masked, we search for additional nucleotide positions within the tRNA molecule that potentially contain information for amino acid identification. Analyzing 3936 sequences of tRNA genes from 86 archaeal species, we show that the tRNAs' cognate amino acids can be identified by the information embedded in the tRNAs' nucleotide positions without relying on the anticodon information. We present a small set of six to 10 informative positions along the tRNA, which allow for amino acid identification accuracy of 90.6% to 97.4%, respectively. We inspected tRNAs for each of the 20 amino acid types for such informative positions and found that tRNA genes for some amino acids are distinguishable from others by as few as one or two positions. The informative nucleotide positions are in agreement with nucleotide positions that were experimentally shown to affect the loaded amino acid identity. Interestingly, the knowledge gained from the tRNA genes of one archaeal phylum does not extrapolate well to another phylum. Furthermore, each species has a unique ensemble of nucleotides in the informative tRNA positions, and the similarity between the sets of positions of two distinct species reflects their evolutionary distance. Hence, we term this set of informative positions a "tRNA cipher." It is tempting to suggest that the diverging code identified here might also serve the aminoacyl tRNA synthetase in the task of tRNA recognition.

Many Phenotypes Without Many False Discoveries: Error Controlling Strategies for Multitrait Association Studies.

The genetic basis of multiple phenotypes such as gene expression, metabolite levels, or imaging features is often investigated by testing a large collection of hypotheses, probing the existence of association between each of the traits and hundreds of thousands of genotyped variants. Appropriate multiplicity adjustment is crucial to guarantee replicability of findings, and the false discovery rate (FDR) is frequently adopted as a measure of global error. In the interest of interpretability, results are often summarized so that reporting focuses on variants discovered to be associated to some phenotypes. We show that applying FDR-controlling procedures on the entire collection of hypotheses fails to control the rate of false discovery of associated variants as well as the expected value of the average proportion of false discovery of phenotypes influenced by such variants. We propose a simple hierarchical testing procedure that allows control of both these error rates and provides a more reliable basis for the identification of variants with functional effects. We demonstrate the utility of this approach through simulation studies comparing various error rates and measures of power for genetic association studies of multiple traits. Finally, we apply the proposed method to identify genetic variants that impact flowering phenotypes in Arabidopsis thaliana, expanding the set of discoveries.

Deciding whether follow-up studies have replicated findings in a preliminary large-scale omics study.

We propose a formal method to declare that findings from a primary study have been replicated in a follow-up study. Our proposal is appropriate for primary studies that involve large-scale searches for rare true positives (i.e., needles in a haystack). Our proposal assigns an r value to each finding; this is the lowest false discovery rate at which the finding can be called replicated. Examples are given and software is available.

Psychophysiological whole-brain network clustering based on connectivity dynamics analysis in naturalistic conditions.

We introduce a novel method for delineating context-dependent functional brain networks whose connectivity dynamics are synchronized with the occurrence of a specific psychophysiological process of interest. In this method of context-related network dynamics analysis (CRNDA), a continuous psychophysiological index serves as a reference for clustering the whole-brain into functional networks. We applied CRNDA to fMRI data recorded during the viewing of a sadness-inducing film clip. The method reliably demarcated networks in which temporal patterns of connectivity related to the time series of reported emotional intensity. Our work successfully replicated the link between network connectivity and emotion rating in an independent sample group for seven of the networks. The demarcated networks have clear common functional denominators. Three of these networks overlap with distinct empathy-related networks, previously identified in distinct sets of studies. The other networks are related to sensorimotor processing, language, attention, and working memory. The results indicate that CRNDA, a data-driven method for network clustering that is sensitive to transient connectivity patterns, can productively and reliably demarcate networks that follow psychologically meaningful processes. Hum Brain Mapp 37:4654-4672, 2016. © 2016 Wiley Periodicals, Inc.

Measuring behavior of animal models: faults and remedies.

Widely used behavioral assays need re-evaluation and validation against their intended use. We focus here on measures of chronic anxiety in mouse models and posit that widely used assays such as the open-field test are performed at the wrong time, for inadequate durations and using inappropriate mouse strains. We propose that behavioral assays be screened for usefulness on the basis of their replicability across laboratories.

Sharing clinical trial data on patient level: opportunities and challenges.

In recent months one of the most controversially discussed topics among regulatory agencies, the pharmaceutical industry, journal editors, and academia has been the sharing of patient-level clinical trial data. Several projects have been started such as the European Medicines Agency´s (EMA) "proactive publication of clinical trial data", the BMJ open data campaign, or the AllTrials initiative. The executive director of the EMA, Dr. Guido Rasi, has recently announced that clinical trial data on patient level will be published from 2014 onwards (although it has since been delayed). The EMA draft policy on proactive access to clinical trial data was published at the end of June 2013 and open for public consultation until the end of September 2013. These initiatives will change the landscape of drug development and publication of medical research. They provide unprecedented opportunities for research and research synthesis, but pose new challenges for regulatory authorities, sponsors, scientific journals, and the public. Besides these general aspects, data sharing also entails intricate biostatistical questions such as problems of multiplicity. An important issue in this respect is the interpretation of multiple statistical analyses, both prospective and retrospective. Expertise in biostatistics is needed to assess the interpretation of such multiple analyses, for example, in the context of regulatory decision-making by optimizing procedural guidance and sophisticated analysis methods.

Quantifying the buildup in extent and complexity of free exploration in mice.

To obtain a perspective on an animal's own functional world, we study its behavior in situations that allow the animal to regulate the growth rate of its behavior and provide us with the opportunity to quantify its moment-by-moment developmental dynamics. Thus, we are able to show that mouse exploratory behavior consists of sequences of repeated motion: iterative processes that increase in extent and complexity, whose presumed function is a systematic active management of input acquired during the exploration of a novel environment. We use this study to demonstrate our approach to quantifying behavior: targeting aspects of behavior that are shown to be actively managed by the animal, and using measures that are discriminative across strains and treatments and replicable across laboratories.

A novel performance scoring quantification framework for stress test set-ups.

Stress tests, e.g., the cardiac stress test, are standard clinical screening tools aimed to unmask clinical pathology. As such stress tests indirectly measure physiological reserves. The term reserve has been developed to account for the dis-junction, often observed, between pathology and clinical manifestation. It describes a physiological capacity that is utilized in demanding situations. However, developing a new and reliable stress test based screening tool is complex, prolonged, and relies extensively on domain knowledge. We propose a novel distributional-free machine-learning framework, the Stress Test Performance Scoring (STEPS) framework, to model expected performance in a stress test. A performance scoring function is trained with measures taken during the performance in a given task while exploiting information regarding the stress test set-up and subjects' medical state. Multiple ways of aggregating performance scores at different stress levels are suggested and are examined with an extensive simulation study. When applied to a real-world data example, an AUC of 84.35[95%CI: 70.68 - 95.13] was obtained for the STEPS framework to distinguish subjects with neurodegeneration from controls. In summary, STEPS improved screening by exploiting existing domain knowledge and state-of-the-art clinical measures. The STEPS framework can ease and speed up the production of new stress tests.

Decoding human spontaneous spiking activity in medial temporal lobe from scalp EEG.

Linking scalp electroencephalography (EEG) signals and spontaneous firing activity from deep nuclei in humans is not trivial. To examine this, we analyzed simultaneous recordings of scalp EEG and unit activity in deeply located sites recorded overnight from patients undergoing pre-surgical invasive monitoring. We focused on modeling the within-subject average unit activity of two medial temporal lobe areas: amygdala and hippocampus. Linear regression model correlates the units' average firing activity to spectral features extracted from the EEG during wakefulness or non-REM sleep. We show that changes in mean firing activity in both areas and states can be estimated from EEG (Pearson r > 0.2, p≪0.001). Region specificity was shown with respect to other areas. Both short- and long-term fluctuations in firing rates contributed to the model accuracy. This demonstrates that scalp EEG frequency modulations can predict changes in neuronal firing rates, opening a new horizon for non-invasive neurological and psychiatric interventions.

Developing a length of stay prediction model for newborns, achieving better accuracy with greater usability.

BACKGROUND: One in ten newborn children is born prematurely. The elongated length of stay (LOS) of these children in the Neonatal Intensive Care Unit (NICU) has important implications on hospital occupancy figures, healthcare and management costs, as well as the psychology of parents. In order to allow accurate planning and resource allocation, this study aims to create a generalizable and robust model to predict the NICU LOS of preterm newborns. METHODS: Data were collected from a large tertiary center NICU between 2011 and 2018 and relates to 5,362 newborns. The selected model was externally validated using a data set of 8,768 newborns from another tertiary center NICU. This report compares several models, such as Random Forest (RF), quantile RF, and other feature selection methods, including LASSO and AIC step-forward selection. In addition, a novel step-forward selection based on False Discovery Rate (FDR) for quantile regression is presented and evaluated. RESULTS: A high-orderquantile regression model for predicting preterm newborns' LOS that uses only four features available at birth had more attractive properties than other richer ones. The model achieved a Mean Absolute Error (MAE) of 6.26 days on the internal validation set (average LOS 27.04) and an MAE of 6.04 days on the external validation set (average LOS 29.32). The suggested model surpassed the accuracy obtained by models in the literature. It is shown empirically that the FDR-based selection has better properties than the AIC-based step-forward selection approach. CONCLUSION: This paper demonstrates a process to create a predictive model for NICU LOS in preterm newborns, where each step is reasoned. We obtain a simple and robust model for NICU LOS prediction, which achieves far better results than the current model used for financing NICUs. Utilizing this model, we have created an easy-to-use online web application to ease parents' worries and to assist NICU management: https://tzviel.shinyapps.io/calcuLOS.

Revisiting the operational RNA code for amino acids: Ensemble attributes and their implications.

It has been suggested that tRNA acceptor stems specify an operational RNA code for amino acids. In the last 20 years several attributes of the putative code have been elucidated for a small number of model organisms. To gain insight about the ensemble attributes of the code, we analyzed 4925 tRNA sequences from 102 bacterial and 21 archaeal species. Here, we used a classification and regression tree (CART) methodology, and we found that the degrees of degeneracy or specificity of the RNA codes in both Archaea and Bacteria differ from those of the genetic code. We found instances of taxon-specific alternative codes, i.e., identical acceptor stem determinants encrypting different amino acids in different species, as well as instances of ambiguity, i.e., identical acceptor stem determinants encrypting two or more amino acids in the same species. When partitioning the data by class of synthetase, the degree of code ambiguity was significantly reduced. In cryptographic terms, a plausible interpretation of this result is that the class distinction in synthetases is an essential part of the decryption rules for resolving the subset of RNA code ambiguities enciphered by identical acceptor stem determinants of tRNAs acylated by enzymes belonging to the two classes. In evolutionary terms, our findings lend support to the notion that in the pre-DNA world, interactions between tRNA acceptor stems and synthetases formed the basis for the distinction between the two classes; hence, ambiguities in the ancient RNA code were pivotal for the fixation of these enzymes in the genomes of ancestral prokaryotes.

Freedom of movement and the stability of its unfolding in free exploration of mice.

Exploration is a central component of human and animal behavior that has been studied in rodents for almost a century. The measures used by neuroscientists to characterize full-blown exploration are limited in exposing the dynamics of the exploratory process, leaving the morphogenesis of its structure and meaning hidden. By unfettering exploration from constraints imposed by hunger, thirst, coercion, and the confines of small cage and short session, using advanced computational tools, we reveal its meaning in the operational world of the mouse. Exploration consists of reiterated roundtrips of increasing amplitude and freedom, involving an increase in the number of independent dimensions along which the mouse moves (macro degrees of freedom). This measurable gradient can serve as a standard reference scale for the developmental dynamics of some aspects of the mouse's emotional-cognitive state and for the study of the interface between behavior and the neurophysiologic and genetic processes mediating it.