A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction.

The utility of mouse and rat studies critically depends on their replicability in other laboratories. A widely advocated approach to improving replicability is through the rigorous control of predefined animal or experimental conditions, known as standardization. However, this approach limits the generalizability of the findings to only to the standardized conditions and is a potential cause rather than solution to what has been called a replicability crisis. Alternative strategies include estimating the heterogeneity of effects across laboratories, either through designs that vary testing conditions, or by direct statistical analysis of laboratory variation. We previously evaluated our statistical approach for estimating the interlaboratory replicability of a single laboratory discovery. Those results, however, were from a well-coordinated, multi-lab phenotyping study and did not extend to the more realistic setting in which laboratories are operating independently of each other. Here, we sought to test our statistical approach as a realistic prospective experiment, in mice, using 152 results from 5 independent published studies deposited in the Mouse Phenome Database (MPD). In independent replication experiments at 3 laboratories, we found that 53 of the results were replicable, so the other 99 were considered non-replicable. Of the 99 non-replicable results, 59 were statistically significant (at 0.05) in their original single-lab analysis, putting the probability that a single-lab statistical discovery was made even though it is non-replicable, at 59.6%. We then introduced the dimensionless "Genotype-by-Laboratory" (GxL) factor-the ratio between the standard deviations of the GxL interaction and the standard deviation within groups. Using the GxL factor reduced the number of single-lab statistical discoveries and alongside reduced the probability of a non-replicable result to be discovered in the single lab to 12.1%. Such reduction naturally leads to reduced power to make replicable discoveries, but this reduction was small (from 87% to 66%), indicating the small price paid for the large improvement in replicability. Tools and data needed for the above GxL adjustment are publicly available at the MPD and will become increasingly useful as the range of assays and testing conditions in this resource increases.

Reproducibility and replicability of rodent phenotyping in preclinical studies.

The scientific community is increasingly concerned with the proportion of published "discoveries" that are not replicated in subsequent studies. The field of rodent behavioral phenotyping was one of the first to raise this concern, and to relate it to other methodological issues: the complex interaction between genotype and environment; the definitions of behavioral constructs; and the use of laboratory mice and rats as model species for investigating human health and disease mechanisms. In January 2015, researchers from various disciplines gathered at Tel Aviv University to discuss these issues. The general consensus was that the issue is prevalent and of concern, and should be addressed at the statistical, methodological and policy levels, but is not so severe as to call into question the validity and the usefulness of model organisms as a whole. Well-organized community efforts, coupled with improved data and metadata sharing, have a key role in identifying specific problems and promoting effective solutions. Replicability is closely related to validity, may affect generalizability and translation of findings, and has important ethical implications.

Mining mouse behavior for patterns predicting psychiatric drug classification.

RATIONALE: In psychiatric drug discovery, a critical step is predicting the psychopharmacological effect and therapeutic potential of novel (or repurposed) compounds early in the development process. This process is hampered by the need to utilize multiple disorder-specific and labor-intensive behavioral assays. OBJECTIVES: This study aims to investigate the feasibility of a single high-throughput behavioral assay to classify psychiatric drugs into multiple psychopharmacological classes. METHODS: Using Pattern Array, a procedure for data mining exploratory behavior in mice, we mined ~100,000 complex movement patterns for those that best predict psychopharmacological class and dose. The best patterns were integrated into a classification model that assigns psychopharmacological compounds to one of six clinically relevant classes--antipsychotic, antidepressant, opioids, psychotomimetic, psychomotor stimulant, and α-adrenergic. RESULTS: Surprisingly, only a small number of well-chosen behaviors were required for successful class prediction. One of them, a behavior termed "universal drug detector", was dose-dependently decreased by drugs from all classes, thus providing a sensitive index of psychopharmacological activity. In independent validation in a blind fashion, simulating the process of in vivo pre-clinical drug screening, the classification model correctly classified nine out of 11 "unknown" compounds. Interestingly, even "misclassifications" match known alternate therapeutic indications, illustrating drug "repurposing" potential. CONCLUSIONS: Unlike standard animal models, the discovered classification model can be systematically updated to improve its predictive power and add therapeutic classes and subclasses with each additional diversification of the database. Our study demonstrates the power of data mining approaches for behavior analysis, using multiple measures in parallel for drug screening and behavioral phenotyping.

Ten ways to improve the quality of descriptions of whole-animal movement.

The demand for replicability of behavioral results across laboratories is viewed as a burden in behavior genetics. We demonstrate how it can become an asset offering a quantitative criterion that guides the design of better ways to describe behavior. Passing the high benchmark dictated by the replicability demand requires less stressful and less restraining experimental setups, less noisy data, individually customized cutoff points between the building blocks of movement, and less variable yet discriminative dynamic representations that would capture more faithfully the nature of the behavior, unmasking similarities and differences and revealing novel animal-centered measures. Here we review ten tools that enhance replicability without compromising discrimination. While we demonstrate the usefulness of these tools in the context of inbred mouse exploratory behavior they can readily be used in any study involving a high-resolution analysis of spatial behavior. Viewing replicability as a design concept and using the ten methodological improvements may prove useful in many fields not necessarily related to spatial behavior.

Drug discovery in psychiatric illness: mining for gold.

The discovery of truly efficacious treatments that lead to full recovery is a daunting task in psychiatric illness. A systems-based orientation to in vivo pharmacology has been suggested as a way to transform psychiatric drug discovery and development. A critical catalyst in the success of recent systems biology efforts has been the incorporation of data mining strategies. Our approach to the drug discovery problem has been to utilize the whole animal to provide a systems response that is subsequently mined for predictive attributes with known psychopharmacological value. Our in vivo data mining approach, termed Pattern Array, establishes a framework for screening novel chemical entities based upon a response that represents the net pharmacological effect on the system of interest, namely the central nervous system (CNS). Large scale screening of small molecules by non-conventional approaches such as this at a systems level may improve the identification of novel chemical entities with psychiatric utility. This type of approach will compliment the more labor-intensive models based upon construct validity. It will take the collective effort of many disciplines and numerous strategies in close association with clinical colleagues to address quality of life issues and breakthrough treatment barriers in psychiatric illness.

A data mining approach to in vivo classification of psychopharmacological drugs.

Data mining is a powerful bioinformatics strategy that has been successfully applied in vitro to screen for gene-expression profiles predicting toxicological or carcinogenic response ('class predictors'). In this report we used a data mining algorithm named Pattern Array (PA) in vivo to analyze mouse open-field behavior and characterize the psychopharmacological effects of three drug classes--psychomotor stimulant, opioid, and psychotomimetic. PA represents rodent movement with approximately 100,000 complex patterns, defined as multiple combinations of several ethologically relevant variables, and mines them for those that maximize any effect of interest, such as the difference between drug classes. We show that PA can discover behavioral predictors of all three drug classes, thus developing a reliable drug-classification scheme in small group sizes. The discovered predictors showed orderly dose dependency despite being explicitly mined only for class differences, with the high doses scoring 4-10 standard deviations from the vehicle group. Furthermore, these predictors correctly classified in a dose-dependent manner four 'unknown' drugs (ie that were not used in the training process), and scored a mixture of a psychomotor stimulant and an opioid as being intermediate between these two classes. The isolated behaviors were highly heritable (h(2)>50%) and replicable as determined in 10 inbred strains across three laboratories. PA can in principle be applied for mining behaviors predicting additional properties, such as within-class differences between drugs and within-drug dose-response, all of which can be measured automatically in a single session per animal in an open-field arena, suggesting a high potential as a tool in psychotherapeutic drug discovery.

Data mining in a behavioral test detects early symptoms in a model of amyotrophic lateral sclerosis.

"What's wrong with my genetically engineered animal?" is a common yet often difficult to answer question in behavioral phenotyping. We present here a method termed Pattern Array for mining movement patterns and isolating those that best capture an effect of a genetic manipulation. We demonstrate the method by searching for early motor symptoms in the open-field behavior of SOD1 mutant rats, an animal model of amyotrophic lateral sclerosis. Pattern Array was able to identify a unique motor pattern that differentiated the SOD1 mutants from the wild-type controls 2 months before disease onset. This pattern included heavy braking while moving near the arena wall but turning away from it. SOD1 mutants performed this pattern significantly less than wild-type controls in 2 independent data sets. At such early age the SOD1 mutants could not be differentiated from the controls by standard behavioral measures or by subjective observation. The early discovered symptom may enable investigators to test therapies aimed for intervention rather than remediation. Our results demonstrate the feasibility and potential of detecting subtle behavioral effects using data mining strategies.

Associating quantitative behavioral traits with gene expression in the brain: searching for diamonds in the hay.

UNLABELLED: Gene expression and phenotypic functionality can best be associated when they are measured quantitatively within the same experiment. The analysis of such a complex experiment is presented, searching for associations between measures of exploratory behavior in mice and gene expression in brain regions. The analysis of such experiments raises several methodological problems. First and foremost, the size of the pool of potential discoveries being screened is enormous yet only few biologically relevant findings are expected, making the problem of multiple testing especially severe. We present solutions based on screening by testing related hypotheses, then testing the hypotheses of interest. In one variant the subset is selected directly, in the other one a tree of hypotheses is tested hierarchical; both variants control the False Discovery Rate (FDR). Other problems in such experiments are in the fact that the level of data aggregation may be different for the quantitative traits (one per animal) and gene expression measurements (pooled across animals); in that the association may not be linear; and in the resolution of interest only few replications exist. We offer solutions to these problems as well. The hierarchical FDR testing strategies presented here can serve beyond the structure of our motivating example study to any complex microarray study. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Combined application of behavior genetics and microarray analysis to identify regional expression themes and gene-behavior associations.

In this report we link candidate genes to complex behavioral phenotypes by using a behavior genetics approach. Gene expression signatures were generated for the prefrontal cortex, ventral striatum, temporal lobe, periaqueductal gray, and cerebellum in eight inbred strains from priority group A of the Mouse Phenome Project. Bioinformatic analysis of regionally enriched genes that were conserved across all strains revealed both functional and structural specialization of particular brain regions. For example, genes encoding proteins with demonstrated anti-apoptotic function were over-represented in the cerebellum, whereas genes coding for proteins associated with learning and memory were enriched in the ventral striatum, as defined by the Expression Analysis Systematic Explorer (EASE) application. Association of regional gene expression with behavioral phenotypes was exploited to identify candidate behavioral genes. Phenotypes that were investigated included anxiety, drug-naive and ethanol-induced distance traveled across a grid floor, and seizure susceptibility. Several genes within the glutamatergic signaling pathway (i.e., NMDA/glutamate receptor subunit 2C, calmodulin, solute carrier family 1 member 2, and glutamine synthetase) were identified in a phenotype-dependent and region-specific manner. In addition to supporting evidence in the literature, many of the genes that were identified could be mapped in silico to surrogate behavior-related quantitative trait loci. The approaches and data set described herein serve as a valuable resource to investigate the genetic underpinning of complex behaviors.

Genotype-environment interactions in mouse behavior: a way out of the problem.

In behavior genetics, behavioral patterns of mouse genotypes, such as inbred strains, crosses, and knockouts, are characterized and compared to associate them with particular gene loci. Such genotype differences, however, are usually established in single-laboratory experiments, and questions have been raised regarding the replicability of the results in other laboratories. A recent multilaboratory experiment found significant laboratory effects and genotype x laboratory interactions even after rigorous standardization, raising the concern that results are idiosyncratic to a particular laboratory. This finding may be regarded by some critics as a serious shortcoming in behavior genetics. A different strategy is offered here: (i) recognize that even after investing much effort in identifying and eliminating causes for laboratory differences, genotype x laboratory interaction is an unavoidable fact of life. (ii) Incorporate this understanding into the statistical analysis of multilaboratory experiments using the mixed model. Such a statistical approach sets a higher benchmark for finding significant genotype differences. (iii) Develop behavioral assays and endpoints that are able to discriminate genetic differences even over the background of the interaction. (iv) Use the publicly available multilaboratory results in single-laboratory experiments. We use software-based strategy for exploring exploration (see) to analyze the open-field behavior in eight genotypes across three laboratories. Our results demonstrate that replicable behavioral measures can be practically established. Even though we address the replicability problem in behavioral genetics, our strategy is also applicable in other areas where concern about replicability has been raised.

Activity density in the open field: a measure for differentiating the effect of psychostimulants.

Traditional open-field activity measures do not provide a sharp behavioral differentiation across psychomotor stimulants such as d-amphetamine (AMPH) and cocaine (COC) in the mouse. We used Software for the Exploration of Exploration (SEE) to investigate and develop a novel behavioral endpoint to characterize the "structure" of AMPH- and COC-induced locomotor behavior in two inbred strains of mouse, C57BL/6 (B6) and DBA/2 (D2). We suggest a measure we term "activity density" as a means to differentiate the behavioral effects of COC and AMPH. Activity density is defined as the activity divided by the range over which it took place. It characterizes the restriction of behavioral repertoire that does not result merely from inactivity. In both the B6 and D2 mice, AMPH increased activity density in a dose-dependent fashion by restricting the range of activity compared with COC doses producing the same level of activity. While AMPH restricted the range in both genotypes, characterizing the geographical region in which the restriction took place further differentiated the genotypes. The newly developed activity density measure thus provides a more general measure than stereotypy of the path, and can differentiate the effects of AMPH and COC both within and across genotypes.

New replicable anxiety-related measures of wall vs center behavior of mice in the open field.

Anxiety is a widely studied psychiatric disorder and is thought to be a complex and multidimensional phenomenon. Sensitive behavioral discrimination of animal models of anxiety is crucial for the elucidation of the behavioral components of anxiety and the physiological processes that mediate them. Commonly used behavior paradigms of anxiety usually include only a few automatically collected measures; these do not exhaust the behavioral richness exhibited by animals, thus perhaps missing important differences between preparations. The aim of the present study was to expand the repertoire of automatically collected measures in a classical test of anxiety: behavior in relation to the wall in the open field. We present an algorithm, based on the Software for the Exploration of Exploration strategy, which automatically partitions the mouse path into intrinsically defined patterns of movement near the wall and in the center. These patterns are used to design new end points, which provide an articulated description of various aspects of behavior near the wall and in the center. Sixteen new end points were designed with data from C57BL/6J and DBA/2J mice tested in three laboratories. The strain differences in all end points were evaluated on another data set to assess their validity and were found to remain stable. Ten of the sixteen end points were found to discriminate between the two strains in a replicable manner. The entire set of end points can be used on various genetic and pharmacological models of anxiety with good prospects of providing fine discrimination in a replicable manner.

The dynamics of spatial behavior: how can robust smoothing techniques help?

A variety of setups and paradigms are used in the neurosciences for automatically tracking the location of an animal in an experiment and for extracting features of interest out of it. Many of these features, however, are critically sensitive to the unavoidable noise and artifacts of tracking. Here, we examine the relevant properties of several smoothing methods and suggest a combination of methods for retrieving locations and velocities and recognizing arrests from time series of coordinates of an animal's center of gravity. We accomplish these by using robust nonparametric methods, such as Running Median (RM) and locally weighted regression methods. The smoothed data may, subsequently, be segmented to obtain discrete behavioral units with proven ethological relevance. New parameters such as the length, duration, maximal speed, and acceleration of these units provide a wealth of measures for, e.g., mouse behavioral phenotyping, studies on spatial orientation in vertebrates and invertebrates, and studies on rodent hippocampal function. This methodology may have implications for many tests of spatial behavior.

Extending SEE for large-scale phenotyping of mouse open-field behavior.

SEE (Software for the Exploration of Exploration) is a visualization and analysis tool designed for the study of open-field behavior in rodents. In this paper, I present new extensions of SEE that were designed to facilitate its use for mouse behavioral phenotyping and, especially, for the problems of discrimination of genotypes and the replication of results across laboratories and experimental conditions. These extensions were specifically designed to promote a new approach in behavioral phenotyping, reminiscent of the approach that has been successfully employed in bioinformatics during recent years. The path coordinates of all animals from many experiments are stored in a database. SEE can be used to query, visualize, and analyze any desirable subsection of this database and to design new measures (endpoints) with increasingly better discriminative power and replicability. The use of the new extensions is demonstrated here in the analysis of results from several experiments and laboratories, with an emphasis on this approach.

SEE locomotor behavior test discriminates C57BL/6J and DBA/2J mouse inbred strains across laboratories and protocol conditions.

Conventional tests of behavioral phenotyping frequently have difficulties differentiating certain genotypes and replicating these differences across laboratories and protocol conditions. This study explores the hypothesis that automated tests can be designed to quantify ethologically relevant behavior patterns that more readily characterize heritable and replicable phenotypes. It used SEE (Strategy for the Exploration of Exploration) to phenotype the locomotor behavior of the C57BL/6 and DBA/2 mouse inbred strains across 3 laboratories. The 2 genotypes differed in 15 different measures of behavior, none of which had a significant genotype-laboratory interaction. Within the same laboratory, most of these differences were replicated in additional experiments despite the test photoperiod phase being changed and saline being injected. Results suggest that well-designed tests may considerably enhance replicability across laboratories.

Darting behavior: a quantitative movement pattern designed for discrimination and replicability in mouse locomotor behavior.

In the open-field behavior of rodents, Software for Exploring Exploration (SEE) can be used for an explicit design of behavioral endpoints with high genotype discrimination and replicability across laboratories. This ability is demonstrated here in the development of a measure for darting behavior. The behavior of two common mouse inbred strains, C57BL/6J (B6) and DBA/2J (D2), was analyzed across three different laboratories, and under the effect of cocaine or amphetamine. "Darting" was defined as having higher acceleration during progression segments while moving less during stops. D2 mice darted significantly more than B6 mice in each laboratory, despite being significantly less active. These differences were maintained following cocaine administration (up to 20mg/kg) and only slightly altered by amphetamine (up to 5mg/kg) despite a several fold increase in activity. The replicability of darting behavior was confirmed in additional experiments distinct from those used for its design. The strategy leading to the darting measure may be used to develop additional discriminative and replicable endpoints of open-field behavior.

Repeated maternal separation does not alter sucrose-reinforced and open-field behaviors.

Repeated separation of rat pups from their mothers has been reported to increase behavioral fearfulness and hypothalamic-pituitary-adrenal (HPA) response to stress. Recently, it was suggested that it might also alter behavioral responses to natural and drug rewards. Here, we studied whether maternal separation (MS) would alter behavioral responses to a sucrose reward. We also tested whether MS would alter behavioral responses in an open-field test using a novel method of analysis [Software for the Exploration of Exploration (SEE)]. Long-Evans rat pups were exposed to either 180 min of MS, 15 min of separation [early handling (EH)] or left undisturbed [nonhandled (NH)] from postnatal day (PND) 3 to 14. The adult male offspring were tested for sucrose solution preference using a two-bottle free-choice test, operant response for sucrose under fixed ratio and progressive ratio (PR) schedules of reinforcement and response to a novel environment (open-field test). MS had no effect on sucrose preference or operant responding for sucrose reward. In the open-field test, NH rats showed a brief decrease in locomotor response, but MS rats did not differ from the NH and EH groups in the other behavioral measures. Thus, under the conditions of the present study, MS did not appear to alter reward-related processes and also had a minimal effect on open-field behavior.