Heterogeneity of animal experiments and how to deal with it.

Heterogeneity of study samples is ubiquitous in animal experiments. Here, we discuss the different options of how to deal with heterogeneity in the statistical analysis of a single experiment. Specifically, data from different sub-groups (e.g. sex, strain, age cohorts) may be analysed separately, heterogenization factors may be ignored and data pooled for analysis, or heterogenization factors may be included as additional variables in the statistical model. The cost of ignoring a heterogenization factor is an inflated estimate of the variance and a consequent loss of statistical power. Therefore, it is usually preferable to include the heterogenization factor in the statistical model, especially if the heterogenization factor has been introduced intentionally (e.g. using both sexes). If heterogenization factors are included, they can be treated either as fixed factors in an analysis of variance design or sometimes as random effects in mixed effects regression models. Finally, for an appropriate sample size estimation, it is necessary to decide whether to treat heterogenization factors as nuisance variables, or whether the experiment should be powered to be able to detect not only the main effect of the treatment but also interactions between heterogenization factors and the treatment variable.

Evidence of HARKing in mouse behavioural tests of anxiety.

Over the last decades, behavioural tests in animals, especially rodents, have been a standard screening method to determine the mechanisms of action and efficacy of psychopharmacological compounds. Yet, recently the reproducibility of some of these tests has been questioned. Based on a systematic review of the sensitivity of mouse behavioural tests to anxiolytic drugs, we analysed behavioural outcomes extracted from 206 studies testing the effect of diazepam in either the open-field test or the hole-board test. Surprisingly, we found that both the rationale given for using the test, whether to detect anxiolytic or sedative effects, and the predicted effect of diazepam, anxiolytic or sedative, strongly depended on the reported test results. The most likely explanation for such strong dependency is post hoc reasoning, also called hypothesizing after the results are known (HARKing). HARKing can invalidate study outcomes and hampers evidence synthesis by inflating effect sizes. It may also lead researchers into blind alleys, and waste animals, time and resources for inconclusive research.

The impact of cage dividers on mouse aggression, dominance and hormone levels.

Home cage aggression in group-housed male mice is a major welfare concern and may compromise animal research. Conventional cages prevent flight or retreat from sight, increasing the risk that agonistic encounters will result in injury. Moreover, depending on social rank, mice vary in their phenotype, and these effects seem highly variable and dependent on the social context. Interventions that reduce aggression, therefore, may reduce not only injuries and stress, but also variability between cage mates. Here we housed male mice (Balb/c and SWISS, group sizes of three and five) with or without partial cage dividers for two months. Mice were inspected for wounding weekly and home cages were recorded during housing and after 6h isolation housing, to assess aggression and assign individual social ranks. Fecal boli and fur were collected to quantify steroid levels. We found no evidence that the provision of cage dividers improves the welfare of group housed male mice; The prevalence of injuries and steroid levels was similar between the two housing conditions and aggression was reduced only in Balb/c strain. However, mice housed with cage dividers developed less despotic hierarchies and had more stable social ranks. We also found a relationship between hormone levels and social rank depending on housing type. Therefore, addition of cage dividers may play a role in stabilizing social ranks and modulating the activation of hypothalamic-pituitary-adrenal (HPA) and hypothalamic-pituitary-gonadal (HPG) axes, thus reducing phenotypic variability between mice of different ranks.

Using mice from different breeding sites fails to improve replicability of results from single-laboratory studies.

Theoretical and empirical evidence indicates that low external validity due to rigorous standardization of study populations is a cause of poor replicability in animal research. Here we report a multi-laboratory study aimed at investigating whether heterogenization of study populations by using animals from different breeding sites increases the replicability of results from single-laboratory studies. We used male C57BL/6J mice from six different breeding sites to test a standardized against a heterogenized (HET) study design in six independent replicate test laboratories. For the standardized design, each laboratory ordered mice from a single breeding site (each laboratory from a different one), while for the HET design, each laboratory ordered proportionate numbers of mice from the five remaining breeding sites. To test our hypothesis, we assessed 14 outcome variables, including body weight, behavioral measures obtained from a single session on an elevated plus maze, and clinical blood parameters. Both breeding site and test laboratory affected variation in outcome variables, but the effect of test laboratory was more pronounced for most outcome variables. Moreover, heterogenization of study populations by breeding site (HET) did not reduce variation in outcome variables between test laboratories, which was most likely due to the fact that breeding site had only little effect on variation in outcome variables, thereby limiting the scope for HET to reduce between-lab variation. We conclude that heterogenization of study populations by breeding site has limited capacity for improving the replicability of results from single-laboratory animal studies.

Practical solutions for including sex as a biological variable (SABV) in preclinical neuropsychopharmacological research.

Recently, many funding agencies have released guidelines on the importance of considering sex as a biological variable (SABV) as an experimental factor, aiming to address sex differences and avoid possible sex biases to enhance the reproducibility and translational relevance of preclinical research. In neuroscience and pharmacology, the female sex is often omitted from experimental designs, with researchers generalizing male-driven outcomes to both sexes, risking a biased or limited understanding of disease mechanisms and thus potentially ineffective therapeutics. Herein, we describe key methodological aspects that should be considered when sex is factored into in vitro and in vivo experiments and provide practical knowledge for researchers to incorporate SABV into preclinical research. Both age and sex significantly influence biological and behavioral processes due to critical changes at different timepoints of development for males and females and due to hormonal fluctuations across the rodent lifespan. We show that including both sexes does not require larger sample sizes, and even if sex is included as an independent variable in the study design, a moderate increase in sample size is sufficient. Moreover, the importance of tracking hormone levels in both sexes and the differentiation between sex differences and sex-related strategy in behaviors are explained. Finally, the lack of robust data on how biological sex influences the pharmacokinetic (PK), pharmacodynamic (PD), or toxicological effects of various preclinically administered drugs to animals due to the exclusion of female animals is discussed, and methodological strategies to enhance the rigor and translational relevance of preclinical research are proposed.

Development of a data-driven method for assessing health and welfare in the most common livestock species in Switzerland: The Smart Animal Health project.

Improving animal health and welfare in livestock systems depends on reliable proxies for assessment and monitoring. The aim of this project was to develop a novel method that relies on animal-based indicators and data-driven metrics for assessing health and welfare at farm level for the most common livestock species in Switzerland. Method development followed a uniform multi-stage process for each species. Scientific literature was systematically reviewed to identify potential health and welfare indicators for cattle, sheep, goats, pigs and poultry. Suitable indicators were applied in the field and compared with outcomes of the Welfare Quality® scores of a given farm. To identify farms at risk for violations of animal welfare regulations, several agricultural and animal health databases were interconnected and various supervised machine-learning techniques were applied to model the status of farms. Literature reviews identified a variety of indicators, some of which are well established, while others lack reliability or practicability, or still need further validation. Data quality and availability strongly varied among animal species, with most data available for dairy cows and pigs. Data-based indicators were almost exclusively limited to the categories "Animal health" and "Husbandry and feeding". The assessment of "Appropriate behavior" and "Freedom from pain, suffering, harm and anxiety" depended largely on indicators that had to be assessed and monitored on-farm. The different machine-learning techniques used to identify farms for risk-based animal welfare inspections reached similar classification performances with sensitivities above 80%. Features with the highest predictive weights were: Participation in federal ecological and animal welfare programs, farm demographics and farmers' notification discipline for animal movements. A common method with individual sets of indicators for each species was developed. The results show that, depending on data availability for the individual animal categories, models based on proxy data can achieve high correlations with animal health and welfare assessed on-farm. Nevertheless, for sufficient validity, a combination of data-based indicators and on-farm assessments is currently required. For a broad implementation of the methods, alternatives to extensive manual on-farm assessments are needed, whereby smart farming technologies have great potential to support the assessment if the specific monitoring goals are defined.

Handling method affects measures of anxiety, but not chronic stress in mice.

Studies in mice have shown that less aversive handling methods (e.g. tunnel or cup handling) can reduce behavioural measures of anxiety in comparison to picking mice up by their tail. Despite such evidence, tail handling continues to be used routinely. Besides resistance to change accustomed procedures, this may also be due to the fact that current evidence in support of less aversive handling is mostly restricted to effects of extensive daily handling, which may not apply to routine husbandry practices. The aim of our study was to assess whether, and to what extent, different handling methods during routine husbandry induce differences in behavioural and physiological measures of stress in laboratory mice. To put the effects of handling method in perspective with chronic stress, we compared handling methods to a validated paradigm of unpredictable chronic mild stress (UCMS). We housed mice of two strains (Balb/c and C57BL/6) and both sexes either under standard laboratory conditions (CTRL) or under UCMS. Half of the animals from each housing condition were tail handled and half were tunnel handled twice per week, once during a cage change and once for a routine health check. We found strain dependent effects of handling method on behavioural measures of anxiety: tunnel handled Balb/c mice interacted with the handler more than tail handled conspecifics, and tunnel handled CTRL mice showed increased open arm exploration in the elevated plus-maze. Mice undergoing UCMS showed increased plasma corticosterone levels and reduced sucrose preference. However, we found no effect of handling method on these stress-associated measures. Our results therefore indicate that routine tail handling can affect behavioural measures of anxiety, but may not be a significant source of chronic husbandry stress. Our results also highlight strain dependent responses to handling methods.

Correction: Do multiple experimenters improve the reproducibility of animal studies?

[This corrects the article DOI: 10.1371/journal.pbio.3001564.].

Systematic assessment of the replicability and generalizability of preclinical findings: Impact of protocol harmonization across laboratory sites.

The influence of protocol standardization between laboratories on their replicability of preclinical results has not been addressed in a systematic way. While standardization is considered good research practice as a means to control for undesired external noise (i.e., highly variable results), some reports suggest that standardized protocols may lead to idiosyncratic results, thus undermining replicability. Through the EQIPD consortium, a multi-lab collaboration between academic and industry partners, we aimed to elucidate parameters that impact the replicability of preclinical animal studies. To this end, 3 experimental protocols were implemented across 7 laboratories. The replicability of results was determined using the distance travelled in an open field after administration of pharmacological compounds known to modulate locomotor activity (MK-801, diazepam, and clozapine) in C57BL/6 mice as a worked example. The goal was to determine whether harmonization of study protocols across laboratories improves the replicability of the results and whether replicability can be further improved by systematic variation (heterogenization) of 2 environmental factors (time of testing and light intensity during testing) within laboratories. Protocols were tested in 3 consecutive stages and differed in the extent of harmonization across laboratories and standardization within laboratories: stage 1, minimally aligned across sites (local protocol); stage 2, fully aligned across sites (harmonized protocol) with and without systematic variation (standardized and heterogenized cohort); and stage 3, fully aligned across sites (standardized protocol) with a different compound. All protocols resulted in consistent treatment effects across laboratories, which were also replicated within laboratories across the different stages. Harmonization of protocols across laboratories reduced between-lab variability substantially compared to each lab using their local protocol. In contrast, the environmental factors chosen to introduce systematic variation within laboratories did not affect the behavioral outcome. Therefore, heterogenization did not reduce between-lab variability further compared to the harmonization of the standardized protocol. Altogether, these findings demonstrate that subtle variations between lab-specific study protocols may introduce variation across independent replicate studies even after protocol harmonization and that systematic heterogenization of environmental factors may not be sufficient to account for such between-lab variation. Differences in replicability of results within and between laboratories highlight the ubiquity of study-specific variation due to between-lab variability, the importance of transparent and fine-grained reporting of methodologies and research protocols, and the importance of independent study replication.

Reliability of common mouse behavioural tests of anxiety: A systematic review and meta-analysis on the effects of anxiolytics.

The validity of widely used rodent behavioural tests of anxiety has been questioned, as they often fail to produce consistent results across independent replicate studies. In this study, we assessed the sensitivity of common behavioural tests of anxiety in mice to detect anxiolytic effects of drugs prescribed to treat anxiety in humans. We conducted a pre-registered systematic review of 814 studies reporting effects of 25 anxiolytic compounds using common behavioural tests for anxiety. Meta-analyses of effect sizes of treatments showed that only two out of 17 commonly used test measures reliably detected effects of anxiolytic compounds. We report considerable between-study variation in size and even direction of effects of most anxiolytics on most outcome variables. Our findings indicate a general lack of sensitivity of those behavioural tests and cast serious doubt on both construct and predictive validity of most of these tests. In view of scientifically valid and ethically responsible research, we call for a revision of behavioural tests of anxiety in mice and the development of more predictive tests.

The rearing environment persistently modulates mouse phenotypes from the molecular to the behavioural level.

The phenotype of an organism results from its genotype and the influence of the environment throughout development. Even when using animals of the same genotype, independent studies may test animals of different phenotypes, resulting in poor replicability due to genotype-by-environment interactions. Thus, genetically defined strains of mice may respond differently to experimental treatments depending on their rearing environment. However, the extent of such phenotypic plasticity and its implications for the replicability of research findings have remained unknown. Here, we examined the extent to which common environmental differences between animal facilities modulate the phenotype of genetically homogeneous (inbred) mice. We conducted a comprehensive multicentre study, whereby inbred C57BL/6J mice from a single breeding cohort were allocated to and reared in 5 different animal facilities throughout early life and adolescence, before being transported to a single test laboratory. We found persistent effects of the rearing facility on the composition and heterogeneity of the gut microbial community. These effects were paralleled by persistent differences in body weight and in the behavioural phenotype of the mice. Furthermore, we show that environmental variation among animal facilities is strong enough to influence epigenetic patterns in neurons at the level of chromatin organisation. We detected changes in chromatin organisation in the regulatory regions of genes involved in nucleosome assembly, neuronal differentiation, synaptic plasticity, and regulation of behaviour. Our findings demonstrate that common environmental differences between animal facilities may produce facility-specific phenotypes, from the molecular to the behavioural level. Furthermore, they highlight an important limitation of inferences from single-laboratory studies and thus argue that study designs should take environmental background into account to increase the robustness and replicability of findings.

Do multiple experimenters improve the reproducibility of animal studies?

The credibility of scientific research has been seriously questioned by the widely claimed "reproducibility crisis". In light of this crisis, there is a growing awareness that the rigorous standardisation of experimental conditions may contribute to poor reproducibility of animal studies. Instead, systematic heterogenisation has been proposed as a tool to enhance reproducibility, but a real-life test across multiple independent laboratories is still pending. The aim of this study was therefore to test whether heterogenisation of experimental conditions by using multiple experimenters improves the reproducibility of research findings compared to standardised conditions with only one experimenter. To this end, we replicated the same animal experiment in 3 independent laboratories, each employing both a heterogenised and a standardised design. Whereas in the standardised design, all animals were tested by a single experimenter; in the heterogenised design, 3 different experimenters were involved in testing the animals. In contrast to our expectation, the inclusion of multiple experimenters in the heterogenised design did not improve the reproducibility of the results across the 3 laboratories. Interestingly, however, a variance component analysis indicated that the variation introduced by the different experimenters was not as high as the variation introduced by the laboratories, probably explaining why this heterogenisation strategy did not bring the anticipated success. Even more interestingly, for the majority of outcome measures, the remaining residual variation was identified as an important source of variance accounting for 41% (CI95 [34%, 49%]) to 72% (CI95 [58%, 88%]) of the observed total variance. Despite some uncertainty surrounding the estimated numbers, these findings argue for systematically including biological variation rather than eliminating it in animal studies and call for future research on effective improvement strategies.

A reaction norm perspective on reproducibility.

Reproducibility in biomedical research, and more specifically in preclinical animal research, has been seriously questioned. Several cases of spectacular failures to replicate findings published in the primary scientific literature have led to a perceived reproducibility crisis. Diverse threats to reproducibility have been proposed, including lack of scientific rigour, low statistical power, publication bias, analytical flexibility and fraud. An important aspect that is generally overlooked is the lack of external validity caused by rigorous standardization of both the animals and the environment. Here, we argue that a reaction norm approach to phenotypic variation, acknowledging gene-by-environment interactions, can help us seeing reproducibility of animal experiments in a new light. We illustrate how dominating environmental effects can affect inference and effect size estimates of studies and how elimination of dominant factors through standardization affects the nature of the expected phenotype variation through the reaction norms of small effect. Finally, we discuss the consequences of reaction norms of small effect for statistical analysis, specifically for random effect latent variable models and the random lab model.

'Puppy Dog Eyes' Are Associated With Eye Movements, Not Communication.

The inner brow raiser is a muscle movement that increases the size of the orbital cavity, leading to the appearance of so-called 'puppy dog eyes'. In domestic dogs, this expression was suggested to be enhanced by artificial selection and to play an important role in the dog-human relationship. Production of the inner brow raiser has been shown to be sensitive to the attentive stance of a human, suggesting a possible communicative function. However, it has not yet been examined whether it is sensitive to human presence. In the current study, we aimed to test whether the inner brow raiser differs depending on the presence or absence of an observer. We used two versions of a paradigm in an equivalent experimental setting in which dogs were trained to expect a reward; however, the presence/absence of a person in the test apparatus was varied. In the social context, a human facing the dog delivered the reward; in the non-social context, reward delivery was automatized. If the inner brow raiser has a communicative function and dogs adjust its expression to an audience, we expect it to be shown more frequently in the social context (when facing a person in the apparatus) than in the non-social context (when facing the apparatus without a person inside). The frequency of the inner brow raiser differed between the two contexts, but contrary to the prediction, it was shown more frequently in the non-social context. We further demonstrate that the inner brow raiser is strongly associated with eye movements and occurs independently in only 6% of cases. This result challenges the hypothesis that the inner brow raiser has a communicative function in dog-human interactions and suggests a lower-level explanation for its production, namely an association with eye movements.

Internal consistency and compatibility of the 3Rs and 3Vs principles for project evaluation of animal research.

Using animals for research raises ethical concerns that are addressed in project evaluation by weighing expected harm to animals against expected benefit to society. A harm-benefit analysis (HBA) relies on two preconditions: (a) the study protocol is scientifically suitable and (b) the use of (sentient) animals and harm imposed on them are necessary for achieving the study's aims. The 3Rs (Replace, Reduce and Refine) provide a guiding principle for evaluating whether the use of animals, their number and the harm imposed on them are necessary. A similar guiding principle for evaluating whether a study protocol is scientifically suitable has recently been proposed: the 3Vs principle referring to the three main aspects of scientific validity in animal research (construct, internal and external validity). Here, we analyse the internal consistency and compatibility of these two principles, address conflicts within and between the 3Rs and 3Vs principles and discuss their implications for project evaluation. We show that a few conflicts and trade-offs exist, but that these can be resolved either by appropriate study designs or by ethical deliberation in the HBA. In combination, the 3Vs, 3Rs and the HBA thus offer a coherent framework for a logically structured evaluation procedure to decide about the legitimacy of animal research projects.

The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research.

Reproducible science requires transparent reporting. The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments) were originally developed in 2010 to improve the reporting of animal research. They consist of a checklist of information to include in publications describing in vivo experiments to enable others to scrutinise the work adequately, evaluate its methodological rigour, and reproduce the methods and results. Despite considerable levels of endorsement by funders and journals over the years, adherence to the guidelines has been inconsistent, and the anticipated improvements in the quality of reporting in animal research publications have not been achieved. Here, we introduce ARRIVE 2.0. The guidelines have been updated and information reorganised to facilitate their use in practice. We used a Delphi exercise to prioritise and divide the items of the guidelines into 2 sets, the "ARRIVE Essential 10," which constitutes the minimum requirement, and the "Recommended Set," which describes the research context. This division facilitates improved reporting of animal research by supporting a stepwise approach to implementation. This helps journal editors and reviewers verify that the most important items are being reported in manuscripts. We have also developed the accompanying Explanation and Elaboration (E&E) document, which serves (1) to explain the rationale behind each item in the guidelines, (2) to clarify key concepts, and (3) to provide illustrative examples. We aim, through these changes, to help ensure that researchers, reviewers, and journal editors are better equipped to improve the rigour and transparency of the scientific process and thus reproducibility.

Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0.

Improving the reproducibility of biomedical research is a major challenge. Transparent and accurate reporting is vital to this process; it allows readers to assess the reliability of the findings and repeat or build upon the work of other researchers. The ARRIVE guidelines (Animal Research: Reporting In Vivo Experiments) were developed in 2010 to help authors and journals identify the minimum information necessary to report in publications describing in vivo experiments. Despite widespread endorsement by the scientific community, the impact of ARRIVE on the transparency of reporting in animal research publications has been limited. We have revised the ARRIVE guidelines to update them and facilitate their use in practice. The revised guidelines are published alongside this paper. This explanation and elaboration document was developed as part of the revision. It provides further information about each of the 21 items in ARRIVE 2.0, including the rationale and supporting evidence for their inclusion in the guidelines, elaboration of details to report, and examples of good reporting from the published literature. This document also covers advice and best practice in the design and conduct of animal studies to support researchers in improving standards from the start of the experimental design process through to publication.