Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology.

Collaborative efforts to directly replicate empirical studies in the medical and social sciences have revealed alarmingly low rates of replicability, a phenomenon dubbed the 'replication crisis'. Poor replicability has spurred cultural changes targeted at improving reliability in these disciplines. Given the absence of equivalent replication projects in ecology and evolutionary biology, two inter-related indicators offer the opportunity to retrospectively assess replicability: publication bias and statistical power. This registered report assesses the prevalence and severity of small-study (i.e., smaller studies reporting larger effect sizes) and decline effects (i.e., effect sizes decreasing over time) across ecology and evolutionary biology using 87 meta-analyses comprising 4,250 primary studies and 17,638 effect sizes. Further, we estimate how publication bias might distort the estimation of effect sizes, statistical power, and errors in magnitude (Type M or exaggeration ratio) and sign (Type S). We show strong evidence for the pervasiveness of both small-study and decline effects in ecology and evolution. There was widespread prevalence of publication bias that resulted in meta-analytic means being over-estimated by (at least) 0.12 standard deviations. The prevalence of publication bias distorted confidence in meta-analytic results, with 66% of initially statistically significant meta-analytic means becoming non-significant after correcting for publication bias. Ecological and evolutionary studies consistently had low statistical power (15%) with a 4-fold exaggeration of effects on average (Type M error rates = 4.4). Notably, publication bias reduced power from 23% to 15% and increased type M error rates from 2.7 to 4.4 because it creates a non-random sample of effect size evidence. The sign errors of effect sizes (Type S error) increased from 5% to 8% because of publication bias. Our research provides clear evidence that many published ecological and evolutionary findings are inflated. Our results highlight the importance of designing high-power empirical studies (e.g., via collaborative team science), promoting and encouraging replication studies, testing and correcting for publication bias in meta-analyses, and adopting open and transparent research practices, such as (pre)registration, data- and code-sharing, and transparent reporting.

Towards open, reliable, and transparent ecology and evolutionary biology.

Unreliable research programmes waste funds, time, and even the lives of the organisms we seek to help and understand. Reducing this waste and increasing the value of scientific evidence require changing the actions of both individual researchers and the institutions they depend on for employment and promotion. While ecologists and evolutionary biologists have somewhat improved research transparency over the past decade (e.g. more data sharing), major obstacles remain. In this commentary, we lift our gaze to the horizon to imagine how researchers and institutions can clear the path towards more credible and effective research programmes.

Replicating research in ecology and evolution: feasibility, incentives, and the cost-benefit conundrum.

We believe that replicating studies in ecology and evolution is extremely valuable, but replication within species and systems is troublingly rare, and even 'quasi-replications' in different systems are often insufficient. We make a case for supporting multiple types of replications and point out that the current incentive structure needs to change if ecologists and evolutionary biologist are to value scientific replication sufficiently.

Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process.

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.

Preferred reporting items for systematic reviews and meta-analyses in ecology and evolutionary biology: a PRISMA extension.

Since the early 1990s, ecologists and evolutionary biologists have aggregated primary research using meta-analytic methods to understand ecological and evolutionary phenomena. Meta-analyses can resolve long-standing disputes, dispel spurious claims, and generate new research questions. At their worst, however, meta-analysis publications are wolves in sheep's clothing: subjective with biased conclusions, hidden under coats of objective authority. Conclusions can be rendered unreliable by inappropriate statistical methods, problems with the methods used to select primary research, or problems within the primary research itself. Because of these risks, meta-analyses are increasingly conducted as part of systematic reviews, which use structured, transparent, and reproducible methods to collate and summarise evidence. For readers to determine whether the conclusions from a systematic review or meta-analysis should be trusted - and to be able to build upon the review - authors need to report what they did, why they did it, and what they found. Complete, transparent, and reproducible reporting is measured by 'reporting quality'. To assess perceptions and standards of reporting quality of systematic reviews and meta-analyses published in ecology and evolutionary biology, we surveyed 208 researchers with relevant experience (as authors, reviewers, or editors), and conducted detailed evaluations of 102 systematic review and meta-analysis papers published between 2010 and 2019. Reporting quality was far below optimal and approximately normally distributed. Measured reporting quality was lower than what the community perceived, particularly for the systematic review methods required to measure trustworthiness. The minority of assessed papers that referenced a guideline (~16%) showed substantially higher reporting quality than average, and surveyed researchers showed interest in using a reporting guideline to improve reporting quality. The leading guideline for improving reporting quality of systematic reviews is the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement. Here we unveil an extension of PRISMA to serve the meta-analysis community in ecology and evolutionary biology: PRISMA-EcoEvo (version 1.0). PRISMA-EcoEvo is a checklist of 27 main items that, when applicable, should be reported in systematic review and meta-analysis publications summarising primary research in ecology and evolutionary biology. In this explanation and elaboration document, we provide guidance for authors, reviewers, and editors, with explanations for each item on the checklist, including supplementary examples from published papers. Authors can consult this PRISMA-EcoEvo guideline both in the planning and writing stages of a systematic review and meta-analysis, to increase reporting quality of submitted manuscripts. Reviewers and editors can use the checklist to assess reporting quality in the manuscripts they review. Overall, PRISMA-EcoEvo is a resource for the ecology and evolutionary biology community to facilitate transparent and comprehensively reported systematic reviews and meta-analyses.

The role of replication studies in ecology.

Recent large-scale projects in other disciplines have shown that results often fail to replicate when studies are repeated. The conditions contributing to this problem are also present in ecology, but there have not been any equivalent replication projects. Here, we survey ecologists' understanding of and opinions about replication studies. The majority of ecologists in our sample considered replication studies to be important (97%), not prevalent enough (91%), worth funding even given limited resources (61%), and suitable for publication in all journals (62%). However, there is a disconnect between this enthusiasm and the prevalence of direct replication studies in the literature which is much lower (0.023%: Kelly 2019) than our participants' median estimate of 10%. This may be explained by the obstacles our participants identified including the difficulty of conducting replication studies and of funding and publishing them. We conclude by offering suggestions for how replications could be better integrated into ecological research.

Nest size is predicted by female identity and the local environment in the blue tit (Cyanistes caeruleus), but is not related to the nest size of the genetic or foster mother.

The potential for animals to respond to changing climates has sparked interest in intraspecific variation in avian nest structure since this may influence nest microclimate and protect eggs and offspring from inclement weather. However, there have been relatively few large-scale attempts to examine variation in nests or the determinates of individual variation in nest structure within populations. Using a set of mostly pre-registered analyses, we studied potential predictors of variation in the size of a large sample (803) of blue tit (Cyanistes caeruleus) nests across three breeding seasons at Wytham Woods, UK. While our pre-registered analyses found that individual females built very similar nests across years, there was no evidence in follow-up (post hoc) analyses that their nest size correlated to that of their genetic mother or, in a cross-fostering experiment, to the nest where they were reared. In further pre-registered analyses, spatial environmental variability explained nest size variability at relatively broad spatial scales, and especially strongly at the scale of individual nest boxes. Our study indicates that nest structure is a characteristic of individuals, but is not strongly heritable, indicating that it will not respond rapidly to selection. Explaining the within-individual and within-location repeatability we observed requires further study.

Empowering peer reviewers with a checklist to improve transparency.

Peer review is widely considered fundamental to maintaining the rigour of science, but it often fails to ensure transparency and reduce bias in published papers, and this systematically weakens the quality of published inferences. In part, this is because many reviewers are unaware of important questions to ask with respect to the soundness of the design and analyses, and the presentation of the methods and results; also some reviewers may expect others to be responsible for these tasks. We therefore present a reviewers' checklist of ten questions that address these critical components. Checklists are commonly used by practitioners of other complex tasks, and we see great potential for the wider adoption of checklists for peer review, especially to reduce bias and facilitate transparency in published papers. We expect that such checklists will be well received by many reviewers.

Detecting and avoiding likely false-positive findings - a practical guide.

Recently there has been a growing concern that many published research findings do not hold up in attempts to replicate them. We argue that this problem may originate from a culture of 'you can publish if you found a significant effect'. This culture creates a systematic bias against the null hypothesis which renders meta-analyses questionable and may even lead to a situation where hypotheses become difficult to falsify. In order to pinpoint the sources of error and possible solutions, we review current scientific practices with regard to their effect on the probability of drawing a false-positive conclusion. We explain why the proportion of published false-positive findings is expected to increase with (i) decreasing sample size, (ii) increasing pursuit of novelty, (iii) various forms of multiple testing and researcher flexibility, and (iv) incorrect P-values, especially due to unaccounted pseudoreplication, i.e. the non-independence of data points (clustered data). We provide examples showing how statistical pitfalls and psychological traps lead to conclusions that are biased and unreliable, and we show how these mistakes can be avoided. Ultimately, we hope to contribute to a culture of 'you can publish if your study is rigorous'. To this end, we highlight promising strategies towards making science more objective. Specifically, we enthusiastically encourage scientists to preregister their studies (including a priori hypotheses and complete analysis plans), to blind observers to treatment groups during data collection and analysis, and unconditionally to report all results. Also, we advocate reallocating some efforts away from seeking novelty and discovery and towards replicating important research findings of one's own and of others for the benefit of the scientific community as a whole. We believe these efforts will be aided by a shift in evaluation criteria away from the current system which values metrics of 'impact' almost exclusively and towards a system which explicitly values indices of scientific rigour.

Transparency in Ecology and Evolution: Real Problems, Real Solutions.

To make progress scientists need to know what other researchers have found and how they found it. However, transparency is often insufficient across much of ecology and evolution. Researchers often fail to report results and methods in detail sufficient to permit interpretation and meta-analysis, and many results go entirely unreported. Further, these unreported results are often a biased subset. Thus the conclusions we can draw from the published literature are themselves often biased and sometimes might be entirely incorrect. Fortunately there is a movement across empirical disciplines, and now within ecology and evolution, to shape editorial policies to better promote transparency. This can be done by either requiring more disclosure by scientists or by developing incentives to encourage disclosure.

Genetic benefits of mate choice separated from differential maternal investment in red junglefowl (Gallus gallus).

Females may choose more attractive mates to obtain better viability or attractiveness genes for their offspring. A number of studies have demonstrated a positive relationship between paternal attractiveness and offspring quality. However, this pattern could be due to inheritance of paternal genes and/or it could be due to increased maternal investment in the offspring of more attractive males. To isolate female responses to male appearance from paternal genetic effects, I housed female red junglefowl (Gallus gallus) with vasectomized (sterile) males and artificially inseminated them. Male junglefowl with larger combs are more attractive to females. Females laid more eggs when housed with a large-combed, as opposed to a small-combed, vasectomized mate. Neither egg volume nor offspring body condition was associated with comb size of the mother's vasectomized mate. Paternal genetics appeared important. Body condition and comb size were greater for the sons of large-combed sperm donor males. This is consistent with the hypothesis that genetic benefits to offspring maintain female preference for the most ornate males. It is possible that greater body condition and comb size in sons of large-combed sires was not caused by genetic differences, but instead was due to compounds in the ejaculate of large-combed sperm donors inducing greater reproductive investment from females. However, females artificially inseminated by large-combed males did not produce more or larger eggs than females artificially inseminated by small-combed males, and thus there is no other evidence consistent with ejaculate-induced differential investment. Furthermore, only in older chicks was body condition significantly related to sire comb size, suggesting genetic rather than differential investment mechanisms.

What do we really know about the signalling role of plumage colour in blue tits? A case study of impediments to progress in evolutionary biology.

Evolutionary biologists seek to explain the origin and maintenance of phenotypes, and a substantial portion of this research is accomplished by thorough study of individual species. For instance, many researchers study individual species to understand evolution of ornamental traits which appear to be products of sexual selection. I explored our understanding of sexual ornaments in a well-studied vertebrate species that may serve as a case study for research programs in evolutionary biology. I attempted to located all published papers examining plumage colour and variables related to sexual selection hypotheses in a common European songbird, the blue tit (Cyanistes caeruleus). Researchers have estimated over 1200 statistical relationships with plumage colour of blue tits in 52 studies. However, of the approximately 1000 main-effect relationships from the 48 studies that are candidates for inclusion in this meta-analysis, more than 400 were reported without details of strength and direction. Circumstantial evidence suggests that an unknown number of other estimated effects remain unpublished. Missing information is a substantial barrier to interpretation of these papers and to meta-analytic synthesis. Examination and analysis of funnel plots indicated that unpublished effects may be a biased sample of all effects, especially for comparisons of plumage colour to age and individual quality, and possibly also to measures of mate choice. Further, type I error was likely elevated by the large number of statistical comparisons evaluated, the frequent use of iterative model-building procedures, and a willingness to interpret a wide variety of results as support for a hypothesis. Type I errors were made more problematic because blue tit plumage researchers only rarely have attempted to replicate important findings in their own work or that of others. Replication is essential to drawing robust scientific conclusions, especially in probabilistic systems with moderate to weak effects or a likelihood of bias. Last, researchers studying blue tit plumage have often developed ad hoc explanations for deviations of results from their predictions. Revising hypotheses in light of data is appropriate, but these revised hypotheses were rarely tested with new data. The only highly robust conclusion supported by meta-analysis is that male blue tits have plumage that reflects more light in the ultraviolet and yellow wavelengths than the plumage of females. Various other effects, including condition-dependence of plumage colour expression and a tendency for females to adjust the sex ratio of their offspring in response to male colour, remain uncertain. These obstacles to progress in the blue tit plumage literature are likely common in evolutionary biology, and so I recommend changes to incentive structures which may improve progress towards scientific understanding in this discipline.