Consensus meetings will outperform integrative experiments.

We expect that consensus meetings, where researchers come together to discuss their theoretical viewpoints, prioritize the factors they agree are important to study, standardize their measures, and determine a smallest effect size of interest, will prove to be a more efficient solution to the lack of coordination and integration of claims in science than integrative experiments.

Publication bias, statistical power and reporting practices in the Journal of Sports Sciences: potential barriers to replicability.

Two factors that decrease the replicability of studies in the scientific literature are publication bias and studies with underpowered desgins. One way to ensure that studies have adequate statistical power to detect the effect size of interest is by conducting a-priori power analyses. Yet, a previous editorial published in the Journal of Sports Sciences reported a median sample size of 19 and the scarce usage of a-priori power analyses. We meta-analysed 89 studies from the same journal to assess the presence and extent of publication bias, as well as the average statistical power, by conducting a z-curve analysis. In a larger sample of 174 studies, we also examined a) the usage, reporting practices and reproducibility of a-priori power analyses; and b) the prevalence of reporting practices of t-statistic or F-ratio, degrees of freedom, exact p-values, effect sizes and confidence intervals. Our results indicate that there was some indication of publication bias and the average observed power was low (53% for significant and non-significant findings and 61% for only significant findings). Finally, the usage and reporting practices of a-priori power analyses as well as statistical results including test statistics, effect sizes and confidence intervals were suboptimal.

Myths and methodologies: The use of equivalence and non-inferiority tests for interventional studies in exercise physiology and sport science.

Exercise physiology and sport science have traditionally made use of the null hypothesis of no difference to make decisions about experimental interventions. In this article, we aim to review current statistical approaches typically used by exercise physiologists and sport scientists for the design and analysis of experimental interventions and to highlight the importance of including equivalence and non-inferiority studies, which address different research questions from deciding whether an effect is present. Initially, we briefly describe the most common approaches, along with their rationale, to investigate the effects of different interventions. We then discuss the main steps involved in the design and analysis of equivalence and non-inferiority studies, commonly performed in other research fields, with worked examples from exercise physiology and sport science scenarios. Finally, we provide recommendations to exercise physiologists and sport scientists who would like to apply the different approaches in future research. We hope this work will promote the correct use of equivalence and non-inferiority designs in exercise physiology and sport science whenever the research context, conditions, applications, researchers' interests or reasonable beliefs justify these approaches.

There is no generalizability crisis.

Falsificationist and confirmationist approaches provide two well-established ways of evaluating generalizability. Yarkoni rejects both and invents a third approach we call neo-operationalism. His proposal cannot work for the hypothetical concepts psychologists use, because the universe of operationalizations is impossible to define, and hypothetical concepts cannot be reduced to their operationalizations. We conclude that he is wrong in his generalizability-crisis diagnosis.

Improving natural product research translation: From source to clinical trial.

While great interest in health effects of natural product (NP) including dietary supplements and foods persists, promising preclinical NP research is not consistently translating into actionable clinical trial (CT) outcomes. Generally considered the gold standard for assessing safety and efficacy, CTs, especially phase III CTs, are costly and require rigorous planning to optimize the value of the information obtained. More effective bridging from NP research to CT was the goal of a September, 2018 transdisciplinary workshop. Participants emphasized that replicability and likelihood of successful translation depend on rigor in experimental design, interpretation, and reporting across the continuum of NP research. Discussions spanned good practices for NP characterization and quality control; use and interpretation of models (computational through in vivo) with strong clinical predictive validity; controls for experimental artefacts, especially for in vitro interrogation of bioactivity and mechanisms of action; rigorous assessment and interpretation of prior research; transparency in all reporting; and prioritization of research questions. Natural product clinical trials prioritized based on rigorous, convergent supporting data and current public health needs are most likely to be informative and ultimately affect public health. Thoughtful, coordinated implementation of these practices should enhance the knowledge gained from future NP research.

Correction: Theory Integration for Lifestyle Behavior Change in the Digital Age: An Adaptive Decision-Making Framework.

[This corrects the article DOI: 10.2196/17127.].

Theory Integration for Lifestyle Behavior Change in the Digital Age: An Adaptive Decision-Making Framework.

Despite the growing popularity of digital health interventions, limitations of traditional behavior change theories and a lack of theory integration hinder theory-driven behavior change applications. In this paper, we aim to review theories relevant to lifestyle behavior change from the broader psychology literature and then integrate these theories into a new theoretical framework called adaptive decision-making to address two specific problems. First, our framework represents lifestyle behaviors at two levels-one of individual daily decisions (action level) and one of larger behavioral episodes (reflection level)-to more closely match the temporal characteristics of lifestyle behaviors and their associated digital data. Second, the framework connects decision-making theories and learning theories to explain how behaviors and cognitive constructs dynamically influence each other, making it a suitable scaffold for building computational models. We map common digital intervention techniques onto the behavioral and cognitive processes in the framework and discuss possible contributions of the framework to both theory development and digital intervention design.

The costs and benefits of replication studies.

The debate about whether replication studies should become mainstream is essentially driven by disagreements about their costs and benefits and the best ways to allocate limited resources. Determining when replications are worthwhile requires quantifying their expected utility. We argue that a formalized framework for such evaluations can be useful for both individual decision-making and collective discussions about replication.

Exploring a formal approach to selecting studies for replication: A feasibility study in social neuroscience.

Replication of published results is crucial for ensuring the robustness and self-correction of research, yet replications are scarce in many fields. Replicating researchers will therefore often have to decide which of several relevant candidates to target for replication. Formal strategies for efficient study selection have been proposed, but none have been explored for practical feasibility - a prerequisite for validation. Here we move one step closer to efficient replication study selection by exploring the feasibility of a particular selection strategy that estimates replication value as a function of citation impact and sample size (Isager, van 't Veer, & Lakens, 2021). We tested our strategy on a sample of fMRI studies in social neuroscience. We first report our efforts to generate a representative candidate set of replication targets. We then explore the feasibility and reliability of estimating replication value for the targets in our set, resulting in a dataset of 1358 studies ranked on their value of prioritising them for replication. In addition, we carefully examine possible measures, test auxiliary assumptions, and identify boundary conditions of measuring value and uncertainty. We end our report by discussing how future validation studies might be designed. Our study demonstrates the importance of investigating how to implement study selection strategies in practice. Our sample and study design can be extended to explore the feasibility of other formal study selection strategies that have been proposed.

Post-selection Inference in Multiverse Analysis (PIMA): An Inferential Framework Based on the Sign Flipping Score Test.

When analyzing data, researchers make some choices that are either arbitrary, based on subjective beliefs about the data-generating process, or for which equally justifiable alternative choices could have been made. This wide range of data-analytic choices can be abused and has been one of the underlying causes of the replication crisis in several fields. Recently, the introduction of multiverse analysis provides researchers with a method to evaluate the stability of the results across reasonable choices that could be made when analyzing data. Multiverse analysis is confined to a descriptive role, lacking a proper and comprehensive inferential procedure. Recently, specification curve analysis adds an inferential procedure to multiverse analysis, but this approach is limited to simple cases related to the linear model, and only allows researchers to infer whether at least one specification rejects the null hypothesis, but not which specifications should be selected. In this paper, we present a Post-selection Inference approach to Multiverse Analysis (PIMA) which is a flexible and general inferential approach that considers for all possible models, i.e., the multiverse of reasonable analyses. The approach allows for a wide range of data specifications (i.e., preprocessing) and any generalized linear model; it allows testing the null hypothesis that a given predictor is not associated with the outcome, by combining information from all reasonable models of multiverse analysis, and provides strong control of the family-wise error rate allowing researchers to claim that the null hypothesis can be rejected for any specification that shows a significant effect. The inferential proposal is based on a conditional resampling procedure. We formally prove that the Type I error rate is controlled, and compute the statistical power of the test through a simulation study. Finally, we apply the PIMA procedure to the analysis of a real dataset on the self-reported hesitancy for the COronaVIrus Disease 2019 (COVID-19) vaccine before and after the 2020 lockdown in Italy. We conclude with practical recommendations to be considered when implementing the proposed procedure.

Brightness differences influence the evaluation of affective pictures.

We explored the possibility of a general brightness bias: brighter pictures are evaluated more positively, while darker pictures are evaluated more negatively. In Study 1 we found that positive pictures are brighter than negative pictures in two affective picture databases (the IAPS and the GAPED). Study 2 revealed that because researchers select affective pictures on the extremity of their affective rating without controlling for brightness differences, pictures used in positive conditions of experiments were on average brighter than those used in negative conditions. Going beyond correlational support for our hypothesis, Studies 3 and 4 showed that brighter versions of neutral pictures were evaluated more positively than darker versions of the same picture. Study 5 revealed that people categorised positive words more quickly than negative words after a bright picture prime, and vice versa for negative pictures. Together, these studies provide strong support for the hypotheses that picture brightness influences evaluations.

Shifting the Level of Selection in Science.

Criteria for recognizing and rewarding scientists primarily focus on individual contributions. This creates a conflict between what is best for scientists' careers and what is best for science. In this article, we show how the theory of multilevel selection provides conceptual tools for modifying incentives to better align individual and collective interests. A core principle is the need to account for indirect effects by shifting the level at which selection operates from individuals to the groups in which individuals are embedded. This principle is used in several fields to improve collective outcomes, including animal husbandry, team sports, and professional organizations. Shifting the level of selection has the potential to ameliorate several problems in contemporary science, including accounting for scientists' diverse contributions to knowledge generation, reducing individual-level competition, and promoting specialization and team science. We discuss the difficulties associated with shifting the level of selection and outline directions for future development in this domain.

Not All Effects Are Indispensable: Psychological Science Requires Verifiable Lines of Reasoning for Whether an Effect Matters.

To help move researchers away from heuristically dismissing "small" effects as unimportant, recent articles have revisited arguments to defend why seemingly small effect sizes in psychological science matter. One argument is based on the idea that an observed effect size may increase in impact when generalized to a new context because of processes of accumulation over time or application to large populations. However, the field is now in danger of heuristically accepting all effects as potentially important. We aim to encourage researchers to think thoroughly about the various mechanisms that may both amplify and counteract the importance of an observed effect size. Researchers should draw on the multiple amplifying and counteracting mechanisms that are likely to simultaneously apply to the effect when that effect is being generalized to a new and likely more dynamic context. In this way, researchers should aim to transparently provide verifiable lines of reasoning to justify their claims about an effect's importance or unimportance. This transparency can help move psychological science toward a more rigorous assessment of when psychological findings matter for the contexts that researchers want to generalize to.

Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints.

Robust scientific knowledge is contingent upon replication of original findings. However, replicating researchers are constrained by resources, and will almost always have to choose one replication effort to focus on from a set of potential candidates. To select a candidate efficiently in these cases, we need methods for deciding which out of all candidates considered would be the most useful to replicate, given some overall goal researchers wish to achieve. In this article we assume that the overall goal researchers wish to achieve is to maximize the utility gained by conducting the replication study. We then propose a general rule for study selection in replication research based on the replication value of the set of claims considered for replication. The replication value of a claim is defined as the maximum expected utility we could gain by conducting a replication of the claim, and is a function of (a) the value of being certain about the claim, and (b) uncertainty about the claim based on current evidence. We formalize this definition in terms of a causal decision model, utilizing concepts from decision theory and causal graph modeling. We discuss the validity of using replication value as a measure of expected utility gain, and we suggest approaches for deriving quantitative estimates of replication value. Our goal in this article is not to define concrete guidelines for study selection, but to provide the necessary theoretical foundations on which such concrete guidelines could be built. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Are Small Effects the Indispensable Foundation for a Cumulative Psychological Science? A Reply to Götz et al. (2022).

In the January 2022 issue of Perspectives, Götz et al. argued that small effects are "the indispensable foundation for a cumulative psychological science." They supported their argument by claiming that (a) psychology, like genetics, consists of complex phenomena explained by additive small effects; (b) psychological-research culture rewards large effects, which means small effects are being ignored; and (c) small effects become meaningful at scale and over time. We rebut these claims with three objections: First, the analogy between genetics and psychology is misleading; second, p values are the main currency for publication in psychology, meaning that any biases in the literature are (currently) caused by pressure to publish statistically significant results and not large effects; and third, claims regarding small effects as important and consequential must be supported by empirical evidence or, at least, a falsifiable line of reasoning. If accepted uncritically, we believe the arguments of Götz et al. could be used as a blanket justification for the importance of any and all "small" effects, thereby undermining best practices in effect-size interpretation. We end with guidance on evaluating effect sizes in relative, not absolute, terms.

Replication concerns in sports and exercise science: a narrative review of selected methodological issues in the field.

Known methodological issues such as publication bias, questionable research practices and studies with underpowered designs are known to decrease the replicability of study findings. The presence of such issues has been widely established across different research fields, especially in psychology. Their presence raised the first concerns that the replicability of study findings could be low and led researchers to conduct large replication projects. These replication projects revealed that a significant portion of original study findings could not be replicated, giving rise to the conceptualization of the replication crisis. Although previous research in the field of sports and exercise science has identified the first warning signs, such as an overwhelming proportion of significant findings, small sample sizes and lack of data availability, their possible consequences for the replicability of our field have been overlooked. We discuss the consequences of the above issues on the replicability of our field and offer potential solutions to improve replicability.

Leaving academia: PhD attrition and unhealthy research environments.

This study investigates PhD candidates' (N = 391) perceptions about their research environment at a Dutch university in terms of the research climate, (un)ethical supervisory practices, and questionable research practices. We assessed whether their perceptions are related to career considerations. We gathered quantitative self-report estimations of the perceptions of PhD candidates using an online survey tool and then conducted descriptive and within-subject correlation analysis of the results. While most PhD candidates experience fair evaluation processes, openness, integrity, trust, and freedom in their research climate, many report lack of time and support, insufficient supervision, and witness questionable research practices. Results based on Spearman correlations indicate that those who experience a less healthy research environment (including experiences with unethical supervision, questionable practices, and barriers to responsible research), more often consider leaving academia and their current PhD position.