RESUMEN
We expect that consensus meetings, where researchers come together to discuss their theoretical viewpoints, prioritize the factors they agree are important to study, standardize their measures, and determine a smallest effect size of interest, will prove to be a more efficient solution to the lack of coordination and integration of claims in science than integrative experiments.
Asunto(s)
ConsensoRESUMEN
Two factors that decrease the replicability of studies in the scientific literature are publication bias and studies with underpowered desgins. One way to ensure that studies have adequate statistical power to detect the effect size of interest is by conducting a-priori power analyses. Yet, a previous editorial published in the Journal of Sports Sciences reported a median sample size of 19 and the scarce usage of a-priori power analyses. We meta-analysed 89 studies from the same journal to assess the presence and extent of publication bias, as well as the average statistical power, by conducting a z-curve analysis. In a larger sample of 174 studies, we also examined a) the usage, reporting practices and reproducibility of a-priori power analyses; and b) the prevalence of reporting practices of t-statistic or F-ratio, degrees of freedom, exact p-values, effect sizes and confidence intervals. Our results indicate that there was some indication of publication bias and the average observed power was low (53% for significant and non-significant findings and 61% for only significant findings). Finally, the usage and reporting practices of a-priori power analyses as well as statistical results including test statistics, effect sizes and confidence intervals were suboptimal.
Asunto(s)
Proyectos de Investigación , Humanos , Sesgo de Publicación , Reproducibilidad de los Resultados , Tamaño de la Muestra , SesgoRESUMEN
Exercise physiology and sport science have traditionally made use of the null hypothesis of no difference to make decisions about experimental interventions. In this article, we aim to review current statistical approaches typically used by exercise physiologists and sport scientists for the design and analysis of experimental interventions and to highlight the importance of including equivalence and non-inferiority studies, which address different research questions from deciding whether an effect is present. Initially, we briefly describe the most common approaches, along with their rationale, to investigate the effects of different interventions. We then discuss the main steps involved in the design and analysis of equivalence and non-inferiority studies, commonly performed in other research fields, with worked examples from exercise physiology and sport science scenarios. Finally, we provide recommendations to exercise physiologists and sport scientists who would like to apply the different approaches in future research. We hope this work will promote the correct use of equivalence and non-inferiority designs in exercise physiology and sport science whenever the research context, conditions, applications, researchers' interests or reasonable beliefs justify these approaches.
Asunto(s)
Deportes , Ejercicio Físico , Humanos , Proyectos de InvestigaciónRESUMEN
Falsificationist and confirmationist approaches provide two well-established ways of evaluating generalizability. Yarkoni rejects both and invents a third approach we call neo-operationalism. His proposal cannot work for the hypothetical concepts psychologists use, because the universe of operationalizations is impossible to define, and hypothetical concepts cannot be reduced to their operationalizations. We conclude that he is wrong in his generalizability-crisis diagnosis.
RESUMEN
While great interest in health effects of natural product (NP) including dietary supplements and foods persists, promising preclinical NP research is not consistently translating into actionable clinical trial (CT) outcomes. Generally considered the gold standard for assessing safety and efficacy, CTs, especially phase III CTs, are costly and require rigorous planning to optimize the value of the information obtained. More effective bridging from NP research to CT was the goal of a September, 2018 transdisciplinary workshop. Participants emphasized that replicability and likelihood of successful translation depend on rigor in experimental design, interpretation, and reporting across the continuum of NP research. Discussions spanned good practices for NP characterization and quality control; use and interpretation of models (computational through in vivo) with strong clinical predictive validity; controls for experimental artefacts, especially for in vitro interrogation of bioactivity and mechanisms of action; rigorous assessment and interpretation of prior research; transparency in all reporting; and prioritization of research questions. Natural product clinical trials prioritized based on rigorous, convergent supporting data and current public health needs are most likely to be informative and ultimately affect public health. Thoughtful, coordinated implementation of these practices should enhance the knowledge gained from future NP research.
Asunto(s)
Productos Biológicos/farmacología , Investigación Biomédica Traslacional/normas , Animales , Evaluación Preclínica de Medicamentos , Etnobotánica , HumanosAsunto(s)
Betacoronavirus , Infecciones por Coronavirus , Pandemias , Neumonía Viral , Control de Calidad , Proyectos de Investigación/normas , Investigadores/normas , Informe de Investigación/normas , Investigación/normas , Sesgo , COVID-19 , Infecciones por Coronavirus/epidemiología , Infecciones por Coronavirus/virología , Retroalimentación Formativa , Humanos , Difusión de la Información , Motivación , Revisión de la Investigación por Pares/métodos , Neumonía Viral/epidemiología , Neumonía Viral/virología , Reproducibilidad de los Resultados , Investigación/economía , Investigadores/economía , SARS-CoV-2 , Factores de TiempoRESUMEN
[This corrects the article DOI: 10.2196/17127.].
RESUMEN
Despite the growing popularity of digital health interventions, limitations of traditional behavior change theories and a lack of theory integration hinder theory-driven behavior change applications. In this paper, we aim to review theories relevant to lifestyle behavior change from the broader psychology literature and then integrate these theories into a new theoretical framework called adaptive decision-making to address two specific problems. First, our framework represents lifestyle behaviors at two levels-one of individual daily decisions (action level) and one of larger behavioral episodes (reflection level)-to more closely match the temporal characteristics of lifestyle behaviors and their associated digital data. Second, the framework connects decision-making theories and learning theories to explain how behaviors and cognitive constructs dynamically influence each other, making it a suitable scaffold for building computational models. We map common digital intervention techniques onto the behavioral and cognitive processes in the framework and discuss possible contributions of the framework to both theory development and digital intervention design.
Asunto(s)
Estilo de Vida , HumanosRESUMEN
The debate about whether replication studies should become mainstream is essentially driven by disagreements about their costs and benefits and the best ways to allocate limited resources. Determining when replications are worthwhile requires quantifying their expected utility. We argue that a formalized framework for such evaluations can be useful for both individual decision-making and collective discussions about replication.
Asunto(s)
Toma de Decisiones , Análisis Costo-BeneficioRESUMEN
Replication of published results is crucial for ensuring the robustness and self-correction of research, yet replications are scarce in many fields. Replicating researchers will therefore often have to decide which of several relevant candidates to target for replication. Formal strategies for efficient study selection have been proposed, but none have been explored for practical feasibility - a prerequisite for validation. Here we move one step closer to efficient replication study selection by exploring the feasibility of a particular selection strategy that estimates replication value as a function of citation impact and sample size (Isager, van 't Veer, & Lakens, 2021). We tested our strategy on a sample of fMRI studies in social neuroscience. We first report our efforts to generate a representative candidate set of replication targets. We then explore the feasibility and reliability of estimating replication value for the targets in our set, resulting in a dataset of 1358 studies ranked on their value of prioritising them for replication. In addition, we carefully examine possible measures, test auxiliary assumptions, and identify boundary conditions of measuring value and uncertainty. We end our report by discussing how future validation studies might be designed. Our study demonstrates the importance of investigating how to implement study selection strategies in practice. Our sample and study design can be extended to explore the feasibility of other formal study selection strategies that have been proposed.
Asunto(s)
Neurociencia Cognitiva , Humanos , Estudios de Factibilidad , Reproducibilidad de los Resultados , Incertidumbre , Proyectos de InvestigaciónRESUMEN
When analyzing data, researchers make some choices that are either arbitrary, based on subjective beliefs about the data-generating process, or for which equally justifiable alternative choices could have been made. This wide range of data-analytic choices can be abused and has been one of the underlying causes of the replication crisis in several fields. Recently, the introduction of multiverse analysis provides researchers with a method to evaluate the stability of the results across reasonable choices that could be made when analyzing data. Multiverse analysis is confined to a descriptive role, lacking a proper and comprehensive inferential procedure. Recently, specification curve analysis adds an inferential procedure to multiverse analysis, but this approach is limited to simple cases related to the linear model, and only allows researchers to infer whether at least one specification rejects the null hypothesis, but not which specifications should be selected. In this paper, we present a Post-selection Inference approach to Multiverse Analysis (PIMA) which is a flexible and general inferential approach that considers for all possible models, i.e., the multiverse of reasonable analyses. The approach allows for a wide range of data specifications (i.e., preprocessing) and any generalized linear model; it allows testing the null hypothesis that a given predictor is not associated with the outcome, by combining information from all reasonable models of multiverse analysis, and provides strong control of the family-wise error rate allowing researchers to claim that the null hypothesis can be rejected for any specification that shows a significant effect. The inferential proposal is based on a conditional resampling procedure. We formally prove that the Type I error rate is controlled, and compute the statistical power of the test through a simulation study. Finally, we apply the PIMA procedure to the analysis of a real dataset on the self-reported hesitancy for the COronaVIrus Disease 2019 (COVID-19) vaccine before and after the 2020 lockdown in Italy. We conclude with practical recommendations to be considered when implementing the proposed procedure.
Asunto(s)
Psicometría , Humanos , Psicometría/métodos , Modelos Estadísticos , Interpretación Estadística de Datos , COVID-19/epidemiología , Modelos Lineales , Simulación por ComputadorRESUMEN
Affective responses during stressful, high-stakes situations can play an important role in shaping performance. For example, feeling shaky and nervous at a job interview can undermine performance, whereas feeling excited during that same interview can optimize performance. Thus, affect regulation-the way people influence their affective responses-might play a key role in determining high-stakes outcomes. To test this idea, we adapted a synergistic mindsets intervention (SMI) (Yeager et al. 2022 Nature 607, 512-520 (doi:10.1038/s41586-022-04907-7)) to a high-stakes esports context. Our approach was motivated by the idea that (i) mindsets both about situations and one's stress responses to situations can be shaped to help optimize stress responses, and (ii) challenge versus threat stress responses will be associated with improved outcomes. After a baseline performance task, we randomly assigned gamers (n = 300) either to SMI or a control condition in which they learned brain facts. After two weeks of daily gaming, gamers competed in a cash-prize tournament. We measured affective experiences before the matches and cardiovascular responses before and throughout the matches. Contrary to predictions, gamers did not experience negative affect (including feeling stressed), thus limiting the capacity for the intervention to regulate physiological responses and optimize performance. Compared with the control participants, synergistic mindsets participants did not show greater challenge responses or improved performance outcomes. Though our adaptation of Yeager et al.'s SMI did not optimize esports performance, our findings point to important considerations regarding the suitability of an intervention such as this to different performance contexts of varying degrees of stressfulness.
RESUMEN
We explored the possibility of a general brightness bias: brighter pictures are evaluated more positively, while darker pictures are evaluated more negatively. In Study 1 we found that positive pictures are brighter than negative pictures in two affective picture databases (the IAPS and the GAPED). Study 2 revealed that because researchers select affective pictures on the extremity of their affective rating without controlling for brightness differences, pictures used in positive conditions of experiments were on average brighter than those used in negative conditions. Going beyond correlational support for our hypothesis, Studies 3 and 4 showed that brighter versions of neutral pictures were evaluated more positively than darker versions of the same picture. Study 5 revealed that people categorised positive words more quickly than negative words after a bright picture prime, and vice versa for negative pictures. Together, these studies provide strong support for the hypotheses that picture brightness influences evaluations.
Asunto(s)
Afecto , Estimulación Luminosa , Percepción Visual , Adulto , Bases de Datos Factuales , Femenino , Humanos , MasculinoRESUMEN
Criteria for recognizing and rewarding scientists primarily focus on individual contributions. This creates a conflict between what is best for scientists' careers and what is best for science. In this article, we show how the theory of multilevel selection provides conceptual tools for modifying incentives to better align individual and collective interests. A core principle is the need to account for indirect effects by shifting the level at which selection operates from individuals to the groups in which individuals are embedded. This principle is used in several fields to improve collective outcomes, including animal husbandry, team sports, and professional organizations. Shifting the level of selection has the potential to ameliorate several problems in contemporary science, including accounting for scientists' diverse contributions to knowledge generation, reducing individual-level competition, and promoting specialization and team science. We discuss the difficulties associated with shifting the level of selection and outline directions for future development in this domain.
RESUMEN
To help move researchers away from heuristically dismissing "small" effects as unimportant, recent articles have revisited arguments to defend why seemingly small effect sizes in psychological science matter. One argument is based on the idea that an observed effect size may increase in impact when generalized to a new context because of processes of accumulation over time or application to large populations. However, the field is now in danger of heuristically accepting all effects as potentially important. We aim to encourage researchers to think thoroughly about the various mechanisms that may both amplify and counteract the importance of an observed effect size. Researchers should draw on the multiple amplifying and counteracting mechanisms that are likely to simultaneously apply to the effect when that effect is being generalized to a new and likely more dynamic context. In this way, researchers should aim to transparently provide verifiable lines of reasoning to justify their claims about an effect's importance or unimportance. This transparency can help move psychological science toward a more rigorous assessment of when psychological findings matter for the contexts that researchers want to generalize to.
Asunto(s)
Disentimientos y Disputas , Solución de Problemas , HumanosRESUMEN
In the January 2022 issue of Perspectives, Götz et al. argued that small effects are "the indispensable foundation for a cumulative psychological science." They supported their argument by claiming that (a) psychology, like genetics, consists of complex phenomena explained by additive small effects; (b) psychological-research culture rewards large effects, which means small effects are being ignored; and (c) small effects become meaningful at scale and over time. We rebut these claims with three objections: First, the analogy between genetics and psychology is misleading; second, p values are the main currency for publication in psychology, meaning that any biases in the literature are (currently) caused by pressure to publish statistically significant results and not large effects; and third, claims regarding small effects as important and consequential must be supported by empirical evidence or, at least, a falsifiable line of reasoning. If accepted uncritically, we believe the arguments of Götz et al. could be used as a blanket justification for the importance of any and all "small" effects, thereby undermining best practices in effect-size interpretation. We end with guidance on evaluating effect sizes in relative, not absolute, terms.
Asunto(s)
Psicología , HumanosRESUMEN
Robust scientific knowledge is contingent upon replication of original findings. However, replicating researchers are constrained by resources, and will almost always have to choose one replication effort to focus on from a set of potential candidates. To select a candidate efficiently in these cases, we need methods for deciding which out of all candidates considered would be the most useful to replicate, given some overall goal researchers wish to achieve. In this article we assume that the overall goal researchers wish to achieve is to maximize the utility gained by conducting the replication study. We then propose a general rule for study selection in replication research based on the replication value of the set of claims considered for replication. The replication value of a claim is defined as the maximum expected utility we could gain by conducting a replication of the claim, and is a function of (a) the value of being certain about the claim, and (b) uncertainty about the claim based on current evidence. We formalize this definition in terms of a causal decision model, utilizing concepts from decision theory and causal graph modeling. We discuss the validity of using replication value as a measure of expected utility gain, and we suggest approaches for deriving quantitative estimates of replication value. Our goal in this article is not to define concrete guidelines for study selection, but to provide the necessary theoretical foundations on which such concrete guidelines could be built. (PsycInfo Database Record (c) 2023 APA, all rights reserved).
Asunto(s)
Conocimiento , Modelos Teóricos , Humanos , IncertidumbreRESUMEN
Known methodological issues such as publication bias, questionable research practices and studies with underpowered designs are known to decrease the replicability of study findings. The presence of such issues has been widely established across different research fields, especially in psychology. Their presence raised the first concerns that the replicability of study findings could be low and led researchers to conduct large replication projects. These replication projects revealed that a significant portion of original study findings could not be replicated, giving rise to the conceptualization of the replication crisis. Although previous research in the field of sports and exercise science has identified the first warning signs, such as an overwhelming proportion of significant findings, small sample sizes and lack of data availability, their possible consequences for the replicability of our field have been overlooked. We discuss the consequences of the above issues on the replicability of our field and offer potential solutions to improve replicability.