What's in a Badge? A Computational Reproducibility Investigation of the Open Data Badge Policy in One Issue of Psychological Science.

In April 2019, Psychological Science published its first issue in which all Research Articles received the Open Data badge. We used that issue to investigate the effectiveness of this badge, focusing on the adherence to its aim at Psychological Science: sharing both data and code to ensure reproducibility of results. Twelve researchers of varying experience levels attempted to reproduce the results of the empirical articles in the target issue (at least three researchers per article). We found that all 14 articles provided at least some data and six provided analysis code, but only one article was rated to be exactly reproducible, and three were rated as essentially reproducible with minor deviations. We suggest that researchers should be encouraged to adhere to the higher standard in force at Psychological Science. Moreover, a check of reproducibility during peer review may be preferable to the disclosure method of awarding badges.

Simpson's Paradox is suppression, but Lord's Paradox is neither: clarification of and correction to Tu, Gunnell, and Gilthorpe (2008).

Tu et al. (Emerg Themes Epidemiol 5:2, 2008. https://doi.org/10.1186/1742-7622-5-2) asserted that suppression, Simpson's Paradox, and Lord's Paradox are all the same phenomenon-the reversal paradox. In the reversal paradox, the association between an outcome variable and an explanatory (predictor) variable is reversed when another explanatory variable is added to the analysis. More specifically, Tu et al. (2008) purported to demonstrate that these three paradoxes are different manifestations of the same phenomenon, differently named depending on the scaling of the outcome variable, the explanatory variable, and the third variable. According to Tu et al. (2008), when all three variables are continuous, the phenomenon is called suppression; when all three variables are categorical, the phenomenon is called Simpson's Paradox; and when the outcome variable and the third variable are continuous but the explanatory variable is categorical, the phenomenon is called Lord's Paradox. We show that (a) the strong form of Simpson's Paradox is equivalent to negative suppression for a 2 × 2 × 2 contingency table, (b) the weak form of Simpson's Paradox is equivalent to classical suppression for a 2 × 2 × 2 contingency table, and (c) Lord's Paradox is not the same phenomenon as suppression or Simpson's Paradox.

Implications of Debunking the "Critical Positivity Ratio" for Humanistic Psychology: Introduction to Special Issue.

An extraordinary claim was made by one of the leading researchers within positive psychology, namely, there is a universal-invariant ratio between positive to negative emotions that serves as a unique tipping point between flourishing and languishing in individuals, marriages, organizations, and other human systems across all cultures and times. Known as the "critical positivity ratio," this finding was supposedly derived from the famous Lorenz equation in physics by using the mathematics of nonlinear dynamic systems, and was defined precisely as "2.9013." This exact number was widely touted as a great discovery by many leaders of positive psychology, had tremendous impact in various applied areas of psychology, and, more broadly, and was extensively cited in both the scientific literature and in the global popular media. However, this finding has been demonstrated to be bogus. Since its advent as a relatively new subdiscipline, positive psychology has claimed superiority to its precursor, the subdiscipline of humanistic psychology, in terms of supposedly both using more rigorous science and avoiding popularizing nonsense. The debunking of the critical positivity ratio demonstrates that positive psychology did not live up to these claims, and this has important implications, which are discussed in terms of "romantic scientism" and "voodoo science." In addition, articles in the special issue on the "Implications of Debunking the 'Critical Positivity Ratio' for Humanistic Psychology" are introduced, as they also delve into these concerns.

A critical reanalysis of the relationship between genomics and well-being.

Fredrickson et al. [Fredrickson BL, et al. (2013) Proc Natl Acad Sci USA 110(33):13684-13689] claimed to have observed significant differences in gene expression related to hedonic and eudaimonic dimensions of well-being. Having closely examined both their claims and their data, we draw substantially different conclusions. After identifying some important conceptual and methodological flaws in their argument, we report the results of a series of reanalyses of their dataset. We first applied a variety of exploratory and confirmatory factor analysis techniques to their self-reported well-being data. A number of plausible factor solutions emerged, but none of these corresponded to Fredrickson et al.'s claimed hedonic and eudaimonic dimensions. We next examined the regression analyses that purportedly yielded distinct differential profiles of gene expression associated with the two well-being dimensions. Using the best-fitting two-factor solution that we identified, we obtained effects almost twice as large as those found by Fredrickson et al. using their questionable hedonic and eudaimonic factors. Next, we conducted regression analyses for all possible two-factor solutions of the psychometric data; we found that 69.2% of these gave statistically significant results for both factors, whereas only 0.25% would be expected to do so if the regression process was really able to identify independent differential gene expression effects. Finally, we replaced Fredrickson et al.'s psychometric data with random numbers and continued to find very large numbers of apparently statistically significant effects. We conclude that Fredrickson et al.'s widely publicized claims about the effects of different dimensions of well-being on health-related gene expression are merely artifacts of dubious analyses and erroneous methodology.

Aortic augmentation index in endurance athletes: a role for cardiorespiratory fitness.

PURPOSE: Endurance exercise improves cardiovascular health and reduces mortality risk. Augmentation index (AIx) reflects adverse loading exerted on the heart and large arteries and predicts future cardiovascular disease. The purpose of this study was to establish whether endurance athletes possess lower AIx and aortic blood pressure compared to healthy controls, and to determine the association between AIx and cardiorespiratory fitness. METHODS: Forty-six endurance athletes and 43 healthy controls underwent central BP and AIx measurements by non-invasive applanation tonometry before a maximal exercise test. Peak oxygen uptake ([Formula: see text]) was assessed by pulmonary analysis. RESULTS: Relative to controls, athletes had significantly lower brachial diastolic blood pressure (BP, -4.8 mmHg, p < 0.01), central systolic BP (-3.5 mmHg, p = 0.07), and AIx at a heart rate of 75 beats min(-1) (AIx@75, -11.9 %, p < 0.001). No AIx@75 differences were observed between athletes and controls when adjusted for age and [Formula: see text] [athletes vs controls mean (%) ± SE: -6.9 ± 2.2 vs -5.7 ± 2.3, p = 0.76]. Relative to men with low [Formula: see text], those with moderate and high [Formula: see text] had lower age-adjusted AIx@75 (p < 0.001). In women, those with high [Formula: see text] had lower AIx@75 than those with low and moderate [Formula: see text] (p < 0.01). CONCLUSIONS: The lower AIx@75 in endurance athletes is partly mediated by [Formula: see text]. While an inverse relationship between AIx@75 and [Formula: see text] was found in men, women with the highest [Formula: see text] possessed lowest AIx@75 compared to females with moderate or poor cardiorespiratory fitness. We recommend aerobic training aimed at achieving a minimum [Formula: see text] of 45 ml kg(-1) min(-1) to decrease the risk of future cardiovascular events and all-cause mortality.

Sublethal effects of imidacloprid-contaminated honey stores on colony performance, queens, and worker activities in fall and early winter colonies.

Neonicotinoid-contaminated sugar stores can have both near term and long term effects on honey bees due to their persistence in honey stores. Effects of imidacloprid food stores contaminants were examined in subtropical colonies that experience reduced brood rearing and foraging during overwintering. Colonies were given treatment sugar syrup containing 0 ppb (control), 20 ppb (field relevant), or 100 ppb (above field relevant) imidacloprid over six weeks to simulate contaminated fall nectar. Colonies were evaluated immediately (post-treatment) and 10 weeks (mid-winter) after treatment to compare proximal and latent effects. Post-treatment 0 ppb and 20 ppb colonies had more workers than 100 ppb colonies while 0 ppb colonies more brood than 20 ppb or 100 ppb colonies. Mid-winter 0 ppb and 20 ppb colonies had more workers than 100 ppb colonies and 0 ppb colonies more brood than 100 ppb colonies. Colonies experienced seasonal declines in stored pollen but no treatment effects. Lower 100 ppb colony performance was associated with reduced effort rather than lifespan. RFID (Radio Frequency Identification) tracking revealed that workers had similar adult lifespans across treatments; however, 100 ppb workers engaged in activities outside the colony for less time than 0 ppb workers. Imidacloprid exposure affected queen but not worker nutritional physiology. Nurses retained well-developed hypopharyngeal glands (as indicated by head protein) across treatments. Mid-winter queens from 0 ppb colonies had marginally higher ovary protein than queens from 100 ppb colonies and more ovary lipids than queens from 20 ppb colonies. However, queen nutrient stores in non-reproductive tissues (fat bodies) did not differ across treatments. Queens from different treatments were attended by comparable numbers of retinue workers and had similar gland contents of four QMP (Queen Mandibular Pheromone) components essential to queen care. High levels of imidacloprid in sugar stores can negatively affect colony performance months after initial storage.

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions.

INTRODUCTION: Randomised controlled trials (RCTs) inform healthcare decisions. It is now apparent that some published RCTs contain false data and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs that have been conducted on a given topic. While it is usual to assess methodological features of the RCTs in the process of undertaking a systematic review, it is not usual to consider whether the RCTs contain false data. Studies containing false data therefore go unnoticed and contribute to systematic review conclusions. The INveStigating ProblEmatic Clinical Trials in Systematic Reviews (INSPECT-SR) project will develop a tool to assess the trustworthiness of RCTs in systematic reviews of healthcare-related interventions. METHODS AND ANALYSIS: The INSPECT-SR tool will be developed using expert consensus in combination with empirical evidence, over five stages: (1) a survey of experts to assemble a comprehensive list of checks for detecting problematic RCTs, (2) an evaluation of the feasibility and impact of applying the checks to systematic reviews, (3) a Delphi survey to determine which of the checks are supported by expert consensus, culminating in, (4) a consensus meeting to select checks to be included in a draft tool and to determine its format and (5) prospective testing of the draft tool in the production of new health systematic reviews, to allow refinement based on user feedback. We anticipate that the INSPECT-SR tool will help researchers to identify problematic studies and will help patients by protecting them from the influence of false data on their healthcare. ETHICS AND DISSEMINATION: The University of Manchester ethics decision tool was used, and this returned the result that ethical approval was not required for this project (30 September 2022), which incorporates secondary research and surveys of professionals about subjects relating to their expertise. Informed consent will be obtained from all survey participants. All results will be published as open-access articles. The final tool will be made freely available.

A survey of experts to identify methods to detect problematic studies: Stage 1 of the INSPECT-SR Project.

Background: Randomised controlled trials (RCTs) inform healthcare decisions. Unfortunately, some published RCTs contain false data, and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs which have been conducted on a given topic. This means that any of these 'problematic studies' are likely to be included, but there are no agreed methods for identifying them. The INSPECT-SR project is developing a tool to identify problematic RCTs in systematic reviews of healthcare-related interventions. The tool will guide the user through a series of 'checks' to determine a study's authenticity. The first objective in the development process is to assemble a comprehensive list of checks to consider for inclusion. Methods: We assembled an initial list of checks for assessing the authenticity of research studies, with no restriction to RCTs, and categorised these into five domains: Inspecting results in the paper; Inspecting the research team; Inspecting conduct, governance, and transparency; Inspecting text and publication details; Inspecting the individual participant data. We implemented this list as an online survey, and invited people with expertise and experience of assessing potentially problematic studies to participate through professional networks and online forums. Participants were invited to provide feedback on the checks on the list, and were asked to describe any additional checks they knew of, which were not featured in the list. Results: Extensive feedback on an initial list of 102 checks was provided by 71 participants based in 16 countries across five continents. Fourteen new checks were proposed across the five domains, and suggestions were made to reword checks on the initial list. An updated list of checks was constructed, comprising 116 checks. Many participants expressed a lack of familiarity with statistical checks, and emphasized the importance of feasibility of the tool. Conclusions: A comprehensive list of trustworthiness checks has been produced. The checks will be evaluated to determine which should be included in the INSPECT-SR tool.

Honey bee retinue workers respond similarly to queens despite seasonal differences in Queen Mandibular Pheromone (QMP) signaling.

Honey bee colonies maintain viable queens in part through communication with Queen Mandibular Pheromone (QMP), a mixture that signals the queen's presence and reproductive quality to workers. In turn, workers are thought to provide retinue queen care or replace queens partially based on QMP profiles. We examined the effects of seasonal dearth (overwintering in a warm subtropical location) on queen-worker interactions. Retinue worker responses to continuously ovipositing queens were considered in view of QMP signaling and queen reproductive quality. QMP signaling was estimated from QMP residues recovered from nest worker bodies, which is the primary mode of QMP transfer from the queen to the colony at large. QMP residues varied seasonally but not at all with queen reproductive quality (spermatheca sperm storage, ovary protein and lipid contents). 9-HDA and 9-ODA were lower in January than other months. HOB decreased from July to January, while HVA, a component associated with mated queens, increased sharply in January. Despite these seasonal signaling differences, retinue workers attended queens at similar levels through the months. In terms of reproductive quality, queens did not differ over the months in matedness (spermatheca sperm storage) or physiological age (protein carbonyl content), but varied in nutrient allocation to reproductive and non-reproductive tissues. Queen ovaries contained more protein in September than in November, and more lipid in July and September than in November and January. Queen fat bodies had more protein in July than September or November, but less lipid in July and September than November or January. Retinue worker responses did not vary with seasonal QMP changes, but reflected overall continuous brood rearing efforts and queen matedness throughout the year. The absence of seasonal differences in worker responses to QMP should be considered in the broader context of continuous reproductive efforts in warm subtropical colonies.

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions.

Introduction: Randomised controlled trials (RCTs) inform healthcare decisions. It is now apparent that some published RCTs contain false data and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs that have been conducted on a given topic. While it is usual to assess methodological features of the RCTs in the process of undertaking a systematic review, it is not usual to consider whether the RCTs contain false data. Studies containing false data therefore go unnoticed and contribute to systematic review conclusions. The INSPECT-SR project will develop a tool to assess the trustworthiness of RCTs in systematic reviews of healthcare related interventions. Methods and analysis: The INSPECT-SR tool will be developed using expert consensus in combination with empirical evidence, over five stages: 1) a survey of experts to assemble a comprehensive list of checks for detecting problematic RCTs, 2) an evaluation of the feasibility and impact of applying the checks to systematic reviews, 3) a Delphi survey to determine which of the checks are supported by expert consensus, culminating in 4) a consensus meeting to select checks to be included in a draft tool and to determine its format, 5) prospective testing of the draft tool in the production of new health systematic reviews, to allow refinement based on user feedback. We anticipate that the INSPECT-SR tool will help researchers to identify problematic studies, and will help patients by protecting them from the influence of false data on their healthcare.

Is the future of peer review automated?

The rising rate of preprints and publications, combined with persistent inadequate reporting practices and problems with study design and execution, have strained the traditional peer review system. Automated screening tools could potentially enhance peer review by helping authors, journal editors, and reviewers to identify beneficial practices and common problems in preprints or submitted manuscripts. Tools can screen many papers quickly, and may be particularly helpful in assessing compliance with journal policies and with straightforward items in reporting guidelines. However, existing tools cannot understand or interpret the paper in the context of the scientific literature. Tools cannot yet determine whether the methods used are suitable to answer the research question, or whether the data support the authors' conclusions. Editors and peer reviewers are essential for assessing journal fit and the overall quality of a paper, including the experimental design, the soundness of the study's conclusions, potential impact and innovation. Automated screening tools cannot replace peer review, but may aid authors, reviewers, and editors in improving scientific papers. Strategies for responsible use of automated tools in peer review may include setting performance criteria for tools, transparently reporting tool performance and use, and training users to interpret reports.

Does Twitter language reliably predict heart disease? A commentary on Eichstaedt et al. (2015a).

We comment on Eichstaedt et al.'s (2015a) claim to have shown that language patterns among Twitter users, aggregated at the level of US counties, predicted county-level mortality rates from atherosclerotic heart disease (AHD), with "negative" language being associated with higher rates of death from AHD and "positive" language associated with lower rates. First, we examine some of Eichstaedt et al.'s apparent assumptions about the nature of AHD, as well as some issues related to the secondary analysis of online data and to considering counties as communities. Next, using the data files supplied by Eichstaedt et al., we reproduce their regression- and correlation-based models, substituting mortality from an alternative cause of death-namely, suicide-as the outcome variable, and observe that the purported associations between "negative" and "positive" language and mortality are reversed when suicide is used as the outcome variable. We identify numerous other conceptual and methodological limitations that call into question the robustness and generalizability of Eichstaedt et al.'s claims, even when these are based on the results of their ridge regression/machine learning model. We conclude that there is no good evidence that analyzing Twitter data in bulk in this way can add anything useful to our ability to understand geographical variation in AHD mortality rates.

Methodological problems in a study of fetal visual perception.

Reid et al.[1] analysed data from 39 third-trimester fetuses, concluding that they showed a preferential head-orienting reaction towards lights projected through the uterine wall in a face-like arrangement, as opposed to an inverted triangle of dots. These results imply not only that assessment of visual-perceptive responses is possible in prenatal subjects, but also that a measurable preference for faces exists before birth. However, we have identified three substantial problems with Reid et al.'s [1] method and analyses, which we outline here.

Emodiversity: Robust predictor of outcomes or statistical artifact?

This article examines the concept of emodiversity, put forward by Quoidbach et al. (2014) as a novel source of information about "the health of the human emotional ecosystem" (p. 2057). Quoidbach et al. drew an analogy between emodiversity as a desirable property of a person's emotional make-up and biological diversity as a desirable property of an ecosystem. They claimed that emodiversity was an independent predictor of better mental and physical health outcomes in two large-scale studies. Here, we show that Quoidbach et al.'s construct of emodiversity suffers from several theoretical and practical deficiencies, which make these authors' use of Shannon's (1948) entropy formula to measure emodiversity highly questionable. Our reanalysis of Quoidbach et al.'s two studies shows that the apparently substantial effects that these authors reported are likely due to a failure to conduct appropriate hierarchical regression in one case and to suppression effects in the other. It appears that Quoidbach et al.'s claims about emodiversity may reduce to little more than a set of computational and statistical artifacts. (PsycINFO Database Record

Statistical heartburn: an attempt to digest four pizza publications from the Cornell Food and Brand Lab.

BACKGROUND: We present the results of a reanalysis of four articles from the Cornell Food and Brand Lab based on data collected from diners at an Italian restaurant buffet. METHOD: We calculated whether the means, standard deviations, and test statistics were compatible with the sample size. Test statistics and p values were recalculated. We also applied deductive logic to see whether the claims made in each article were compatible with the claims made in the others. We have so far been unable to obtain the data from the authors of the four articles. RESULTS: A thorough reading of the articles and careful reanalysis of the results revealed a wide range of problems. The sample sizes for the number of diners in each condition are incongruous both within and between the four articles. In some cases, the degrees of freedom of between-participant test statistics are larger than the sample size, which is impossible. Many of the computed F and t statistics are inconsistent with the reported means and standard deviations. In some cases, the number of possible inconsistencies for a single statistic was such that we were unable to determine which of the components of that statistic were incorrect. Our Appendix reports approximately 150 inconsistencies in these four articles, which we were able to identify from the reported statistics alone. CONCLUSIONS: We hope that our analysis will encourage readers, using and extending the simple methods that we describe, to undertake their own efforts to verify published results, and that such initiatives will improve the accuracy and reproducibility of the scientific literature. We also anticipate that the editors of the journals that published these four articles may wish to consider whether any corrective action is required.

More Questions than Answers: Continued Critical Reanalysis of Fredrickson et al.'s Studies of Genomics and Well-Being.

We critically re-examine Fredrickson et al.'s renewed claims concerning the differential relationship between hedonic and eudaimonic forms of well-being and gene expression, namely that people who experience a preponderance of eudaimonic well-being have gene expression profiles that are associated with more favorable health outcomes. By means of an extensive reanalysis of their data, we identify several discrepancies between what these authors claimed and what their data support; we further show that their different analysis models produce mutually contradictory results. We then show how Fredrickson et al.'s most recent article on this topic not only fails to adequately address our previously published concerns about their earlier related work, but also introduces significant further problems, including inconsistency in their hypotheses. Additionally, we demonstrate that regardless of which statistical model is used to analyze their data, Fredrickson et al.'s method can be highly sensitive to the inclusion (or exclusion) of data from a single subject. We reiterate our previous conclusions, namely that there is no evidence that Fredrickson et al. have established a reliable empirical distinction between their two delineated forms of well-being, nor that eudaimonic well-being provides any overall health benefits over hedonic well-being.

Increased expression of telomere-regulating genes in endurance athletes with long leukocyte telomeres.

Leukocyte telomeres shorten with age, and excessive shortening is associated with age-related cardiometabolic diseases. Exercise training may prevent disease through telomere length maintenance although the optimal amount of exercise that attenuates telomere attrition is unknown. Furthermore, the underlying molecular mechanisms responsible for the enhanced telomere maintenance observed in endurance athletes is poorly understood. We quantified the leukocyte telomere length and analyzed the expression of telomere-regulating genes in endurance athletes and healthy controls (both n = 61), using quantitative PCR. We found endurance athletes have significantly longer (7.1%, 208-416 nt) leukocyte telomeres and upregulated TERT (2.0-fold) and TPP1 (1.3-fold) mRNA expression compared with controls in age-adjusted analysis. The telomere length and telomere-regulating gene expression differences were no longer statistically significant after adjustment for resting heart rate and relative V̇O(2 max) (all P > 0.05). Resting heart rate emerged as an independent predictor of leukocyte telomere length and TERT and TPP1 mRNA expression in stepwise regression models. To gauge whether volume of exercise was associated with leukocyte telomere length, we divided subjects into running and cycling tertiles (distance covered per week) and found individuals in the middle and highest tertiles had longer telomeres than individuals in the lowest tertile. These data emphasize the importance of cardiorespiratory fitness and exercise training in the prevention of biological aging. They also support the concept that moderate amounts of exercise training protects against biological aging, while higher amounts may not elicit additional benefits.

Does sadness impair color perception? Flawed evidence and faulty methods.

In their 2015 paper, Thorstenson, Pazda, and Elliot offered evidence from two experiments that perception of colors on the blue-yellow axis was impaired if the participants had watched a sad movie clip, compared to participants who watched clips designed to induce a happy or neutral mood. Subsequently, these authors retracted their article, citing a mistake in their statistical analyses and a problem with the data in one of their experiments. Here, we discuss a number of other methodological problems with Thorstenson et al.'s experimental design, and also demonstrate that the problems with the data go beyond what these authors reported. We conclude that repeating one of the two experiments, with the minor revisions proposed by Thorstenson et al., will not be sufficient to address the problems with this work.