Ensuring the quality and specificity of preregistrations.

Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of "researcher degrees of freedom" aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called "OSF Preregistration," http://osf.io/prereg/). The Prereg Challenge format was a "structured" workflow with detailed instructions and an independent review to confirm completeness; the "Standard" format was "unstructured" with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the "structured" format restricted the opportunistic use of researcher degrees of freedom better (Cliff's Delta = 0.49) than the "unstructured" format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research.

Preregistration in practice: A comparison of preregistered and non-preregistered studies in psychology.

Preregistration has gained traction as one of the most promising solutions to improve the replicability of scientific effects. In this project, we compared 193 psychology studies that earned a Preregistration Challenge prize or preregistration badge to 193 related studies that were not preregistered. In contrast to our theoretical expectations and prior research, we did not find that preregistered studies had a lower proportion of positive results (Hypothesis 1), smaller effect sizes (Hypothesis 2), or fewer statistical errors (Hypothesis 3) than non-preregistered studies. Supporting our Hypotheses 4 and 5, we found that preregistered studies more often contained power analyses and typically had larger sample sizes than non-preregistered studies. Finally, concerns about the publishability and impact of preregistered studies seem unwarranted, as preregistered studies did not take longer to publish and scored better on several impact measures. Overall, our data indicate that preregistration has beneficial effects in the realm of statistical power and impact, but we did not find robust evidence that preregistration prevents p-hacking and HARKing (Hypothesizing After the Results are Known).

Group autonomy enhancing treatment versus cognitive behavioral therapy for anxiety disorders: A cluster-randomized clinical trial.

BACKGROUND: Although cognitive behavioral therapy (CBT) is effective in the treatment of anxiety disorders, few evidence-based alternatives exist. Autonomy enhancing treatment (AET) aims to decrease the vulnerability for anxiety disorders by targeting underlying autonomy deficits and may therefore have similar effects on anxiety as CBT, but yield broader effects. METHODS: A multicenter cluster-randomized clinical trial was conducted including 129 patients with DSM-5 anxiety disorders, on average 33.66 years of age (SD = 12.57), 91 (70.5%) female, and most (92.2%) born in the Netherlands. Participants were randomized over 15-week groupwise AET or groupwise CBT and completed questionnaires on anxiety, general psychopathology, depression, quality of life, autonomy-connectedness and self-esteem, pre-, mid-, and posttreatment, and after 3, 6, and 12 months (six measurements). RESULTS: Contrary to the hypotheses, effects on the broader outcome measures did not differ between AET and CBT (d = .16 or smaller at post-test). Anxiety reduction was similar across conditions (d = .059 at post-test) and neither therapy was superior on long term. CONCLUSION: This was the first clinical randomized trial comparing AET to CBT. The added value of AET does not seem to lie in enhanced effectiveness on broader outcome measures or on long term compared to CBT. However, the study supports the effectiveness of AET and thereby contributes to extended treatment options for anxiety disorders.

Associations between lifestyle factors and multidimensional frailty: a cross-sectional study among community-dwelling older people.

BACKGROUND: Multidimensional frailty, including physical, psychological, and social components, is associated to disability, lower quality of life, increased healthcare utilization, and mortality. In order to prevent or delay frailty, more knowledge of its determinants is necessary; one of these determinants is lifestyle. The aim of this study is to determine the association between lifestyle factors smoking, alcohol use, nutrition, physical activity, and multidimensional frailty. METHODS: This cross-sectional study was conducted in two samples comprising in total 45,336 Dutch community-dwelling individuals aged 65 years or older. These samples completed a questionnaire including questions about smoking, alcohol use, physical activity, sociodemographic factors (both samples), and nutrition (one sample). Multidimensional frailty was assessed with the Tilburg Frailty Indicator (TFI). RESULTS: Higher alcohol consumption, physical activity, healthy nutrition, and less smoking were associated with less total, physical, psychological and social frailty after controlling for effects of other lifestyle factors and sociodemographic characteristics of the participants (age, gender, marital status, education, income). Effects of physical activity on total and physical frailty were up to considerable, whereas the effects of other lifestyle factors on frailty were small. CONCLUSIONS: The four lifestyle factors were not only associated with physical frailty but also with psychological and social frailty. The different associations of frailty domains with lifestyle factors emphasize the importance of assessing frailty broadly and thus to pay attention to the multidimensional nature of this concept. The findings offer healthcare professionals starting points for interventions with the purpose to prevent or delay the onset of frailty, so community-dwelling older people have the possibility to aging in place accompanied by a good quality of life.

The Prevalence of Marginally Significant Results in Psychology Over Time.

We examined the percentage of p values (.05 < p ≤ .10) reported as marginally significant in 44,200 articles, across nine psychology disciplines, published in 70 journals belonging to the American Psychological Association between 1985 and 2016. Using regular expressions, we extracted 42,504 p values between .05 and .10. Almost 40% of p values in this range were reported as marginally significant, although there were considerable differences between disciplines. The practice is most common in organizational psychology (45.4%) and least common in clinical psychology (30.1%). Contrary to what was reported by previous researchers, our results showed no evidence of an increasing trend in any discipline; in all disciplines, the percentage of p values reported as marginally significant was decreasing or constant over time. We recommend against reporting these results as marginally significant because of the low evidential value of p values between .05 and .10.

Associations Between Secret-Keeping and Quality of Life in Older Adults.

This study examined the effects of secrecy on quality of life in a sample consisting of older adults (>50 years; N = 301). Three key components of secrecy were examined with the Tilburg Secrecy Scale-25 (TSS25; possession of a secret, self-concealment, and cognitive preoccupation). The TSS25 distinguishes between the tendency to conceal personal information (self-concealment) and the tendency to worry or ruminate about the secret (cognitive preoccupation), thereby enabling investigation of the effects of secrecy on quality of life in detail. Confirming previous findings in younger samples, we found a positive effect of possession of a secret on quality of life, after controlling for both TSS25's self-concealment and cognitive preoccupation. This suggests that keeping secrets may have a positive association with quality of life in older adults as well, as long as they do not have the tendency to self-conceal and are not cognitively preoccupied with their secret.

Modeling the micro-macro link: Understanding macro-level outcomes using randomization tests on micro-level data.

Analytical sociology explains macro-level outcomes by referring to micro-level behaviors, and its hypotheses thus take macro-level entities (e.g. groups) as their units of analysis. The statistical analysis of these macro-level units is problematic, since macro units are often few in number, leading to low statistical power. Additionally, micro-level processes take place within macro units, but tests on macro-level units cannot adequately deal with these processes. Consequently, much analytical sociology focuses on testing micro-level predictions. We propose a better alternative; a method to test macro hypotheses on micro data, using randomization tests. The advantages of our method are (i) increased statistical power, (ii) possibilities to control for micro covariates, and (iii) the possibility to test macro hypotheses without macro units. We provide a heuristic description of our method and illustrate it with data from a published study. Data and R-scripts for this paper are available in the Open Science Framework (https://osf.io/scfx3/).

Examining reproducibility in psychology: A hybrid method for combining a statistically significant original study and a replication.

The unrealistically high rate of positive results within psychology has increased the attention to replication research. However, researchers who conduct a replication and want to statistically combine the results of their replication with a statistically significant original study encounter problems when using traditional meta-analysis techniques. The original study's effect size is most probably overestimated because it is statistically significant, and this bias is not taken into consideration in traditional meta-analysis. We have developed a hybrid method that does take the statistical significance of an original study into account and enables (a) accurate effect size estimation, (b) estimation of a confidence interval, and (c) testing of the null hypothesis of no effect. We analytically approximate the performance of the hybrid method and describe its statistical properties. By applying the hybrid method to data from the Reproducibility Project: Psychology (Open Science Collaboration, 2015), we demonstrate that the conclusions based on the hybrid method are often in line with those of the replication, suggesting that many published psychological studies have smaller effect sizes than those reported in the original study, and that some effects may even be absent. We offer hands-on guidelines for how to statistically combine an original study and replication, and have developed a Web-based application ( https://rvanaert.shinyapps.io/hybrid ) for applying the hybrid method.

Chemosignalling effects of human tears revisited: Does exposure to female tears decrease males' perception of female sexual attractiveness?

Gelstein et al. reported the results of three experiments suggesting a dampening influence of inhalation of female emotional tears on males' arousal and perception of female sexual attractiveness, specifically in non-sexual situations. This prompted the hypothesis that crying exerts its influence on others not only via the auditory and visual mode but also via chemosignals. In three studies, we attempted to replicate and extend Gelstein et al.'s findings by including an additional condition with irritant tears, by using pictures of sexually attractive women, and by testing related hypotheses on the pro-social effects of exposure to tears. All three studies, separately or combined in a meta-analysis, failed to replicate the original inhibitory effects of tears. In addition, sniffing tears did not affect measures of connectedness, aggression and pro-social behaviour. It is concluded that the effects of female tears on male arousal and perception of female sexual attractiveness, if any, are very weak at best. Rather, it seems that crying exerts its strong inter-personal effects through the visual and auditory sensory channels.

Psychometric properties of the Dutch WHOQOL-OLD.

BACKGROUND: To assess the internal consistency reliability and construct validity of the Dutch version of the World Health Organization Quality of Life Instrument-Older Adults Module (WHOQOL-OLD). METHODS: The psychometric properties of the Dutch WHOQOL-OLD were examined in a cross-sectional study using a sample of 1,340 people aged 60 years or older. Participants completed a Web-based questionnaire, the 'Senioren Barometer'. Reliability was evaluated using Cronbach's alpha and corrected item-total correlations. Construct validity of the Dutch WHOQOL-OLD was evaluated with confirmatory factor analyses, and correlations within and between scales, using scales WHOQOL-BREF, Short Form Health Survey (SF-12), Tilburg Frailty Indicator (TFI), and the Emotional and Social Loneliness Scale (ESLS). RESULTS: The reliabilities of the six WHOQOL-OLD facets or subscales were sufficient to good (.66-.91). The convergent validity of the WHOQOL-OLD was good, whereas our findings on the divergent validity of the WHOQOL-OLD were somewhat mixed. Findings corroborating the divergent validity were that the 6-factor model fitted better than the second-order factor model, and WHOQOL-OLD facets sensory abilities, past, present and future activities, death and dying, intimacy correlated more strongly with similar than dissimilar scales. Not fully supporting divergent validity were the extremely high correlations between the factors corresponding to autonomy, past, present and future activities, and social participation. CONCLUSION: We offer Dutch healthcare and social workers an instrument with good psychometric properties for measuring quality of life in older people. Further research on interrelations between WHOQOL-OLD facets is recommended.

The prevalence of statistical reporting errors in psychology (1985-2013).

This study documents reporting errors in a sample of over 250,000 p-values reported in eight major psychology journals from 1985 until 2013, using the new R package "statcheck." statcheck retrieved null-hypothesis significance testing (NHST) results from over half of the articles from this period. In line with earlier research, we found that half of all published psychology papers that use NHST contained at least one p-value that was inconsistent with its test statistic and degrees of freedom. One in eight papers contained a grossly inconsistent p-value that may have affected the statistical conclusion. In contrast to earlier findings, we found that the average prevalence of inconsistent p-values has been stable over the years or has declined. The prevalence of gross inconsistencies was higher in p-values reported as significant than in p-values reported as nonsignificant. This could indicate a systematic bias in favor of significant results. Possible solutions for the high prevalence of reporting inconsistencies could be to encourage sharing data, to let co-authors check results in a so-called "co-pilot model," and to use statcheck to flag possible inconsistencies in one's own manuscript or during the review process.

Associations of individual chronic diseases and multimorbidity with multidimensional frailty.

OBJECTIVE: To examine the associations between individual chronic diseases and multidimensional frailty comprising physical, psychological, and social frailty. METHODS: Dutch individuals (N = 47,768) age ≥ 65 years completed a general health questionnaire sent by the Public Health Services (response rate of 58.5 %), including data concerning self-reported chronic diseases, multidimensional frailty, and sociodemographic characteristics. Multidimensional frailty was assessed with the Tilburg Frailty Indicator (TFI). Total frailty and each frailty domain were regressed onto background characteristics and the six most prevalent chronic diseases: diabetes mellitus, cancer, hypertension, arthrosis, urinary incontinence, and severe back disorder. Multimorbidity was defined as the presence of combinations of these six diseases. RESULTS: The six chronic diseases had medium and strong associations with total ((f2 = 0.122) and physical frailty (f2 = 0.170), respectively, and weak associations with psychological (f2 = 0.023) and social frailty (f2 = 0.008). The effects of the six diseases on the frailty variables differed strongly across diseases, with urinary incontinence and severe back disorder impairing frailty most. No synergetic effects were found; the effects of a disease on frailty did not get noteworthy stronger in the presence of another disease. CONCLUSIONS: Chronic diseases, in particular urinary incontinence and severe back disorder, were associated with frailty. We thus recommend assigning different weights to individual chronic diseases in a measure of multimorbidity that aims to examine effects of multimorbidity on multidimensional frailty. Because there were no synergetic effects of chronic diseases, the measure does not need to include interactions between diseases.

Psychological characteristics of BDSM practitioners.

INTRODUCTION: It has been generally thought that the practice of bondage-discipline, dominance-submission, sadism-masochism (BDSM) is in some form associated with psychopathology. However, several more recent studies suggest a relative good psychological health of BDSM practitioners. AIM: The aim of this study was to compare scores of BDSM practitioners and a control group on various fundamental psychological characteristics. METHODS: For this aim, 902 BDSM and 434 control participants completely filled out online questionnaires. Associations were examined using χ(2) tests of independence with φ and Cramer's V as effect size measures and eta or Pearson's correlation. Group differences were tested using analysis of covariance, with partial η(2) as effect size measure. A priori contrasts were tested using α = 0.01 to correct for multiple testing; for all other tests we used α = 0.05, two tailed. MAIN OUTCOME MEASURES: The study used Big Five personality dimensions (NEO Five-Factor Inventory), attachment styles (Attachment Styles Questionnaire), rejection sensitivity (Rejection Sensitivity Questionnaire), and subjective well-being (World Health Organization-Five Well-being Index). RESULTS: The results mostly suggest favorable psychological characteristics of BDSM practitioners compared with the control group; BDSM practitioners were less neurotic, more extraverted, more open to new experiences, more conscientious, less rejection sensitive, had higher subjective well-being, yet were less agreeable. Comparing the four groups, if differences were observed, BDSM scores were generally more favorably for those with a dominant than a submissive role, with least favorable scores for controls. CONCLUSION: We conclude that BDSM may be thought of as a recreational leisure, rather than the expression of psychopathological processes.

Anxiety and depression in patients with an implantable cardioverter defibrillator and their partners: a longitudinal study.

BACKGROUND: Little is known about the relation between emotional distress of patients with an implantable cardioverter defibrillator (ICD) and distress of their partners. This longitudinal study aimed to determine the association between patient and partner distress, and to explain distress in patients and partners using demographic, clinical, and psychological characteristics of patients and partners. METHODS: In total, 343 patients (84% male, mean age = 63.2 ± 9.2 years) and partners (18% male, mean age = 60.3 ± 9.4 years) completed self-report measures on anxiety and depressive symptoms at the time of implantation and at 2, 12, and 18 months follow-up. Type D personality (DS14; nonexpression of negative emotions) was assessed at baseline. Clinical information was gathered from medical records. Multilevel modeling was applied. RESULTS: The correlations between distress in patients and partners were positive (range r = 0.19-0.43; all P's < 0.001). Generally, partners experienced more anxiety and patients more depression. Patient and partner characteristics explained about half of the variance of distress of patients and partners. As expected, psychological characteristics had the strongest effect on distress. Patient anxiety was associated with patient depression and Type D, and to the corresponding distress variable of the partner (anxiety). Similar results were found for patient depression, partner anxiety, and partner depression. CONCLUSION: Distress of ICD patients is associated with distress of their partners. Future studies need to investigate mechanisms of these patient-partner dynamics, such as communication and behavioral issues. It may be timely to not only focus emotional support on patients, but also on their partners.

Explaining quality of life of older people in the Netherlands using a multidimensional assessment of frailty.

PURPOSE: Although frailty was originally a medical concept, nowadays more and more researchers are convinced of its multidimensional nature, including a psychological and social domain of frailty as well as a physical domain. The objective of this study was to test the hypothesis that the prediction of quality of life by physical frailty components is improved by adding psychological and social frailty components. METHODS: This cross-sectional study was carried out with a sample of Dutch citizens. A total of 1,031 people aged 65 years and older completed a Web-based questionnaire containing the Tilburg Frailty Indicator for measuring physical, psychological, and social frailty, and the WHOQOL-BREF for measuring four quality of life domains (physical health, psychological, social relations, environmental). RESULTS: The findings show that the prediction of all quality of life domains by eight physical components of frailty was improved after adding four psychological and three social frailty components. The psychological frailty component 'feeling down' significantly improved the prediction of all four quality of life domains, after controlling for the effects of background characteristics and all other frailty components. CONCLUSION: This study emphasizes the importance of a multidimensional assessment of frailty in the prediction of quality of life in older people.

How do psychology researchers interpret the results of multiple replication studies?

Employing two vignette studies, we examined how psychology researchers interpret the results of a set of four experiments that all test a given theory. In both studies, we found that participants' belief in the theory increased with the number of statistically significant results, and that the result of a direct replication had a stronger effect on belief in the theory than the result of a conceptual replication. In Study 2, we additionally found that participants' belief in the theory was lower when they assumed the presence of p-hacking, but that belief in the theory did not differ between preregistered and non-preregistered replication studies. In analyses of individual participant data from both studies, we examined the heuristics academics use to interpret the results of four experiments. Only a small proportion (Study 1: 1.6%; Study 2: 2.2%) of participants used the normative method of Bayesian inference, whereas many of the participants' responses were in line with generally dismissed and problematic vote-counting approaches. Our studies demonstrate that many psychology researchers overestimate the evidence in favor of a theory if one or more results from a set of replication studies are statistically significant, highlighting the need for better statistical education.

The dire disregard of measurement invariance testing in psychological science.

Self-report scales are widely used in psychology to compare means in latent constructs across groups, experimental conditions, or time points. However, for these comparisons to be meaningful and unbiased, the scales must demonstrate measurement invariance (MI) across compared time points or (experimental) groups. MI testing determines whether the latent constructs are measured equivalently across groups or time, which is essential for meaningful comparisons. We conducted a systematic review of 426 psychology articles with openly available data, to (a) examine common practices in conducting and reporting of MI testing, (b) assess whether we could reproduce the reported MI results, and (c) conduct MI tests for the comparisons that enabled sufficiently powerful MI testing. We identified 96 articles that contained a total of 929 comparisons. Results showed that only 4% of the 929 comparisons underwent MI testing, and the tests were generally poorly reported. None of the reported MI tests were reproducible, and only 26% of the 174 newly performed MI tests reached sufficient (scalar) invariance, with MI failing completely in 58% of tests. Exploratory analyses suggested that in nearly half of the comparisons where configural invariance was rejected, the number of factors differed between groups. These results indicate that MI tests are rarely conducted and poorly reported in psychological studies. We observed frequent violations of MI, suggesting that reported differences between (experimental) groups may not be solely attributed to group differences in the latent constructs. We offer recommendations aimed at improving reporting and computational reproducibility practices in psychology. (PsycInfo Database Record (c) 2024 APA, all rights reserved).