Correcting for outcome reporting bias in a meta-analysis: A meta-regression approach.

Outcome reporting bias (ORB) refers to the biasing effect caused by researchers selectively reporting outcomes within a study based on their statistical significance. ORB leads to inflated effect size estimates in meta-analysis if only the outcome with the largest effect size is reported due to ORB. We propose a new method (CORB) to correct for ORB that includes an estimate of the variability of the outcomes' effect size as a moderator in a meta-regression model. An estimate of the variability of the outcomes' effect size can be computed by assuming a correlation among the outcomes. Results of a Monte-Carlo simulation study showed that the effect size in meta-analyses may be severely overestimated without correcting for ORB. Estimates of CORB are close to the true effect size when overestimation caused by ORB is the largest. Applying the method to a meta-analysis on the effect of playing violent video games on aggression showed that the effect size estimate decreased when correcting for ORB. We recommend to routinely apply methods to correct for ORB in any meta-analysis. We provide annotated R code and functions to help researchers apply the CORB method.

Type D Personality as a Risk Factor for Adverse Outcome in Patients With Cardiovascular Disease: An Individual Patient-Data Meta-analysis.

OBJECTIVE: Type D personality, a joint tendency toward negative affectivity and social inhibition, has been linked to adverse events in patients with heart disease, although with inconsistent findings. Here, we apply an individual patient-data meta-analysis to data from 19 prospective cohort studies ( N = 11,151) to investigate the prediction of adverse outcomes by type D personality in patients with acquired cardiovascular disease. METHOD: For each outcome (all-cause mortality, cardiac mortality, myocardial infarction, coronary artery bypass grafting, percutaneous coronary intervention, major adverse cardiac event, any adverse event), we estimated type D's prognostic influence and the moderation by age, sex, and disease type. RESULTS: In patients with cardiovascular disease, evidence for a type D effect in terms of the Bayes factor (BF) was strong for major adverse cardiac event (BF = 42.5; odds ratio [OR] = 1.14) and any adverse event (BF = 129.4; OR = 1.15). Evidence for the null hypothesis was found for all-cause mortality (BF = 45.9; OR = 1.03), cardiac mortality (BF = 23.7; OR = 0.99), and myocardial infarction (BF = 16.9; OR = 1.12), suggesting that type D had no effect on these outcomes. This evidence was similar in the subset of patients with coronary artery disease (CAD), but inconclusive for patients with heart failure (HF). Positive effects were found for negative affectivity on cardiac and all-cause mortality, with the latter being more pronounced in male than female patients. CONCLUSION: Across 19 prospective cohort studies, type D predicts adverse events in patients with CAD, whereas evidence in patients with HF was inconclusive. In both patients with CAD and HF, we found evidence for a null effect of type D on cardiac and all-cause mortality.

Ensuring the quality and specificity of preregistrations.

Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of "researcher degrees of freedom" aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called "OSF Preregistration," http://osf.io/prereg/). The Prereg Challenge format was a "structured" workflow with detailed instructions and an independent review to confirm completeness; the "Standard" format was "unstructured" with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the "structured" format restricted the opportunistic use of researcher degrees of freedom better (Cliff's Delta = 0.49) than the "unstructured" format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research.

Psychometric evaluation of the computerized battery for neuropsychological evaluation of children (BENCI) among school aged children in the context of HIV in an urban Kenyan setting.

INTRODUCTION: Culturally validated neurocognitive measures for children in Low- and Middle-Income Countries are important in the timely and correct identification of neurocognitive impairments. Such measures can inform development of interventions for children exposed to additional vulnerabilities like HIV infection. The Battery for Neuropsychological Evaluation of Children (BENCI) is an openly available, computerized neuropsychological battery specifically developed to evaluate neurocognitive impairment. This study adapted the BENCI and evaluated its reliability and validity in Kenya. METHODOLOGY: The BENCI was adapted using translation and back-translation from Spanish to English. The psychometric properties were evaluated in a case-control study of 328 children (aged 6 - 14 years) living with HIV and 260 children not living with HIV in Kenya. We assessed reliability, factor structure, and measurement invariance with respect to HIV. Additionally, we examined convergent validity of the BENCI using tests from the Kilifi Toolkit. RESULTS: Internal consistencies (0.49 < α < 0.97) and test-retest reliabilities (-.34 to .81) were sufficient-to-good for most of the subtests. Convergent validity was supported by significant correlations between the BENCI's Verbal memory and Kilifi's Verbal List Learning (r = .41), the BENCI's Visual memory and Kilifi's Verbal List Learning (r = .32) and the BENCI's Planning total time test and Kilifi's Tower Test (r = -.21) and the BENCI's Abstract Reasoning test and Kilifi's Raven's Progressive Matrix (r = .21). The BENCI subtests highlighted meaningful differences between children living with HIV and those not living with HIV. After some minor adaptions, a confirmatory four-factor model consisting of flexibility, fluency, reasoning and working memory fitted well (χ2 = 135.57, DF = 51, N = 604, p < .001, RMSEA = .052, CFI = .944, TLI = .914) and was partially scalar invariant between HIV positive and negative groups. CONCLUSION: The English version of the BENCI formally translated for use in Kenya can be further adapted and integrated in clinical and research settings as a valid and reliable cognitive test battery.

Preregistration in practice: A comparison of preregistered and non-preregistered studies in psychology.

Preregistration has gained traction as one of the most promising solutions to improve the replicability of scientific effects. In this project, we compared 193 psychology studies that earned a Preregistration Challenge prize or preregistration badge to 193 related studies that were not preregistered. In contrast to our theoretical expectations and prior research, we did not find that preregistered studies had a lower proportion of positive results (Hypothesis 1), smaller effect sizes (Hypothesis 2), or fewer statistical errors (Hypothesis 3) than non-preregistered studies. Supporting our Hypotheses 4 and 5, we found that preregistered studies more often contained power analyses and typically had larger sample sizes than non-preregistered studies. Finally, concerns about the publishability and impact of preregistered studies seem unwarranted, as preregistered studies did not take longer to publish and scored better on several impact measures. Overall, our data indicate that preregistration has beneficial effects in the realm of statistical power and impact, but we did not find robust evidence that preregistration prevents p-hacking and HARKing (Hypothesizing After the Results are Known).

Modeling Interactions Between Latent Variables in Research on Type D Personality: A Monte Carlo Simulation and Clinical Study of Depression and Anxiety.

Several approaches exist to model interactions between latent variables. However, it is unclear how these perform when item scores are skewed and ordinal. Research on Type D personality serves as a good case study for that matter. In Study 1, we fitted a multivariate interaction model to predict depression and anxiety with Type D personality, operationalized as an interaction between its two subcomponents negative affectivity (NA) and social inhibition (SI). We constructed this interaction according to four approaches: (1) sum score product; (2) single product indicator; (3) matched product indicators; and (4) latent moderated structural equations (LMS). In Study 2, we compared these interaction models in a simulation study by assessing for each method the bias and precision of the estimated interaction effect under varying conditions. In Study 1, all methods showed a significant Type D effect on both depression and anxiety, although this effect diminished after including the NA and SI quadratic effects. Study 2 showed that the LMS approach performed best with respect to minimizing bias and maximizing power, even when item scores were ordinal and skewed. However, when latent traits were skewed LMS resulted in more false-positive conclusions, while the Matched PI approach adequately controlled the false-positive rate.

THIS (METHOD) IS (NOT) FINE.

In their response to my criticism of their recent article in Journal of Biosocial Science (te Nijenhuis et al., 2017), te Nijenhuis and van den Hoek (2018) raise four points none of which concerns my main point that the method of correlated vectors (MCV) applied to item-level data represents a flawed method. Here, I discuss te Nijenhuis and van den Hoek's four points. First, I argue that my previous application of MCV to item-level data showed that the method can yield nonsensical results. Second, I note that meta-analytic corrections for sampling error, imperfect measures, restriction of range and unreliability of the vectors are futile and cannot help fix the method. Third, I note that even with perfect data, the method can yield negative correlations. Fourth, I highlight the irrelevance of te Nijenhuis and van den Hoek (2018)'s point that my comment had not been published in a peerreviewed journal by referring to my articles in 2009 and 2017 on MCV in peer-reviewed journals.

IGNORING PSYCHOMETRIC PROBLEMS IN THE STUDY OF GROUP DIFFERENCES IN COGNITIVE TEST PERFORMANCE.

In a recent study, te Nijenhuis et al. (2017) used a version of Jensen's method of correlated vectors to study the nature of ethnic group differences on Raven's Progressive Matrices test. In this comment, the author points out that this method has been shown to be psychometrically inappropriate in studying group differences in performance on dichotomous (correctly or incorrectly scored) items. Specifically, the method uses item statistics like the item-total correlation that necessarily differ across groups differing in ability and employs a linear model to test inherent non-linear relations. Wicherts (2017) showed that this method can provide correlations far exceeding r=0.44 in cases where the group differences cannot possibly be on g because the items measure different traits across the groups. The psychometric problems with their method cast serious doubts on te Nijenhuis et al.'s conclusions concerning the role of g in the studied group difference in cognitive test performance.

Verify original results through reanalysis before replicating.

In determining the need to directly replicate, it is crucial to first verify the original results through independent reanalysis of the data. Original results that appear erroneous and that cannot be reproduced by reanalysis offer little evidence to begin with, thereby diminishing the need to replicate. Sharing data and scripts is essential to ensure reproducibility.

Researchers' Intuitions About Power in Psychological Research.

Many psychology studies are statistically underpowered. In part, this may be because many researchers rely on intuition, rules of thumb, and prior practice (along with practical considerations) to determine the number of subjects to test. In Study 1, we surveyed 291 published research psychologists and found large discrepancies between their reports of their preferred amount of power and the actual power of their studies (calculated from their reported typical cell size, typical effect size, and acceptable alpha). Furthermore, in Study 2, 89% of the 214 respondents overestimated the power of specific research designs with a small expected effect size, and 95% underestimated the sample size needed to obtain .80 power for detecting a small effect. Neither researchers' experience nor their knowledge predicted the bias in their self-reported power intuitions. Because many respondents reported that they based their sample sizes on rules of thumb or common practice in the field, we recommend that researchers conduct and report formal power analyses for their studies.

The prevalence of statistical reporting errors in psychology (1985-2013).

This study documents reporting errors in a sample of over 250,000 p-values reported in eight major psychology journals from 1985 until 2013, using the new R package "statcheck." statcheck retrieved null-hypothesis significance testing (NHST) results from over half of the articles from this period. In line with earlier research, we found that half of all published psychology papers that use NHST contained at least one p-value that was inconsistent with its test statistic and degrees of freedom. One in eight papers contained a grossly inconsistent p-value that may have affected the statistical conclusion. In contrast to earlier findings, we found that the average prevalence of inconsistent p-values has been stable over the years or has declined. The prevalence of gross inconsistencies was higher in p-values reported as significant than in p-values reported as nonsignificant. This could indicate a systematic bias in favor of significant results. Possible solutions for the high prevalence of reporting inconsistencies could be to encourage sharing data, to let co-authors check results in a so-called "co-pilot model," and to use statcheck to flag possible inconsistencies in one's own manuscript or during the review process.

On the nature and nurture of intelligence and specific cognitive abilities: the more heritable, the more culture dependent.

To further knowledge concerning the nature and nurture of intelligence, we scrutinized how heritability coefficients vary across specific cognitive abilities both theoretically and empirically. Data from 23 twin studies (combined N = 7,852) showed that (a) in adult samples, culture-loaded subtests tend to demonstrate greater heritability coefficients than do culture-reduced subtests; and (b) in samples of both adults and children, a subtest's proportion of variance shared with general intelligence is a function of its cultural load. These findings require an explanation because they do not follow from mainstream theories of intelligence. The findings are consistent with our hypothesis that heritability coefficients differ across cognitive abilities as a result of differences in the contribution of genotype-environment covariance. The counterintuitive finding that the most heritable abilities are the most culture-dependent abilities sheds a new light on the long-standing nature-nurture debate of intelligence.

The effects of height-for-age and HIV on cognitive development of school-aged children in Nairobi, Kenya: a structural equation modelling analysis.

Background: Empirical evidence indicates that both HIV infection and stunting impede cognitive functions of school-going children. However, there is less evidence on how these two risk factors amplify each other's negative effects. This study aimed to examine the direct effects of stunting on cognitive outcomes and the extent to which stunting (partially) mediates the effects of HIV, age, and gender on cognitive outcomes. Methodology: We applied structural equation modelling to cross-sectional data from 328 children living with HIV and 260 children living without HIV aged 6-14 years from Nairobi, Kenya to test the mediating effect of stunting and predictive effects of HIV, age, and gender on cognitive latent variables flexibility, fluency, reasoning, and verbal memory. Results: The model predicting the cognitive outcomes fitted well (RMSEA = 0.041, CFI = 0.966, χ2 = 154.29, DF = 77, p < 0.001). Height-for-age (a continuous indicator of stunting) predicted fluency (ß = 0.14) and reasoning (ß = 0.16). HIV predicted height-for-age (ß = -0.24) and showed direct effects on reasoning (ß = -0.66), fluency (ß = -0.34), flexibility (ß = 0.26), and verbal memory (ß = -0.22), highlighting that the effect of HIV on cognitive variables was partly mediated by height-for-age. Conclusion: In this study, we found evidence that stunting partly explains the effects of HIV on cognitive outcomes. The model suggests there is urgency to develop targeted preventative and rehabilitative nutritional interventions for school children with HIV as part of a comprehensive set of interventions to improve cognitive functioning in this high-risk group of children. Being infected or having been born to a mother who is HIV positive poses a risk to normal child development.

Are Speeded Tests Unfair? Modeling the Impact of Time Limits on the Gender Gap in Mathematics.

When cognitive and educational tests are administered under time limits, tests may become speeded and this may affect the reliability and validity of the resulting test scores. Prior research has shown that time limits may create or enlarge gender gaps in cognitive and academic testing. On average, women complete fewer items than men when a test is administered with a strict time limit, whereas gender gaps are frequently reduced when time limits are relaxed. In this study, we propose that gender differences in test strategy might inflate gender gaps favoring men, and relate test strategy to stereotype threat effects under which women underperform due to the pressure of negative stereotypes about their performance. First, we applied a Bayesian two-dimensional item response theory (IRT) model to data obtained from two registered reports that investigated stereotype threat in mathematics, and estimated the latent correlation between underlying test strategy (here, completion factor, a proxy for working speed) and mathematics ability. Second, we tested the gender gap and assessed potential effects of stereotype threat on female test performance. We found a positive correlation between the completion factor and mathematics ability, such that more able participants dropped out later in the test. We did not observe a stereotype threat effect but found larger gender differences on the latent completion factor than on latent mathematical ability, suggesting that test strategies affect the gender gap in timed mathematics performance. We argue that if the effect of time limits on tests is not taken into account, this may lead to test unfairness and biased group comparisons, and urge researchers to consider these effects in either their analyses or study planning.

How do psychology researchers interpret the results of multiple replication studies?

Employing two vignette studies, we examined how psychology researchers interpret the results of a set of four experiments that all test a given theory. In both studies, we found that participants' belief in the theory increased with the number of statistically significant results, and that the result of a direct replication had a stronger effect on belief in the theory than the result of a conceptual replication. In Study 2, we additionally found that participants' belief in the theory was lower when they assumed the presence of p-hacking, but that belief in the theory did not differ between preregistered and non-preregistered replication studies. In analyses of individual participant data from both studies, we examined the heuristics academics use to interpret the results of four experiments. Only a small proportion (Study 1: 1.6%; Study 2: 2.2%) of participants used the normative method of Bayesian inference, whereas many of the participants' responses were in line with generally dismissed and problematic vote-counting approaches. Our studies demonstrate that many psychology researchers overestimate the evidence in favor of a theory if one or more results from a set of replication studies are statistically significant, highlighting the need for better statistical education.

The dire disregard of measurement invariance testing in psychological science.

Self-report scales are widely used in psychology to compare means in latent constructs across groups, experimental conditions, or time points. However, for these comparisons to be meaningful and unbiased, the scales must demonstrate measurement invariance (MI) across compared time points or (experimental) groups. MI testing determines whether the latent constructs are measured equivalently across groups or time, which is essential for meaningful comparisons. We conducted a systematic review of 426 psychology articles with openly available data, to (a) examine common practices in conducting and reporting of MI testing, (b) assess whether we could reproduce the reported MI results, and (c) conduct MI tests for the comparisons that enabled sufficiently powerful MI testing. We identified 96 articles that contained a total of 929 comparisons. Results showed that only 4% of the 929 comparisons underwent MI testing, and the tests were generally poorly reported. None of the reported MI tests were reproducible, and only 26% of the 174 newly performed MI tests reached sufficient (scalar) invariance, with MI failing completely in 58% of tests. Exploratory analyses suggested that in nearly half of the comparisons where configural invariance was rejected, the number of factors differed between groups. These results indicate that MI tests are rarely conducted and poorly reported in psychological studies. We observed frequent violations of MI, suggesting that reported differences between (experimental) groups may not be solely attributed to group differences in the latent constructs. We offer recommendations aimed at improving reporting and computational reproducibility practices in psychology. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Comparing the prevalence of statistical reporting inconsistencies in COVID-19 preprints and matched controls: a registered report.

The COVID-19 outbreak has led to an exponential increase of publications and preprints about the virus, its causes, consequences, and possible cures. COVID-19 research has been conducted under high time pressure and has been subject to financial and societal interests. Doing research under such pressure may influence the scrutiny with which researchers perform and write up their studies. Either researchers become more diligent, because of the high-stakes nature of the research, or the time pressure may lead to cutting corners and lower quality output. In this study, we conducted a natural experiment to compare the prevalence of incorrectly reported statistics in a stratified random sample of COVID-19 preprints and a matched sample of non-COVID-19 preprints. Our results show that the overall prevalence of incorrectly reported statistics is 9-10%, but frequentist as well as Bayesian hypothesis tests show no difference in the number of statistical inconsistencies between COVID-19 and non-COVID-19 preprints. In conclusion, the literature suggests that COVID-19 research may on average have more methodological problems than non-COVID-19 research, but our results show that there is no difference in the statistical reporting quality.