Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty.

This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

Airborne human papillomavirus (HPV) transmission risk during ablation procedures: A systematic review and meta-analysis.

BACKGROUND: Human Papillomavirus (HPV) is associated with development of oropharyngeal cancer. Aim of this review was to assess airborne transmission risk of infectious particles from HPV lesions to airway mucosa of medical staff during established ablation procedures. METHODS: A systematic review of human and animal studies, published before 09/2020, relevant to airborne HPV transmission. Controlled studies reporting prevalence of HPV-associated upper airway (nasal/oral/pharyngeal) disease in staff performing ablation procedures (laser, loop electrosurgical excision [LEEP], cryosurgery) on HPV lesions were included in meta-analysis. Additionally, we aimed for a comprehensive systematic overview of studies regarding occupational risk of airborne HPV transmission and safety measures during ablation procedures. RESULTS: A total of n = 30 original studies report outcomes related to HPV transmission risk in medical staff conducting ablation procedures. HPV DNA detection in ablation smoke (n = 7), matching HPV genotypes on ablated HPV lesions and face/airways of medical staff after ablation (n = 2), and evidence for infectivity of papillomavirus in ablation smoke (n = 3, animal models only) were reported. Three case reports describe occupational HPV disease of upper airway mucosa. Three controlled studies assessed warts (in CO2 laser-users only); when pooling all controls (general population, non-laser users), nasal/oral/pharyngeal lesion sites were more common amongst laser-users (OR = 5.75; 95%CI[1.55, 21.38]; p < .001). DISCUSSION: Airborne HPV dispersal with matching "high-risk" HPV-genotypes in airways of medical staff after ablations (LEEP and CO2-laser) and cases of HPV-associated upper airways neoplasms based on exposure to laser and LEEP smoke are documented. Upper airway mucosa is a more common anatomical site for warts in CO2 laser users compared to controls. Simple safety measures greatly reduce HPV contamination and transmission risk.

Web-based and mixed-mode cognitive large-scale assessments in higher education: An evaluation of selection bias, measurement bias, and prediction bias.

Educational large-scale studies typically adopt highly standardized settings to collect cognitive data on large samples of respondents. Increasing costs alongside dwindling response rates in these studies necessitate exploring alternative assessment strategies such as unsupervised web-based testing. Before respective assessment modes can be implemented on a broad scale, their impact on cognitive measurements needs to be quantified. Therefore, an experimental study on N = 17,473 university students from the German National Educational Panel Study has been conducted. Respondents were randomly assigned to a supervised paper-based, a supervised computerized, and an unsupervised web-based mode to work on a test of scientific literacy. Mode-specific effects on selection bias, measurement bias, and predictive bias were examined. The results showed a higher response rate in web-based testing as compared to the supervised modes, without introducing a pronounced mode-specific selection bias. Analyses of differential test functioning showed systematically larger test scores in paper-based testing, particularly among low to medium ability respondents. Prediction bias for web-based testing was observed for one out of four criteria on study-related success factors. Overall, the results indicate that unsupervised web-based testing is not strictly equivalent to other assessment modes. However, the respective bias introduced by web-based testing was generally small. Thus, unsupervised web-based assessments seem to be a feasible option in cognitive large-scale studies in higher education.

Opinion leadership types or continuous opinion leadership traits?

Opinion leadership is typically conceptualised as a continuous personality trait. However, many authors adhere to the view of qualitatively different opinion leadership types and apply arbitrary criteria to split continuous trait scores into two groups (i.e., opinion leaders vs. non-leaders). The present study is the first to empirically evaluate this approach. A sample of N = 3812 adults (67% women) was administered a validated opinion leadership scale. Finite mixture models examined whether the latent trait distribution can be represented by a set of discrete trait levels that reflected distinct opinion leadership types. The results did not give support to a discrete typology that distinguished leaders from non-leaders. Rather, opinion leadership was best characterised as a continuous trait.

Narcissism and Social Networking Behavior: A Meta-Analysis.

OBJECTIVE: The increasing popularity of social networking sites (SNS) such as Facebook and Twitter has given rise to speculations that the intensity of using these platforms is associated with narcissistic tendencies. However, recent research on this issue has been all but conclusive. METHOD: We present a three-level, random effects meta-analysis including 289 effect sizes from 57 studies (total N = 25,631) on the association between trait narcissism and social networking behavior. RESULTS: The meta-analysis identified a small to moderate effect of ρ = .17 (τ = .11), 95% CI [.13, .21], for grandiose narcissism that replicated across different social networking platforms, respondent characteristics, and time. Moderator analyses revealed pronounced cultural differences, with stronger associations in power-distant cultures. Moreover, social networking behaviors geared toward self-presentation and the number of SNS friends exhibited stronger effects than usage durations. CONCLUSIONS: Overall, the study not only supported but also refined the notion of a relationship between engaging in social networking sites and narcissistic personality traits.

Modeling competence development in the presence of selection bias.

A major challenge for representative longitudinal studies is panel attrition, because some respondents refuse to continue participating across all measurement waves. Depending on the nature of this selection process, statistical inferences based on the observed sample can be biased. Therefore, statistical analyses need to consider a missing-data mechanism. Because each missing-data model hinges on frequently untestable assumptions, sensitivity analyses are indispensable to gauging the robustness of statistical inferences. This article highlights contemporary approaches for applied researchers to acknowledge missing data in longitudinal, multilevel modeling and shows how sensitivity analyses can guide their interpretation. Using a representative sample of N = 13,417 German students, the development of mathematical competence across three years was examined by contrasting seven missing-data models, including listwise deletion, full-information maximum likelihood estimation, inverse probability weighting, multiple imputation, selection models, and pattern mixture models. These analyses identified strong selection effects related to various individual and context factors. Comparative analyses revealed that inverse probability weighting performed rather poorly in growth curve modeling. Moreover, school-specific effects should be acknowledged in missing-data models for educational data. Finally, we demonstrated how sensitivity analyses can be used to gauge the robustness of the identified effects.

Psychological effects of (non)employment: A cross-national comparison of the United States and Japan.

The involuntary loss of employment has been shown to deteriorate subjective well-being. Adopting a cross-cultural perspective on Jahoda's (1982) deprivation model this study examines several latent and manifest benefits of work that were expected to mediate the effects of employment status on well-being. It was hypothesized that in more collectivistic societies the decline in subjective well-being would be a consequence of a diminished sense of collective purpose for the non-employed, whereas in individualistic societies the crucial factors would be a loss of social status and financial benefits. The findings from two representative national surveys conducted in the United States (N = 1,093) and Japan (N = 647) provided partial support for these hypotheses. Cultural differences moderated the effects of employment status on the benefits of work. As a consequence, different processes mediated the decline in well-being for the non-employed in the two countries. These results are embedded within the wider discourse on culture and its effect on unemployment.

Disclosure of sensitive behaviors across self-administered survey modes: a meta-analysis.

In surveys, individuals tend to misreport behaviors that are in contrast to prevalent social norms or regulations. Several design features of the survey procedure have been suggested to counteract this problem; particularly, computerized surveys are supposed to elicit more truthful responding. This assumption was tested in a meta-analysis of survey experiments reporting 460 effect sizes (total N =125,672). Self-reported prevalence rates of several sensitive behaviors for which motivated misreporting has been frequently observed were compared across self-administered paper-and-pencil versus computerized surveys. The results revealed that computerized surveys led to significantly more reporting of socially undesirable behaviors than comparable surveys administered on paper. This effect was strongest for highly sensitive behaviors and surveys administered individually to respondents. Moderator analyses did not identify interviewer effects or benefits of audio-enhanced computer surveys. The meta-analysis highlighted the advantages of computerized survey modes for the assessment of sensitive topics.

Remote Testing of Reading Comprehension in 8-Year-Old Children: Mode and Setting Effects.

Proctored remote testing of cognitive abilities in the private homes of test-takers is becoming an increasingly popular alternative to standard psychological assessments in test centers or classrooms. Because these tests are administered under less standardized conditions, differences in computer devices or situational contexts might contribute to measurement biases that impede fair comparisons between test-takers. Because it is unclear whether cognitive remote testing might be a feasible assessment approach for young children, the present study (N = 1,590) evaluated a test of reading comprehension administered to children at the age of 8 years. To disentangle mode from setting effects, the children finished the test either in the classroom on paper or computer or remotely on tablets or laptops. Analyses of differential response functioning found notable differences between assessment conditions for selected items. However, biases in test scores were largely negligible. Only for children with below-average reading comprehension small setting effects between on-site and remote testing were observed. Moreover, response effort was higher in the three computerized test versions, among which, reading on tablets most strongly resembled the paper condition. Overall, these results suggest that, on average, even for young children remote testing introduces little measurement bias.

Accuracy and precision of fixed and random effects in meta-analyses of randomized control trials for continuous outcomes.

Meta-analyses of treatment effects in randomized control trials are often faced with the problem of missing information required to calculate effect sizes and their sampling variances. Particularly, correlations between pre- and posttest scores are frequently not available. As an ad-hoc solution, researchers impute a constant value for the missing correlation. As an alternative, we propose adopting a multivariate meta-regression approach that models independent group effect sizes and accounts for the dependency structure using robust variance estimation or three-level modeling. A comprehensive simulation study mimicking realistic conditions of meta-analyses in clinical and educational psychology suggested that imputing a fixed correlation 0.8 or adopting a multivariate meta-regression with robust variance estimation work well for estimating the pooled effect but lead to slightly distorted between-study heterogeneity estimates. In contrast, three-level meta-regressions resulted in largely unbiased fixed effects but more inconsistent prediction intervals. Based on these results recommendations for meta-analytic practice and future meta-analytic developments are provided.

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators.

Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In this study, three process data variables-text reread, item revisit, and answer change-were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed.

Attitudes towards AI: measurement and associations with personality.

Artificial intelligence (AI) has become an integral part of many contemporary technologies, such as social media platforms, smart devices, and global logistics systems. At the same time, research on the public acceptance of AI shows that many people feel quite apprehensive about the potential of such technologies-an observation that has been connected to both demographic and sociocultural user variables (e.g., age, previous media exposure). Yet, due to divergent and often ad-hoc measurements of AI-related attitudes, the current body of evidence remains inconclusive. Likewise, it is still unclear if attitudes towards AI are also affected by users' personality traits. In response to these research gaps, we offer a two-fold contribution. First, we present a novel, psychologically informed questionnaire (ATTARI-12) that captures attitudes towards AI as a single construct, independent of specific contexts or applications. Having observed good reliability and validity for our new measure across two studies (N1 = 490; N2 = 150), we examine several personality traits-the Big Five, the Dark Triad, and conspiracy mentality-as potential predictors of AI-related attitudes in a third study (N3 = 298). We find that agreeableness and younger age predict a more positive view towards artificially intelligent technology, whereas the susceptibility to conspiracy beliefs connects to a more negative attitude. Our findings are discussed considering potential limitations and future directions for research and practice.

The Predictive Validity of Item Effect Variables in the Satisfaction With Life Scale for Psychological and Physical Health.

Although the Satisfaction with Life Scale strives to capture a single dimension, describing respondents' satisfaction with life as a whole, individual items might also capture unique aspects of life satisfaction leading to some form of multidimensionality. Such systematic item-specific variance can be viewed as a content-laden secondary trait. Information on the nomological net and predictive validity can be useful to aid the interpretation of these item-specific effects. Therefore, the present study on N = 2,543 Dutch respondents adopts revised latent state-trait theory to disentangle common construct variance, random measurement error, and person-specific item effects in the Satisfaction with Life Scale across three measurement occasions. The reported analyses not only demonstrate how to examine item-specific multidimensionality in longitudinal data but also emphasize how different identification constraints for the latent variable lead to different interpretations. Moreover, the predictive validity of item effect variables for the prediction of psychological and physical health is examined. A cross-validation with the same sample at a later measurement period and robustness checks with incomplete data, support our findings on the substantive value of a multidimensional specification of the Satisfaction with Life Scale for substantive analyses. Finally, the contributions of person-specific item effects for psychological assessments are discussed.

Studying item-effect variables and their correlation patterns with multi-construct multi-state models.

Method effects on the item level can be modeled as latent difference variables in longitudinal data. These item-effect variables represent interindividual differences associated with responses to a specific item when assessing a common construct with multi-item scales. In latent variable analyses, their inclusion substantially improves model fits in comparison to classical unidimensional measurement models. More importantly, covariations between different item-effect variables and with other constructs can provide valuable insights, for example, into the structure of the studied instrument or the response process. Therefore, we introduce a multi-construct multi-state model with item-effect variables for systematic investigations of these correlation patterns within and between constructs. The implementation of this model is demonstrated using a sample of N = 2,529 Dutch respondents that provided measures of life satisfaction and positive affect at five measurement occasions. Our results confirm non-negligible item effects in two ostensibly unidimensional scales, indicating the importance of modeling interindividual differences on the item level. The correlation pattern between constructs indicated rather specific effects for individual items and no common causes, but the correlations within a construct align with the item content and support a substantive meaning. These analyses exemplify how multi-construct multi-state models allow the systematic examination of item effects to improve substantive and psychometric research.

The Dimensionality of the Brief COPE Before and During the COVID-19 Pandemic.

The Brief COPE (Coping Orientation to Problems Experienced) is a frequently used questionnaire assessing 14 theoretically derived coping mechanisms, but psychometric research has suggested inconsistent results concerning its factor structure. The aim of this study was to investigate primary and secondary order factor structures of the Brief COPE during the COVID-19 pandemic by testing 11 different models by confirmatory factor analyses and to assess differences between sex, age groups, and relationship status. Altogether, 529 respondents from Austria and Germany participated in a web-based survey. Results supported the originally hypothesized 14-factor structure but did not support previously described higher-order structures. However, bass-ackwards analyses suggested systematic overlap between different factors, which might have contributed to different factor solutions in previous research. Measurement invariance across sex, age groups, and relationship status could be confirmed. Findings suggest that cultural and situational aspects as well as the functional level should be considered in research on theoretical framing of coping behavior.

No effects of short-term exposure to mobile phone electromagnetic fields on human cognitive performance: a meta-analysis.

During recent years, a large number of studies on the effects of electromagnetic fields emitted by cellular mobile phones on human cognitive performance have been carried out. However, the results have been ambiguous. We carried out the current meta-analysis in order to investigate the impact of electromagnetic fields emitted by mobile phones on human cognition. Seventeen studies were included in the meta-analysis as they fulfill several requirements such as single- or double-blind experimental study design, and documentation of means and standard deviations of dependent variables. The meta-analysis was carried out as a group comparison between exposed and non-exposed subjects. No significant effects of electromagnetic fields emitted by Global System for Mobile Communications (GSM) and Universal Mobile Telecommunications System (UMTS) mobile phones were found. Cognitive abilities seem to be neither impaired nor facilitated. Results of the meta-analysis suggest that a substantial short-term impact of high frequency electromagnetic fields emitted by mobile phones on cognitive performance can essentially be ruled out.

A short measure of the need for affect.

Two studies are presented to introduce a 10-item short form of the Need for Affect Questionnaire (NAQ-S; cf. Maio & Esses, 2001 ). Study 1 was based on 4 independent samples (German or English language; N (total) = 2,151) and demonstrated the expected factorial structure of the NAQ-S; its measurement invariance with respect to gender, age, and education; and the predicted associations with relevant personality measures. A latent state-trait analysis conducted in Study 2 (N = 140) suggests that most of the reliable variance of the NAQ-S represents stable individual differences.

Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting.

Careless responding is a bias in survey responses that disregards the actual item content, constituting a threat to the factor structure, reliability, and validity of psychological measurements. Different approaches have been proposed to detect aberrant responses such as probing questions that directly assess test-taking behavior (e.g., bogus items), auxiliary or paradata (e.g., response times), or data-driven statistical techniques (e.g., Mahalanobis distance). In the present study, gradient boosted trees, a state-of-the-art machine learning technique, are introduced to identify careless respondents. The performance of the approach was compared with established techniques previously described in the literature (e.g., statistical outlier methods, consistency analyses, and response pattern functions) using simulated data and empirical data from a web-based study, in which diligent versus careless response behavior was experimentally induced. In the simulation study, gradient boosting machines outperformed traditional detection mechanisms in flagging aberrant responses. However, this advantage did not transfer to the empirical study. In terms of precision, the results of both traditional and the novel detection mechanisms were unsatisfactory, although the latter incorporated response times as additional information. The comparison between the results of the simulation and the online study showed that responses in real-world settings seem to be much more erratic than can be expected from the simulation studies. We critically discuss the generalizability of currently available detection methods and provide an outlook on future research on the detection of aberrant response patterns in survey research.

Exploring the myth of the chubby gamer: A meta-analysis on sedentary video gaming and body mass.

RATIONALE: High body mass and obesity are frequently linked to the use of sedentary media, like television (TV) or non-active video games. Empirical evidence regarding video gaming, however, has been mixed, and theoretical considerations explaining a relationship between general screen time and body mass may not generalize to non-active video gaming. OBJECTIVE: The current meta-analysis had two main goals. First, we wanted to provide an estimate of the average effect size of the relationship between sedentary video gaming and body mass. In doing so we acknowledged several context variables to gauge the stability of the average effect. Second, to provide additional evidence on processes, we tested the displacement effect of physical activity by video gaming time with the help of a meta-analytic structural equation model (MASEM). METHOD: Published and unpublished studies were identified through keyword searches in different databases and references in relevant reports were inspected for further studies. We present a random-effects, three-level meta-analysis based on 20 studies (total N = 38,097) with 32 effect sizes. RESULTS: The analyses revealed a small positive relationship between non-active video game use and body mass, ρˆ=.09, 95% CI [0.03, 0.14], indicating that they shared less than 1% in variance. The studies showed significant heterogeneity, Q (31) = 593.03, p < .001, I2 = 95.13. Moderator analyses revealed that the relationship was more pronounced for adults, ρˆ=.22, 95% CI [0.04, 0.40], as compared to adolescents, ρˆ=.01, 95% CI [-0.21, 0.23], or children, ρˆ=.09, 95% CI [-0.07, 0.25]. Meta-analytic structural equation modeling found little evidence for a displacement of physical activity through time spent on video gaming. CONCLUSION: These results do not corroborate the assumption of a strong link between video gaming and body mass as respective associations are small and primarily observed among adults.

The Structure of the Toronto Alexithymia Scale (TAS-20): A Meta-Analytic Confirmatory Factor Analysis.

Alexithymia is defined as the inability of persons to describe their emotional states, to identify the feelings of others, and a utilitarian type of thinking. The most popular instrument to assess alexithymia is the Toronto Alexithymia Scale (TAS-20). Despite its widespread use, an ongoing controversy pertains to its internal structure. The TAS-20 was originally constructed to capture three different factors, but several studies suggested different factor solutions, including bifactor models and models with a method factor for the reversely keyed items. The present study examined the dimensionality of the TAS-20 using summary data of 88 samples from 62 studies (total N = 69,722) with meta-analytic structural equation modeling. We found support for the originally proposed three-dimensional solution, whereas more complex models produced inconsistent factor loadings. Because a major source of misfit stems from translated versions, the results are discussed with respect to generalizations across languages and cultural contexts.