Adventitious Error and Its Implications for Testing Relations Between Variables and for Composite Measurement Outcomes.

Wu and Browne (Psychometrika 80(3):571-600, 2015. https://doi.org/10.1007/s11336-015-9451-3 ; henceforth W &B) introduced the notion of adventitious error to explicitly take into account approximate goodness of fit of covariance structure models (CSMs). Adventitious error supposes that observed covariance matrices are not directly sampled from a theoretical population covariance matrix but from an operational population covariance matrix. This operational matrix is randomly distorted from the theoretical matrix due to differences in study implementations. W &B showed how adventitious error is linked to the root mean square error of approximation (RMSEA) and how the standard errors (SEs) of parameter estimates are augmented. Our contribution is to consider adventitious error as a general phenomenon and to illustrate its consequences. Using simulations, we illustrate that its impact on SEs can be generalized to pairwise relations between variables beyond the CSM context. Using derivations, we conjecture that heterogeneity of effect sizes across studies and overestimation of statistical power can both be interpreted as stemming from adventitious error. We also show that adventitious error, if it occurs, has an impact on the uncertainty of composite measurement outcomes such as factor scores and summed scores. The results of a simulation study show that the impact on measurement uncertainty is rather small although larger for factor scores than for summed scores. Adventitious error is an assumption about the data generating mechanism; the notion offers a statistical framework for understanding a broad range of phenomena, including approximate fit, varying research findings, heterogeneity of effects, and overestimates of power.

Joint Latent Space Model for Social Networks with Multivariate Attributes.

In social, behavioral and economic sciences, researchers are interested in modeling a social network among a group of individuals, along with their attributes. The attributes can be responses to survey questionnaires and are often high dimensional. We propose a joint latent space model (JLSM) that summarizes information from the social network and the multivariate attributes in a person-attribute joint latent space. We develop a variational Bayesian expectation-maximization estimation algorithm to estimate the attribute and person locations in the joint latent space. This methodology allows for effective integration, informative visualization and prediction of social networks and attributes. Using JLSM, we explore the French financial elites based on their social networks and their career, political views and social status. We observe a division in the social circles of the French elites in accordance with the differences in their attributes. We analyze user networks and behaviors in multimodal social media systems like YouTube. A R package "jlsm" is developed to fit the models proposed in this paper and is publicly available from the CRAN repository https://cran.r-project.org/web/packages/jlsm/jlsm.pdf .

Characterizing Sleep Problems in 16p11.2 Deletion and Duplication.

Studies of 16p11.2 copy number variants (CNVs) provide an avenue to identify mechanisms of impairment and develop targeted treatments for individuals with neurodevelopmental disorders. 16p11.2 deletion and duplication phenotypes are currently being ascertained; however, sleep disturbances are minimally described. In this study, we examine sleep disturbance in a well-characterized national sample of 16p11.2 CNVs, the Simons Foundation Autism Research Initiative (SFARI) database of youth and adults (n = 692). Factor analyses and multilevel models of derived sleep questionnaires for youth (n = 345) and adults (n = 347) indicate that 16p11.2 carriers show elevated sleep disturbance relative to community controls. Non-carrier family members also show elevated sleep disturbance. However, sleep duration does not differ between carriers and controls. Further studies of sleep in 16p11.2 are needed.

Impact of word properties on list learning: An explanatory item analysis.

OBJECTIVE: A variety of factors affect list learning performance and relatively few studies have examined the impact of word selection on these tests. This study examines the effect of both language and memory processing of individual words on list learning. METHOD: Item-response data from 1,219 participants, Mage = 74.41 (SD = 7.13), Medu = 13.30 (SD = 2.72), in the Harmonized Cognitive Assessment Protocol were used. A Bayesian generalized (non)linear multilevel modeling framework was used to specify the measurement and explanatory item-response theory models. Explanatory effects on items due to learning over trials, serial position of words, and six word properties obtained through the English Lexicon Project were modeled. RESULTS: A two parameter logistic (2PL) model with trial-specific learning effects produced the best measurement fit. Evidence of the serial position effect on word learning was observed. Robust positive effects on word learning were observed for body-object integration while robust negative effects were observed for word frequency, concreteness, and semantic diversity. A weak negative effect of average age of acquisition and a weak positive effect for the number of phonemes in the word were also observed. CONCLUSIONS: Results demonstrate that list learning performance depends on factors beyond the repetition of words. Identification of item factors that predict learning could extend to a range of test development problems including translation, form equating, item revision, and item bias. In data harmonization efforts, these methods can also be used to help link tests via shared item features and testing of whether these features are equally explanatory across samples. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Accelerating Psychological Science With Metastudies: A Demonstration Using the Risky-Choice Framing Effect.

A metastudy is a set of many tiny studies (microstudies) created from a much larger collection of possibilities. Metastudies can yield many of the benefits of time-consuming replications and meta-analyses but more efficiently and with greater attention to generalizability and the causal effects of moderators. Statistical precision and power are higher than in studies with the same total sample size but with fewer conditions and more participants per condition. In this article, we describe metastudies and their benefits, demonstrate how to conduct a metastudy using the well-known risky-choice framing effect as an example, and report on the generalizability of that effect. In three metastudies (total N = 2,338), the framing effect generalized well across most of the potential moderators tested, as was expected. Surprisingly, however, the effect was up to twice as large when the certain option was replaced with a slightly risky option; prospect theory predicts the opposite, and fuzzy-trace theory predicts no difference. Metastudies provide a relatively quick and not-so-painful way of examining an effect's generalizability without waiting for a meta-analysis. Both individual labs and multilab networks are encouraged to shift from traditional studies to metastudies.

Space-time modeling of intensive binary time series eye-tracking data using a generalized additive logistic regression model.

Eye-tracking has emerged as a popular method for empirical studies of cognitive processes across multiple substantive research areas. Eye-tracking systems are capable of automatically generating fixation-location data over time at high temporal resolution. Often, the researcher obtains a binary measure of whether or not, at each point in time, the participant is fixating on a critical interest area or object in the real world or in a computerized display. Eye-tracking data are characterized by spatial-temporal correlations and random variability, driven by multiple fine-grained observations taken over small time intervals (e.g., every 10 ms). Ignoring these data complexities leads to biased inferences for the covariates of interest such as experimental condition effects. This article presents a novel application of a generalized additive logistic regression model for intensive binary time series eye-tracking data from a between- and within-subjects experimental design. The model is formulated as a generalized additive mixed model (GAMM) and implemented in the mgcv R package. The generalized additive logistic regression model was illustrated using an empirical data set aimed at understanding the accommodation of regional accents in spoken language processing. Accuracy of parameter estimates and the importance of modeling the spatial-temporal correlations in detecting the experimental condition effects were shown in conditions similar to our empirical data set via a simulation study. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Level-specific residuals and diagnostic measures, plots, and tests for random effects selection in multilevel and mixed models.

Multilevel data structures are often found in multiple substantive research areas, and multilevel models (MLMs) have been widely used to allow for such multilevel data structures. One important step when applying MLM is the selection of an optimal set of random effects to account for variability and heteroscedasticity in multilevel data. Literature reviews on current practices in applying MLM showed that diagnostic plots are only rarely used for model selection and for model checking. In this study, possible random effects and a generic description of the random effects were provided to guide researchers to select necessary random effects. In addition, based on extensive literature reviews, level-specific diagnostic plots were presented using various kinds of level-specific residuals, and diagnostic measures and statistical tests were suggested to select a set of random effects. Existing and newly proposed methods were illustrated using two data sets: a cross-sectional data set and a longitudinal data set. Along with the illustration, we discuss the methods and provide guidelines to select necessary random effects in model-building steps. R code was provided for the analyses.

Understanding the role of subpopulations and reliability in between-group studies.

The replication crisis has led to a renewed discussion about the impacts of measurement quality on the precision of psychology research. High measurement quality is associated with low measurement error, yet the role of reliability in the quality of experimental research is not always well understood. In this study, we attempt to understand the role of reliability through its relationship with power while focusing on between-group designs for experimental studies. We outline a latent variable framework to investigate this nuanced relationship through equations. An under-evaluated aspect of the relationship is the variance and homogeneity of the subpopulation from which the study sample is drawn. Higher homogeneity implies a lower reliability, but yields higher power. We proceed to demonstrate the impact of this relationship between reliability and power by imitating different scenarios of large-scale replications with between-group designs. We find negative correlations between reliability and power when there are sizable differences in the latent variable variance and negligible differences in the other parameters across studies. Finally, we analyze the data from the replications of the ego depletion effect (Hagger et al., 2016) and the replications of the grammatical aspect effect (Eerland et al., 2016), each time with between-group designs, and the results align with previous findings. The applications show that a negative relationship between reliability and power is a realistic possibility with consequences for applied work. We suggest that more attention be given to the homogeneity of the subpopulation when study-specific reliability coefficients are reported in between-group studies.

Modeling Conditional Dependence of Response Accuracy and Response Time with the Diffusion Item Response Theory Model.

In this paper, we propose a model-based method to study conditional dependence between response accuracy and response time (RT) with the diffusion IRT model (Tuerlinckx and De Boeck in Psychometrika 70(4):629-650, 2005, https://doi.org/10.1007/s11336-000-0810-3 ; van der Maas et al. in Psychol Rev 118(2):339-356, 2011, https://doi.org/10.1080/20445911.2011.454498 ). We extend the earlier diffusion IRT model by introducing variability across persons and items in cognitive capacity (drift rate in the evidence accumulation process) and variability in the starting point of the decision processes. We show that the extended model can explain the behavioral patterns of conditional dependency found in the previous studies in psychometrics. Variability in cognitive capacity can predict positive and negative conditional dependency and their interaction with the item difficulty. Variability in starting point can account for the early changes in the response accuracy as a function of RT given the person and item effects. By the combination of the two variability components, the extended model can produce the curvilinear conditional accuracy functions that have been observed in psychometric data. We also provide a simulation study to validate the parameter recovery of the proposed model and present two empirical applications to show how to implement the model to study conditional dependency underlying data response accuracy and RTs.

From the Outgoing Editor.

The Journal of Intelligence was founded in 2013, eight years ago [...].

Not all DIF is shaped similarly.

In response to the target article by Teresi et al. (2021), we explain why the article is useful and we also present a different approach. An alternative category of differential item functioning (DIF) is presented with a corresponding way of modeling DIF, based on random person and random item effects and explanatory covariates.

Modeling Within-Item Dependencies in Parallel Data on Test Responses and Brain Activation.

In this paper, we propose a joint modeling approach to analyze dependency in parallel response data. We define two types of dependency: higher-level dependency and within-item conditional dependency. While higher-level dependency can be estimated with common latent variable modeling approaches, within-item conditional dependency is a unique kind of information that is often not captured with extant methods, despite its potential to shed new insights into the relationship between the two types of response data. We differentiate three ways of modeling within-item conditional dependency by conditioning on raw values, expected values, or residual values of the response data, which have different implications in terms of response processes. The proposed approach is illustrated with the example of analyzing parallel data on response accuracy and brain activations from a Theory of Mind assessment. The consequence of ignoring within-item conditional dependency is investigated with empirical and simulation studies in comparison to conventional dependency analysis that focuses exclusively on relationships between latent variables.

Does planning help for execution? The complex relationship between planning and execution.

Planning and execution are two important parts of the problem-solving process. Based on related research, it is expected that planning speed and execution speed are positively correlated because of underlying individual differences in general mental speed. While there could also be a direct negative dependency of execution time on planning time, given the hypothesis that an investment in planning contributes to more efficient execution. The positive correlation and negative dependency are not contradictory since the former is a relationship across individuals (at the latent variable level) and the latter is a relationship within individuals (at the manifest variable level) after controlling for across-individual relationships. With two linear mixed model analyses and a factor model analysis, these two different kinds of relationships were examined using dependency analysis. The results supported the above hypotheses. The correlation between the latent variables of planning and execution was found to be positive and the dependency of execution time on planning time was found to be negative in all analyses. Moreover, the negative dependency varied among items and to some extent among persons as well. In summary, this study provides a clearer picture of the relationship between planning and execution and suggests that analyses at different levels may reveal different relationships.

Trivariate Theory of Mind Data Analysis with a Conditional Joint Modeling Approach.

Theory of mind (ToM) is an essential social-cognitive ability to understand one's own and other people's mental states. Neural data as well as behavior data have been utilized in ToM research, but the two types of data have rarely been analyzed together, creating a large gap in the literature. In this paper, we propose and apply a novel joint modeling approach to analyze brain activations with two types of behavioral data, response times and response accuracy, obtained from a multi-item ToM assessment, with the intention to shed new light on the nature of the underlying process of ToM reasoning. Our trivariate data analysis suggested that different levels or kinds of processes might be involved during the ToM assessment, which seem to differ in terms of cognitive efficiency and sensitivity to ToM items and the correctness of item responses. Additional details on the trivariate data analysis results are provided with discussions on their implications for ToM research.

The Many Faces of Intelligence: A Discussion of Geary's Mitochondrial Functioning Theory on General Intelligence.

David Geary's article on intelligence (Geary, 2018) and the summary of his theory in Journal of Intelligence offer a refreshing and inspirational view on intelligence [...].

Factors associated with sensitive regression weights: A fungible parameter approach.

Sensitive parameters serve as a weak foundation for scientific inferences, because they provide less certainty about the accuracy and trustworthiness of the estimated model. Fungible weights may be used to examine parameter sensitivity by looking at how much sets of interchangeable, slightly suboptimal linear regression weights, all of which yield an identical, slightly reduced value of R2, differ from the optimal OLS weights. We find that in the two-predictor case, the range of a predictor's fungible weights is almost completely explained by the absolute value of the correlation of the other predictor with the criterion variable (R2 = .990); an interaction with the variance inflation factor (VIF) yields R2 = 1. In the more complicated three-predictor case, the effects of the other two correlations yield R2 = .839, and including the predictor's VIF and its interactions yields R2 = .910. The effects observed occur because alternative predictors with a high correlation with the criterion, or with each other, can compensate for the changes to a predictor's weight while still yielding similar predicted values. An R function is provided to calculate the range of fungible weights for a given covariance matrix. We close with a discussion of some important implications of our results regarding parameter sensitivity and the trustworthiness of effect estimates.

Standard Setting of Competency in Mastoidectomy for the Cross-Institutional Mastoidectomy Assessment Tool.

OBJECTIVE: Competency-based surgical training involves progressive autonomy given to the trainee. This requires systematic and evidence-based assessment with well-defined standards of proficiency. The objective of this study is to develop standards for the cross-institutional mastoidectomy assessment tool to inform decisions regarding whether a resident demonstrates sufficient skill to perform a mastoidectomy with or without supervision. METHODS: A panel of fellowship-trained content experts in mastoidectomy was surveyed in relation to the 16 items of the assessment tool to determine the skills needed for supervised and unsupervised surgery. We examined the consensus score to investigate the degree of agreement among respondents for each survey item as well as additional analyses to determine whether the reported skill level required for each survey item was significantly different for the supervised versus unsupervised level. RESULTS: Ten panelists representing different US training programs responded. There was considerable consensus on cut-off scores for each item and trainee level between panelists, with moderate (0.62) to very high (0.95) consensus scores depending on assessment item. Further analyses demonstrated that the difference between supervised and unsupervised skill levels was significantly meaningful for all items. Finally, minimum-passing scores for each item was established. CONCLUSION: We defined performance standards for the cross-institutional mastoidectomy assessment tool using the Angoff method. These cut-off scores that can be used to determine when trainees can progress from performance under supervision to performance without supervision. This can be used to guide training in a competency-based training curriculum.

Evaluation on types of invariance in studying extreme response bias with an IRTree approach.

In recent years, item response tree (IRTree) approaches have received increasing attention in the response style literature for their ability to partial out response style latent variables as well as associated item parameters. When an IRTree approach is adopted to measure extreme response styles, directional and content invariance could be assumed at the latent variable and item parameter levels. In this study, we propose to evaluate the empirical validity of these invariance assumptions by employing a general IRTree model with relaxed invariance assumptions. This would allow us to examine extreme response biases, beyond extreme response styles. With three empirical applications of the proposed evaluation, we find that relaxing some of the invariance assumptions improves the model fit, which suggests that not all assumed invariances are empirically supported. Specifically, at the latent variable level, we find reasonable evidence for directional invariance but mixed evidence for content invariance, although we also find that estimated correlations between content-specific extreme response latent variables are high, hinting at the potential presence of a general extreme response tendency. At the item parameter level, we find no directional or content invariance for thresholds and no content invariance for slopes. We discuss how the variant item parameter estimates obtained from a general IRTree model can offer useful insight to help us understand response bias related to extreme responding measured within the IRTree framework.

Collaborative Problem Solving: Processing Actions, Time, and Performance.

This study is based on one collaborative problem solving task from an international assessment: the Xandar task. It was developed and delivered by the Organization for Economic Co-operation and Development Program for International Student Assessment (OECD PISA) 2015. We have investigated the relationship of problem solving performance with invested time and number of actions in collaborative episodes for the four parts of the Xandar task. The parts require the respondent to collaboratively plan a process for problem solving, implement the process, reach a solution, and evaluate the solution (For a full description, see the Materials and Methods section, "Parts of the Xandar Task.") Examples of an action include posting to a chat log, accessing a shared resource, or conducting a search on a map tool. Actions taken in each part of the task were identified by PISA and recorded in the data set numerically. A confirmatory factor analysis (CFA) model looks at two types of relationship: at the level of latent variables (the factors) and at extra dependencies, which here are direct effects and correlated residuals (independent of the factors). The model, which is well-fitting, has three latent variables: actions (A), times (T), and level of performance (P). Evidence for the uni-dimensionality of performance level is also found in a separate analysis of the binary items. On the whole for the entire task, participants with more activities are less successful and faster, based on the United States data set employed in the analysis. By contrast, successful participants take more time. By task part, the model also investigates relationships between activities, time, and performance level within the parts. This was done because one can expect dependencies within parts of such a complex task. Results indicate some general and some specific relationships within the parts, see the full manuscript for more detail. We conclude with a discussion of what the investigated relationships may reveal. We also describe why such investigations may be important to consider when preparing students for improved skills in collaborative problem solving, considered a key aspect of successful 21st century skills in the workplace and in everyday life in many countries.

An Overview of Models for Response Times and Processes in Cognitive Tests.

Response times (RTs) are a natural kind of data to investigate cognitive processes underlying cognitive test performance. We give an overview of modeling approaches and of findings obtained with these approaches. Four types of models are discussed: response time models (RT as the sole dependent variable), joint models (RT together with other variables as dependent variable), local dependency models (with remaining dependencies between RT and accuracy), and response time as covariate models (RT as independent variable). The evidence from these approaches is often not very informative about the specific kind of processes (other than problem solving, information accumulation, and rapid guessing), but the findings do suggest dual processing: automated processing (e.g., knowledge retrieval) vs. controlled processing (e.g., sequential reasoning steps), and alternative explanations for the same results exist. While it seems well-possible to differentiate rapid guessing from normal problem solving (which can be based on automated or controlled processing), further decompositions of response times are rarely made, although possible based on some of model approaches.