Strong intercorrelations among global graph-theoretic indices of structural connectivity in the human brain.

Graph-theoretic metrics derived from neuroimaging data have been heralded as powerful tools for uncovering neural mechanisms of psychological traits, psychiatric disorders, and neurodegenerative diseases. In N = 8,185 human structural connectomes from UK Biobank, we examined the extent to which 11 commonly-used global graph-theoretic metrics index distinct versus overlapping information with respect to interindividual differences in brain organization. Using unthresholded, FA-weighted networks we found that all metrics other than Participation Coefficient were highly intercorrelated, both with each other (mean |r| = 0.788) and with a topologically-naïve summary index of brain structure (mean edge weight; mean |r| = 0.873). In a series of sensitivity analyses, we found that overlap between metrics is influenced by the sparseness of the network and the magnitude of variation in edge weights. Simulation analyses representing a range of population network structures indicated that individual differences in global graph metrics may be intrinsically difficult to separate from mean edge weight. In particular, Closeness, Characteristic Path Length, Global Efficiency, Clustering Coefficient, and Small Worldness were nearly perfectly collinear with one another (mean |r| = 0.939) and with mean edge weight (mean |r| = 0.952) across all observed and simulated conditions. Global graph-theoretic measures are valuable for their ability to distill a high-dimensional system of neural connections into summary indices of brain organization, but they may be of more limited utility when the goal is to index separable components of interindividual variation in specific properties of the human structural connectome.

Pay Attention to the Ignorable Missing Data Mechanisms! An Exploration of Their Impact on the Efficiency of Regression Coefficients.

The use of modern missing data techniques has become more prevalent with their increasing accessibility in statistical software. These techniques focus on handling data that are missing at random (MAR). Although all MAR mechanisms are routinely treated as the same, they are not equal. The impact of missing data on the efficiency of parameter estimates can differ for different MAR variations, even when the amount of missing data is held constant; yet, in current practice, only the rate of missing data is reported. The impact of MAR on the loss of efficiency can instead be more directly measured by the fraction of missing information (FMI). In this article, we explore this impact using FMIs in regression models with one and two predictors. With the help of a Shiny application, we demonstrate that efficiency loss due to missing data can be highly complex and is not always intuitive. We recommend substantive researchers who work with missing data report estimates of FMIs in addition to the rate of missingness. We also encourage methodologists to examine FMIs when designing simulation studies with missing data, and to explore the behavior of efficiency loss under MAR using FMIs in more complex models.

On Penalty Parameter Selection for Estimating Network Models.

Network models are gaining popularity as a way to estimate direct effects among psychological variables and investigate the structure of constructs. A key feature of network estimation is determining which edges are likely to be non-zero. In psychology, this is commonly achieved through the graphical lasso regularization method that estimates a precision matrix of Gaussian variables using an ℓ1-penalty to push small values to zero. A tuning parameter, λ, controls the sparsity of the network. There are many methods to select λ, which can lead to vastly different graphs. The most common approach in psychological network applications is to minimize the extended Bayesian information criterion, but the consistency of this method for model selection has primarily been examined in high dimensional settings (i.e., n < p) that are uncommon in psychology. Further, there is some evidence that alternative selection methods may have superior performance. Here, using simulation, we compare four different methods for selecting λ, including the stability approach to regularization selection (StARS), K-fold cross-validation, the rotation information criterion (RIC), and the extended Bayesian information criterion (EBIC). Our results demonstrate that penalty parameter selection should be made based on data characteristics and the inferential goal (e.g., to increase sensitivity versus to avoid false positives). We end with recommendations for selecting the penalty parameter when using the graphical lasso.

Latent Variable Models and Networks: Statistical Equivalence and Testability.

Networks are gaining popularity as an alternative to latent variable models for representing psychological constructs. Whereas latent variable approaches introduce unobserved common causes to explain the relations among observed variables, network approaches posit direct causal relations between observed variables. While these approaches lead to radically different understandings of the psychological constructs of interest, recent articles have established mathematical equivalences that hold between network models and latent variable models. We argue that the fact that for any model from one class there is an equivalent model from the other class does not mean that both models are equally plausible accounts of the data-generating mechanism. In many cases the constraints that are meaningful in one framework translate to constraints in the equivalent model that lack a clear interpretation in the other framework. Finally, we discuss three diverging predictions for the relation between zero-order correlations and partial correlations implied by sparse network models and unidimensional factor models. We propose a test procedure that compares the likelihoods of these models in light of these diverging implications. We use an empirical example to illustrate our argument.

Investigating the Utility of Fixed-margin Sampling in Network Psychometrics.

Steinley, Hoffman, Brusco, and Sher (2017) proposed a new method for evaluating the performance of psychological network models: fixed-margin sampling. The authors investigated LASSO regularized Ising models (eLasso) by generating random datasets with the same margins as the original binary dataset, and concluded that many estimated eLasso parameters are not distinguishable from those that would be expected if the data were generated by chance. We argue that fixed-margin sampling cannot be used for this purpose, as it generates data under a particular null-hypothesis: a unidimensional factor model with interchangeable indicators (i.e., the Rasch model). We show this by discussing relevant psychometric literature and by performing simulation studies. Results indicate that while eLasso correctly estimated network models and estimated almost no edges due to chance, fixed-margin sampling performed poorly in classifying true effects as "interesting" (Steinley et al. 2017, p. 1004). Further simulation studies indicate that fixed-margin sampling offers a powerful method for highlighting local misfit from the Rasch model, but performs only moderately in identifying global departures from the Rasch model. We conclude that fixed-margin sampling is not up to the task of assessing if results from estimated Ising models or other multivariate psychometric models are due to chance.

Robust data and power in infant research: A case study of the effect of number of infants and number of trials in visual preference procedures.

As in many areas of science, infant research suffers from low power. The problem is further compounded in infant research because of the difficulty in recruiting and testing large numbers of infant participants. Researchers have been searching for a solution and, as illustrated by this special section, have been focused on getting the most out of infant data. We illustrate one solution by showing how we can increase power in visual preference tasks by increasing the amount of data obtained from each infant. We discuss issues of power and present work examining how, under some circumstances, power is increased by increasing the precision of measurement. We report the results of a series of simulations based on a sample of visual preference task data collected from three infant laboratories showing how more powerful research designs can be achieved by including more trials per infant. Implications for infant procedures in general are discussed.

Two-stage maximum likelihood approach for item-level missing data in regression.

Psychologists use scales comprised of multiple items to measure underlying constructs. Missing data on such scales often occur at the item level, whereas the model of interest to the researcher is at the composite (scale score) level. Existing analytic approaches cannot easily accommodate item-level missing data when models involve composites. A very common practice in psychology is to average all available items to produce scale scores. This approach, referred to as available-case maximum likelihood (ACML), may produce biased parameter estimates. Another approach researchers use to deal with item-level missing data is scale-level full information maximum likelihood (SL-FIML), which treats the whole scale as missing if any item is missing. SL-FIML is inefficient and it may also exhibit bias. Multiple imputation (MI) produces the correct results using a simulation-based approach. We study a new analytic alternative for item-level missingness, called two-stage maximum likelihood (TSML; Savalei & Rhemtulla, Journal of Educational and Behavioral Statistics, 42(4), 405-431. 2017). The original work showed the method outperforming ACML and SL-FIML in structural equation models with parcels. The current simulation study examined the performance of ACML, SL-FIML, MI, and TSML in the context of univariate regression. We demonstrated performance issues encountered by ACML and SL-FIML when estimating regression coefficients, under both MCAR and MAR conditions. Aside from convergence issues with small sample sizes and high missingness, TSML performed similarly to MI in all conditions, showing negligible bias, high efficiency, and good coverage. This fast analytic approach is therefore recommended whenever it achieves convergence. R code and a Shiny app to perform TSML are provided.

On Nonregularized Estimation of Psychological Networks.

An important goal for psychological science is developing methods to characterize relationships between variables. Customary approaches use structural equation models to connect latent factors to a number of observed measurements, or test causal hypotheses between observed variables. More recently, regularized partial correlation networks have been proposed as an alternative approach for characterizing relationships among variables through off-diagonal elements in the precision matrix. While the graphical Lasso (glasso) has emerged as the default network estimation method, it was optimized in fields outside of psychology with very different needs, such as high dimensional data where the number of variables (p) exceeds the number of observations (n). In this article, we describe the glasso method in the context of the fields where it was developed, and then we demonstrate that the advantages of regularization diminish in settings where psychological networks are often fitted ( p≪n ). We first show that improved properties of the precision matrix, such as eigenvalue estimation, and predictive accuracy with cross-validation are not always appreciable. We then introduce nonregularized methods based on multiple regression and a nonparametric bootstrap strategy, after which we characterize performance with extensive simulations. Our results demonstrate that the nonregularized methods can be used to reduce the false-positive rate, compared to glasso, and they appear to provide consistent performance across sparsity levels, sample composition (p/n), and partial correlation size. We end by reviewing recent findings in the statistics literature that suggest alternative methods often have superior performance than glasso, as well as suggesting areas for future research in psychology. The nonregularized methods have been implemented in the R package GGMnonreg.

Normal Theory Two-Stage ML Estimator When Data Are Missing at the Item Level.

In many modeling contexts, the variables in the model are linear composites of the raw items measured for each participant; for instance, regression and path analysis models rely on scale scores, and structural equation models often use parcels as indicators of latent constructs. Currently, no analytic estimation method exists to appropriately handle missing data at the item level. Item-level multiple imputation (MI), however, can handle such missing data straightforwardly. In this article, we develop an analytic approach for dealing with item-level missing data-that is, one that obtains a unique set of parameter estimates directly from the incomplete data set and does not require imputations. The proposed approach is a variant of the two-stage maximum likelihood (TSML) methodology, and it is the analytic equivalent of item-level MI. We compare the new TSML approach to three existing alternatives for handling item-level missing data: scale-level full information maximum likelihood, available-case maximum likelihood, and item-level MI. We find that the TSML approach is the best analytic approach, and its performance is similar to item-level MI. We recommend its implementation in popular software and its further study.

The application of a network approach to Health-Related Quality of Life (HRQoL): introducing a new method for assessing HRQoL in healthy adults and cancer patients.

PURPOSE: Health-Related Quality of Life (HRQoL) research has typically adopted either a formative approach, in which HRQoL is the common effect of its observables, or a reflective approach--defining HRQoL as a latent variable that determines observable characteristics of HRQoL. Both approaches, however, do not take into account the complex organization of these characteristics. The objective of this study was to introduce a new approach for analyzing HRQoL data, namely a network model (NM). An NM, as opposed to traditional research strategies, accounts for interactions among observables and offers a complementary analytic approach. METHODS: We applied the NM to samples of Dutch cancer patients (N = 485) and Dutch healthy adults (N = 1742) who completed the 36-item Short Form Health Survey (SF-36). Networks were constructed for both samples separately and for a combined sample with diagnostic status added as an extra variable. We assessed the network structures and compared the structures of the two separate samples on the item and domain levels. The relative importance of individual items in the network structures was determined using centrality analyses. RESULTS: We found that the global structure of the SF-36 is dominant in all networks, supporting the validity of questionnaire's subscales. Furthermore, results suggest that the network structure of both samples was highly similar. Centrality analyses revealed that maintaining a daily routine despite one's physical health predicts HRQoL levels best. CONCLUSIONS: We concluded that the NM provides a fruitful alternative to classical approaches used in the psychometric analysis of HRQoL data.

Search for efficient complete and planned missing data designs for analysis of change.

The design of longitudinal data collection is an essential component of any study of change. A well-designed study will maximize the efficiency of statistical tests and minimize the cost of available resources (e.g., budget). Two families of designs have been used to collect longitudinal data: complete data (CD) and planned missing (PM) designs. This article proposes a systematic and flexible procedure named SEEDMC (SEarch for Efficient Designs using Monte Carlo Simulation) to search for efficient CD and PM designs for growth-curve modeling under budget constraints. This procedure allows researchers to identify efficient designs for multiple effects separately and simultaneously, and designs that are robust to MCAR attrition. SEEDMC is applied to identify efficient designs for key change parameters in linear and quadratic growth models. The identified efficient designs are summarized and the strengths and possible extensions of SEEDMC are discussed.

Using Principal Components as Auxiliary Variables in Missing Data Estimation.

To deal with missing data that arise due to participant nonresponse or attrition, methodologists have recommended an "inclusive" strategy where a large set of auxiliary variables are used to inform the missing data process. In practice, the set of possible auxiliary variables is often too large. We propose using principal components analysis (PCA) to reduce the number of possible auxiliary variables to a manageable number. A series of Monte Carlo simulations compared the performance of the inclusive strategy with eight auxiliary variables (inclusive approach) to the PCA strategy using just one principal component derived from the eight original variables (PCA approach). We examined the influence of four independent variables: magnitude of correlations, rate of missing data, missing data mechanism, and sample size on parameter bias, root mean squared error, and confidence interval coverage. Results indicate that the PCA approach results in unbiased parameter estimates and potentially more accuracy than the inclusive approach. We conclude that using the PCA strategy to reduce the number of auxiliary variables is an effective and practical way to reap the benefits of the inclusive strategy in the presence of many possible auxiliary variables.

The InterModel Vigorish as a Lens for Understanding (and Quantifying) the Value of Item Response Models for Dichotomously Coded Items.

The deployment of statistical models-such as those used in item response theory-necessitates the use of indices that are informative about the degree to which a given model is appropriate for a specific data context. We introduce the InterModel Vigorish (IMV) as an index that can be used to quantify accuracy for models of dichotomous item responses based on the improvement across two sets of predictions (i.e., predictions from two item response models or predictions from a single such model relative to prediction based on the mean). This index has a range of desirable features: It can be used for the comparison of non-nested models and its values are highly portable and generalizable. We use this fact to compare predictive performance across a variety of simulated data contexts and also demonstrate qualitative differences in behavior between the IMV and other common indices (e.g., the AIC and RMSEA). We also illustrate the utility of the IMV in empirical applications with data from 89 dichotomous item response datasets. These empirical applications help illustrate how the IMV can be used in practice and substantiate our claims regarding various aspects of model performance. These findings indicate that the IMV may be a useful indicator in psychometrics, especially as it allows for easy comparison of predictions across a variety of contexts.

A graph theory based similarity metric enables comparison of subpopulation psychometric networks.

Network psychometrics leverages pairwise Markov random fields to depict conditional dependencies among a set of psychological variables as undirected edge-weighted graphs. Researchers often intend to compare such psychometric networks across subpopulations, and recent methodological advances provide invariance tests of differences in subpopulation networks. What remains missing, though, is an analogue to an effect size measure that quantifies differences in psychometric networks. We address this gap by complementing recent advances for investigating whether psychometric networks differ with an intuitive similarity measure quantifying the extent to which networks differ. To this end, we build on graph-theoretic approaches and propose a similarity measure based on the Frobenius norm of differences in psychometric networks' weighted adjacency matrices. To assess this measure's utility for quantifying differences between psychometric networks, we study how it captures differences in subpopulation network models implied by both latent variable models and Gaussian graphical models. We show that a wide array of network differences translates intuitively into the proposed measure, while the same does not hold true for customary correlation-based comparisons. In a simulation study on finite-sample behavior, we show that the proposed measure yields trustworthy results when population networks differ and sample sizes are sufficiently large, but fails to identify exact similarity when population networks are the same. From these results, we derive a strong recommendation to only use the measure as a complement to a significant test for network similarity. We illustrate potential insights from quantifying psychometric network similarities through cross-country comparisons of human values networks. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Gene-by-socioeconomic status interaction on school readiness.

In previous work with a nationally representative sample of over 1,400 monozygotic and dizygotic twins born in the US, Tucker-Drob et al. (Psychological Science, 22, 125-133, 2011) uncovered a gene × environment interaction on scores on the Bayley Short Form test of mental ability (MA) at 2 years of age-higher socioeconomic status (SES) was associated not only with higher MA, but also with larger genetic contributions to individual differences in MA. The current study examined gene × SES interactions in mathematics skill and reading skill at 4 years of age (preschool age) in the same sample of twins, and further examined whether interactions detected at 4 years could be attributed to the persistence of the interaction previously observed at 2 years. For early mathematics skill but not early reading skill, genetic influences were more pronounced at higher levels of SES. This interaction was not accounted for by the interaction observed at 2 years. These findings indicate that SES moderates the etiological influences on certain cognitive functions at multiple stages of child development.

Treating random effects as observed versus latent predictors: The bias-variance tradeoff in small samples.

Random effects in longitudinal multilevel models represent individuals' deviations from population means and are indicators of individual differences. Researchers are often interested in examining how these random effects predict outcome variables that vary across individuals. This can be done via a two-step approach in which empirical Bayes (EB) estimates of the random effects are extracted and then treated as observed predictor variables in follow-up regression analyses. This approach ignores the unreliability of EB estimates, leading to underestimation of regression coefficients. As such, previous studies have recommended a multilevel structural equation modeling (ML-SEM) approach that treats random effects as latent variables. The current study uses simulation and empirical data to show that a bias-variance tradeoff exists when selecting between the two approaches. ML-SEM produces generally unbiased regression coefficient estimates but also larger standard errors, which can lead to lower power than the two-step approach. Implications of the results for model selection and alternative solutions are discussed.

A causal theory of error scores.

In modern test theory, response variables are a function of a common latent variable that represents the measured attribute, and error variables that are unique to the response variables. While considerable thought goes into the interpretation of latent variables in these models (e.g., validity research), the interpretation of error variables is typically left implicit (e.g., describing error variables as residuals). Yet, many psychometric assumptions are essentially assumptions about error and thus being able to reason about psychometric models requires the ability to reason about errors. We propose a causal theory of error as a framework that enables researchers to reason about errors in terms of the data-generating mechanism. In this framework, the error variable reflects myriad causes that are specific to an item and, together with the latent variable, determine the scores on that item. We distinguish two types of item-specific causes: characteristic variables that differ between people (e.g., familiarity with words used in the item), and circumstance variables that vary over occasions in which the item is administered (e.g., a distracting noise). We show that different assumptions about these unique causes (a) imply different psychometric models; (b) have different implications for the chance experiment that makes these models probabilistic models; and (c) have different consequences for item bias, local homogeneity, and reliability coefficient α and the test-retest correlation. The ability to reason about the causes that produce error variance puts researchers in a better position to motivate modeling choices. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

What do participants think of our research practices? An examination of behavioural psychology participants' preferences.

What research practices should be considered acceptable? Historically, scientists have set the standards for what constitutes acceptable research practices. However, there is value in considering non-scientists' perspectives, including research participants'. 1873 participants from MTurk and university subject pools were surveyed after their participation in one of eight minimal-risk studies. We asked participants how they would feel if (mostly) common research practices were applied to their data: p-hacking/cherry-picking results, selective reporting of studies, Hypothesizing After Results are Known (HARKing), committing fraud, conducting direct replications, sharing data, sharing methods, and open access publishing. An overwhelming majority of psychology research participants think questionable research practices (e.g. p-hacking, HARKing) are unacceptable (68.3-81.3%), and were supportive of practices to increase transparency and replicability (71.4-80.1%). A surprising number of participants expressed positive or neutral views toward scientific fraud (18.7%), raising concerns about data quality. We grapple with this concern and interpret our results in light of the limitations of our study. Despite the ambiguity in our results, we argue that there is evidence (from our study and others') that researchers may be violating participants' expectations and should be transparent with participants about how their data will be used.

Ubiquitous bias and false discovery due to model misspecification in analysis of statistical interactions: The role of the outcome's distribution and metric properties.

Studies of interaction effects are of great interest because they identify crucial interplay between predictors in explaining outcomes. Previous work has considered several potential sources of statistical bias and substantive misinterpretation in the study of interactions, but less attention has been devoted to the role of the outcome variable in such research. Here, we consider bias and false discovery associated with estimates of interaction parameters as a function of the distributional and metric properties of the outcome variable. We begin by illustrating that, for a variety of noncontinuously distributed outcomes (i.e., binary and count outcomes), attempts to use the linear model for recovery leads to catastrophic levels of bias and false discovery. Next, focusing on transformations of normally distributed variables (i.e., censoring and noninterval scaling), we show that linear models again produce spurious interaction effects. We provide explanations offering geometric and algebraic intuition as to why interactions are a challenge for these incorrectly specified models. In light of these findings, we make two specific recommendations. First, a careful consideration of the outcome's distributional properties should be a standard component of interaction studies. Second, researchers should approach research focusing on interactions with heightened levels of scrutiny. (PsycInfo Database Record (c) 2022 APA, all rights reserved).