Informing Harmonization Decisions in Integrative Data Analysis: Exploring the Measurement Multiverse.

Combining datasets in an integrative data analysis (IDA) requires researchers to make a number of decisions about how best to harmonize item responses across datasets. This entails two sets of steps: logical harmonization, which involves combining items which appear similar across datasets, and analytic harmonization, which involves using psychometric models to find and account for cross-study differences in measurement. Embedded in logical and analytic harmonization are many decisions, from deciding whether items can be combined prima facie to how best to find covariate effects on specific items. Researchers may not have specific hypotheses about these decisions, and each individual choice may seem arbitrary, but the cumulative effects of these decisions are unknown. In the current study, we conducted an IDA of the relationship between alcohol use and delinquency using three datasets (total N = 2245). For analytic harmonization, we used moderated nonlinear factor analysis (MNLFA) to generate factor scores for delinquency. We conducted both logical and analytic harmonization 72 times, each time making a different set of decisions. We assessed the cumulative influence of these decisions on MNLFA parameter estimates, factor scores, and estimates of the relationship between delinquency and alcohol use. There were differences across paths in MNLFA parameter estimates, but fewer differences in estimates of factor scores and regression parameters linking delinquency to alcohol use. These results suggest that factor scores may be relatively robust to subtly different decisions in data harmonization, and measurement model parameters are less so.

Effects of Training on Use of Stimulant Diversion Prevention Strategies by Pediatric Primary Care Providers: Results from a Cluster-Randomized Trial.

Pediatric primary care is a promising setting for reducing diversion of stimulant medications for ADHD. We tested if training pediatric primary care providers (PCPs) increased use of diversion prevention strategies with adolescents with ADHD. The study was a cluster-randomized trial in 7 pediatric primary care practices. Participants were pediatric PCPs (N = 76) at participating practices. Practices were randomized to a 1-h training in stimulant diversion prevention or treatment-as-usual. At baseline, 6 months, 12 months, and 18 months, PCPs rated how often they used four categories of strategies: patient/family education, medication management/monitoring, assessment of mental health symptoms/functioning, and assessment of risky behaviors. They completed measures of attitudes, implementation climate, knowledge/skill, and resource constraints. Generalized Estimating Equations estimated differences in outcomes by condition. Mediation analyses tested if changes in knowledge/skill mediated training effects on strategy use. PCPs in the intervention condition reported significantly greater use of patient/family education strategies at all follow-up time points. There were no differences between conditions in medication management, assessment of mental health symptoms/functioning, or assessment of risky behaviors. At 6 months, PCPs in the intervention condition reported more positive attitudes toward diversion prevention, stronger implementation climate, greater knowledge/skill, and less resource constraints. Differences in knowledge/skill persisted at 12 months and 18 months. Brief training in stimulant diversion had substantial and enduring effects on PCPs' self-reported knowledge/skill and use of patient/family education strategies to prevent diversion. Training had modest effects on attitudes, implementation climate, and resource constraints and did not change use of strategies related to medication management and assessment of mental health symptoms/functioning and risky behaviors. Changes in knowledge/skill accounted for 49% of the total effect of training on use of patient/family education strategies. Trial registration This trial is registered on ClinicalTrials.gov (NCT03080259). Posted March 15, 2017.

Reducing Incidence of Nonpositive Definite Covariance Matrices in Mixed Effect Models.

Deciding which random effects to retain is a central decision in mixed effect models. Recent recommendations advise a maximal structure whereby all theoretically relevant random effects are retained. Nonetheless, including many random effects often leads to nonpositive definiteness. A typical remedy is to simplify the random effect structure by removing random effects or associated covariances. However, this practice is known to bias estimates of remaining covariance parameters and compromise fixed effect inferences. Cholesky decompositions frequently are suggested as an alternative and are automatically implemented in some software. Instead of Cholesky decompositions, we describe factor analytic structures as an approach to avoid nonpositive definiteness. This approach is occasionally employed in biosciences like plant breeding, but, ironically, has not been established in behavioral sciences despite the close historical connection with factor analysis in these fields. We discuss how a factor analytic structure facilitates estimation and conduct simulations to compare convergence and performance to simplifying the random effects structure or Cholesky decomposition approaches. Results show a lower rate of nonpositive definiteness with the factor analytic structure than Cholesky decomposition and suggest that factor analytic covariance structure may be useful to combating nonpositive definiteness, especially in models with many random effects.

Harmonizing altered measures in integrative data analysis: A methods analogue study.

In the current study, we used an analogue integrative data analysis (IDA) design to test optimal scoring strategies for harmonizing alcohol- and drug-use consequence measures with varying degrees of alteration across four study conditions. We evaluated performance of mean, confirmatory factor analysis (CFA), and moderated nonlinear factor analysis (MNLFA) scores based on traditional indices of reliability (test-retest, internal, and score recovery or parallel forms) and validity. Participants in the analogue study included 854 college students (46% male; 21% African American, 5% Hispanic/Latino, 56% European American) who completed two versions of the altered measures at two sessions, separated by 2 weeks. As expected, mean, CFA, and MNLFA scores all resulted in scales with lower reliability given increasing scale alteration (with less fidelity to formerly developed scales) and shorter scale length. MNLFA and CFA scores, however, showed greater validity than mean scores, demonstrating stronger relationships with external correlates. Implications for measurement harmonization in the context of IDA are discussed.

Assessing the Robustness of Mixture Models to Measurement Noninvariance.

Recent work reframes direct effects of covariates on items in mixture models as differential item functioning (DIF) and shows that, when present in the data but omitted from the fitted latent class model, DIF can lead to overextraction of classes. However, less is known about the effects of DIF on model performance-including parameter bias, classification accuracy, and distortion of class-specific response profiles-once the correct number of classes is chosen. First, we replicate and extend prior findings relating DIF to class enumeration using a comprehensive simulation study. In a second simulation study using the same parameters, we show that, while the performance of LCA is robust to the misspecification of DIF effects, it is degraded when DIF is omitted entirely. Moreover, the robustness of LCA to omitted DIF differs widely based on the degree of class separation. Finally, simulation results are contextualized by an empirical example.

Suicide ideation among high-risk adolescent females: Examining the interplay between parasympathetic regulation and friendship support.

Poor physiological self-regulation has been proposed as a potential biological vulnerability for adolescent suicidality. This study tested this hypothesis by examining the effect of parasympathetic stress responses on future suicide ideation. In addition, drawing from multilevel developmental psychopathology theories, the interplay between parasympathetic regulation and friendship support, conceptualized as an external source of regulation, was examined. At baseline, 132 adolescent females (M age = 14.59, SD = 1.39) with a history of mental health concerns participated in an in vivo interpersonal stressor (a laboratory speech task) and completed self-report measures of depressive symptoms and perceived support within a close same-age female friendship. Respiratory sinus arrhythmia (RSA) was measured before and during the speech task. Suicide ideation was assessed at baseline and at 3, 6, and 9 months follow-up. The results revealed that females with greater relative RSA decreases to the laboratory stressor were at higher risk for reporting suicide ideation over the subsequent 9 months. Moreover, parasympathetic responses moderated the effect of friendship support on suicide ideation; among females with mild changes or higher relative increases in RSA, but not more pronounced RSA decreases, friendship support reduced risk for future suicide ideation. Findings highlight the crucial role of physiological and external regulation sources as protective factors for youth suicidality.

Friendship Dissolution Within Social Networks Modeled Through Multilevel Event History Analysis.

A social network perspective can bring important insight into the processes that shape human behavior. Longitudinal social network data, measuring relations between individuals over time, has become increasingly common-as have the methods available to analyze such data. A friendship duration model utilizing discrete-time multilevel survival analysis with a multiple membership random effect structure is developed and applied here to study the processes leading to undirected friendship dissolution within a larger social network. While the modeling framework is introduced in terms of understanding friendship dissolution, it can be used to understand microlevel dynamics of a social network more generally. These models can be fit with standard generalized linear mixed-model software, after transforming the data to a pair-period data set. An empirical example highlights how the model can be applied to understand the processes leading to friendship dissolution between high school students, and a simulation study is used to test the use of the modeling framework under representative conditions that would be found in social network data. Advantages of the modeling framework are highlighted, and potential limitations and future directions are discussed.

Addiction History Associates with the Propensity to Form Habits.

Learned habitual responses to environmental stimuli allow efficient interaction with the environment, freeing cognitive resources for more demanding tasks. However, when the outcome of such actions is no longer a desired goal, established stimulus-response (S-R) associations or habits must be overcome. Among people with substance use disorders (SUDs), difficulty in overcoming habitual responses to stimuli associated with their addiction in favor of new, goal-directed behaviors contributes to relapse. Animal models of habit learning demonstrate that chronic self-administration of drugs of abuse promotes habitual responding beyond the domain of compulsive drug seeking. However, whether a similar propensity toward domain-general habitual responding occurs in humans with SUDs has remained unclear. To address this question, we used a visuomotor S-R learning and relearning task, the Hidden Association between Images Task, which employs abstract visual stimuli and manual responses. This task allows us to measure new S-R association learning and well-learned S-R association execution and includes a response contingency change manipulation to quantify the degree to which responding is habit-based, rather than goal-directed. We find that people with SUDs learn new S-R associations as well as healthy control participants do. Moreover, people with an SUD history slightly outperform controls in S-R execution. In contrast, people with SUDs are specifically impaired in overcoming well-learned S-R associations; those with SUDs make a significantly greater proportion of perseverative errors during well-learned S-R replacement, indicating the more habitual nature of their responses. Thus, with equivalent training and practice, people with SUDs appear to show enhanced domain-general habit formation.

Proximal and time-varying effects of cigarette, alcohol, marijuana and other hard drug use on adolescent dating aggression.

Although numerous studies have established a link between substance use and adult partner violence, little research has examined the relationship during adolescence and most extant research has not examined multiple substance use types. The current study used hierarchical growth modeling to simultaneously examine proximal (between-person) and time-varying (within-person) relations between cigarette, alcohol, marijuana and hard drug use and physical dating aggression across grades 8 through 12 while controlling for demographic covariates and shared risk factors. Proximal effects of marijuana use on dating aggression were found for girls and proximal effects of hard drug use on dating aggression were found for boys. Time-varying effects were found for alcohol for both boys and girls and for hard drug use for boys only. Overall, findings suggest that alcohol, marijuana and hard drug use predict whether and when adolescents engage in dating aggression and should be targeted by prevention interventions.

A Moderated Nonlinear Factor Model for the Development of Commensurate Measures in Integrative Data Analysis.

Integrative data analysis (IDA) is a methodological framework that allows for the fitting of models to data that have been pooled across two or more independent sources. IDA offers many potential advantages including increased statistical power, greater subject heterogeneity, higher observed frequencies of low base-rate behaviors, and longer developmental periods of study. However, a core challenge is the estimation of valid and reliable psychometric scores that are based on potentially different items with different response options drawn from different studies. In Bauer and Hussong (2009) we proposed a method for obtaining scores within an IDA called moderated nonlinear factor analysis (MNLFA). Here we move significantly beyond this work in the development of a general framework for estimating MNLFA models and obtaining scale scores across a variety of settings. We propose a five step procedure and demonstrate this approach using data drawn from n=1972 individuals ranging in age from 11 to 34 years pooled across three independent studies to examine the factor structure of 17 binary items assessing depressive symptomatology. We offer substantive conclusions about the factor structure of depression, use this structure to compute individual-specific scale scores, and make recommendations for the use of these methods in practice.

How should we model the effect of "change"-Or should we?

There have been long and bitter debates between those who advocate for the use of residualized change as the foundation of longitudinal models versus those who utilize difference scores. However, these debates have focused primarily on modeling change in the outcome variable. Here, we extend these same ideas to the covariate side of the change equation, finding similar issues arise when using lagged versus difference scores as covariates of interest in models of change. We derive a system of relationships that emerge across models differing in how time-varying covariates are represented, and then demonstrate how the set of logical transformations emerges in applied longitudinal settings. We conclude by considering the practical implications of a synthesized understanding of the effects of difference scores as both outcomes and predictors, with specific consequences for mediation analysis within multivariate longitudinal models. Our results suggest that there is reason for caution when using difference scores as time-varying covariates, given their propensity for inducing apparent inferential inversions within different analyses. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Modeling construct change over time amidst potential changes in construct measurement: A longitudinal moderated factor analysis approach.

In analyzing longitudinal data with growth curve models, a critical assumption is that changes in the observed measures reflect construct changes and not changes in the manifestation of the construct over time. However, growth curve models are often fit to a repeated measure constructed as a sum or mean of scale items, making an implicit assumption of constancy of measurement. This practice risks confounding actual construct change with changes in measurement (i.e., differential item functioning [DIF]), threatening the validity of conclusions. An improved method that avoids such confounding is the second-order growth curve (SGC) model. It specifies a measurement model at each occasion of measurement that can be evaluated for invariance over time. The applicability of the SGC model is hindered by key limitations: (a) the SGC model treats time as continuous when modeling construct growth but as discrete when modeling measurement, reducing interpretability and parsimony; (b) the evaluation of DIF becomes increasingly error-prone given multiple timepoints and groups; (c) DIF associated with continuous covariates is difficult to incorporate. Drawing on moderated nonlinear factor analysis, we propose an alternative approach that provides a parsimonious framework for including many time points and DIF from different types of covariates. We implement this model through Bayesian estimation, allowing for incorporation of regularizing priors to facilitate efficient evaluation of DIF. We demonstrate a two-step workflow of measurement evaluation and growth modeling, with an empirical example examining changes in adolescent delinquency over time. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Interpersonal dynamics of vocal fundamental frequency in couples: Depressive symptoms, anxiety symptoms, and relationship distress.

Given the bidirectional association between psychopathology and relationship distress, an in-depth understanding of couples' interaction processes that contribute to psychopathology is needed. This study examined the interpersonal dynamics of vocally-encoded emotional arousal (fundamental frequency, f0) during couple conversations and their associations with depressive symptoms, anxiety symptoms, and relationship distress. Data from eight samples were pooled (N = 404 couples) to examine (a) overall trajectories of f0 across the interaction and (b) moment-by-moment intraindividual changes in and interpersonal reactivity to partners' f0. Multilevel growth models and repeated-measures actor-partner interdependence models demonstrated that individuals with more severe depression showed more synchronizing reactivity to their partners' f0 on a moment-by-moment basis, and their overall baseline level of f0 was lower. More severe relationship distress was associated with more steeply increasing trajectories of f0 and with greater synchronizing reactivity to partners' f0. Relative differences in depressive symptoms between the two members of a couple were associated with interpersonal dynamics of f0 as well. There were no associations with anxiety symptoms. Thus, depressive symptoms were associated with characteristic interpersonal dynamics of vocally-encoded emotional arousal; yet, most consistent associations emerged for relationship distress, which future studies on individual psychopathology should take into account.

Integrative data analysis in clinical psychology research.

Integrative data analysis (IDA), a novel framework for conducting the simultaneous analysis of raw data pooled from multiple studies, offers many advantages including economy (i.e., reuse of extant data), power (i.e., large combined sample sizes), the potential to address new questions not answerable by a single contributing study (e.g., combining longitudinal studies to cover a broader swath of the lifespan), and the opportunity to build a more cumulative science (i.e., examining the similarity of effects across studies and potential reasons for dissimilarities). There are also methodological challenges associated with IDA, including the need to account for sampling heterogeneity across studies, to develop commensurate measures across studies, and to account for multiple sources of study differences as they impact hypothesis testing. In this review, we outline potential solutions to these challenges and describe future avenues for developing IDA as a framework for studies in clinical psychology.

Psychometric properties of the mini-IPIP in a large, nationally representative sample of young adults.

Drawing on a large, nationally representative sample of young adults (the National Longitudinal Study of Adolescent Health; N = 15,701; M age = 29.10), we evaluated the psychometric properties of the Mini-IPIP, a 20-item inventory designed to concisely assess the 5 factors of personality. The results suggest that the Mini-IPIP has a 5-factor structure; most of the scales have acceptable reliability; all the scales have partial or full metric invariance; and the scales exhibit some degree of criterion validity. However, the absence of scalar invariance for many of the scales suggests caution when comparing personality scores among groups defined by sex or race and ethnicity. We offer practical considerations for researchers interested in using this inventory with this sample, and also suggestions for modification of the Mini-IPIP.

Enhancing measurement validity in diverse populations: Modern approaches to evaluating differential item functioning.

When developing and evaluating psychometric measures, a key concern is to ensure that they accurately capture individual differences on the intended construct across the entire population of interest. Inaccurate assessments of individual differences can occur when responses to some items reflect not only the intended construct but also construct-irrelevant characteristics, like a person's race or sex. Unaccounted for, this item bias can lead to apparent differences on the scores that do not reflect true differences, invalidating comparisons between people with different backgrounds. Accordingly, empirically identifying which items manifest bias through the evaluation of differential item functioning (DIF) has been a longstanding focus of much psychometric research. The majority of this work has focused on evaluating DIF across two (or a few) groups. Modern conceptualizations of identity, however, emphasize its multi-determined and intersectional nature, with some aspects better represented as dimensional than categorical. Fortunately, many model-based approaches to modelling DIF now exist that allow for simultaneous evaluation of multiple background variables, including both continuous and categorical variables, and potential interactions among background variables. This paper provides a comparative, integrative review of these new approaches to modelling DIF and clarifies both the opportunities and challenges associated with their application in psychometric research.

Nonconvergence, covariance constraints, and class enumeration in growth mixture models.

Growth mixture models (GMMs) are a popular method to identify latent classes of growth trajectories. One shortcoming of GMMs is nonconvergence, which often leads researchers to apply covariance equality constraints to simplify estimation, though this may be a dubious assumption. Alternative model specifications have been proposed to reduce nonconvergence without imposing covariance equality constraints. These methods perform well when the correct number of classes is known, but research has not yet examined their use when the number of classes is unknown. Given the importance of selecting the number of classes, more information about class enumeration performance is crucial to assess the potential utility of these methods. We conducted an extensive simulation to explore class enumeration and classification accuracy of model specifications that are more robust to nonconvergence. Results show that the typical approach of applying covariance equality constraints performs quite poorly. Instead, we recommended covariance pattern GMMs because they (a) had the highest convergence rates, (b) were most likely to identify the correct number of classes, and (c) had the highest classification accuracy in many conditions, even with modest sample sizes. An analysis of empirical posttraumatic stress disorder (PTSD) data is provided to show that the typical four-class solution found in many empirical PTSD studies may be an artifact of the covariance equality constraint method that has permeated this literature. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Bayesian penalty methods for evaluating measurement invariance in moderated nonlinear factor analysis.

Measurement invariance (MI) is one of the main psychometric requirements for analyses that focus on potentially heterogeneous populations. MI allows researchers to compare latent factor scores across persons from different subgroups, whereas if a measure is not invariant across all items and persons then such comparisons may be misleading. If full MI does not hold further testing may identify problematic items showing differential item functioning (DIF). Most methods developed to test DIF focused on simple scenarios often with comparisons across two groups. In practical applications, this is an oversimplification if many grouping variables (e.g., gender, race) or continuous covariates (e.g., age) exist that might influence the measurement properties of items; these variables are often correlated, making traditional tests that consider each variable separately less useful. Here, we propose the application of Bayesian Moderated Nonlinear Factor Analysis to overcome limitations of traditional approaches to detect DIF. We investigate how modern Bayesian shrinkage priors can be used to identify DIF items in situations with many groups and continuous covariates. We compare the performance of lasso-type, spike-and-slab, and global-local shrinkage priors (e.g., horseshoe) to standard normal and small variance priors. Results indicate that spike-and-slab and lasso priors outperform the other priors. Horseshoe priors provide slightly lower power compared to lasso and spike-and-slab priors. Small variance priors result in very low power to detect DIF with sample sizes below 800, and normal priors may produce severely inflated type I error rates. We illustrate the approach with data from the PISA 2018 study. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Modeling individual differences in the timing of change onset and offset.

Individual differences in the timing of developmental processes are often of interest in longitudinal studies, yet common statistical approaches to modeling change cannot directly estimate the timing of when change occurs. The time-to-criterion framework was recently developed to incorporate the timing of a prespecified criterion value; however, this framework has difficulty accommodating contexts where the criterion value differs across people or when the criterion value is not known a priori, such as when the interest is in individual differences in when change starts or stops. This article combines aspects of reparameterized quadratic models and multiphase models to provide information on the timing of change. We first consider the more common situation of modeling decelerating change to an offset point, defined as the point in time at which change ceases. For increasing trajectories, the offset occurs when the criterion attains its maximum ("inverted J-shaped" trajectories). For decreasing trajectories, offset instead occurs at the minimum. Our model allows for individual differences in both the timing of offset and ultimate level of the outcome. The same model, reparameterized slightly, captures accelerating change from a point of onset ("J-shaped" trajectories). We then extend the framework to accommodate "S-shaped" curves where both the onset and offset of change are within the observation window. We provide demonstrations that span neuroscience, educational psychology, developmental psychology, and cognitive science, illustrating the applicability of the modeling framework to a variety of research questions about individual differences in the timing of change. (PsycInfo Database Record (c) 2023 APA, all rights reserved).