Enhancing measurement validity in diverse populations: Modern approaches to evaluating differential item functioning.

When developing and evaluating psychometric measures, a key concern is to ensure that they accurately capture individual differences on the intended construct across the entire population of interest. Inaccurate assessments of individual differences can occur when responses to some items reflect not only the intended construct but also construct-irrelevant characteristics, like a person's race or sex. Unaccounted for, this item bias can lead to apparent differences on the scores that do not reflect true differences, invalidating comparisons between people with different backgrounds. Accordingly, empirically identifying which items manifest bias through the evaluation of differential item functioning (DIF) has been a longstanding focus of much psychometric research. The majority of this work has focused on evaluating DIF across two (or a few) groups. Modern conceptualizations of identity, however, emphasize its multi-determined and intersectional nature, with some aspects better represented as dimensional than categorical. Fortunately, many model-based approaches to modelling DIF now exist that allow for simultaneous evaluation of multiple background variables, including both continuous and categorical variables, and potential interactions among background variables. This paper provides a comparative, integrative review of these new approaches to modelling DIF and clarifies both the opportunities and challenges associated with their application in psychometric research.

Bayesian penalty methods for evaluating measurement invariance in moderated nonlinear factor analysis.

Measurement invariance (MI) is one of the main psychometric requirements for analyses that focus on potentially heterogeneous populations. MI allows researchers to compare latent factor scores across persons from different subgroups, whereas if a measure is not invariant across all items and persons then such comparisons may be misleading. If full MI does not hold further testing may identify problematic items showing differential item functioning (DIF). Most methods developed to test DIF focused on simple scenarios often with comparisons across two groups. In practical applications, this is an oversimplification if many grouping variables (e.g., gender, race) or continuous covariates (e.g., age) exist that might influence the measurement properties of items; these variables are often correlated, making traditional tests that consider each variable separately less useful. Here, we propose the application of Bayesian Moderated Nonlinear Factor Analysis to overcome limitations of traditional approaches to detect DIF. We investigate how modern Bayesian shrinkage priors can be used to identify DIF items in situations with many groups and continuous covariates. We compare the performance of lasso-type, spike-and-slab, and global-local shrinkage priors (e.g., horseshoe) to standard normal and small variance priors. Results indicate that spike-and-slab and lasso priors outperform the other priors. Horseshoe priors provide slightly lower power compared to lasso and spike-and-slab priors. Small variance priors result in very low power to detect DIF with sample sizes below 800, and normal priors may produce severely inflated type I error rates. We illustrate the approach with data from the PISA 2018 study. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Nonconvergence, covariance constraints, and class enumeration in growth mixture models.

Growth mixture models (GMMs) are a popular method to identify latent classes of growth trajectories. One shortcoming of GMMs is nonconvergence, which often leads researchers to apply covariance equality constraints to simplify estimation, though this may be a dubious assumption. Alternative model specifications have been proposed to reduce nonconvergence without imposing covariance equality constraints. These methods perform well when the correct number of classes is known, but research has not yet examined their use when the number of classes is unknown. Given the importance of selecting the number of classes, more information about class enumeration performance is crucial to assess the potential utility of these methods. We conducted an extensive simulation to explore class enumeration and classification accuracy of model specifications that are more robust to nonconvergence. Results show that the typical approach of applying covariance equality constraints performs quite poorly. Instead, we recommended covariance pattern GMMs because they (a) had the highest convergence rates, (b) were most likely to identify the correct number of classes, and (c) had the highest classification accuracy in many conditions, even with modest sample sizes. An analysis of empirical posttraumatic stress disorder (PTSD) data is provided to show that the typical four-class solution found in many empirical PTSD studies may be an artifact of the covariance equality constraint method that has permeated this literature. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Modeling individual differences in the timing of change onset and offset.

Individual differences in the timing of developmental processes are often of interest in longitudinal studies, yet common statistical approaches to modeling change cannot directly estimate the timing of when change occurs. The time-to-criterion framework was recently developed to incorporate the timing of a prespecified criterion value; however, this framework has difficulty accommodating contexts where the criterion value differs across people or when the criterion value is not known a priori, such as when the interest is in individual differences in when change starts or stops. This article combines aspects of reparameterized quadratic models and multiphase models to provide information on the timing of change. We first consider the more common situation of modeling decelerating change to an offset point, defined as the point in time at which change ceases. For increasing trajectories, the offset occurs when the criterion attains its maximum ("inverted J-shaped" trajectories). For decreasing trajectories, offset instead occurs at the minimum. Our model allows for individual differences in both the timing of offset and ultimate level of the outcome. The same model, reparameterized slightly, captures accelerating change from a point of onset ("J-shaped" trajectories). We then extend the framework to accommodate "S-shaped" curves where both the onset and offset of change are within the observation window. We provide demonstrations that span neuroscience, educational psychology, developmental psychology, and cognitive science, illustrating the applicability of the modeling framework to a variety of research questions about individual differences in the timing of change. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Informing Harmonization Decisions in Integrative Data Analysis: Exploring the Measurement Multiverse.

Combining datasets in an integrative data analysis (IDA) requires researchers to make a number of decisions about how best to harmonize item responses across datasets. This entails two sets of steps: logical harmonization, which involves combining items which appear similar across datasets, and analytic harmonization, which involves using psychometric models to find and account for cross-study differences in measurement. Embedded in logical and analytic harmonization are many decisions, from deciding whether items can be combined prima facie to how best to find covariate effects on specific items. Researchers may not have specific hypotheses about these decisions, and each individual choice may seem arbitrary, but the cumulative effects of these decisions are unknown. In the current study, we conducted an IDA of the relationship between alcohol use and delinquency using three datasets (total N = 2245). For analytic harmonization, we used moderated nonlinear factor analysis (MNLFA) to generate factor scores for delinquency. We conducted both logical and analytic harmonization 72 times, each time making a different set of decisions. We assessed the cumulative influence of these decisions on MNLFA parameter estimates, factor scores, and estimates of the relationship between delinquency and alcohol use. There were differences across paths in MNLFA parameter estimates, but fewer differences in estimates of factor scores and regression parameters linking delinquency to alcohol use. These results suggest that factor scores may be relatively robust to subtly different decisions in data harmonization, and measurement model parameters are less so.

A critical period for pandemic adaptation: The evolution of depressive symptomatology in a representative sample of adults across a 17-month period during COVID-19.

This 17-month longitudinal study on a representative sample of 4,361 Norwegian adults employs an observational ABAB design across 6 repeated assessments and 3 pandemic waves to systematically investigate the evolution of depressive symptomatology across all modifications of social distancing protocols (SDPs) from their onset to termination. Using Latent Change Score Models to analyze 26,166 observations, the study empirically corroborates that critical fluctuations in depressive symptomatology within and across individuals occur during the first 3 months of the pandemic, after which symptom profiles are predominantly consolidated throughout the pandemic period. Contrary to established belief, female sex, young age, lower education and preexisting psychiatric diagnosis only served as adequate predictors of the initial shocks to symptomatology observed during the onset of the pandemic and did not adequately predict subsequent change observed in symptoms within and across individuals. Population-level analyses demonstrated that symptom levels strongly covaried with the presence and strictness of SDPs and were unrelated to COVID-19 incidence rates. Upon predominant termination of SDPs, population-level symptoms began declining, while large heterogeneity was present across the adult population. Detrimental long-term adversities were revealed by 10% of the adults. These individuals displayed chaotic adaptation to the pandemic and its SDPs, exhibiting substantial increases in clinical levels of symptomatology ensuing partial reopening of society and through the remainder of the pandemic, with these deleterious symptoms projected to remain heightened ahead. Frequency of quarantine exposure was incrementally tied with increases in contemporaneously experienced and long-term depressive adversities, with information obtainment through unmonitored sources further associated with contemporaneous and long-term states of heightened symptomatology. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Anxiety in the adult population from the onset to termination of social distancing protocols during the COVID-19: a 20-month longitudinal study.

The social distancing protocols (SDPs) implemented as a response to the COVID-19 pandemic may seriously influence peoples' mental health. We used a sample of 4361 Norwegian adults recruited online and stratified to be nationally representative to investigate the evolution of anxiety following each modification in national SDPs across a 20-month period from the onset of the pandemic to the reopening of society and discontinuation of SDPs. The mean anxiety level fluctuated throughout the observation period and these fluctuations were related to the stringency of the modified SDPs. Those with a high initial level almost in unison showed a substantial and lasting decrease of anxiety after the first lifting of SDPs. A sub-group of 9% had developed a persistent anxiety state during the first 3 months. Younger age, pre-existing psychiatric diagnosis, and use of unverified information platforms proved to predict marked higher anxiety in the long run. In conclusion, individuals with a high level of anxiety at the outbreak of the pandemic improved when the social distancing protocols were lifted. By contrast, a sizeable subgroup developed lasting clinical levels of anxiety during the first 3 months of the pandemic and is vulnerable to prolonged anxiety beyond the pandemic period.

Effects of Training on Use of Stimulant Diversion Prevention Strategies by Pediatric Primary Care Providers: Results from a Cluster-Randomized Trial.

Pediatric primary care is a promising setting for reducing diversion of stimulant medications for ADHD. We tested if training pediatric primary care providers (PCPs) increased use of diversion prevention strategies with adolescents with ADHD. The study was a cluster-randomized trial in 7 pediatric primary care practices. Participants were pediatric PCPs (N = 76) at participating practices. Practices were randomized to a 1-h training in stimulant diversion prevention or treatment-as-usual. At baseline, 6 months, 12 months, and 18 months, PCPs rated how often they used four categories of strategies: patient/family education, medication management/monitoring, assessment of mental health symptoms/functioning, and assessment of risky behaviors. They completed measures of attitudes, implementation climate, knowledge/skill, and resource constraints. Generalized Estimating Equations estimated differences in outcomes by condition. Mediation analyses tested if changes in knowledge/skill mediated training effects on strategy use. PCPs in the intervention condition reported significantly greater use of patient/family education strategies at all follow-up time points. There were no differences between conditions in medication management, assessment of mental health symptoms/functioning, or assessment of risky behaviors. At 6 months, PCPs in the intervention condition reported more positive attitudes toward diversion prevention, stronger implementation climate, greater knowledge/skill, and less resource constraints. Differences in knowledge/skill persisted at 12 months and 18 months. Brief training in stimulant diversion had substantial and enduring effects on PCPs' self-reported knowledge/skill and use of patient/family education strategies to prevent diversion. Training had modest effects on attitudes, implementation climate, and resource constraints and did not change use of strategies related to medication management and assessment of mental health symptoms/functioning and risky behaviors. Changes in knowledge/skill accounted for 49% of the total effect of training on use of patient/family education strategies. Trial registration This trial is registered on ClinicalTrials.gov (NCT03080259). Posted March 15, 2017.

Reducing Incidence of Nonpositive Definite Covariance Matrices in Mixed Effect Models.

Deciding which random effects to retain is a central decision in mixed effect models. Recent recommendations advise a maximal structure whereby all theoretically relevant random effects are retained. Nonetheless, including many random effects often leads to nonpositive definiteness. A typical remedy is to simplify the random effect structure by removing random effects or associated covariances. However, this practice is known to bias estimates of remaining covariance parameters and compromise fixed effect inferences. Cholesky decompositions frequently are suggested as an alternative and are automatically implemented in some software. Instead of Cholesky decompositions, we describe factor analytic structures as an approach to avoid nonpositive definiteness. This approach is occasionally employed in biosciences like plant breeding, but, ironically, has not been established in behavioral sciences despite the close historical connection with factor analysis in these fields. We discuss how a factor analytic structure facilitates estimation and conduct simulations to compare convergence and performance to simplifying the random effects structure or Cholesky decomposition approaches. Results show a lower rate of nonpositive definiteness with the factor analytic structure than Cholesky decomposition and suggest that factor analytic covariance structure may be useful to combating nonpositive definiteness, especially in models with many random effects.

Adolescents Treated for Attention-Deficit/Hyperactivity Disorder in Pediatric Primary Care: Characterizing Risk for Stimulant Diversion.

OBJECTIVE: To describe the clinical and psychosocial characteristics, and their hypothesized interrelations, as it pertains to risk for stimulant diversion (sharing, selling, or trading) for adolescents in pediatric primary care treatment for attention-deficit/hyperactivity disorder. METHODS: Baseline data for 341 adolescents in a cluster-randomized controlled trial of stimulant diversion prevention in pediatric primary care (NCT_03080259) were used to (1) characterize diversion and newly measured risk factors, (2) examine their associations with age and sex, and (3) test whether associations among risk factors were consistent with model-implied predictions. Data were collected through multi-informant electronic surveys from adolescents and parents. RESULTS: Diversion was rare (1%) in this sample (Mage = 15, SD = 1.5, 74% male participants). Older age was associated with being approached to divert (r = 0.25, p < 0.001) and higher risk on variables pertinent to stimulant treatment, such as treatment disclosure (r = 0.12, p < 0.05), tolerance for stimulant misuse and diversion (r = 0.17, p < 0.05), and peer norms favorable to stimulant misuse and diversion (r values = 0.15-0.34, p < 0.001). Sex differences were minimal. Variables from our conceptual model and specific to stimulants (e.g., perceived likelihood of negative consequences from diversion and schoolmate stimulant misuse/diversion) were related in multivariable regressions to hypothesized immediate precursors of diversion (e.g., diversion intentions). CONCLUSION: Although diversion was rare for these primary care-treated adolescents, risk levels appear to be higher for older adolescents. Prevention may be most effective by capitalizing on current psychosocial strengths and discussing stimulant-specific attitudes, behaviors, and social norms before vulnerability to diversion increases in the final years of high school and into college.

Psychometric Models for Scoring Multiple Reporter Assessments: Applications to Integrative Data Analysis in Prevention Science and Beyond.

Conducting valid and reliable empirical research in the prevention sciences is an inherently difficult and challenging task. Chief among these is the need to obtain numerical scores of underlying theoretical constructs for use in subsequent analysis. This challenge is further exacerbated by the increasingly common need to consider multiple reporter assessments, particularly when using integrative data analysis to fit models to data that have been pooled across two or more independent samples. The current paper uses both simulated and real data to examine the utility of a recently proposed psychometric model for multiple reporter data called the trifactor model (TFM) in settings that might be commonly found in prevention research. Results suggest that numerical scores obtained using the TFM are superior to more traditional methods, particularly when pooling samples that contribute different reporter perspectives.

A Deep Learning Algorithm for High-Dimensional Exploratory Item Factor Analysis.

Marginal maximum likelihood (MML) estimation is the preferred approach to fitting item response theory models in psychometrics due to the MML estimator's consistency, normality, and efficiency as the sample size tends to infinity. However, state-of-the-art MML estimation procedures such as the Metropolis-Hastings Robbins-Monro (MH-RM) algorithm as well as approximate MML estimation procedures such as variational inference (VI) are computationally time-consuming when the sample size and the number of latent factors are very large. In this work, we investigate a deep learning-based VI algorithm for exploratory item factor analysis (IFA) that is computationally fast even in large data sets with many latent factors. The proposed approach applies a deep artificial neural network model called an importance-weighted autoencoder (IWAE) for exploratory IFA. The IWAE approximates the MML estimator using an importance sampling technique wherein increasing the number of importance-weighted (IW) samples drawn during fitting improves the approximation, typically at the cost of decreased computational efficiency. We provide a real data application that recovers results aligning with psychological theory across random starts. Via simulation studies, we show that the IWAE yields more accurate estimates as either the sample size or the number of IW samples increases (although factor correlation and intercepts estimates exhibit some bias) and obtains similar results to MH-RM in less time. Our simulations also suggest that the proposed approach performs similarly to and is potentially faster than constrained joint maximum likelihood estimation, a fast procedure that is consistent when the sample size and the number of items simultaneously tend to infinity.

Harmonizing altered measures in integrative data analysis: A methods analogue study.

In the current study, we used an analogue integrative data analysis (IDA) design to test optimal scoring strategies for harmonizing alcohol- and drug-use consequence measures with varying degrees of alteration across four study conditions. We evaluated performance of mean, confirmatory factor analysis (CFA), and moderated nonlinear factor analysis (MNLFA) scores based on traditional indices of reliability (test-retest, internal, and score recovery or parallel forms) and validity. Participants in the analogue study included 854 college students (46% male; 21% African American, 5% Hispanic/Latino, 56% European American) who completed two versions of the altered measures at two sessions, separated by 2 weeks. As expected, mean, CFA, and MNLFA scores all resulted in scales with lower reliability given increasing scale alteration (with less fidelity to formerly developed scales) and shorter scale length. MNLFA and CFA scores, however, showed greater validity than mean scores, demonstrating stronger relationships with external correlates. Implications for measurement harmonization in the context of IDA are discussed.

Simplifying the Assessment of Measurement Invariance over Multiple Background Variables: Using Regularized Moderated Nonlinear Factor Analysis to Detect Differential Item Functioning.

Determining whether measures are equally valid for all individuals is a core component of psychometric analysis. Traditionally, the evaluation of measurement invariance (MI) involves comparing independent groups defined by a single categorical covariate (e.g., men and women) to determine if there are any items that display differential item functioning (DIF). More recently, Moderated Nonlinear Factor Analysis (MNLFA) has been advanced as an approach for evaluating MI/DIF simultaneously over multiple background variables, categorical and continuous. Unfortunately, conventional procedures for detecting DIF do not scale well to the more complex MNLFA. The current manuscript therefore proposes a regularization approach to MNLFA estimation that penalizes the likelihood for DIF parameters (i.e., rewarding sparse DIF). This procedure avoids the pitfalls of sequential inference tests, is automated for end users, and is shown to perform well in both a small-scale simulation and an empirical validation study.

Improving the assessment of measurement invariance: Using regularization to select anchor items and identify differential item functioning.

A common challenge in the behavioral sciences is evaluating measurement invariance, or whether the measurement properties of a scale are consistent for individuals from different groups. Measurement invariance fails when differential item functioning (DIF) exists, that is, when item responses relate to the latent variable differently across groups. To identify DIF in a scale, many data-driven procedures iteratively test for DIF one item at a time while assuming other items have no DIF. The DIF-free items are used to anchor the scale of the latent variable across groups, identifying the model. A major drawback to these iterative testing procedures is that they can fail to select the correct anchor items and identify true DIF, particularly when DIF is present in many items. We propose an alternative method for selecting anchors and identifying DIF. Namely, we use regularization, a machine learning technique that imposes a penalty function during estimation to remove parameters that have little impact on the fit of the model. We focus specifically here on a lasso penalty for group differences in the item parameters within the two-parameter logistic item response theory model. We compare lasso regularization with the more commonly used likelihood ratio test method in a 2-group DIF analysis. Simulation and empirical results show that when large amounts of DIF are present and sample sizes are large, lasso regularization has far better control of Type I error than the likelihood ratio test method with little decrement in power. This provides strong evidence that lasso regularization is a promising alternative for testing DIF and selecting anchors. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Stimulant Diversion Risk Among College Students Treated for ADHD: Primary Care Provider Prevention Training.

OBJECTIVE: To address increasing rates of stimulant misuse in college students, this study developed an evidence-based, brief clinical practice intervention for primary care providers (PCPs) to reduce stimulant medication diversion among young adults with attention-deficit/hyperactivity disorder (ADHD). METHODS: College students (N-114; 18-25 years; 68% attending universities; 24% attending community college) treated for ADHD with a stimulant and their PCPs across six practices participated in this initial, uncontrolled study of pre- to post-intervention change. An educational workshop providing strategies aimed at reducing stimulant diversion was developed and delivered to providers and staff across all practices (50% pediatric; 50% family medicine). Patients and providers completed baseline and post intervention surveys. RESULTS: Diversion was relatively infrequent, 16.7% at baseline and 14.9% post-intervention, respectively. Statistically significant decreases from baseline to post-intervention were found for three diversion risk factors: (1) number of times approached to divert, (2) intent to share, sell, or trade stimulants, and (3) disclosure of stimulant use. Providers and staff reported mostly high satisfaction with the training. CONCLUSIONS: This study provides initial evidence for a PCP-delivered intervention to reduce stimulant diversion. Research is needed on the efficacy of targeting college students directly, working with pharmacies and student health centers, and preventing misuse among teenagers.

Assessing the Robustness of Mixture Models to Measurement Noninvariance.

Recent work reframes direct effects of covariates on items in mixture models as differential item functioning (DIF) and shows that, when present in the data but omitted from the fitted latent class model, DIF can lead to overextraction of classes. However, less is known about the effects of DIF on model performance-including parameter bias, classification accuracy, and distortion of class-specific response profiles-once the correct number of classes is chosen. First, we replicate and extend prior findings relating DIF to class enumeration using a comprehensive simulation study. In a second simulation study using the same parameters, we show that, while the performance of LCA is robust to the misspecification of DIF effects, it is degraded when DIF is omitted entirely. Moreover, the robustness of LCA to omitted DIF differs widely based on the degree of class separation. Finally, simulation results are contextualized by an empirical example.

Distress tolerance trajectories following substance use treatment.

OBJECTIVE: Distress tolerance (DT), the ability to withstand aversive internal states, represents an important risk factor for substance use relapse and a potential treatment target. Neurobiological research in substance using populations suggests that continued substance use could erode DT, whereas abstinence could bolster it. The current study characterized trajectories of behavioral and self-reported indices of DT and examined the prospective effect of substance use on DT trajectories among those seeking treatment for substance use. METHOD: Individuals (N = 263, Mage = 42.68, SD = 11.8, 70.7% male, 94.7% African American) in residential substance use treatment completed subjective (Distress Tolerance Scale) and behavioral (Mirror Tracing Persistence Task-computerized version) DT measures, as well as report of daily substance use (timeline follow-back) over 5 assessment time-points from pretreatment to 12 months posttreatment. Latent curve modeling estimated DT trajectories and their associations with substance use behavior, including abstinence duration (days until first use) and substance use frequency (percentage of substance use days between assessments). RESULTS: Self-reported and behavioral DT indicators both exhibited positive, nonlinear change over time (standardized slope parameter estimates: Distress Tolerance Scale ß = 0.61, p < .01; Mirror Tracing Persistence Task ß = 0.34, p < .01). Abstinence duration was associated with greater improvement in behavioral (ß = .20, p = .03) DT specifically. Frequency of use was statistically significantly associated with attenuated behavioral DT at 6-month (ß = -.12, p = .03) and 12-month follow-ups (ß = -.08, p = .045). CONCLUSIONS: DT appears to improve appreciably posttreatment, and return to substance use may shape the degree of this improvement. Collectively, these findings support the conceptualization of DT as a malleable treatment target and emphasize the benefit of abstinence on improvement in DT. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Simplifying the implementation of modern scale scoring methods with an automated R package: Automated moderated nonlinear factor analysis (aMNLFA).

When generating scores to represent latent constructs, analysts have a choice between applying psychometric approaches that are principled but that can be complicated and time-intensive versus applying simple and fast, but less precise approaches, such as sum or mean scoring. We explain the reasons for preferring modern psychometric approaches: namely, use of unequal item weights and severity parameters, the ability to account for local dependence and differential item functioning, and the use of covariate information to more efficiently estimate factor scores. We describe moderated nonlinear factor analysis (MNLFA), a relatively new, highly flexible approach that allows analysts to develop precise factor score estimates that address limitations of sum score, mean score, and traditional factor analytic approaches to scoring. We then outline the steps involved in using the MNLFA scoring approach and discuss the circumstances in which this approach is preferred. To overcome the difficulty of implementing MNLFA models in practice, we developed an R package, aMNLFA, that automates much of the rule-based scoring process. We illustrate the use of aMNLFA with an empirical example of scoring alcohol involvement in a longitudinal study of 6998 adolescents and compare performance of MNLFA scores with traditional factor analysis and sum scores based on the same set of 12 items. MNLFA scores retain more meaningful variation than other approaches. We conclude with practical guidelines for scoring.