A comparison of random forest-based missing imputation methods for covariates in propensity score analysis.

Propensity score analysis (PSA) is a prominent method to alleviate selection bias in observational studies, but missing data in covariates is prevalent and must be dealt with during propensity score estimation. Through Monte Carlo simulations, this study evaluates the use of imputation methods based on multiple random forests algorithms to handle missing data in covariates: multivariate imputation by chained equations-random forest (Caliber), proximity imputation (PI), and missForest. The results indicated that PI and missForest outperformed other methods with respect to bias of average treatment effect regardless of sample size and missing mechanisms. A demonstration of these five methods with PSA to evaluate the effect of participation in center-based care on children's reading ability is provided using data from the Early Childhood Longitudinal Study, Kindergarten Class of 2010-2011. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Unreliable Continuous Treatment Indicators in Propensity Score Analysis.

Propensity score analyses (PSA) of continuous treatments often operationalize the treatment as a multi-indicator composite, and its composite reliability is unreported. Latent variables or factor scores accounting for this unreliability are seldom used as alternatives to composites. This study examines the effects of the unreliability of indicators of a latent treatment in PSA using the generalized propensity score (GPS). A Monte Carlo simulation study was conducted varying composite reliability, continuous treatment representation, variability of factor loadings, sample size, and number of treatment indicators to assess whether Average Treatment Effect (ATE) estimates differed in their relative bias, Root Mean Squared Error, and coverage rates. Results indicate that low composite reliability leads to underestimation of the ATE of latent continuous treatments, while the number of treatment indicators and variability of factor loadings show little effect on ATE estimates, after controlling for overall composite reliability. The results also show that, in correctly specified GPS models, the effects of low composite reliability can be somewhat ameliorated by using factor scores that were estimated including covariates. An illustrative example is provided using survey data to estimate the effect of teacher adoption of a workbook related to a virtual learning environment in the classroom.

A Comparison of Person-Fit Indices to Detect Social Desirability Bias.

Social desirability bias (SDB) has been a major concern in educational and psychological assessments when measuring latent variables because it has the potential to introduce measurement error and bias in assessments. Person-fit indices can detect bias in the form of misfitted response vectors. The objective of this study was to compare the performance of 14 person-fit indices to identify SDB in simulated responses. The area under the curve (AUC) of receiver operating characteristic (ROC) curve analysis was computed to evaluate the predictive power of these statistics. The findings showed that the agreement statistic (A) outperformed all other person-fit indices, while the disagreement statistic (D), dependability statistic (E), and the number of Guttman errors (G) also demonstrated high AUCs to detect SDB. Recommendations for practitioners to use these fit indices are provided.

Examining gender and the longitudinal effect of weight conscious drinking dimensions on body mass index among a college freshman cohort.

Objective: This study aims to: (1) examine gender differences for weight conscious drinking among college students accounting for the broader phenomenon (e.g. including the Alcohol Effects dimension); and (2) longitudinally examine the effect of weight conscious drinking behaviors on body mass index (BMI). Participants: United States freshmen students from eight participating universities (N= 1,149). Methods: Structural equation modeling was used to model the effect of gender on weight conscious drinking dimensions at 7-month follow-up. Results: Findings suggest a significant effect of gender on Alcohol Effects (ß = -.15, SE = .05, p = .005) at 7-month follow-up among college freshmen. Weight conscious drinking dimensions predicted no significant change in BMI at 7-month follow-up among college freshmen. Conclusion: Findings contribute to weight conscious drinking theory and provide campus weight conscious drinking prevention initiatives with evidence to tailor their programming to address female tendencies to engage in compensatory strategies to enhance the psychoactive effects of alcohol.

Assessing Ability Recovery of the Sequential IRT Model With Unstructured Multiple-Attempt Data.

The unstructured multiple-attempt (MA) item response data in virtual learning environments (VLEs) are often from student-selected assessment data sets, which include missing data, single-attempt responses, multiple-attempt responses, and unknown growth ability across attempts, leading to a complex and complicated scenario for using this kind of data set as a whole in the practice of educational measurement. It is critical that methods be available for measuring ability from VLE data to improve VLE systems, monitor student progress in instructional settings, and conduct educational research. The purpose of this study is to explore the ability recovery of the multidimensional sequential 2-PL IRT model in unstructured MA data from VLEs. We conduct a simulation study to evaluate the effects of the magnitude of ability growth and the proportion of students who make two attempts, as well as the moderated effects of sample size, test length, and missingness, on the bias and root mean square error of ability estimates. Results show that the model poses promise for evaluating ability in unstructured VLE data, but that some data conditions can result in biased ability estimates.

Neural Networks to Estimate Generalized Propensity Scores for Continuous Treatment Doses.

BACKGROUND: The generalized propensity score (GPS) addresses selection bias due to observed confounding variables and provides a means to demonstrate causality of continuous treatment doses with propensity score analyses. Estimating the GPS with parametric models obliges researchers to meet improbable conditions such as correct model specification, normal distribution of variables, and large sample sizes. OBJECTIVES: The purpose of this Monte Carlo simulation study is to examine the performance of neural networks as compared to full factorial regression models to estimate GPS in the presence of Gaussian and skewed treatment doses and small to moderate sample sizes. RESEARCH DESIGN: A detailed conceptual introduction of neural networks is provided, as well as an illustration of selection of hyperparameters to estimate GPS. An example from public health and nutrition literature uses residential distance as a treatment variable to illustrate how neural networks can be used in a propensity score analysis to estimate a dose-response function of grocery spending behaviors. RESULTS: We found substantially higher correlations and lower mean squared error values after comparing true GPS with the scores estimated by neural networks. The implication is that more selection bias was removed using GPS estimated with neural networks than using GPS estimated with classical regression. CONCLUSIONS: This study proposes a new methodological procedure, neural networks, to estimate GPS. Neural networks are not sensitive to the assumptions of linear regression and other parametric models and have been shown to be a contender against parametric approaches to estimate propensity scores for continuous treatments.

A Comparison of Metaheuristic Optimization Algorithms for Scale Short-Form Development.

This study compares automated methods to develop short forms of psychometric scales. Obtaining a short form that has both adequate internal structure and strong validity with respect to relationships with other variables is difficult with traditional methods of short-form development. Metaheuristic algorithms can select items for short forms while optimizing on several validity criteria, such as adequate model fit, composite reliability, and relationship to external variables. Using a Monte Carlo simulation study, this study compared existing implementations of the ant colony optimization, Tabu search, and genetic algorithm to select short forms of scales, as well as a new implementation of the simulated annealing algorithm. Selection of short forms of scales with unidimensional, multidimensional, and bifactor structure were evaluated, with and without model misspecification and/or an external variable. The results showed that when the confirmatory factor analysis model of the full form of the scale was correctly specified or had only minor misspecification, the four algorithms produced short forms with good psychometric qualities that maintained the desired factor structure of the full scale. Major model misspecification resulted in worse performance for all algorithms, but including an external variable only had minor effects on results. The simulated annealing algorithm showed the best overall performance as well as robustness to model misspecification, while the genetic algorithm produced short forms with worse fit than the other algorithms under conditions with model misspecification.

Examining the effect of weight conscious drinking on binge drinking frequency among college freshmen.

OBJECTIVE: To examine the effect of weight-conscious drinking and compensatory behavior temporality on binge drinking frequency of college freshmen. Participants: Freshmen (n = 1149) from eight US universities, Fall 2015. Methods: Participants completed the Compensatory Eating Behaviors in Response to Alcohol Consumption Scale and Alcohol Use Disorders Identification Test-Consumption. Structural equation modeling was used to model the effect of weight-conscious drinking constructs on binge drinking frequency. Results: Bulimia, Dietary Restraint and Exercise, Restriction, proactive Alcohol Effects, during Alcohol Effects, and proactive Dietary Restraint and Exercise factors significantly predicted binge drinking frequency. Conclusion: Weight-conscious drinking among this cohort of college students comprises temporal factors significantly associated with binge drinking frequency. Relationships between Bulimia, Dietary Restraint and Exercise, and Restriction compensatory behaviors and binge drinking should be considered in interventions to address binge drinking among college students.

An evaluation of the use of covariates to assist in class enumeration in linear growth mixture modeling.

This study examined whether the inclusion of covariates that predict class membership improves class identification in a growth mixture modeling (GMM). We manipulated the degree of class separation, sample size, the magnitude of covariate effect on class membership, the covariance between the intercept and the slope, and fit two models with covariates and an unconditional model. We concluded that correct class identification in GMM requires large sample sizes and class separation, and that unconditional GMM performs better than GMM with covariates if the sample size and class separation are sufficiently large. With small sample sizes, GMM with covariates outperformed unconditional GMM, but the percentage of correct class enumeration was low across different fit criteria.

Assessing Change in Latent Skills Across Time With Longitudinal Cognitive Diagnosis Modeling: An Evaluation of Model Performance.

Cognitive diagnosis models are diagnostic models used to classify respondents into homogenous groups based on multiple categorical latent variables representing the measured cognitive attributes. This study aims to present longitudinal models for cognitive diagnosis modeling, which can be applied to repeated measurements in order to monitor attribute stability of individuals and to account for respondent dependence. Models based on combining latent transition analysis modeling and the DINA and DINO cognitive diagnosis models were developed and then evaluated through a Monte Carlo simulation study. The study results indicate that the proposed models provide adequate convergence and correct classification rates.

Older Adults' Engagement During an Intervention Involving Off-the-Shelf Videogame.

OBJECTIVE: The overall goal of our current study was to examine older adults' experience of Flow (i.e., subjective engagement) during the course of a home-based cognitive training program. MATERIALS AND METHODS: In this study, participants took part in a home-based training program. They were randomized to one of the two training groups. One group played an off-the-shelf videogame (i.e., Crazy Taxi), and the other group played a brain training game (i.e., Insight). Training consisted of 60 training sessions of 1 hour each, which were completed in 3 months (5 hours a week). After each training session, participants completed a Flow questionnaire to measure their engagement with the training. RESULTS: The analysis was performed with a linear growth curve model. The results indicate that on average, there was no change in flow for the Insight group between time points. There was no difference between the initial flow status of the Insight group and the Crazy Taxi group. However, the interaction between group membership and time was statistically significant, indicating that the participants in the Crazy Taxi group increased their scores at each week at a rate that was 0.99 larger than those in the Insight group. CONCLUSION: The analyses revealed that both groups experienced increase in Flow over the period, but only participants in the Crazy Taxi group significantly improved in Flow. This has long-term implications since we would expect participation to go beyond 12 weeks in a real-world scenario.

The Effects of Including Observed Means or Latent Means as Covariates in Multilevel Models for Cluster Randomized Trials.

We investigated methods of including covariates in two-level models for cluster randomized trials to increase power to detect the treatment effect. We compared multilevel models that included either an observed cluster mean or a latent cluster mean as a covariate, as well as the effect of including Level 1 deviation scores in the model. A Monte Carlo simulation study was performed manipulating effect sizes, cluster sizes, number of clusters, intraclass correlation of the outcome, patterns of missing data, and the squared correlations between Level 1 and Level 2 covariates and the outcome. We found no substantial difference between models with observed means or latent means with respect to convergence, Type I error rates, coverage, and bias. However, coverage could fall outside of acceptable limits if a latent mean is included as a covariate when cluster sizes are small. In terms of statistical power, models with observed means performed similarly to models with latent means, but better when cluster sizes were small. A demonstration is provided using data from a study of the Tools for Getting Along intervention.

An Evaluation of Weighting Methods Based on Propensity Scores to Reduce Selection Bias in Multilevel Observational Studies.

Observational studies of multilevel data to estimate treatment effects must consider both the nonrandom treatment assignment mechanism and the clustered structure of the data. We present an approach for implementation of four propensity score (PS) methods with multilevel data involving creation of weights and three types of weight scaling (normalized, cluster-normalized and effective), followed by estimation of multilevel models with the multilevel pseudo-maximum likelihood estimation method. Using a Monte Carlo simulation study, we found that the multilevel model provided unbiased estimates of the Average Treatment Effect on the Treated (ATT) and its standard error across manipulated conditions and combinations of PS model, PS method, and type of weight scaling. Estimates of between-cluster variances of the ATT were biased, but improved as cluster sizes increased. We provide a step-by-step demonstration of how to combine PS methods and multilevel modeling to estimate treatment effects using multilevel data from the Early Childhood Longitudinal Study-Kindergarten Cohort (ECLS-K).

The Consequences of Ignoring Variability in Measurement Occasions Within Data Collection Waves in Latent Growth Models.

In longitudinal data collection, it is common that each wave of collection spans several months. However, researchers using latent growth models commonly ignore variability in data collection occasions within a wave. In this study, we investigated the consequences of ignoring within-wave variability in measurement occasions using a Monte Carlo simulation and an empirical study. The results of the simulation study showed that ignoring heterogeneity resulted in biased estimates for some parameters, especially when heterogeneity was large and assessment dates had a skewed distribution. Models constructed on person-specific time points yielded precise estimates and more adequate model fit. In the empirical study, we demonstrated different time coding strategies with a subsample taken from Early Childhood Longitudinal Study Kindergarten Cohort.

Differential item functioning in quality of life measure between children with and without special health-care needs.

OBJECTIVES: Limited studies consider the effect of differential item functioning (DIF) on health-related quality of life (HRQOL) comparisons between ill and health children. The objective is to assess DIF and compare HRQOL between children with special health-care needs (CSHCN) and children without needs. METHODS: Data were collected from 1195 families of children enrolled in Florida's public insurance programs. HRQOL was measured using physical, emotional, social, and school functioning of the Pediatric Quality of Life Inventory (PedsQL). We identified CSHCN using the CSHCN Screener and assessed DIF related to CSHCN using a multiple group-multiple indicators-multiple causes method (MG-MIMIC). We assessed the impact of DIF by examining expected item/test scores and item/test information function. We tested the discrepancy between underlying HRQOL scores of both groups before and after DIF calibration (allowing parameters of DIF items to be different and DIF-free items to be the same across both groups). RESULTS: Two (25%) and three items (60%) of physical and school functioning, respectively, were identified with nonuniform DIF, and two items (40%) of social functioning were identified with uniform DIF. Expected item/test scores and item/test information function suggest that the impact of DIF is minimal. Before DIF calibration, HRQOL in CSHCN was more impaired than in children without needs (effect sizes -1.04, -0.74, -0.96, and -0.98 for physical, emotional, social, and school functioning, respectively). After DIF calibration, the discrepancy was increased slightly. CONCLUSIONS: Although 30% of items on the PedsQL were identified with DIF related to CSHCN status, the impact of DIF is minimal.

Detecting Social Desirability Bias Using Factor Mixture Models.

Based on the conceptualization that social desirable bias (SDB) is a discrete event resulting from an interaction between a scale's items, the testing situation, and the respondent's latent trait on a social desirability factor, we present a method that makes use of factor mixture models to identify which examinees are most likely to provide biased responses, which items elicit the most socially desirable responses, and which external variables predict SDB. Problems associated with the common use of correlation coefficients based on scales' total scores to diagnose SDB and partial correlations to correct for SDB are discussed. The method is demonstrated with an analysis of SDB in the Attitude toward Interprofessional Service-Learning scale with a sample of students from health-related fields.

Addressing ceiling effects in health status measures: a comparison of techniques applied to measures for people with HIV disease.

OBJECTIVES: To compare different approaches to address ceiling effects when predicting EQ-5D index scores from the 10 subscales of the MOS-HIV Health Survey. STUDY DESIGN: Data were collected from an HIV treatment trial. Statistical methods included ordinary least squares (OLS) regression, the censored least absolute deviations (CLAD) approach, a standard two-part model (TPM), a TPM with a log-transformed EQ-5D index, and a latent class model (LCM). Predictive accuracy was evaluated using percentage of absolute error (R(1)) and squared error (R(2)) predicted by statistical methods. FINDINGS: A TPM with a log-transformed EQ-5D index performed best on R(1); a LCM performed best on R(2). In contrast, the CLAD was worst. Performance of the OLS and a standard TPM were intermediate. Values for R(1) ranged from 0.33 (CLAD) to 0.42 (TPM-L); R(2) ranged from 0.37 (CLAD) to 0.53 (LCM). CONCLUSIONS: The LCM and TPM with a log-transformed dependent variable are superior to other approaches in handling data with ceiling effects.

Item Selection for the Development of Short Forms of Scales Using an Ant Colony Optimization Algorithm.

This article presents the use of an ant colony optimization (ACO) algorithm for the development of short forms of scales. An example 22-item short form is developed for the Diabetes-39 scale, a quality-of-life scale for diabetes patients, using a sample of 265 diabetes patients. A simulation study comparing the performance of the ACO algorithm and traditionally used methods of item selection is also presented. It is shown that the ACO algorithm outperforms the largest factor loadings and maximum test information item selection methods. The results demonstrate the capabilities of using ACO for creating short-form scales.