Evaluating SEM Model Fit with Small Degrees of Freedom.

Research has revealed that the performance of root mean square error of approximation (RMSEA) in assessing structural equation models with small degrees of freedom (df) is suboptimal, often resulting in the rejection of correctly specified or closely fitted models. This study investigates the performance of standardized root mean square residual (SRMR) and comparative fit index (CFI) in small df models with various levels of factor loadings, sample sizes, and model misspecifications. We find that, in comparison with RMSEA, population SRMR and CFI are less susceptible to the effects of df. In small df models, the sample SRMR and CFI could provide more useful information to differentiate models with various levels of misfit. The confidence intervals and p-values of a close fit were generally accurate for all three fit indices. We recommend researchers use caution when interpreting RMSEA for models with small df and to rely more on SRMR and CFI.

Comparing Informants for Mental Health Screening at the Preschool Level.

Universal screening for mental health in preschools provides the opportunity for early identification and early intervention, but guidance regarding which informants to use is needed. Preschoolers' (N = 535) parent and teacher reports across two screening forms were analyzed to determine similarities and discrepancies for classification results and screener scores. The analyses also examined if an additional rater provided incrementally valid information to the prediction of longitudinal kindergarten outcomes. Parents' and teachers' screening scores were significantly correlated across forms by rater and across raters. However, categorical classification results indicated that teachers were more likely than parents to rate preschoolers in at-risk ranges across forms. Finally, hierarchical regression analyses revealed teacher ratings were predictive of kindergarten social-emotional, cognitive, and academic outcomes, and that the addition of parent ratings did not significantly improve prediction of outcomes. Implications are discussed in the context of multiple raters within multiple-gating screening procedures.

Moving towards Implementation of Universal Mental Health Screening by Examining Attitudes toward School-based Practices.

Universal mental health screening is a proactive approach to identify students who may benefit from prevention or early intervention services. Despite known benefits, few schools are engaging in screening efforts and it is critical to examine factors that may impede or enhance implementation. Following implementation of a universal screening program across five preschools and elementary schools, this study investigated the attitudes of teachers (N = 40) and parents (N = 330) and found strong agreement among stakeholders about the acceptability and appropriateness of universal mental health screening. Teachers and parents expressed less willingness to regularly complete screening forms, yet teachers reported that the Behavior Assessment System for Children - Third Edition: Behavioral Emotional Screening System was a usable screening tool. Implications and future directions to enhance implementation efforts are discussed.

Using a linear mixed-effect model framework to estimate multivariate generalizability theory parameters in R.

Multivariate generalizability theory (mG-theory) is an important framework in many behavioral and educational studies, as it describes useful psychometric properties of multidimensional assessments. Nevertheless, the use of mG-theory estimation is limited due to the lack of available software for carrying out the necessary calculations: users rely heavily on independent software programs such as mGENOVA and the BUGS/JAGS suite of programs. Considering the prevalence of R software, this paper presents a solution using the glmmTMB package to accomplish the estimation task. Users adopting the proposed method may find it more convenient for conducting both applied investigation and simulation studies without the need to switch between different software programs.

Evaluating Factorial Invariance: An Interval Estimation Approach Using Bayesian Structural Equation Modeling.

In this study, we introduce an interval estimation approach based on Bayesian structural equation modeling to evaluate factorial invariance. For each tested parameter, the size of noninvariance with an uncertainty interval (i.e. highest density interval [HDI]) is assessed via Bayesian parameter estimation. By comparing the most credible values (i.e. 95% HDI) with a region of practical equivalence (ROPE), the Bayesian approach allows researchers to (1) support the null hypothesis of practical invariance, and (2) examine the practical importance of the noninvariant parameter. Compared to the traditional likelihood ratio test, simulation results suggested that the proposed Bayesian approach could offer additional insight into evaluating factorial invariance, thus, leading to more informative conclusions. We provide an empirical example to demonstrate the procedures necessary to implement the proposed method in applied research. The importance of and influences on the choice of an appropriate ROPE are discussed.

School and Family Factors Predicting Adolescent Cognition Regarding Bystander Intervention in Response to Bullying and Victim Retaliation.

Youth aggression occurs at high rates. Aggressive acts can be curbed through bystander intervention; yet, little is known about school and family factors that predict bystander intervention in response to both aggression and victim retaliation. This research examines school and family factors related to standing up to aggression and intervening before possible retaliation occurs. Participants included 6th and 9th graders (N = 896, 52.8% female), who evaluated how likely they would be to intervene if they observed aggression and if they heard the victim was planning to retaliate. Family and school factors are important predictors of bystander intervention, with higher family management, and more positive school climate associated with greater likelihood of intervention and higher feelings of social exclusion and teacher and peer discrimination associated with inactive responses to aggression and retaliation. Thus, a complex constellation of factors relate to the likelihood of intervening if someone is being victimized or considering retaliation in response to victimization. The results provide guidance and new directions for possible school- and family-based interventions to encourage bystander intervention in instances of aggression.

The Relationship Between the Standardized Root Mean Square Residual and Model Misspecification in Factor Analysis Models.

We argue that the definition of close fitting models should embody the notion of substantially ignorable misspecifications (SIM). A SIM model is a misspecified model that might be selected, based on parsimony, over the true model should knowledge of the true model be available. Because in applications the true model (i.e., the data generating mechanism) is unknown, we investigate the relationship between the population standardized root mean square residual (SRMR) values and various model misspecifications in factor analysis models to better understand the magnitudes of the SRMR. Summary effect sizes of misfit such as the SRMR are necessarily insensitive to some non-ignorable localized misspecifications (i.e., the presence of a few large residual correlations in large models). Localized misspecifications may be identified by examining the largest standardized residual covariance. Based on the findings, our population reference values for close fit are based on a two-index strategy: (1) largest absolute value of standardized residual covariance ≤0.10, and (2) SRMR ≤0.05× R¯2 the average R2 of the manifest variables; for acceptable fit our values are 0.15 and 0.10× R¯2 , respectively.

Examining Class Differences in Method Effects Related to Negative Wording: An Example using Rasch Mixture Modeling.

This study presents a mixture Rasch-based approach to investigating method effects associated with negatively worded items to illustrate how responses to this method effect vary depending on group characteristics. Using college students' responses on the Rosenberg Self-Esteem scale (Rosenberg, M., 1989), four latent classes were identified using six personality measures associated with the presence of this method effect. In addition, Rasch-based parameter estimates suggested latent classes differed in their use of the scale, showing that the method effect associated with negatively worded items may be more prominent for subjects possessing selected personality traits than for others. The mixture model approach to investigating method effects provides a way to address systematic methodological variation that is left unaccounted when a heterogeneous population is analyzed as one group.

Creating a Physical Activity Self-Report Form for Youth Using Rasch Methodology.

Measurement of youth's physical activity levels is recommended to ensure that children are meeting recommended activity guidelines. This article describes the creation of an instrument to measure youth's levels of physical activity, where a strong test validation perspective (Benson, 1998) was followed to create the scale. The development process involved a mixed-method (qualitative followed by quantitative) framework. First, focus groups were conducted, where results informed item creation. Next, three alternative forms were created with different response formats to measure childrens' frequency of participation in various physical activities and intensity of participation. Lastly, a sample of over 500 middle school children was obtained, where three different response scales were investigated. The optimal scale considered measurement of physical activity using a three-point Likert frequency; intensity of activity participation did not strongly contribute to the measurement of children's activity levels. The final version form is thought to be acceptable for use with children in surveillance and large-group studies, as well as in smaller sample applications.

A Note on Comparing the Bifactor and Second-Order Factor Models: Is the Bayesian Information Criterion a Routinely Dependable Index for Model Selection?

This note demonstrates that the widely used Bayesian Information Criterion (BIC) need not be generally viewed as a routinely dependable index for model selection when the bifactor and second-order factor models are examined as rival means for data description and explanation. To this end, we use an empirically relevant setting with multidimensional measuring instrument components, where the bifactor model is found consistently inferior to the second-order model in terms of the BIC even though the data on a large number of replications at different sample sizes were generated following the bifactor model. We therefore caution researchers that routine reliance on the BIC for the purpose of discriminating between these two widely used models may not always lead to correct decisions with respect to model choice.

An examination of personality characteristics related to acquiescence.

Acquiescence, the tendency to agree with statements regardless of content, is often a concern when administering self-report instruments. While there is evidence to support acquiescence as a response style, this reporting tendency may be related to personality factors of individuals. Using a sample of 757 adults, we investigated the Rosenberg Self-Esteem Scale for acquiescence response tendencies by applying the Rasch partial credit model. Results suggested that favorable (i.e., Agree or Strongly Agree) responses were more frequent for the positively worded items than for negatively worded items. Second, the relationship between acquiescence and seven additional personality measures was examined overall and by sex. Among females, acquiescence was correlated with personality measures measuring perceptions by others, whereas acquiescence among males was related to exhibition types of behaviors.

Design Effect in Multilevel Settings: A Commentary on a Latent Variable Modeling Procedure for Its Evaluation.

A latent variable modeling-based procedure is discussed that permits to readily point and interval estimate the design effect index in multilevel settings using widely circulated software. The method provides useful information about the relationship of important parameter standard errors when accounting for clustering effects relative to conducting single-level analyses. The approach can also be employed as an addendum to point and interval estimation of the intraclass correlation coefficient in empirical research. The discussed procedure makes it easily possible to evaluate the design effect in two-level studies by utilizing the popular latent variable modeling methodology and is illustrated with an example.

On Effect Size Measures for Nested Measurement Models.

A class of effect size indices are discussed that evaluate the degree to which two nested confirmatory factor analysis models differ from each other in terms of fit to a set of observed variables. These descriptive effect measures can be used to quantify the impact of parameter restrictions imposed in an initially considered model and are free from an explicit relationship to sample size. The described indices represent the extent to which respective linear combinations of the proportions of explained variance in the manifest variables are changed as a result of introducing the constraints. The indices reflect corresponding aspects of the impact of the restrictions and are independent of their statistical significance or lack thereof. The discussed effect size measures are readily point and interval estimated, using popular software, and their application is illustrated with numerical examples.

A Monte Carlo Study of Confidence Interval Methods for Generalizability Coefficient.

Computing confidence intervals around generalizability coefficients has long been a challenging task in generalizability theory. This is a serious practical problem because generalizability coefficients are often computed from designs where some facets have small sample sizes, and researchers have little guide regarding the trustworthiness of the coefficients. As generalizability theory can be framed to a linear mixed-effect model (LMM), bootstrap and simulation techniques from LMM paradigm can be used to construct the confidence intervals. The purpose of this research is to examine four different LMM-based methods for computing the confidence intervals that have been proposed and to determine their accuracy under six simulated conditions based on the type of test scores (normal, dichotomous, and polytomous data) and data measurement design (p×i×r and p× [i:r]). A bootstrap technique called "parametric methods with spherical random effects" consistently produced more accurate confidence intervals than the three other LMM-based methods. Furthermore, the selected technique was compared with model-based approach to investigate the performance at the levels of variance components via the second simulation study, where the numbers of examines, raters, and items were varied. We conclude with the recommendation generalizability coefficients, the confidence interval should accompany the point estimate.

A randomized controlled trial on the interconnected systems framework for school mental health and PBIS: Focus on proximal variables and school discipline.

This study reviews findings for the first randomized controlled trial (RCT) on the Interconnected Systems Framework (ISF) for school mental health (SMH) and Positive Behavioral Interventions and Supports (PBIS). Since its development in the late 2000s, the ISF has been supported by federally funded centers for SMH and PBIS, and, guided by a national workgroup, is being implemented in >50 communities in the United States. This experimental evaluation of the ISF involved an RCT implemented in 24 schools in two southeastern states, with the ISF implemented in eight schools, PBIS alone implemented in eight schools, and typically co-located PBIS+SMH implemented in eight schools. Related to very poor implementation, documented by two sources of fidelity data, two ISF schools were dropped from major analyses; hence, the study used a treatment on the treated (ToT; Rubin, 1974) as compared to a more traditional Intent-to-Treat approach (ITT; Lachin, 2000). This is the first paper from this large study, with emphasis here on proximal variables and school discipline. Within schools' multi-tiered systems of support (MTSS), ISF schools delivered more Tier 2 (early intervention) and Tier 3 (treatment) interventions to a greater proportion of students than the other two conditions by the second year of the intervention. There was also a dramatic difference in the provision of interventions by community mental health clinicians in ISF schools (almost half of interventions delivered) as compared to PBIS+SMH schools (around 3% of interventions delivered), underscoring the critical role of the ISF in integrating clinicians into MTSS teams and core school functions in SMH. As compared to the other two conditions, ISF schools also had reduced office discipline referrals (ODRs) and in-school suspensions, as well as reduced ODRs and out-of-school suspensions for African American students. Findings are discussed in relation to future directions of education-mental health system partnerships in improving the delivery and impact of SMH programs and services, demonstrated in the ISF.

Evaluating Restrictive Models in Educational and Behavioral Research: Local Misfit Overrides Model Tenability.

The frequent practice of overall fit evaluation for latent variable models in educational and behavioral research is reconsidered. It is argued that since overall plausibility does not imply local plausibility and is only necessary for the latter, local misfit should be considered a sufficient condition for model rejection, even in the case of omnibus model tenability. The argument is exemplified with a comparison of the widely used one-parameter and two-parameter logistic models. A theoretically and practically relevant setting illustrates how discounting local fit and concentrating instead on overall model fit may lead to incorrect model selection, even if a popular information criterion is also employed. The article concludes with the recommendation for routine examination of particular parameter constraints within latent variable models as part of their fit evaluation.

A Short Note on Optimizing Cost-Generalizability via a Machine-Learning Approach.

The costs of an objective structured clinical examination (OSCE) are of concern to health profession educators globally. As OSCEs are usually designed under generalizability theory (G-theory) framework, this article proposes a machine-learning-based approach to optimize the costs, while maintaining the minimum required generalizability coefficient, a reliability-like index in G-theory. The authors adopted G-theory parameters yielded from an OSCE hosted by a medical school, reproduced the generalizability coefficients to prepare for optimizing manipulations, applied simulated annealing algorithm to calculate the number of facet levels minimizing the associated costs, and conducted the analysis in various conditions via computer simulation. With a given generalizability coefficient, the proposed approach, virtually an instrument of decision-making supports, found the optimal solution for the OSCE such that the associated costs were minimized. The computer simulation results showed how the cost reductions varied with different levels of required generalizability coefficients. Machine learning-based approaches can be used in conjunction with psychometric modeling to help planning assessment tasks more scientifically. The proposed approach is easy to adopt into practice and customize in alignment with specific testing designs. While these results are encouraging, the possible pitfalls such as algorithmic convergences' failure and inadequate cost assumptions should also be avoided.

Fitting Latent Growth Models with Small Sample Sizes and Non-normal Missing Data.

This study investigates the performance of robust ML estimators when fitting and evaluating small sample latent growth models (LGM) with non-normal missing data. Results showed that the robust ML methods could be used to account for non-normality even when the sample size is very small (e.g., N < 100). Among the robust ML estimators, "MLR" was the optimal choice, as it was found to be robust to both non-normality and missing data while also yielding more accurate standard error estimates and growth parameter coverage. However, the choice "MLMV" produced the most accurate p values for the Chi-square test statistic under conditions studied. Regarding the goodness of fit indices, as sample size decreased, all three fit indices studied (i.e., CFI, RMSEA, and SRMR) exhibited worse fit. When the sample size was very small (e.g., N < 60), the fit indices would imply that a proposed model fit poorly, when this might not be actually the case in the population.

Using Integrative Data Analysis to Investigate School Climate Across Multiple Informants.

Survey research frequently involves the collection of data from multiple informants. Results, however, are usually analyzed by informant group, potentially ignoring important relationships across groups. When the same construct(s) are measured, integrative data analysis (IDA) allows pooling of data from multiple sources into one data set to examine information from multiple perspectives within the same analysis. Here, the IDA procedure is demonstrated via the examination of pooled data from student and teacher school climate surveys. This study contributes to the sparse literature regarding IDA applications in the social sciences, specifically in education. It also lays the groundwork for future educational researchers interested in the practical applications of the IDA framework to empirical data sets with complex model structures.

School and Teacher Factors That Promote Adolescents' Bystander Responses to Social Exclusion.

Schools may be one important context where adolescents learn and shape the behaviors necessary for promoting global inclusivity in adulthood. Given the importance of bystanders in halting bullying and peer aggression, the focus of this study is on both moral judgments regarding one type of bullying, social exclusion, and factors that are associated with bystander intervention. The study includes 896 adolescents, who were 6th (N = 450, M age = 11.73), and 9th (N = 446, M age = 14.82) graders, approximately evenly divided by gender. Participants were primarily European-American (63.3%). Results revealed that girls and participants who perceived better relationships between students and teachers were more likely to judge exclusion to be wrong. Further, ethnic minority participants, those who were more anxious about being rejected by their teachers and reported more teacher discrimination were less likely to judge exclusion as wrong. Participants who reported more positive student-teacher relationships, perceptions of a more positive school social environment and more prior experiences of teacher discrimination were more likely to report that they would seek help for the victim. On the other hand, participants who reported being more angry about teacher rejection, experiencing either peer or teacher discrimination, and perceiving they are excluded from opportunities at school were less likely to intervene to come to the aid of a peer who is being excluded. The results document the complex interplay of school and teacher factors in shaping adolescents' bystander responses to social exclusion. Our findings suggest that positive school climate can promote intentions to intervene. However, findings indicate that adolescents who are marginalized in their school environments, and who report experiences of rejection, exclusion or discrimination are not willing or likely to intervene to prevent others from experiencing exclusion.