A Comparison of Metaheuristic Optimization Algorithms for Scale Short-Form Development.

This study compares automated methods to develop short forms of psychometric scales. Obtaining a short form that has both adequate internal structure and strong validity with respect to relationships with other variables is difficult with traditional methods of short-form development. Metaheuristic algorithms can select items for short forms while optimizing on several validity criteria, such as adequate model fit, composite reliability, and relationship to external variables. Using a Monte Carlo simulation study, this study compared existing implementations of the ant colony optimization, Tabu search, and genetic algorithm to select short forms of scales, as well as a new implementation of the simulated annealing algorithm. Selection of short forms of scales with unidimensional, multidimensional, and bifactor structure were evaluated, with and without model misspecification and/or an external variable. The results showed that when the confirmatory factor analysis model of the full form of the scale was correctly specified or had only minor misspecification, the four algorithms produced short forms with good psychometric qualities that maintained the desired factor structure of the full scale. Major model misspecification resulted in worse performance for all algorithms, but including an external variable only had minor effects on results. The simulated annealing algorithm showed the best overall performance as well as robustness to model misspecification, while the genetic algorithm produced short forms with worse fit than the other algorithms under conditions with model misspecification.

Between-teacher variance of students' teacher-rated risk for emotional, behavioral, and adaptive functioning.

As schools increasingly adopt universal social, emotional, and behavioral screening, more research is needed to examine the effects of between-teacher differences due to error and bias on students' teacher-rated screening scores. The current study examined predictors of between-teacher differences in students' teacher-rated risk across one global and three narrow domains of behavioral functioning. Participants included 2450 students (52.1% male, 54.2% White) and 160 teachers (92.1% female, 80.3% White) from four elementary schools in one Southeastern U.S. school district. Teachers rated student behavior on the Behavior Assessment System for Children (Third Edition) Behavioral and Emotional Screening System (BESS)-Teacher Form and completed a survey about their training and perspectives of common behavior problems. Results of multilevel linear regression found between-teacher effects to be greater for internalizing risk scores (intraclass correlation = 0.23) than for externalizing risk scores (intraclass correlation = 0.12) or adaptive behavior scores (intraclass correlation = 0.14). Statistically significant student predictors in most models included student grade, gender, race and/or ethnicity, office discipline referrals, and course grades. We also detected effects of several teacher-level variables in one or more of the models, including teacher gender, teacher ratings of problem severity and concern for hypothetical children displaying behavior problems, and the covariance of random teacher intercept and teacher random slopes for students' office discipline referrals. Although these factors explained some teacher-level variance in students' risk scores, a notable amount of variance between teachers remains unexplained. Future research is needed to fully understand, reduce, and account for differences between teacher ratings due to error and bias.

Preliminary development and validation of the Interconnected Systems Framework-Implementation Inventory (ISF-II).

As schools increasingly implement multitiered systems of support, there is a pressing need to develop psychometrically sound implementation fidelity measures. The interconnected systems framework (ISF) is a multitiered model blending systems of positive behavioral interventions and supports with promotion, prevention, and intervention strategies of school mental health. The ISF is being implemented in communities across the United States with ongoing evaluation in several randomized controlled trials. The ISF-Implementation Inventory (ISF-II) was developed to measure fidelity of the ISF within a school building. We conducted a national validation study including completion of the ISF-II by 398 educators in 49 schools, 16 school districts, and 9 states. Results indicate the ISF-II produces scores that are internally consistent and structurally valid when items are organized into a three-tiered model. Additionally, the ISF-II was rated as feasible, acceptable, and beneficial. Limitations of the study, including the need for additional psychometric testing, are discussed in light of these results that suggest educators and researchers, alike, should feel confident in using the ISF-II as a measure of ISF implementation quality. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Student, teacher, and classroom predictors of between-teacher variance of students' teacher-rated behavior.

The current study examined between-teacher variance in teacher ratings of student behavioral and emotional risk to identify student, teacher and classroom characteristics that predict such differences and can be considered in future research and practice. Data were taken from seven elementary schools in one school district implementing universal screening, including 1,241 students rated by 68 teachers. Students were mostly African America (68.5%) with equal gender (female 50.1%) and grade-level distributions. Teachers, mostly White (76.5%) and female (89.7%), completed both a background survey regarding their professional experiences and demographic characteristics and the Behavior Assessment System for Children (Second Edition) Behavioral and Emotional Screening System-Teacher Form for all students in their class, rating an average of 17.69 students each. Extant student data were provided by the district. Analyses followed multilevel linear model stepwise model-building procedures. We detected a significant amount of variance in teachers' ratings of students' behavioral and emotional risk at both student and teacher/classroom levels with student predictors explaining about 39% of student-level variance and teacher/classroom predictors explaining about 20% of between-teacher differences. The final model fit the data (Akaike information criterion = 8,687.709; pseudo-R2 = 0.544) significantly better than the null model (Akaike information criterion = 9,457.160). Significant predictors included student gender, race ethnicity, academic performance and disciplinary incidents, teacher gender, student-teacher gender interaction, teacher professional development in behavior screening, and classroom academic performance. Future research and practice should interpret teacher-rated universal screening of students' behavioral and emotional risk with consideration of the between-teacher variance unrelated to student behavior detected. (PsycINFO Database Record

What Information Works Best?: A Comparison of Routing Methods.

There are many item selection methods proposed for computerized adaptive testing (CAT) applications. However, not all of them have been used in computerized multistage testing (ca-MST). This study uses some item selection methods as a routing method in ca-MST framework. These are maximum Fisher information (MFI), maximum likelihood weighted information (MLWI), maximum posterior weighted information (MPWI), Kullback-Leibler (KL), and posterior Kullback-Leibler (KLP). The main purpose of this study is to examine the performance of these methods when they are used as a routing method in ca-MST applications. These five information methods under four ca-MST panel designs and two test lengths (30 items and 60 items) were tested using the parameters of a real item bank. Results were evaluated with overall findings (mean bias, root mean square error, correlation between true and estimated thetas, and module exposure rates) and conditional findings (conditional absolute bias, standard error of measurement, and root mean square error). It was found that test length affected the outcomes much more than other study conditions. Under 30-item conditions, 1-3 designs outperformed other panel designs. Under 60-item conditions, 1-3-3 designs were better than other panel designs. Each routing method performed well under particular conditions; there was no clear best method in the studied conditions. The recommendations for routing methods in any particular condition were provided for researchers and practitioners as well as the limitations of these results.

Factor analytic replication and model comparison of the BASC-2 Behavioral and Emotional Screening System.

We conducted this study to add to literature of previous conflicting factorial examinations of the BASC-2 Behavioral and Emotional Screening System (BESS), Teacher Form-Child/Adolescent. Data were collected by an urban school district in the southeastern United States including 2,228 students rated by 120 teachers in Fall 2014 and 1,955 students rated by 104 teachers in Spring 2015. In both samples, we replicated and then conceptually and statistically compared factor models to examine the (a) 4-factor structure from which the BESS Teacher Form was developed, and (b) existence of a general factor currently being used. Previous studies examined the 4-factor and bifactor structure of the BESS Teacher Form on separate samples. Our model comparison results support a multidimensional interpretation. We recovered similar fit statistics and standardized factor loadings as previous factor analyses. However, measures of variance accounted for by the general factor were below recommended thresholds of a unidimensional construct. We recommend advancing a factorial model that represents a weighted combination of general and specific factors, but do not support continued use of a unidimensional total T score. Limitations and implications of the study are discussed. (PsycINFO Database Record