Comparison of model- and design-based approaches to detect the treatment effect and covariate by treatment interactions in three-level models for multisite cluster-randomized trials.

In this study, we evaluated the estimation of three important parameters for data collected in a multisite cluster-randomized trial (MS-CRT): the treatment effect, and the treatment by covariate interactions at Levels 1 and 2. The Level 1 and Level 2 interaction parameters are the coefficients for the products of the treatment indicator, with the covariate centered on its Level 2 expected value and with the Level 2 expected value centered on its Level 3 expected value, respectively. A comparison of a model-based approach to design-based approaches was performed using simulation studies. The results showed that both approaches produced similar treatment effect estimates and interaction estimates at Level 1, as well as similar Type I error rates and statistical power. However, the estimate of the Level 2 interaction coefficient for the product of the treatment indicator and an arithmetic mean of the Level 1 covariate was severely biased in most conditions. Therefore, applied researchers should be cautious when using arithmetic means to form a treatment by covariate interaction at Level 2 in MS-CRT data.

Factors Associated with Teacher Delivery of a Classroom-Based Tier 2 Prevention Program.

Teachers sometimes struggle to deliver evidence-based programs designed to prevent and ameliorate chronic problem behaviors of young children with integrity. Identifying factors associated with variations in the quantity and quality of delivery is thus an important goal for the field. This study investigated factors associated with teacher treatment integrity of BEST in CLASS, a tier-2 prevention program designed for young children at risk for developing emotional/behavioral disorders. Ninety-two early childhood teachers and 231 young children at-risk for emotional/behavioral disorders participated in the study. Latent growth curve analyses indicated that both adherence and competence of delivery increased across six observed time points. Results suggest that teacher education and initial levels of classroom quality may be important factors to consider when teachers deliver tier-2 (i.e., targeted to children who are not responsive to universal or tier-1 programming) prevention programs in early childhood settings. Teachers with higher levels of education delivered the program with more adherence and competence initially. Teachers with higher initial scores on the Emotional Support subscale of the Classroom Assessment Scoring System (CLASS) delivered the program with more competence initially and exhibited higher growth in both adherence and competence of delivery across time. Teachers with higher initial scores on the Classroom Organization subscale of the CLASS exhibited lower growth in adherence across time. Contrary to hypotheses, teacher self-efficacy did not predict adherence, and teachers who reported higher initial levels of Student Engagement self-efficacy exhibited lower growth in competence of delivery. Results are discussed in relation to teacher delivery of evidence-based programs in early childhood classrooms.

The Consequences of Ignoring Variability in Measurement Occasions Within Data Collection Waves in Latent Growth Models.

In longitudinal data collection, it is common that each wave of collection spans several months. However, researchers using latent growth models commonly ignore variability in data collection occasions within a wave. In this study, we investigated the consequences of ignoring within-wave variability in measurement occasions using a Monte Carlo simulation and an empirical study. The results of the simulation study showed that ignoring heterogeneity resulted in biased estimates for some parameters, especially when heterogeneity was large and assessment dates had a skewed distribution. Models constructed on person-specific time points yielded precise estimates and more adequate model fit. In the empirical study, we demonstrated different time coding strategies with a subsample taken from Early Childhood Longitudinal Study Kindergarten Cohort.

Efficacy of the social-emotional learning foundations curriculum for kindergarten and first grade students at risk for emotional and behavioral disorders.

Researchers have shown that children's social-emotional growth is inextricably connected to academic learning. We developed the Social-Emotional Learning Foundations (SELF) intervention, a Grade K-1 curriculum merging social-emotional learning (SEL) and literacy instruction, to promote language supported self-regulation, specifically for primary grade children at early risk for emotional or behavioral difficulties. We report findings from a pretest-posttest cluster randomized efficacy trial with one fixed between-subjects factor to test the effects of teacher-delivered SEL instruction against those of business as usual (BAU). We recruited 163 kindergarten (K) and 141 first grade teachers from 52 schools across 11 school districts within one southeastern state. Our student sample (n = 1154) consisted of 627 kindergarteners and 527 first graders identified by teachers as at risk for internalizing or externalizing emotional and behavioral problems using the Systematic Screening for Behavioral Disorders; 613 of these students participated in the SELF condition and 541 participated in the BAU condition. We randomly assigned schools to SELF or BAU and used a multilevel model with three levels (i.e., children, classrooms, schools) to analyze data on subscales of six (four teacher-report and two direct) assessments related to self-regulation, social-emotional learning, social-emotional vocabulary, and general behavioral functioning. We found positive main effects of SELF compared to BAU on all but one measure, with effect sizes (calculated using Hedges' g) ranging from 0.20 to 0.65. Findings provide evidence for guiding future SEL intervention research and informing practice to improve student outcomes, particularly for children at risk for behavior problems.

The impact of omitting the interaction between crossed factors in cross-classified random effects modelling.

Cross-classified random effects modelling (CCREM) is a special case of multi-level modelling where the units of one level are nested within two cross-classified factors. Typically, CCREM analyses omit the random interaction effect of the cross-classified factors. We investigate the impact of the omission of the interaction effect on parameter estimates and standard errors. Results from a Monte Carlo simulation study indicate that, for fixed effects, both coefficients estimates and accompanied standard error estimates are not biased. For random effects, results are affected at level 2 but not at level 1 by the presence of an interaction variance and/or a correlation between the residual of level two factors. Results from the analysis of the Early Childhood Longitudinal Study and the National Educational Longitudinal Study agree with those obtained from simulated data. We recommend that researchers attempt to include interaction effects of cross-classified factors in their models.

A generally robust approach for testing hypotheses and setting confidence intervals for effect sizes.

Standard least squares analysis of variance methods suffer from poor power under arbitrarily small departures from normality and fail to control the probability of a Type I error when standard assumptions are violated. This article describes a framework for robust estimation and testing that uses trimmed means with an approximate degrees of freedom heteroscedastic statistic for independent and correlated groups designs in order to achieve robustness to the biasing effects of nonnormality and variance heterogeneity. The authors describe a nonparametric bootstrap methodology that can provide improved Type I error control. In addition, the authors indicate how researchers can set robust confidence intervals around a robust effect size parameter estimate. In an online supplement, the authors use several examples to illustrate the application of an SAS program to implement these statistical methods.

Multigroup confirmatory factor analysis for the adaptive behavior assessment system-II parent form, ages 5-21.

The AAIDD has promulgated various models of adaptive behavior, including its 1992 model stressing 10 adaptive skills and its 2002 model that highlighted three conceptual domains. In previous studies on the Adaptive Behavior Assessment System-II (ABAS-II), researchers found support for a model including both 10 adaptive skills and three conceptual domains. To extend this review, we examined gender-invariant structure of adaptive behavior using the ABAS-II Parent Form, Ages 5-21, to answer four questions: Do the skill areas in this measure display the same pattern of factor loadings and the same factor loadings? Are intercepts of the observed skill areas equal? Do skill areas measure the corresponding factors with the same accuracy? Results show a similar one-factor structure for males and females.

A comparative study of robust tests for spread: asymmetric trimming strategies.

We examined 633 procedures that can be used to compare the variability of scores across independent groups. The procedures, except for one, were modifications of the procedures suggested by Levene (1960) and O'Brien (1981). We modified their procedures by substituting robust measures of the typical score and variability, rather than relying on classical estimators. The robust measures that we utilized were either based on a priori or empirically determined symmetric or asymmetric trimming strategies. The Levene-type and O'Brien-type transformed scores were used with either the ANOVA F test, a robust test due to Lee and Fung (1985), or the Welch (1951) test. Based on four measures of robustness, we recommend a Levene-type transformation based upon empirically determined 20% asymmetric trimmed means, involving a particular adaptive estimator, where the transformed scores are then used with the ANOVA F test.

Examining young children's social competence using functional ability profiles.

PURPOSE: To explore the use of International Classification of Functioning, Disability, and Health for Children and Youth (ICF-CY) based profiles of children's functional abilities in relation to their social competence. Subgroups based on shared profiles of functional ability were investigated as an alternative or complement to subgroups defined by disability categories. METHODS: Secondary analysis of a nationally representative data set of young children identified for special education services in the United States was used for the present study. Using five subgroups of children with shared profiles of functional ability, derived from latent class analysis in previous work, regression analyses were used to examine the relationships between social competence and functional abilities profile subgroup membership. Differences among the subgroups were examined using standardized effect sizes. R2 values were used to examine explained variance in social competence in relation to subgroup membership, disability category, and these variables in combination. RESULTS: Functional ability profile subgroup membership was moderately related to children's social competence outcomes: social skills and problem behaviors. Effect sizes showed significant differences between subgroups. Subgroup membership accounted for more variance in social competence outcomes than disability category. CONCLUSIONS: The results provide empirical support for the importance of functional ability profiles when examining social competence within a population of young children with disabilities. Implications for Rehabilitation The extent to which children with disabilities experience difficulty with social competence varies by their functional characteristics. Functional ability profiles can provide practitioners and researchers working young children with disabilities important tools to examine social competence and to inform interventions.

Adaptive robust estimation and testing.

We examined nine adaptive methods of trimming, that is, methods that empirically determine when data should be trimmed and the amount to be trimmed from the tails of the empirical distribution. Over the 240 empirical values collected for each method investigated, in which we varied the total percentage of data trimmed, sample size, degree of variance heterogeneity, pairing of variances and group sizes, and population shape, one method resulted in exceptionally good control of Type I errors. However, under less extreme cases of non-normality and variance heterogeneity a number of methods exhibited reasonably good Type I error control. With regard to the power to detect non-null treatment effects, we found that the choice among the methods depended on the degree of non-normality and variance heterogeneity. Recommendations are offered.

Predicting outcome in parent-child interaction therapy: success and attrition.

This study explored predictors of treatment response and attrition in Parent-Child Interaction Therapy (PCIT). Participants were 99 families of 3- to 6-year-old children with disruptive behavior disorders. Multiple logistic regression was used to identify those pretreatment child, family, and accessibility factors that were predictive of success or attrition. For all study participants, waitlist group assignment and maternal age were the significant predictors of outcome. For treatment participants (study participants excluding those who dropped out after the initial evaluation but before treatment began), only maternal ratings of parenting stress and maternal inappropriate behavior during parent-child interactions were significant predictors of treatment outcome. These results suggest that for treatment studies of disruptive preschoolers, the benefits of using a waitlist control group may be outweighed by the disproportionate number of dropouts from this group. Once families begin PCIT, however, parent-related variables become salient in predicting treatment outcome.

The Effects of Including Observed Means or Latent Means as Covariates in Multilevel Models for Cluster Randomized Trials.

We investigated methods of including covariates in two-level models for cluster randomized trials to increase power to detect the treatment effect. We compared multilevel models that included either an observed cluster mean or a latent cluster mean as a covariate, as well as the effect of including Level 1 deviation scores in the model. A Monte Carlo simulation study was performed manipulating effect sizes, cluster sizes, number of clusters, intraclass correlation of the outcome, patterns of missing data, and the squared correlations between Level 1 and Level 2 covariates and the outcome. We found no substantial difference between models with observed means or latent means with respect to convergence, Type I error rates, coverage, and bias. However, coverage could fall outside of acceptable limits if a latent mean is included as a covariate when cluster sizes are small. In terms of statistical power, models with observed means performed similarly to models with latent means, but better when cluster sizes were small. A demonstration is provided using data from a study of the Tools for Getting Along intervention.

An alternative to Cohen's standardized mean difference effect size: a robust parameter and confidence interval in the two independent groups case.

The authors argue that a robust version of Cohen's effect size constructed by replacing population means with 20% trimmed means and the population standard deviation with the square root of a 20% Winsorized variance is a better measure of population separation than is Cohen's effect size. The authors investigated coverage probability for confidence intervals for the new effect size measure. The confidence intervals were constructed by using the noncentral t distribution and the percentile bootstrap. Over the range of distributions and effect sizes investigated in the study, coverage probability was better for the percentile bootstrap confidence interval.

Generalized eta and omega squared statistics: measures of effect size for some common research designs.

The editorial policies of several prominent educational and psychological journals require that researchers report some measure of effect size along with tests for statistical significance. In analysis of variance contexts, this requirement might be met by using eta squared or omega squared statistics. Current procedures for computing these measures of effect often do not consider the effect that design features of the study have on the size of these statistics. Because research-design features can have a large effect on the estimated proportion of explained variance, the use of partial eta or omega squared can be misleading. The present article provides formulas for computing generalized eta and omega squared statistics, which provide estimates of effect size that are comparable across a variety of research designs.

Verbal memory abilities of children with brain tumors.

Several critical neuroanatomical structures and pathways for memory performance are located in the third ventricle region. This led us to predict that verbal memory abilities would be more impaired in children treated for third ventricle tumors compared to those treated for cerebellar tumors. Archival data was obtained from 24 pediatric patients with third ventricle region tumors and 18 pediatric patients with cerebellar tumors. Neuroradiological verifications of tumor involvement and hydrocephalus severity (i.e., Evans Index) on preoperative scans and MRIs proximal to the time of the neuropsychological evaluation were conducted. The potential confounds of hydrocephalus severity, seizure medication, age, radiation treatment, and chemotherapy were addressed. Verbal IQ was comparable between tumor groups and in the Average range. The third ventricle region group performed significantly worse on list learning and delayed list recall compared to the cerebellar group. Their mean performance was in the clinically impaired range on both trials. The third ventricle region tumor group performed better than the cerebellar tumor group on Digit Span, a basic repetition, attention span task. These findings support the hypothesis that pediatric patients with third ventricle region brain tumors are more likely to be impaired on verbal recall tasks compared to pediatric patients with cerebellar brain tumors. In contrast, patients who were treated for cerebellar tumors were more impaired on the basic repetition, attention span task compared to patients who were treated for third ventricle tumors. Future studies should examine the specific neuroanatomical structures and pathways that are damaged and may influence differential cognitive impairments in children.

Conducting power analyses for ANOVA and ANCOVA in between-subjects designs.

Researchers are frequently asked to justify the sample size used in their quantitative inquiries. Such a justification can be provided through a power analysis. Conducting power analyses, however, can raise some difficult issues regarding the specification of the size of the effect, testing for interaction effects, the role of covariates, and the use of an estimated effect size in the power analysis. The authors present methods for conducting power analyses along with a discussion of these issues, and they make available SAS programs that can be used to implement the power analyses that are discussed.

Type I Error Rates For A One Factor Within-Subjects Design With Missing Values.

Missing data are a common problem in educational research. A promising technique, that can be implemented in SAS PROC MIXED and is therefore widely available, is to use maximum likelihood to estimate model parameters and base hypothesis tests on these estimates. However, it is not clear which test statistic in PROC MIXED performs better with missing data. The performance of the Hotelling-Lawley-McKeon and Kenward-Roger omnibus test statistics on the means for a single factor within-subject ANOVA are compared. The results indicate that the Kenward-Roger statistic performed better in terms of keeping the Type I error close to the nominal alpha level.

Use of the color trails test as an embedded measure of performance validity.

One hundred personal injury litigants and disability claimants referred for a forensic neuropsychological evaluation were administered both portions of the Color Trails Test (CTT) as part of a more comprehensive battery of standardized tests. Subjects who failed two or more free-standing tests of cognitive performance validity formed the Failed Performance Validity (FPV) group, while subjects who passed all free-standing performance validity measures were assigned to the Passed Performance Validity (PPV) group. A cutscore of ≥45 seconds to complete Color Trails 1 (CT1) was associated with a classification accuracy of 78%, good sensitivity (66%) and high specificity (90%), while a cutscore of ≥84 seconds to complete Color Trails 2 (CT2) was associated with a classification accuracy of 82%, good sensitivity (74%) and high specificity (90%). A CT1 cutscore of ≥58 seconds, and a CT2 cutscore ≥100 seconds was associated with 100% positive predictive power at base rates from 20 to 50%.

Derivation of the MMPI-2-RF Henry-Heilbronner Index-r (HHI-r) scale.

The 15-item Henry-Heilbronner Index (HHI) was published in 2006 as an MMPI-2 embedded measure of psychological response validity. When the MMPI-2 was revised in 2008 only 11 of the 15 original HHI items were retained on the MMPI-2-RF, prohibiting use of the HHI as an embedded validity indicator on the MMPI-2-RF. Using the original HHI sample an 11-item version of the HHI, the HHI-r, was evaluated for use as an embedded measure of psychological response validity for the MMPI-2-RF. The 11-item HHI-r was very similar to the HHI in classification accuracy. An HHI-r cutoff score of ≥7 was associated with a classification accuracy rate of 84.0%, good sensitivity (68.9%), and high specificity (93.2%) in identifying symptom exaggeration in personal injury and disability litigants versus non-litigating head-injured patients. These preliminary results suggest the HHI-r functions in a manner similar to the original HHI as a measure of psychological response validity, and may be used by psychologists and neuropsychologists as an MMPI-2-RF embedded validity indicator.

Factorial validity of the Behavior Rating Inventory of Executive Function (BRIEF)-Teacher form.

Deficits in executive function (self-regulatory mechanisms) have been linked with many childhood disorders including attention deficit/hyperactivity disorder (ADHD), autism spectrum disorder, and conduct disorder. Executive functioning is typically assessed by individually administering performance-based measures in a clinical setting. However, performance-based methods are inefficient for school psychologists. A more feasibly implemented measure for applied settings is the Behavior Rating Inventory of Executive Function (BRIEF), but researchers have raised questions about the internal validity and the proposed factors. In this study, we examined the factor structure of the teacher form of the BRIEF in a sample of 2,044 general education elementary students and 131 teachers in a multilevel design. Results revealed support for a model with three factors at Level 1 and one general factor at Level 2. The results of our study do not support the current two-factor model of the published BRIEF protocol.