Power Analysis for Moderator Effects in Longitudinal Cluster Randomized Designs.

Cluster randomized control trials often incorporate a longitudinal component where, for example, students are followed over time and student outcomes are measured repeatedly. Besides examining how intervention effects induce changes in outcomes, researchers are sometimes also interested in exploring whether intervention effects on outcomes are modified by moderator variables at the individual (e.g., gender, race/ethnicity) and/or the cluster level (e.g., school urbanicity) over time. This study provides methods for statistical power analysis of moderator effects in two- and three-level longitudinal cluster randomized designs. Power computations take into account clustering effects, the number of measurement occasions, the impact of sample sizes at different levels, covariates effects, and the variance of the moderator variable. Illustrative examples are offered to demonstrate the applicability of the methods.

Incorporating cost in power analysis for three-level cluster-randomized designs.

In experimental designs with nested structures, entire groups (such as schools) are often assigned to treatment conditions. Key aspects of the design in these cluster-randomized experiments involve knowledge of the intraclass correlation structure, the effect size, and the sample sizes necessary to achieve adequate power to detect the treatment effect. However, the units at each level of the hierarchy have a cost associated with them and thus researchers need to decide on sample sizes given a certain budget, when designing their studies. This article provides methods for computing power within an optimal design framework that incorporates costs of units in all three levels for three-level cluster-randomized balanced designs with two levels of nesting at the second and third level. The optimal sample sizes are a function of the variances at each level and the cost of each unit. Overall, larger effect sizes, smaller intraclass correlations at the second and third level, and lower cost of Level 3 and Level 2 units result in higher estimates of power.

Using Quantile Regression to Estimate Intervention Effects Beyond the Mean.

This study discusses quantile regression methodology and its usefulness in education and social science research. First, quantile regression is defined and its advantages vis-à-vis vis ordinary least squares regression are illustrated. Second, specific comparisons are made between ordinary least squares and quantile regression methods. Third, the applicability of quantile regression to empirical work to estimate intervention effects is demonstrated using education data from a large-scale experiment. The estimation of quantile treatment effects at various quantiles in the presence of dropouts is also discussed. Quantile regression is especially suitable in examining predictor effects at various locations of the outcome distribution (e.g., lower and upper tails).

Power Analysis for Models of Change in Cluster Randomized Designs.

Field experiments in education frequently assign entire groups such as schools to treatment or control conditions. These experiments incorporate sometimes a longitudinal component where for example students are followed over time to assess differences in the average rate of linear change, or rate of acceleration. In this study, we provide methods for power analysis in three-level polynomial change models for cluster randomized designs (i.e., treatment assigned to units at the third level). Power computations take into account clustering effects at the second and third levels, the number of measurement occasions, the impact of sample sizes at different levels (e.g., number of schools or students), and covariates effects. An illustrative example that shows how power is influenced by the number of measurement occasions, and sample sizes and covariates at the second or third levels is presented.

Effects of Interim Assessments Across the Achievement Distribution: Evidence From an Experiment.

We use data from a large-scale experiment conducted in Indiana in 2009-2010 to examine the impact of two interim assessment programs (mCLASS and Acuity) across the mathematics and reading achievement distributions. Specifically, we focus on whether the use of interim assessments has a particularly strong effect on improving outcomes for low achievers. Quantile regression is used to estimate treatment effects across the entire achievement distribution (i.e., provide estimates in the lower, middle, or upper tails). Results indicate that in Grades 3 to 8 (particularly third, fifth, and sixth) lower achievers seem to benefit more from interim assessments than higher achieving students.

Separate but correlated: The latent structure of space and mathematics across development.

The relations among various spatial and mathematics skills were assessed in a cross-sectional study of 854 children from kindergarten, third, and sixth grades (i.e., 5 to 13 years of age). Children completed a battery of spatial mathematics tests and their scores were submitted to exploratory factor analyses both within and across domains. In the within domain analyses, all of the measures formed single factors at each age, suggesting consistent, unitary structures across this age range. Yet, as in previous work, the 2 domains were highly correlated, both in terms of overall composite score and pairwise comparisons of individual tasks. When both spatial and mathematics scores were submitted to the same factor analysis, the 2 domain specific factors again emerged, but there also were significant cross-domain factor loadings that varied with age. Multivariate regressions replicated the factor analysis and further revealed that mental rotation was the best predictor of mathematical performance in kindergarten, and visual-spatial working memory was the best predictor of mathematical performance in sixth grade. The mathematical tasks that predicted the most variance in spatial skill were place value (K, 3rd, 6th), word problems (3rd, 6th), calculation (K), fraction concepts (3rd), and algebra (6th). Thus, although spatial skill and mathematics each have strong internal structures, they also share significant overlap, and have particularly strong cross-domain relations for certain tasks. (PsycINFO Database Record

The Impact of Covariates on Statistical Power in Cluster Randomized Designs: Which Level Matters More?

Field experiments with nested structures are becoming increasingly common, especially designs that assign randomly entire clusters such as schools to a treatment and a control group. In such large-scale cluster randomized studies the challenge is to obtain sufficient power of the test of the treatment effect. The objective is to maximize power without adding many clusters that make the study much more expensive. In this article I discuss how power estimates of tests of treatment effects in balanced cluster randomized designs are affected by covariates at different levels. I use third-grade data from Project STAR, a field experiment about class size, to demonstrate how covariates that explain a considerable proportion of variance in outcomes increase power significantly. When lower level covariates are group-mean centered and clustering effects are larger, top-level covariates increase power more than lower level covariates. In contrast, when clustering effects are smaller and lower level covariates are grand-mean centered or uncentered, lower level covariates increase power more than top-level covariates.

Fixed effects and variance components estimation in three-level meta-analysis.

Meta-analytic methods have been widely applied to education, medicine, and the social sciences. Much of meta-analytic data are hierarchically structured because effect size estimates are nested within studies, and in turn, studies can be nested within level-3 units such as laboratories or investigators, and so forth. Thus, multilevel models are a natural framework for analyzing meta-analytic data. This paper discusses the application of a Fisher scoring method in two-level and three-level meta-analysis that takes into account random variation at the second and third levels. The usefulness of the model is demonstrated using data that provide information about school calendar types. sas proc mixed and hlm can be used to compute the estimates of fixed effects and variance components. Copyright © 2011 John Wiley & Sons, Ltd.

How consistent are class size effects?

Thus far researchers have focused on computing average differences in student achievement between smaller and larger classes. In this study, the author focus on the distribution of the small class effects at the school level and compute the inconsistency of the small class effects across schools. The author use data from Project STAR to estimate small class effects for each school on mathematics and reading scores from kindergarten through third grade. Then, all school estimates were combined to calculate an overall weighted average. The results revealed that a large proportion of the school-specific small class effects are positive, while a smaller proportion of the estimates are negative. Although students benefit considerably from being in small classes in many schools, in other schools being in small classes is either not beneficial or is a disadvantage. Small class effects were inconsistent and varied significantly across schools in all grades indicating a small class by school interaction.

Computing Power of Tests of the Variance of Treatment Effects in Designs With Two Levels of Nesting.

Experiments that involve nested structures may assign treatment conditions either to entire groups (such as classrooms or schools) or individuals within groups (such as students). Although typically the interest in field experiments is in determining the significance of the overall treatment effect, it is equally important to examine the inconsistency of the treatment effect in different groups. This study provides methods for computing power of tests for the variability of treatment effects across level-2 and level-3 units in three-level designs, where, for example, students are nested within classrooms and classrooms are nested within schools and random assignment takes place at the first or the second level. The power computations take into account nesting effects at the second (e.g., classroom) and at the third (e.g., school) level as well as sample size effects (e.g., number of level-1 and level-2 units). The methods can also be applied to quasi-experimental studies that examine the significance of the variation of group differences in an outcome or associations between predictors and outcomes across level-2 and level-3 units.

The gender gap reloaded: are school characteristics linked to labor market performance?

This study examines the wage gender gap of young adults in the 1970s, 1980s, and 2000 in the US. Using quantile regression we estimate the gender gap across the entire wage distribution. We also study the importance of high school characteristics in predicting future labor market performance. We conduct analyses for three major racial/ethnic groups in the US: Whites, Blacks, and Hispanics, employing data from two rich longitudinal studies: NLS and NELS. Our results indicate that while some school characteristics are positive and significant predictors of future wages for Whites, they are less so for the two minority groups. We find significant wage gender disparities favoring men across all three surveys in the 1970s, 1980s, and 2000. The wage gender gap is more pronounced in higher paid jobs (90th quantile) for all groups, indicating the presence of a persistent and alarming "glass ceiling."