R-squared Measures for Multilevel Models with Three or More Levels.

Applications of multilevel models (MLMs) with three or more levels have increased alongside expanding software capability and dataset availability. Though researchers often express interest in R-squared measures as effect sizes for MLMs, R-squareds previously proposed for MLMs with three or more levels cover a limited subset of choices for how to quantify explained variance in these models. Additionally, analytic relationships between total and level-specific versions of MLM R-squared measures have not been clarified, despite such relationships becoming increasingly important to understand when there are more levels. Furthermore, the impact of predictor centering strategy on R-squared computation and interpretation has not been explicated for MLMs with any number of levels. To fill these gaps, we extend the Rights and Sterba two-level MLM R-squared framework to three or more levels, providing a general set of measures that includes preexisting three-level measures as special cases and yields additional results not obtainable from existing measures. We mathematically and pedagogically relate total and level-specific R-squareds, and show how all total and level-specific R-squared measures in our framework can be computed under any centering strategy. Finally, we provide and empirically demonstrate software (available in the r2mlm R package) to compute measures and graphically depict results.

On the Common but Problematic Specification of Conflated Random Slopes in Multilevel Models.

For multilevel models (MLMs) with fixed slopes, it has been widely recognized that a level-1 variable can have distinct between-cluster and within-cluster fixed effects, and that failing to disaggregate these effects yields a conflated, uninterpretable fixed effect. For MLMs with random slopes, however, we clarify that two different types of slope conflation can occur: that of the fixed component (termed fixed conflation) and that of the random component (termed random conflation). The latter is rarely recognized and not well understood. Here we explain that a model commonly used to disaggregate the fixed component-the contextual effect model with random slopes-troublingly still yields a conflated random component. Negative consequences of such random conflation have not been demonstrated. Here we show that they include erroneous interpretation and inferences about the substantively important extent of between-cluster differences in slopes, including either underestimating or overestimating such slope heterogeneity. Furthermore, we show that this random conflation can yield inappropriate standard errors for fixed effects. To aid researchers in practice, we delineate which types of random slope specifications yield an unconflated random component. We demonstrate the advantages of these unconflated models in terms of estimating and testing random slope variance (i.e., improved power, Type I error, and bias) and in terms of standard error estimation for fixed effects (i.e., more accurate standard errors), and make recommendations for which specifications to use for particular research purposes.

Latent profile analyses of disordered eating behaviors and nonsuicidal self-injury among Vietnamese adolescents.

OBJECTIVE: Comorbidity of disordered eating (DE) behaviors and nonsuicidal self-injury (NSSI) is linked to increased functional impairment. The present study identified subtypes of DE and NSSI comorbidity in a non-Western, low- and middle-income country where there has been particularly little research in this area. METHOD: Latent profile analyses (LPA) were conducted to identify patterns of subgroup comorbidity in self-reported DE behaviors and NSSI behaviors, in a sample of Vietnamese high-school students (N = 1451, 51% female). Parallel-process LPA was used to predict NSSI subgroup membership from DE subgroup membership. RESULTS: A seven-class LPA model was identified for DE: (1) Low frequencies of all DE behaviors; (2) Frequent levels of all DE behaviors; (3) Frequent fasting and purging behaviors; (4) Frequent binge eating only; (5) Moderate binge eating; (6) Moderate fasting and purging behaviors; (7) Exercise and fasting. A two-class (high versus low) model was identified for NSSI. Odds of membership in the high-NSSI class were significantly increased for all DE classes except the class characterized by moderate binge eating. Odds of belonging to the high-NSSI class were highest for those in DE classes involving purging behaviors. Males predominated in classes characterized by high behavior dysregulation. DISCUSSION: DE and NSSI represent important clinical concerns among Vietnamese youth in urban contexts, and males may show particular vulnerability for certain risk behaviors. The association of classes of DE behaviors to NSSI was particularly strong for classes that involved purging, highlighting the importance of this specific behavior for understanding comorbidity of DE and NSSI. PUBLIC SIGNIFICANCE: Disordered eating and nonsuicidal self-injury (NSSI; hurting oneself without intending to die) commonly co-occur. There has been less research on disordered eating and NSSI among individuals from non-Western or low/middle-income countries. We examined patterns of these behaviors in a school-based sample of Vietnamese adolescents. We found that patterns of disordered eating distinguished by purging showed particular links to NSSI, which could have implications for screening and treatment.

Closing the Word-Problem Achievement Gap in First Grade: Schema-Based Word-Problem Intervention with Embedded Language Comprehension Instruction.

The main purpose of this study was to test the effects of word-problem intervention, with versus without embedded language comprehension instruction, on at-risk 1st graders' word-problem performance. We also isolated the need for a structured approach to word-problem intervention and tested the efficacy of schema-based instruction at 1st grade. Children (n=391; mean age = 6.53, SD = 0.32) were randomly assigned to 4 conditions: schema-based word-problem intervention with embedded language instruction, the same word-problem intervention but without language comprehension instruction, structured number knowledge intervention without a structured word-problem component, and a control group. Each intervention included 45 30-min sessions. Multilevel models, accounting for classroom and school effects, revealed the efficacy of schema-based word-problem intervention at 1st grade, with both word-problem conditions outperforming the number knowledge condition and the control group. Yet, word-problem performance was significantly stronger for the schema-based condition with embedded language comprehension instruction compared to the schema-based condition without language comprehension instruction. Number knowledge intervention conveyed no word-problem advantage over the control group, even though all 3 intervention conditions outperformed the control group on arithmetic. Results demonstrate the importance of a structured approach to word-problem intervention; the efficacy of schema-based instruction at 1st grade; and the added value of language comprehension instruction within word-problem intervention. Results also provide causal evidence on the role of language comprehension in word-problem solving.

Effect size measures for longitudinal growth analyses: Extending a framework of multilevel model R-squareds to accommodate heteroscedasticity, autocorrelation, nonlinearity, and alternative centering strategies.

Developmental researchers commonly utilize multilevel models (MLMs) to describe and predict individual differences in change over time. In such growth model applications, researchers have been widely encouraged to supplement reporting of statistical significance with measures of effect size, such as R-squareds (R2 ) that convey variance explained by terms in the model. An integrative framework for computing R-squareds in MLMs with random intercepts and/or slopes was recently introduced by Rights and Sterba and it subsumed pre-existing MLM R-squareds as special cases. However, this work focused on cross-sectional applications, and hence did not address how the computation and interpretation of MLM R-squareds are affected by modeling considerations typically arising in longitudinal settings: (a) alternative centering choices for time (e.g., centering-at-a-constant vs. person-mean-centering), (b) nonlinear effects of predictors such as time, (c) heteroscedastic level-1 errors and/or (d) autocorrelated level-1 errors. This paper addresses these gaps by extending the Rights and Sterba R-squared framework to longitudinal contexts. We: (a) provide a full framework of total and level-specific R-squared measures for MLMs that utilize any type of centering, and contrast these with Rights and Sterba's measures assuming cluster-mean-centering, (b) explain and derive which measures are applicable for MLMs with nonlinear terms, and extend the R-squared computation to accommodate (c) heteroscedastic and/or (d) autocorrelated errors. Additionally, we show how to use differences in R-squared (ΔR2 ) measures between growth models (adding, for instance, time-varying covariates as level-1 predictors or time-invariant covariates as level-2 predictors) to obtain effects sizes for individual terms. We provide R software (r2MLMlong) and a running pedagogical example analyzing growth in adolescent self-efficacy to illustrate these methodological developments. With these developments, researchers will have greater ability to consider effect size when analyzing and predicting change using MLMs.

New Recommendations on the Use of R-Squared Differences in Multilevel Model Comparisons.

When comparing multilevel models (MLMs) differing in fixed and/or random effects, researchers have had continuing interest in using R-squared differences to communicate effect size and importance of included terms. However, there has been longstanding confusion regarding which R-squared difference measures should be used for which kind of MLM comparisons. Furthermore, several limitations of recent studies on R-squared differences in MLM have led to misleading or incomplete recommendations for practice. These limitations include computing measures that are by definition incapable of detecting a particular type of added term, considering only a subset of the broader class of available R-squared difference measures, and incorrectly defining what a given R-squared difference measure quantifies. The purpose of this paper is to elucidate and resolve these issues. To do so, we define a more general set of total, within-cluster, and between-cluster R-squared difference measures than previously considered in MLM comparisons and give researchers concrete step-by-step procedures for identifying which measure is relevant to which model comparison. We supply simulated and analytic demonstrations of limitations of previous MLM studies on R-squared differences and show how application of our step-by-step procedures and general set of measures overcomes each. Additionally, we provide and illustrate graphical tools and software allowing researchers to automatically compute and visualize our set of measures in an integrated manner. We conclude with recommendations, as well as extensions involving (a) how our framework relates to and can be used to obtain pseudo-R-squareds, and (b) how our framework can accommodate both simultaneous and hierarchical model-building approaches.

Problems with Rationales for Parceling that Fail to Consider Parcel-Allocation Variability.

In structural equation modeling applications, parcels-averages or sums of subsets of item scores-are often used as indicators of latent constructs. Parcel-allocation variability (PAV) is variability in results that arises within sample across alternative item-to-parcel allocations. PAV can manifest in all results of a parcel-level model (e.g., model fit, parameter estimates, standard errors, and inferential decisions). It is a source of uncertainty in parcel-level model results that can be investigated, reported, and accounted for. Failing to do so raises representativeness and replicability concerns. However, in recent methodological literature (Cole, Perkins, & Zelkowitz, 2016 ; Little, Rhemtulla, Gibson, & Shoemann, 2013 ; Marsh, Ludtke, Nagengast, Morin, & von Davier, 2013 ; Rhemtulla, 2016 ) parceling has been justified and recommended in several situations without quantifying or accounting for PAV. In this article, we explain and demonstrate problems with these rationales. Overall, we find that: (1) using a purposive parceling algorithm for a multidimensional construct does not avoid PAV; (2) passing a test of unidimensionality of the item-level model need not avoid PAV; and (3) a desire to improve power for detecting structural misspecification does not warrant parceling without addressing PAV; we show how to simultaneously avoid PAV and obtain even higher power by comparing item-level models differing in structural constraints. Implications for practice are discussed.

Anxiety and Depression in Children of Depressed Parents: Dynamics of Change in a Preventive Intervention.

The current study examined effects of a preventive intervention on patterns of change in symptoms of anxiety and depression in a sample of children of depressed parents. Parents with a history of depression (N = 180) and their children (N = 242; 50% female; Mage = 11.38; 74% Euro-American) enrolled in an intervention to prevent psychopathology in youth. Families were randomized to a family group cognitive behavioral intervention (FGCB) or a written information (WI) control condition. Parents and youth completed the Child Behavior Checklist and Youth Self Report at baseline, 6-, 12-, 18-, and 24-month follow up. Youth in the FGCB intervention reported significantly greater declines in symptoms of both anxiety and depression at 6, 12, and 18 months compared to youth in the WI condition. Youth with higher baseline levels of each symptom (e.g., anxiety) reported greater declines in the other symptom (e.g., depression) from 0 to 6 months in the FGCB intervention only. Changes in anxiety symptoms from 0 to 6 months predicted different patterns of subsequent changes in depressive symptoms from 6 to 12 months for the two conditions, such that declines in anxiety preceded and predicted greater declines in depression for FGCB youth but lesser increases in depression for WI youth. Findings inform transdiagnostic approaches to preventive interventions for at-risk youth, suggesting that both initial symptom levels and initial magnitude of change in symptoms are important to understand subsequent patterns of change in response to intervention.

Explicating the Conditions Under Which Multilevel Multiple Imputation Mitigates Bias Resulting from Random Coefficient-Dependent Missing Longitudinal Data.

Random coefficient-dependent (RCD) missingness is a non-ignorable mechanism through which missing data can arise in longitudinal designs. RCD, for which we cannot test, is a problematic form of missingness that occurs if subject-specific random effects correlate with propensity for missingness or dropout. Particularly when covariate missingness is a problem, investigators typically handle missing longitudinal data by using single-level multiple imputation procedures implemented with long-format data, which ignores within-person dependency entirely, or implemented with wide-format (i.e., multivariate) data, which ignores some aspects of within-person dependency. When either of these standard approaches to handling missing longitudinal data is used, RCD missingness leads to parameter bias and incorrect inference. We explain why multilevel multiple imputation (MMI) should alleviate bias induced by a RCD missing data mechanism under conditions that contribute to stronger determinacy of random coefficients. We evaluate our hypothesis with a simulation study. Three design factors are considered: intraclass correlation (ICC; ranging from .25 to .75), number of waves (ranging from 4 to 8), and percent of missing data (ranging from 20 to 50%). We find that MMI greatly outperforms the single-level wide-format (multivariate) method for imputation under a RCD mechanism. For the MMI analyses, bias was most alleviated when the ICC is high, there were more waves of data, and when there was less missing data. Practical recommendations for handling longitudinal missing data are suggested.

Partially nested designs in psychotherapy trials: A review of modeling developments.

OBJECTIVE: Individually-randomized psychotherapy trials are often partially nested. For instance, individuals assigned to a treatment arm may be clustered into therapy groups for purposes of treatment administration, whereas individuals assigned to a wait-list control are unclustered. The past several years have seen rapid expansion and investigation of methods for analyzing partially nested data. Yet partial nesting often remains ignored in psychotherapy trials. METHODS: This review integrates and disseminates developments in the analysis of partially nested data that are particularly relevant for psychotherapy researchers. RESULTS: First, we differentiate among alternative partially nested designs. Then, we present adaptations of multilevel model specifications that accommodate each design. Next, we address how moderation by treatment as well as mediation of the treatment effect can be investigated in partially nested designs. Model fitting results, annotated software syntax, and illustrative data sets are provided and key methodological issues are discussed. CONCLUSIONS: We emphasize that cluster-level variability in the treatment arm need not be considered a nuisance; it can be modeled to yield insights about the treatment process.

Cautions on the Use of Multiple Imputation When Selecting Between Latent Categorical versus Continuous Models for Psychological Constructs.

Clinical psychology researchers studying adolescents and young adults long have been interested in characterizing the latent categorical (classes/profiles) versus continuous (factors) nature of psychological syndromes. To inform this debate, researchers sometimes compare the fit of finite mixture versus factor analysis models to symptom data. This study explains and evaluates how missing data handling methods can impact results of this important model fit comparison. Via simulation, we assess three missing data-handling methods previously recommended to researchers fitting these models: multiple imputation using a saturated multivariate normal imputation model, multiple imputation using a hypothesized model, or full information maximum likelihood using the EM algorithm (FIML-EM). Results show that, under certain conditions, the method used to handle missing data can interfere with clinical psychologists' ability to accurately discriminate latent classes from continua. For instance, certain imputation methods increase the chance of selecting latent continua when latent classes truly exist. FIML-EM performed best overall. Recommendations for practice are discussed.

Accounting for Parcel-Allocation Variability in Practice: Combining Sources of Uncertainty and Choosing the Number of Allocations.

Item parceling remains widely used under conditions that can lead to parcel-allocation variability in results. Hence, researchers may be interested in quantifying and accounting for parcel-allocation variability within sample. To do so in practice, three key issues need to be addressed. First, how can we combine sources of uncertainty arising from sampling variability and parcel-allocation variability when drawing inferences about parameters in structural equation models? Second, on what basis can we choose the number of repeated item-to-parcel allocations within sample? Third, how can we diagnose and report proportions of total variability per estimate arising due to parcel-allocation variability versus sampling variability? This article addresses these three methodological issues. Developments are illustrated using simulated and empirical examples, and software for implementing them is provided.

Handling Missing Covariates in Conditional Mixture Models Under Missing at Random Assumptions.

Mixture modeling is a popular method that accounts for unobserved population heterogeneity using multiple latent classes that differ in response patterns. Psychologists use conditional mixture models to incorporate covariates into between-class and/or within-class regressions. Although psychologists often have missing covariate data, conditional mixtures are currently fit with a conditional likelihood, treating covariates as fixed and fully observed. Under this exogenous-x approach, missing covariates are handled primarily via listwise deletion. This sacrifices efficiency and does not allow missingness to depend on observed outcomes. Here we describe a modified joint likelihood approach that (a) allows inference about parameters of the exogenous-x conditional mixture even with nonnormal covariates, unlike a conventional multivariate mixture; (b) retains all cases under missing at random assumptions; (c) yields lower bias and higher efficiency than the exogenous-x approach under a variety of conditions with missing covariates; and (d) is straightforward to implement in available commercial software. The proposed approach is illustrated with an empirical analysis predicting membership in latent classes of conduct problems. Recommendations for practice are discussed.

Structural Equation Modeling Approaches for Analyzing Partially Nested Data.

Study designs involving clustering in some study arms, but not all study arms, are common in clinical treatment-outcome and educational settings. For instance, in a treatment arm, persons may be nested in therapy groups, whereas in a control arm there are no groups. Methodological approaches for handling such partially nested designs have recently been developed in a multilevel modeling framework (MLM-PN) and have proved very useful. We introduce two alternative structural equation modeling (SEM) approaches for analyzing partially nested data: a multivariate single-level SEM (SSEM-PN) and a multiple-arm multilevel SEM (MSEM-PN). We show how SSEM-PN and MSEM-PN can produce results equivalent to existing MLM-PNs and can be extended to flexibly accommodate several modeling features that are difficult or impossible to handle in MLM-PNs. For instance, using an SSEM-PN or MSEM-PN, it is possible to specify complex structural models involving cluster-level outcomes, obtain absolute model fit, decompose person-level predictor effects in the treatment arm using latent cluster means, and include traditional factors as predictors/outcomes. Importantly, implementation of such features for partially nested designs differs from that for fully nested designs. An empirical example involving a partially nested depression intervention combines several of these features in an analysis of interest for treatment-outcome studies.

Longitudinal dimensionality of adolescent psychopathology: testing the differentiation hypothesis.

BACKGROUND: The differentiation hypothesis posits that the underlying liability distribution for psychopathology is of low dimensionality in young children, inflating diagnostic comorbidity rates, but increases in dimensionality with age as latent syndromes become less correlated. This hypothesis has not been adequately tested with longitudinal psychiatric symptom data. METHODS: Confirmatory factor analyses of DSM-IV symptoms from seven common Axis I syndromes--major depression, generalized anxiety, separation anxiety, social anxiety, attention deficient hyperactivity, conduct, and oppositional defiant disorders--were conducted longitudinally, from ages 9 to 16, using the general-population Great Smoky Mountains Study sample. RESULTS: An eight-syndrome model fit well at all ages, and in both genders. It included social anxiety, separation anxiety, oppositional defiant, and conduct syndromes, along with a multidimensional attention deficit-hyperactivity syndrome (i.e., inattention, hyperactivity, and impulsivity) and a unidimensional major depression/generalized anxiety syndrome. A high degree of measurement invariance across age was found for all syndromes, except for major depression/generalized anxiety. Major depression and generalized anxiety syndromes slightly diverged at age 14-16, when they also began to explain more symptom variance. Additionally, correlations between some emotional and disruptive syndromes showed slight differentiation. CONCLUSIONS: Marked developmental differentiation of psychopathology, as implied by the orthogenetic principle, is not a prominent cause of preadolescent and adolescent psychiatric comorbidity.

Matching method with theory in person-oriented developmental psychopathology research.

The person-oriented approach seeks to match theories and methods that portray development as a holistic, highly interactional, and individualized process. Over the past decade, this approach has gained popularity in developmental psychopathology research, particularly as model-based varieties of person-oriented methods have emerged. Although these methods allow some principles of person-oriented theory to be tested, little attention has been paid to the fact that these methods cannot test other principles, and may actually be inconsistent with certain principles. Lacking clarification regarding which aspects of person-oriented theory are testable under which person-oriented methods, assumptions of the methods have sometimes been presented as testable hypotheses or interpreted as affirming the theory. This general blurring of the line between person-oriented theory and method has even led to the occasional perception that the method is the theory and vice versa. We review assumptions, strengths, and limitations of model-based person-oriented methods, clarifying which theoretical principles they can test and the compromises and trade-offs required to do so.

Alternative Model-Based and Design-Based Frameworks for Inference From Samples to Populations: From Polarization to Integration.

A model-based framework, due originally to R. A. Fisher, and a design-based framework, due originally to J. Neyman, offer alternative mechanisms for inference from samples to populations. We show how these frameworks can utilize different types of samples (nonrandom or random vs. only random) and allow different kinds of inference (descriptive vs. analytic) to different kinds of populations (finite vs. infinite). We describe the extent of each framework's implementation in observational psychology research. After clarifying some important limitations of each framework, we describe how these limitations are overcome by a newer hybrid model/design-based inferential framework. This hybrid framework allows both kinds of inference to both kinds of populations, given a random sample. We illustrate implementation of the hybrid framework using the High School and Beyond data set.

Quantifying explained variance in multilevel models: An integrative framework for defining R-squared measures.

Researchers often mention the utility and need for R-squared measures of explained variance for multilevel models (MLMs). Although this topic has been addressed by methodologists, the MLM R-squared literature suffers from several shortcomings: (a) analytic relationships among existing measures have not been established so measures equivalent in the population have been redeveloped 2 or 3 times; (b) a completely full partitioning of variance has not been used to create measures, leading to gaps in the availability of measures to address key substantive questions; (c) a unifying approach to interpreting and choosing among measures has not been provided, leading to researchers' difficulty with implementation; and (d) software has inconsistently and infrequently incorporated available measures. We address these issues with the following contributions. We develop an integrative framework of R-squared measures for MLMs with random intercepts and/or slopes based on a completely full decomposition of variance. We analytically relate 10 existing measures from different disciplines as special cases of 5 measures from our framework. We show how our framework fills gaps by supplying additional total and level-specific measures that answer new substantive research questions. To facilitate interpretation, we provide a novel and integrative graphical representation of all the measures in the framework; we use it to demonstrate limitations of current reporting practices for MLM R-squareds, as well as benefits of considering multiple measures from the framework in juxtaposition. We supply and empirically illustrate an R function, r2MLM, that computes all measures in our framework to help researchers in considering effect size and conveying practical significance. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Evaluating Group-Based Interventions When Control Participants Are Ungrouped.

Individually randomized treatments are often administered within a group setting. As a consequence, outcomes for treated individuals may be correlated due to provider effects, common experiences within the group, and/or informal processes of socialization. In contrast, it is often reasonable to regard outcomes for control participants as independent, given that these individuals are not placed into groups. Although this kind of design is common in intervention research, the statistical models applied to evaluate the treatment effects are usually inconsistent with the resulting data structure, potentially leading to biased inferences. This article presents an alternative model that explicitly accounts for the fact that only treated participants are grouped. In addition to providing a useful test of the overall treatment effect, this approach also permits one to formally determine the extent to which treatment effects vary over treatment groups and whether there is evidence that individuals within treatment groups become similar to one another. This strategy is demonstrated with data from the Reconnecting Youth program for high school students at risk of school failure and behavioral disorders.

A framework of R-squared measures for single-level and multilevel regression mixture models.

Psychologists commonly apply regression mixture models in single-level (i.e., unclustered) and multilevel (i.e., clustered) data analysis contexts. Though researchers applying nonmixture regression models typically report R-squared measures of explained variance, there has been no general treatment of R-squared measures for single-level and multilevel regression mixtures. Consequently, it is common for researchers to summarize results of a fitted regression mixture by simply reporting class-specific regression coefficients and their associated p values, rather than considering measures of effect size. In this article, we fill this gap by providing an integrative framework of R-squared measures for single-level regression mixture models and multilevel regression mixture models (with classes at Level-2 or both levels). Specifically, we describe 11 R-squared measures that are distinguished based on what the researcher chooses to consider as outcome variance and what sources the researcher chooses to contribute to predicted variance. We relate these measures analytically and through simulated graphical illustrations. Further, we demonstrate how these R-squared measures can be decomposed in novel ways into substantively meaningful sources of explained variance. We describe and demonstrate new software tools to allow researchers to compute these R-squared measures and decompositions in practice. Using 2 empirical examples, we show how researchers can answer distinct substantive questions with each measure and can gain insights by interpreting the set of measures in juxtaposition to each other. (PsycINFO Database Record