Repeated measures ANOVA and adjusted F-tests when sphericity is violated: which procedure is best?

Introduction: One-way repeated measures ANOVA requires sphericity. Research indicates that violation of this assumption has an important impact on Type I error. Although more advanced alternative procedures exist, most classical texts recommend the use of adjusted F-tests, which are frequently employed because they are intuitive, easy to apply, and available in most statistical software. Adjusted F-tests differ in the procedure used to estimate the corrective factor ε, the most common being the Greenhouse-Geisser (F-GG) and Huynh-Feldt (F-HF) adjustments. Although numerous studies have analyzed the robustness of these procedures, the results are inconsistent, thus highlighting the need for further research. Methods: The aim of this simulation study was to analyze the performance of the F-statistic, F-GG, and F-HF in terms of Type I error and power in one-way designs with normal data under a variety of conditions that may be encountered in real research practice. Values of ε were fixed according to the Greenhouse-Geisser procedure (ε^). We manipulated the number of repeated measures (3, 4, and 6) and sample size (from 10 to 300), with ε^ values ranging from the lower to its upper limit. Results: Overall, the results showed that the F-statistic becomes more liberal as sphericity violation increases, whereas both F-HF and F-GG control Type I error; of the two, F-GG is more conservative, especially with large values of ε^ and small samples. Discussion: If different statistical conclusions follow from application of the two tests, we recommend using F-GG for ε^ values below 0.60, and F-HF for ε^ values equal to or above 0.60.

Non-normal Data in Repeated Measures ANOVA: Impact on Type I Error and Power.

BACKGROUND: Repeated measures designs are commonly used in health and social sciences research. Although there are other, more advanced, statistical analyses, the F-statistic of repeated measures analysis of variance (RM-ANOVA) remains the most widely used procedure for analyzing differences in means. The impact of the violation of normality has been extensively studied for between-subjects ANOVA, but this is not the case for RM-ANOVA. Therefore, studies that extensively and systematically analyze the robustness of RM-ANOVA under the violation of normality are needed. This paper reports the results of two simulation studies aimed at analyzing the Type I error and power of RM-ANOVA when the normality assumption is violated but sphericity is fulfilled. METHOD: Study 1 considered 20 distributions, both known and unknown, and we manipulated the number of repeated measures (3, 4, 6, and 8) and sample size (from 10 to 300). Study 2 involved unequal distributions in each repeated measure. The distributions analyzed represent slight, moderate, and severe deviation from normality. RESULTS: Overall, the results show that the Type I error and power of the F-statistic are not altered by the violation of normality. CONCLUSIONS: RM-ANOVA is generally robust to non-normality when the sphericity assumption is met.

Report Quality of Generalized Linear Mixed Models in Psychology: A Systematic Review.

Generalized linear mixed models (GLMMs) estimate fixed and random effects and are especially useful when the dependent variable is binary, ordinal, count or quantitative but not normally distributed. They are also useful when the dependent variable involves repeated measures, since GLMMs can model autocorrelation. This study aimed to determine how and how often GLMMs are used in psychology and to summarize how the information about them is presented in published articles. Our focus in this respect was mainly on frequentist models. In order to review studies applying GLMMs in psychology we searched the Web of Science for articles published over the period 2014-2018. A total of 316 empirical articles were selected for trend study from 2014 to 2018. We then conducted a systematic review of 118 GLMM analyses from 80 empirical articles indexed in Journal Citation Reports during 2018 in order to evaluate report quality. Results showed that the use of GLMMs increased over time and that 86.4% of articles were published in first- or second-quartile journals. Although GLMMs have, in recent years, been increasingly used in psychology, most of the important information about them was not stated in the majority of articles. Report quality needs to be improved in line with current recommendations for the use of GLMMs.

Manipulating the Alpha Level Cannot Cure Significance Testing.

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p = 0.05 to p = 0.005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of 0.05, 0.01, 0.005, or anything else, is not acceptable.

Current Practices in Data Analysis Procedures in Psychology: What Has Changed?

This paper analyzes current practices in psychology in the use of research methods and data analysis procedures (DAP) and aims to determine whether researchers are now using more sophisticated and advanced DAP than were employed previously. We reviewed empirical research published recently in prominent journals from the USA and Europe corresponding to the main psychological categories of Journal Citation Reports and examined research methods, number of studies, number and type of DAP, and statistical package. The 288 papers reviewed used 663 different DAP. Experimental and correlational studies were the most prevalent, depending on the specific field of psychology. Two-thirds of the papers reported a single study, although those in journals with an experimental focus typically described more. The papers mainly used parametric tests for comparison and statistical techniques for analyzing relationships among variables. Regarding the former, the most frequently used procedure was ANOVA, with mixed factorial ANOVA being the most prevalent. A decline in the use of non-parametric analysis was observed in relation to previous research. Relationships among variables were most commonly examined using regression models, with hierarchical regression and mediation analysis being the most prevalent procedures. There was also a decline in the use of stepwise regression and an increase in the use of structural equation modeling, confirmatory factor analysis, and hierarchical linear modeling. Overall, the results show that recent empirical studies published in journals belonging to the main areas of psychology are employing more varied and advanced statistical techniques of greater computational complexity.

Effect of variance ratio on ANOVA robustness: Might 1.5 be the limit?

Inconsistencies in the research findings on F-test robustness to variance heterogeneity could be related to the lack of a standard criterion to assess robustness or to the different measures used to quantify heterogeneity. In the present paper we use Monte Carlo simulation to systematically examine the Type I error rate of F-test under heterogeneity. One-way, balanced, and unbalanced designs with monotonic patterns of variance were considered. Variance ratio (VR) was used as a measure of heterogeneity (1.5, 1.6, 1.7, 1.8, 2, 3, 5, and 9), the coefficient of sample size variation as a measure of inequality between group sizes (0.16, 0.33, and 0.50), and the correlation between variance and group size as an indicator of the pairing between them (1, .50, 0, -.50, and -1). Overall, the results suggest that in terms of Type I error a VR above 1.5 may be established as a rule of thumb for considering a potential threat to F-test robustness under heterogeneity with unequal sample sizes.

Non-normal data: Is ANOVA still a valid option?

BACKGROUND: The robustness of F-test to non-normality has been studied from the 1930s through to the present day. However, this extensive body of research has yielded contradictory results, there being evidence both for and against its robustness. This study provides a systematic examination of F-test robustness to violations of normality in terms of Type I error, considering a wide variety of distributions commonly found in the health and social sciences. METHOD: We conducted a Monte Carlo simulation study involving a design with three groups and several known and unknown distributions. The manipulated variables were: Equal and unequal group sample sizes; group sample size and total sample size; coefficient of sample size variation; shape of the distribution and equal or unequal shapes of the group distributions; and pairing of group size with the degree of contamination in the distribution. RESULTS: The results showed that in terms of Type I error the F-test was robust in 100% of the cases studied, independently of the manipulated conditions.

Non-normal Distributions Commonly Used in Health, Education, and Social Sciences: A Systematic Review.

Statistical analysis is crucial for research and the choice of analytical technique should take into account the specific distribution of data. Although the data obtained from health, educational, and social sciences research are often not normally distributed, there are very few studies detailing which distributions are most likely to represent data in these disciplines. The aim of this systematic review was to determine the frequency of appearance of the most common non-normal distributions in the health, educational, and social sciences. The search was carried out in the Web of Science database, from which we retrieved the abstracts of papers published between 2010 and 2015. The selection was made on the basis of the title and the abstract, and was performed independently by two reviewers. The inter-rater reliability for article selection was high (Cohen's kappa = 0.84), and agreement regarding the type of distribution reached 96.5%. A total of 262 abstracts were included in the final review. The distribution of the response variable was reported in 231 of these abstracts, while in the remaining 31 it was merely stated that the distribution was non-normal. In terms of their frequency of appearance, the most-common non-normal distributions can be ranked in descending order as follows: gamma, negative binomial, multinomial, binomial, lognormal, and exponential. In addition to identifying the distributions most commonly used in empirical studies these results will help researchers to decide which distributions should be included in simulation studies examining statistical procedures.

Sphericity estimation bias for repeated measures designs in simulation studies.

In this study, we explored the accuracy of sphericity estimation and analyzed how the sphericity of covariance matrices may be affected when the latter are derived from simulated data. We analyzed the consequences that normal and nonnormal data generated from an unstructured population covariance matrix-with low (ε = .57) and high (ε = .75) sphericity-can have on the sphericity of the matrix that is fitted to these data. To this end, data were generated for four types of distributions (normal, slightly skewed, moderately skewed, and severely skewed or log-normal), four sample sizes (very small, small, medium, and large), and four values of the within-subjects factor (K = 4, 6, 8, and 10). Normal data were generated using the Cholesky decomposition of the correlation matrix, whereas the Vale-Maurelli method was used to generate nonnormal data. The results indicate the extent to which sphericity is altered by recalculating the covariance matrix on the basis of simulated data. We concluded that bias is greater with spherical covariance matrices, nonnormal distributions, and small sample sizes, and that it increases in line with the value of K. An interaction was also observed between sample size and K: With very small samples, the observed bias was greater as the value of K increased.

The effect of skewness and kurtosis on the Kenward-Roger approximation when group distributions differ.

BACKGROUND: This study examined the independent effect of skewness and kurtosis on the robustness of the linear mixed model (LMM), with the Kenward-Roger (KR) procedure, when group distributions are different, sample sizes are small, and sphericity cannot be assumed. METHODS: A Monte Carlo simulation study considering a split-plot design involving three groups and four repeated measures was performed. RESULTS: The results showed that when group distributions are different, the effect of skewness on KR robustness is greater than that of kurtosis for the corresponding values. Furthermore, the pairings of skewness and kurtosis with group size were found to be relevant variables when applying this procedure. CONCLUSIONS: With sample sizes of 45 and 60, KR is a suitable option for analyzing data when the distributions are: (a) mesokurtic and not highly or extremely skewed, and (b) symmetric with different degrees of kurtosis. With total sample sizes of 30, it is adequate when group sizes are equal and the distributions are: (a) mesokurtic and slightly or moderately skewed, and sphericity is assumed; and (b) symmetric with a moderate or high/extreme violation of kurtosis. Alternative analyses should be considered when the distributions are highly or extremely skewed and samples sizes are small.

Should we rely on the Kenward-Roger approximation when using linear mixed models if the groups have different distributions?

The study explores the robustness to violations of normality and sphericity of linear mixed models when they are used with the Kenward-Roger procedure (KR) in split-plot designs in which the groups have different distributions and sample sizes are small. The focus is on examining the effect of skewness and kurtosis. To this end, a Monte Carlo simulation study was carried out, involving a split-plot design with three levels of the between-subjects grouping factor and four levels of the within-subjects factor. The results show that: (1) the violation of the sphericity assumption did not affect KR robustness when the assumption of normality was not fulfilled; (2) the robustness of the KR procedure decreased as skewness in the distributions increased, there being no strong effect of kurtosis; and (3) the type of pairing between kurtosis and group size was shown to be a relevant variable to consider when using this procedure, especially when pairing is positive (i.e., when the largest group is associated with the largest value of the kurtosis coefficient and the smallest group with its smallest value). The KR procedure can be a good option for analysing repeated-measures data when the groups have different distributions, provided the total sample sizes are 45 or larger and the data are not highly or extremely skewed.

The effect of skewness and kurtosis on the robustness of linear mixed models.

This study analyzes the robustness of the linear mixed model (LMM) with the Kenward-Roger (KR) procedure to violations of normality and sphericity when used in split-plot designs with small sample sizes. Specifically, it explores the independent effect of skewness and kurtosis on KR robustness for the values of skewness and kurtosis coefficients that are most frequently found in psychological and educational research data. To this end, a Monte Carlo simulation study was designed, considering a split-plot design with three levels of the between-subjects grouping factor and four levels of the within-subjects factor. Robustness is assessed in terms of the probability of type I error. The results showed that (1) the robustness of the KR procedure does not differ as a function of the violation or satisfaction of the sphericity assumption when small samples are used; (2) the LMM with KR can be a good option for analyzing total sample sizes of 45 or larger when their distributions are normal, slightly or moderately skewed, and with different degrees of kurtosis violation; (3) the effect of skewness on the robustness of the LMM with KR is greater than the corresponding effect of kurtosis for common values; and (4) when data are not normal and the total sample size is 30, the procedure is not robust. Alternative analyses should be performed when the total sample size is 30.

[Effect of the violation of normality and sphericity in the linear mixed model in split-plot designs]. / Efecto de la violación de la normalidad y esfericidad en el modelo lineal mixto en diseños split-plot.

This study aimed to evaluate the robustness of the linear mixed model, with the Kenward-Roger correction for degrees of freedom, when implemented in SAS PROC MIXED, using split-plot designs with small sample sizes. A Monte Carlo simulation design involving three groups and four repeated measures was used, assuming an unstructured covariance matrix to generate the data. The study variables were: sphericity, with epsilon values of 0.75 and 0.57; group sizes, equal or unequal; and shape of the distribution. As regards the latter, non-normal distributions were introduced, combining different values of kurtosis in each group. In the case of unbalanced designs, the effect of pairing (positive or negative) the degree of kurtosis with group size was also analysed. The results show that the Kenward-Roger procedure is liberal, particularly for the interaction effect, under certain conditions in which normality is violated. The relationship between the values of kurtosis in the groups and the pairing of kurtosis with group size are found to be relevant variables to take into account when applying this procedure.

Using the linear mixed model to analyze nonnormal data distributions in longitudinal designs.

Using a Monte Carlo simulation and the Kenward-Roger (KR) correction for degrees of freedom, in this article we analyzed the application of the linear mixed model (LMM) to a mixed repeated measures design. The LMM was first used to select the covariance structure with three types of data distribution: normal, exponential, and log-normal. This showed that, with homogeneous between-groups covariance and when the distribution was normal, the covariance structure with the best fit was the unstructured population matrix. However, with heterogeneous between-groups covariance and when the pairing between covariance matrices and group sizes was null, the best fit was shown by the between-subjects heterogeneous unstructured population matrix, which was the case for all of the distributions analyzed. By contrast, with positive or negative pairings, the within-subjects and between-subjects heterogeneous first-order autoregressive structure produced the best fit. In the second stage of the study, the robustness of the LMM was tested. This showed that the KR method provided adequate control of Type I error rates for the time effect with normally distributed data. However, as skewness increased-as occurs, for example, in the log-normal distribution-the robustness of KR was null, especially when the assumption of sphericity was violated. As regards the influence of kurtosis, the analysis showed that the degree of robustness increased in line with the amount of kurtosis.

Drug awareness in adolescents attending a mental health service: analysis of longitudinal data.

One of the procedures used most recently with longitudinal data is linear mixed models. In the context of health research the increasing number of studies that now use these models bears witness to the growing interest in this type of analysis. This paper describes the application of linear mixed models to a longitudinal study of a sample of Spanish adolescents attending a mental health service, the aim being to investigate their knowledge about the consumption of alcohol and other drugs. More specifically, the main objective was to compare the efficacy of a motivational interviewing programme with a standard approach to drug awareness. The models used to analyse the overall indicator of drug awareness were as follows: (a) unconditional linear growth curve model; (b) growth model with subject-associated variables; and (c) individual curve model with predictive variables. The results showed that awareness increased over time and that the variable 'schooling years' explained part of the between-subjects variation. The effect of motivational interviewing was also significant.

Regression-based techniques for statistical decision making in single-case designs.

The present study evaluates the performance of four methods for estimating regression coefficients used to make statistical decisions about intervention effectiveness in single-case designs. Ordinary least square estimation is compared to two correction techniques dealing with general trend and a procedure that eliminates autocorrelation whenever it is present. Type I error rates and statistical power are studied for experimental conditions defined by the presence or absence of treatment effect (change in level or in slope), general trend, and serial dependence. The results show that empirical Type I error rates do not approach the nominal ones in the presence of autocorrelation or general trend when ordinary and generalized least squares are applied. The techniques controlling trend show lower false alarm rates, but prove to be insufficiently sensitive to existing treatment effects. Consequently, the use of the statistical significance of the regression coefficients for detecting treatment effects is not recommended for short data series.

General linear mixed model for analysing longitudinal data in developmental research.

Many areas of psychological, social, and health research are characterised by hierarchically structured data. Growth curves are usually represented by means of a two-level hierarchical structure in which observations are the first-level units nested within subjects, the second-level units. With data such as these, the best option for analysis is the general linear mixed model, which can be used even with longitudinal data series in which intervals are not constant or for which over the passage of time there is loss of data. In this paper an overview is given of the general linear mixed model approach to the analysis of longitudinal data in developmental research. The advantages of this model in comparison with the traditional approaches for analysing longitudinal data are shown, emphasising the usefulness of modelling the covariance structure properly to achieve a precise estimation of the parameters of the model.

[Information and communication technologies in teaching applied and experimental designs]. / Tecnologías de la información y comunicación en la enseñanza de diseños experimentales y aplicados.

In this work, an innovative teaching model applied to methodological contents in psychology is presented. The proposed didactic model includes Information and Communication Technologies (ICT), such as CD-ROMs, web sites and Internet. These resources complement class attendance. In the classes the students are informed, guided and oriented so that they are able to obtain information and reorganize it in a coherent way. The aim of this article is to find out the students' learning preferences and estimate the incorporation of ICT, by means of the ETIM (Evaluation of Teaching Innovation Model) questionnaire. The results show that the students are aware of the need to consult other materials and that ICT helps students to understand the subject from various perspectives. In this way, the students become more autonomous in acquiring learning results.

Evaluating effects in short time series: alternative models of analysis.

In the applied context, short time-series designs are suitable to evaluate a treatment effect. These designs present serious problems given autocorrelation among data and the small number of observations involved. This paper describes analytic procedures that have been applied to data from short time series, and an alternative which is a new version of the generalized least squares method to simplify estimation of the error covariance matrix. Using the results of a simulation study and assuming a stationary first-order autoregressive model, it is proposed that the original observations and the design matrix be transformed by means of the square root or Cholesky factor of the inverse of the covariance matrix. This provides a solution to the problem of estimating the parameters of the error covariance matrix. Finally, the results of the simulation study obtained using the proposed generalized least squares method are compared with those obtained by the ordinary least squares approach. The probability of Type I error associated with the proposed method is close to the nominal value for all values of rho1 and n investigated, especially for positive values of rho1. The proposed generalized least squares method corrects the effect of autocorrelation on the test's power.