The Case for Use of Simple Difference Scores to Test the Significance of Differences in Mean Rates of Change in Controlled Repeated Measurements Designs.

A previous Monte Carlo study examined the relative powers of several simple and more complex procedures for testing the significance of difference in mean rates of change in a controlled, longitudinal, treatment evaluation study. Results revealed that the relative powers depended on the correlation structure of the simulated repeated measurements. Tests on dropout-weighted linear slope coefficients fitted to all of the available measurements for each participant were found to provide superior power in the presence of compound symmetry (CS), but tests of significance applied to simple baseline-to-endpoint difference scores provided superior power in the presence of a strongly autoregressive (AR) correlation structure. Type I error rates appeared in an acceptable range for both of those analyses. Insofar as the previous study considered only two widely disparate correlation structures, the present work was undertaken to examine where along a continuum of correlation structures lying between strongly AR and CS the power balance shifts from favoring the simple endpoint difference-score analysis to favoring a regression analysis that utilizes all of the available repeated measurements for each participant. With power calculated from the relative frequencies of rejecting Ho at different levels of autoregression, the results indicate superior power for the simple endpoint analysis across more than half the distance from strongly AR to CS. To examine replicability of the simulation results using real data from a previously published study, sampling with replacement from a double-blind controlled study examining the treatment of depression was used to create a Monte Carlo data set from which power could be calculated from relative frequencies of rejecting Ho.

Rule-of-thumb adjustment of sample sizes to accommodate dropouts in a two-stage analysis of repeated measurements.

Recent contributions to the statistical literature have provided elegant model-based solutions to the problem of estimating sample sizes for testing the significance of differences in mean rates of change across repeated measures in controlled longitudinal studies with differentially correlated error and missing data due to dropouts. However, the mathematical complexity and model specificity of these solutions make them generally inaccessible to most applied researchers who actually design and undertake treatment evaluation research in psychiatry. In contrast, this article relies on a simple two-stage analysis in which dropout-weighted slope coefficients fitted to the available repeated measurements for each subject separately serve as the dependent variable for a familiar ANCOVA test of significance for differences in mean rates of change. This article is about how a sample of size that is estimated or calculated to provide desired power for testing that hypothesis without considering dropouts can be adjusted appropriately to take dropouts into account. Empirical results support the conclusion that, whatever reasonable level of power would be provided by a given sample size in the absence of dropouts, essentially the same power can be realized in the presence of dropouts simply by adding to the original dropout-free sample size the number of subjects who would be expected to drop from a sample of that original size under conditions of the proposed study.

A two-stage analysis of repeated measurements with dropouts and/or intermittent missing data.

This article is about a simple two-stage analysis that utilizes slope coefficients as the dependent variable for testing the significance of difference in mean rates of change in repeated measurement designs with missing data. The ANCOVA test on the doubly weighted slope coefficients provides power comparable to that of more complex maximum likelihood procedures when data are missing completely at random, requires fewer assumptions and is more generally applicable under realistic nonrandom dropout conditions, and most importantly can be readily understood and explained by those who actually do most controlled clinical research.

Lower effectiveness of divalproex versus valproic acid in a prospective, quasi-experimental clinical trial involving 9,260 psychiatric admissions.

OBJECTIVE: The authors examined clinical differences between divalproex sodium and generic immediate-release valproic acid. METHOD: This 6-year prospective, quasi-experimental clinical trial compared the effectiveness and tolerability of divalproex and valproic acid. The dependent variables were length of hospital stay, rehospitalization rate, and adverse drug reactions in 9,260 psychiatric admissions. RESULTS: Inpatients who initially received divalproex sodium had a 32.7% longer hospital stay and 3.8% higher readmission rate than did patients who initially received valproic acid. Initial treatment with divalproex prolonged length of stay by 30.3% in patients treated with divalproex and valproic acid during different admissions. After other variables were controlled by multiway analysis of variance, the hospital stay of patients who continued the initial medication was 15.2% longer (2.0 days) for divalproex than valproic acid. Switching medications was more common for valproic acid, partly because of study design. Medication intolerance occurred in approximately 6.4% more patients taking valproic acid than divalproex. However, switching from valproic acid to divalproex did not significantly prolong length of stay, over that for continuous divalproex, or increase the rehospitalization rate. CONCLUSIONS: Lower peak valproate concentrations with divalproex sodium may have enhanced tolerability but may also explain the lower effectiveness. Extended-release divalproex could lower effectiveness further and require higher doses. Thus, inpatients are better served by beginning with generic valproic acid and by changing to delayed-release divalproex only if intolerance occurs. This would save up to one-third of inpatient costs and two-thirds of a billion dollars yearly in medication costs.

Use of resampling to select among alternative error structure specifications for GLMM analyses of repeated measurements.

Autocorrelated error and missing data due to dropouts have fostered interest in the flexible general linear mixed model (GLMM) procedures for analysis of data from controlled clinical trials. The user of these adaptable statistical tools must, however, choose among alternative structural models to represent the correlated repeated measurements. The fit of the error structure model specification is important for validity of tests for differences in patterns of treatment effects across time, particularly when maximum likelihood procedures are relied upon. Results can be affected significantly by the error specification that is selected, so a principled basis for selecting the specification is important. As no theoretical grounds are usually available to guide this decision, empirical criteria have been developed that focus on mode fit. The current report proposes alternative empirical criteria that focus on bootstrap estimates of actual type I error an power of tests for treatment effects. Results for model selection before and after the blind is broken are compared. Goodness-of-fit statistics also compare favourably for models fitted to the blinded or unblinded data, although the correspondence to actual type I error and power depends on the particular fit statistic that is considered.

Determining the number of clusters by sampling with replacement.

A split-sample replication criterion originally proposed by J. E. Overall and K. N. Magee (1992) as a stopping rule for hierarchical cluster analysis is applied to multiple data sets generated by sampling with replacement from an original simulated primary data set. An investigation of the validity of this bootstrap procedure was undertaken using different combinations of the true number of latent populations, degrees of overlap, and sample sizes. The bootstrap procedure enhanced the accuracy of identifying the true number of latent populations under virtually all conditions. Increasing the size of the resampled data sets relative to the size of the primary data set further increased accuracy. A computer program to implement the bootstrap stopping rule is made available via a referenced Web site.

Measuring change in controlled longitudinal studies.

This paper examines the implications of the correlational structure of repeated measurements for three indices of change that can be used to evaluate treatment effects in longitudinal studies with scheduled assessment times and fixed total duration. The generalized least squares (GLS) regression of repeated measurements on time, which is usually reserved for complex mixed model solutions, takes the correlational structure of the repeated measurements into account, whereas simple gain scores and ordinary least squares (OLS) regression calculations do not. Nevertheless, the GLS solution is equivalent to OLS under conditions of compound symmetry and is equivalent to the analysis of simple gain scores in the presence of an autoregressive (order 1) correlational structure. The understanding of these relationships is important with regard to the frequently heard criticisms of the simpler definitions of treatment response in repeated measurement designs.

A Comment on the Efficacy of Unilateral Versus Bilateral ECT.

Ten studies comparing the efficacy of unilateral versus bilateral electroconvulsive therapy (ECT) were reexamined. Three different methods of meta-analysis applied to the combined results revealed statistical significance in favor of bilateral ECT for the relief of depression. The recognition that bilateral ECT has some advantage over unilateral is in sharp contrast to an overly strong conclusion to the contrary previously reached by Janicak et al. (1985), in reviewing the same studies. Alternative statistical methods for evaluating the significance of combined results from several independent studies are illustrated with reference to previously published ECT research.