Reliability of multi-category rating scales.

The use of multi-category scales is increasing for the monitoring of IEP goals, classroom and school rules, and Behavior Improvement Plans (BIPs). Although they require greater inference than traditional data counting, little is known about the inter-rater reliability of these scales. This simulation study examined the performance of nine reliability indices applied to six multi-category scales of different gradations (2, 3, 5, 7, 10, and 15 points) all derived from the same quasi-continuous (1-30) data. The researchers find that each index behaves differently and requires its own interpretation; there is no one-best reliability indices as most indices are scale-dependent. Finally, index values do not remain constant when more categories are collapsed to fewer. New guidelines are needed for optimal methods of obtaining reliability with ordinal scales.

Moderating factors of video-modeling with other as model: a meta-analysis of single-case studies.

Video modeling with other as model (VMO) is a more practical method for implementing video-based modeling techniques, such as video self-modeling, which requires significantly more editing. Despite this, identification of contextual factors such as participant characteristics and targeted outcomes that moderate the effectiveness of VMO has not previously been explored. The purpose of this study was to meta-analytically evaluate the evidence base of VMO with individuals with disabilities to determine if participant characteristics and targeted outcomes moderate the effectiveness of the intervention. Findings indicate that VMO is highly effective for participants with autism spectrum disorder (IRD=.83) and moderately effective for participants with developmental disabilities (IRD=.68). However, differential effects are indicated across levels of moderators for diagnoses and targeted outcomes. Implications for practice and future research are discussed.

A meta-analysis of single case research studies on aided augmentative and alternative communication systems with individuals with autism spectrum disorders.

Many individuals with autism cannot speak or cannot speak intelligibly. A variety of aided augmentative and alternative communication (AAC) approaches have been investigated. Most of the research on these approaches has been single-case research, with small numbers of participants. The purpose of this investigation was to meta-analyze the single case research on the use of aided AAC with individuals with autism spectrum disorders (ASD). Twenty-four single-case studies were analyzed via an effect size measure, the Improvement Rate Difference (IRD). Three research questions were investigated concerning the overall impact of AAC interventions on targeted behavioral outcomes, effects of AAC interventions on individual targeted behavioral outcomes, and effects of three types of AAC interventions. Results indicated that, overall, aided AAC interventions had large effects on targeted behavioral outcomes in individuals with ASD. AAC interventions had positive effects on all of the targeted behavioral outcome; however, effects were greater for communication skills than other categories of skills. Effects of the Picture Exchange Communication System and speech-generating devices were larger than those for other picture-based systems, though picture-based systems did have small effects.

Combining nonoverlap and trend for single-case research: Tau-U.

A new index for analysis of single-case research data was proposed, Tau-U, which combines nonoverlap between phases with trend from within the intervention phase. In addition, it provides the option of controlling undesirable Phase A trend. The derivation of Tau-U from Kendall's Rank Correlation and the Mann-Whitney U test between groups is demonstrated. The equivalence of trend and nonoverlap is also shown, with supportive citations from field leaders. Tau-U calculations are demonstrated for simple AB and ABA designs. Tau-U is then field tested on a sample of 382 published data series. Controlling undesirable Phase A trend caused only a modest change from nonoverlap. The inclusion of Phase B trend yielded more modest results than simple nonoverlap. The Tau-U score distribution did not show the artificial ceiling shown by all other nonoverlap techniques. It performed reasonably well with autocorrelated data. Tau-U shows promise for single-case applications, but further study is desirable.

Robust regression for single-case data analysis: how can it help?

This study examined the degree to which outliers were present in a convenience sample of published single-case research. Using a procedure for analyzing single-case data Allison & Gorman (Behaviour Research and Therapy, 31, 621-631, 1993), this study compared the effect of outliers using ordinary least squares (OLS) regression to a robust regression method and attempted to answer four questions: (1) To what degree does outlier detection vary from OLS to robust regression? (2) How much do effect sizes differ from OLS to robust regression? (3) Are the differences produced by robust regression in more or less agreement with visual judgments of treatment effectiveness? (4) What is a typical range of effect sizes for robust regression versus OLS regression for data from "effective interventions"? Results suggest that outliers are common in single-case data. The effects of outliers in single-case data are explored, and the implications for researchers and practitioners using single-case designs are discussed.

Effect size in single-case research: a review of nine nonoverlap techniques.

With rapid advances in the analysis of data from single-case research designs, the various behavior-change indices, that is, effect sizes, can be confusing. To reduce this confusion, nine effect-size indices are described and compared. Each of these indices examines data nonoverlap between phases. Similarities and differences, both conceptual and computational, are highlighted. Seven of the nine indices are applied to a sample of 200 published time series data sets, to examine their distributions. A generic meta-analytic method is presented for combining nonoverlap indices across multiple data series within complex designs.

An improved effect size for single-case research: nonoverlap of all pairs.

Nonoverlap of All Pairs (NAP), an index of data overlap between phases in single-case research, is demonstrated and field tested with 200 published AB contrasts. NAP is a novel application of an established effect size known in various forms as Area Under the Curve (AUC), the Common Language Effect Size (CL), the Probability of Superiority (PS), the Dominance Statistic (DS), Mann-Whitney's U, and Sommers D, among others. NAP was compared with 3 other non-overlap-based indices: PND (percent of nonoverlapping data), PEM (percent of data points exceeding the median), and PAND (percent of all nonoverlapping data), as well as Pearson's R(2). Five questions were addressed about NAP: (a) typical NAP values, (b) its ability to discriminate among typical single-case research results, (c) its power and precision (confidence interval width), (d) its correlation with the established effect size index, R(2), and (e) its relationship with visual judgments. Results were positive, the new index equaling or outperforming the other overlap indices on most criteria.

Median-based overlap analysis for single case data: a second study.

This article takes a further look at the percentage of data points exceeding the median (PEM) analysis method for single-case research data, first presented in this journal by Hsen-Hsing Ma. Ma examined the relationship between PEM and the established percentage of nonoverlapping data (PND) and then applied PEM in a meta-analysis of 61 data sets, correlating their authors' judgments of intervention effectiveness with PEM. The present article covers PEM's historical and statistical context and then applies the new measure in a field test with 165 contrasts between a baseline phase A and a treatment phase B. For comparison, Pearson r , Kruskal-Wallis W, PND, and IRD (improvement rate difference) indices also are calculated and correlated with PEM, and all distributions are examined. Expert visual analysis ratings of the 165 graphs are correlated with all indices. PEM surpassed PND in its validation by other established measures. However, PEM was weaker in distribution shape and visual judgment validation. More strongly validated than either PEM or PND was the new nonparametric measure, IRD.

Useful effect size interpretations for single case research.

An obstacle to broader acceptability of effect sizes in single case research is their lack of intuitive and useful interpretations. Interpreting Cohen's d as "standard deviation units difference" and R2 as "percent of variance accounted for" do not resound with most visual analysts. In fact, the only comparative analysis widely supported in single case research (SCR) is "percent of nonoverlapping data." This article explores five alternative interpretations of Cohen's d and R2 effect sizes that may be more acceptable to the SCR field. They are: (a) Cohen's (Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum) "Percent of Nonoverlapping Data" (CPND), (b) Parker et al.'s (Parker, R.I., Cryer, J., Byrns, G., 2006. Controlling trend in single case research. School Psychology Quarterly, 21, 418-440, Parker, R.I., Hagan-Burke, S., Vannest, K., in press. Percent of all non-overlapping data (PAND): An alternative to PND. Journal of Special Education) "Percent of All Nonoverlapping Data" (PAND), (c) Rosenthal et al. (Rosenthal, R., Rosnow, R., & Rubin, D. (2000). Contrasts and effect sizes in behavioral research: A correlational approach. Cambridge: Cambridge University Press.) "Binomial Effect Size Display" (BESD), (d) "Percentile Rank in Control Group" (PR), and (e) McGraw and Wong's (McGraw, K. O., & Wong, S. P. (1992). A common language effect-size statistic. Psychological Bulletin, 111, 361-365) "Common Language Effect Size" (CLES). Each of the five interpretation schemes are applied to a published data set and are evaluated according to (a) intuitive appeal, (b) relevance to visual analysis, (c) ease of calculation, and (d) technical adequacy. Three of the five appear to be improvements over prevailing practice.

Increased reliability for single-case research results: is the bootstrap the answer?

There is need for objective and reliable single-case research (SCR) results in the movement toward evidence-based interventions (EBI), for inclusion in meta-analyses, and for funding accountability in clinical contexts. Yet SCR deals with data that often do not conform to parametric data assumptions and that yield results of low reliability. A resampling technique, the bootstrap, largely bypasses statistical assumptions and usually yields more reliable results. This study answers questions about the extent of need for the bootstrap in SCR and its impact on effect size reliability. The bootstrap was applied in Allison et al. mean shift analyses (Faith, Allison, & Gorman, 1997) to data from 166 published AB graphs. Results showed the bootstrap improved reliability of 88% of the analyses and reduced reliability of only 3%. The reliability improvement was large enough to be practically useful. The bootstrap was paired with a method for cleansing data of autocorrelation, which also proved effective. Pending replication, the findings encourage broad application within SCR of both the bootstrap and autocorrelation cleansing.

The relationship between visual analysis and five statistical analyses in a simple AB single-case research design.

This study explored some practical issues for single-case researchers who rely on visual analysis of graphed data, but who also may consider supplemental use of promising statistical analysis techniques. The study sought to answer three major questions: (a) What is a typical range of effect sizes from these analytic techniques for data from "effective interventions"? (b) How closely do results from these same analytic techniques concur with visual-analysis-based judgments of effective interventions? and (c) What role does autocorrelation play in interpretation of these analytic results? To answer these questions, five analytic techniques were compared with the judgments of 45 doctoral students and faculty, who rated intervention effectiveness from visual analysis of 35 fabricated AB design graphs. Implications for researchers and practitioners using single-case designs are discussed.

Solution-focused family therapy with three aggressive and oppositional-acting children: an N = 1 empirical study.

The efficacy of Solution-Focused Family Therapy (SFFT) for helping three families with aggressive and oppositional-acting children (aged 8-9) was examined. The N = 1 multiple-baseline design with three replications used validated measures, a treatment manual, and a treatment integrity measure. The interventions lasted from four to five sessions. SFFT appeared to be effective with the families at post-treatment and 3-month follow up.