The effect of measurement error on the positive predictive value of PSW methods for SLD identification: How buffer zones dispel the illusion of inaccuracy.

Patterns of Strengths and Weaknesses (PSW) methods are widely used for identifying specific learning disabilities (SLDs). Several researchers, however, have reported that the diagnostic accuracy of PSW methods is unacceptably low when strict thresholds were used to identify students with SLDs. We believe these findings give a misleading impression of the magnitude of the diagnostic errors that are likely to arise in PSW assessments. In a simulation study of 10 million cases using a simplified PSW method for demonstration, most of what have been called diagnostic errors were cases in which observed scores and true scores fell on opposite sides of a strict threshold but were still within a buffer zone the size of a typical measurement error. Because small score differences do not result in meaningfully different case conceptualizations, the use of buffer zones reveals that previous estimates of the diagnostic accuracy of PSW methods are misleadingly low. We also demonstrate that diagnostic decisions become increasingly reliable when observed scores are comfortably distant from diagnostic thresholds. For practitioners, we present a flowchart and practical guidelines to improve the accuracy and stability of SLD identification decisions.

It's the destination and it's the journey: using multilevel modeling to assess patterns of change in psychotherapy.

BACKGROUND: The pairing of analysis of variance (ANOVA)-based analyses with designs lacking multiple measurement of client change may lead to conclusions that are inconsistent with theoretical expectations about how change will occur. Multilevel modeling (MLM), when used with multiple assessments of client outcomes, provides a flexible set of tools to analyze trajectories of client change. PURPOSE: The authors present an overview of MLM and growth curve analysis as applied to psychotherapy outcome. Results from a simulated data set illustrate how MLM-specifically, growth curve analysis-can be used to test hypotheses from randomized clinical trials. RESULTS: Growth curve analyses were applied to simulated data from an experimental design (treatment versus control group) with multiple assessments of client distress and a client individual difference variable. Results of growth curve analyses confirmed what was found with a pretreatment/posttreatment repeated-measures ANOVA but provided additional information not available from the ANOVA framework. CONCLUSIONS: Given its flexibility, MLM is the recommended method to assess patterns of change in studies of psychotherapy outcome.

A Psychometric Network Analysis of CHC Intelligence Measures: Implications for Research, Theory, and Interpretation of Broad CHC Scores "Beyond g".

For over a century, the structure of intelligence has been dominated by factor analytic methods that presume tests are indicators of latent entities (e.g., general intelligence or g). Recently, psychometric network methods and theories (e.g., process overlap theory; dynamic mutualism) have provided alternatives to g-centric factor models. However, few studies have investigated contemporary cognitive measures using network methods. We apply a Gaussian graphical network model to the age 9-19 standardization sample of the Woodcock-Johnson Tests of Cognitive Ability-Fourth Edition. Results support the primary broad abilities from the Cattell-Horn-Carroll (CHC) theory and suggest that the working memory-attentional control complex may be central to understanding a CHC network model of intelligence. Supplementary multidimensional scaling analyses indicate the existence of possible higher-order dimensions (PPIK; triadic theory; System I-II cognitive processing) as well as separate learning and retrieval aspects of long-term memory. Overall, the network approach offers a viable alternative to factor models with a g-centric bias (i.e., bifactor models) that have led to erroneous conclusions regarding the utility of broad CHC scores in test interpretation beyond the full-scale IQ, g.

Meta-analyses and effect sizes in applied behavior analysis: A review and discussion.

For more than four decades, researchers have used meta-analyses to synthesize data from multiple experimental studies often to draw conclusions that are not supported by individual studies. More recently, single-case experimental design (SCED) researchers have adopted meta-analysis techniques to answer research questions with data gleaned from SCED experiments. Meta-analyses enable researchers to answer questions regarding intervention efficacy, generality, and condition boundaries. Here we discuss meta-analysis techniques, the rationale for their adaptation with SCED studies, and current indices used to quantify the effect of SCED data in applied behavior analysis.

Evaluation of publication bias in response interruption and redirection: A meta-analysis.

Publication bias is the disproportionate representation of studies with large effects and statistically significant findings in the published research literature. If publication bias occurs in single-case research design studies on applied behavior-analytic (ABA) interventions, it can result in inflated estimates of ABA intervention effects. We conducted an empirical evaluation of publication bias on an evidence-based ABA intervention for children diagnosed with autism spectrum disorder, response interruption and redirection (RIRD). We determined effect size estimates for published and unpublished studies using 3 metrics, percentage of nonoverlapping data (PND), Hedges' g, and log response ratios (LRR). Omnibus effect size estimates across all 3 metrics were positive, supporting that RIRD is an effective treatment for reducing problem behavior maintained by nonsocial consequences. We observed larger PND for published compared to unpublished studies, small and nonsignificant differences in LRR for published compared to unpublished studies, and significant differences in Hedges' g for published compared to unpublished studies, with published studies showing slightly larger effect. We found little, if any, difference in methodological quality between published and unpublished studies. While RIRD appears to be an effective intervention for challenging behavior maintained by nonsocial consequences, our results reflect some degree of publication bias present in the RIRD research literature.

The emotional content of client disclosures and session impact: An analogue study.

The authors examined the relation between a client's disclosure of emotional material in an analogue psychotherapy session and the depth and smoothness of that session. Transcripts of 33 prepracticum psychotherapy sessions (3 sessions each from 11 volunteer clients) were coded for client disclosures, and the text of these disclosures was subjected to a computer analysis to count the number of positive and negative emotion words. While controlling for client functioning, sessions in which clients disclosed more and used more positive-emotion words in their disclosures were rated as having more depth. Clients' use of emotion in their disclosures was not related to session smoothness. Findings from this study suggest aspects of client disclosure that are associated with deeper sessions. (PsycINFO Database Record (c) 2010 APA, all rights reserved).

Let's Not Do Away with Comprehensive Cognitive Assessments Just Yet.

We review rational and empirical reasons that comprehensive cognitive assessments are useful sources of information in the evaluation and treatment of learning disabilities. However, the existing evidence base that demonstrates the value of comprehensive cognitive assessments for this purpose is not nearly as strong as it needs to be. Proponents of comprehensive cognitive assessments for learning disability identification must do more to rigorously evaluate their beliefs or else concede the argument to those with better evidence.

Are positive and negative reinforcement "different"? Insights from a free-operant differential outcomes effect.

Although theoretical discussions typically assume that positive and negative reinforcement differ, the literature contains little unambiguous evidence that they produce differential behavioral effects. To test whether the two types of consequences control behavior differently, we pitted money-gain positive reinforcement and money-loss-avoidance negative reinforcement, scheduled through identically programmed variable-cycle schedules, against each other in concurrent schedules. Contingencies of response-produced feedback, normally different in positive and negative reinforcement, were made symmetrical. Steeper matching slopes were produced compared to a baseline consisting of all positive reinforcement. This free-operant differential outcomes effect supports the notion that that stimulus-presentation positive reinforcement and stimulus-elimination negative reinforcement are functionally "different." However, a control experiment showed that the feedback asymmetry of more traditional positive and negative reinforcement schedules also is sufficient to create a "difference" when the type of consequence is held constant. We offer these findings as a small step in meeting the very large challenge of moving negative reinforcement theory beyond decades of relative quiescence.

Accuracy in parameter estimation in cluster randomized designs.

When planning to conduct a study, not only is it important to select a sample size that will ensure adequate statistical power, often it is important to select a sample size that results in accurate effect size estimates. In cluster-randomized designs (CRD), such planning presents special challenges. In CRD studies, instead of assigning individual objects to treatment conditions, objects are grouped in clusters, and these clusters are then assigned to different treatment conditions. Sample size in CRD studies is a function of 2 components: the number of clusters and the cluster size. Planning to conduct a CRD study is difficult because 2 distinct sample size combinations might be associated with similar costs but can result in dramatically different levels of statistical power and accuracy in effect size estimation. Thus, we present a method that assists researchers in finding the least expensive sample size combination that still results in adequate accuracy in effect size estimation. Alternatively, if researchers have a fixed budget, they can select the sample size combination that results in the most precise estimate of effect size. A free computer program that automates these procedures is available.

Changing grasp position on a wielded object provides self-training for the perception of length.

Calibration of perception to environmental properties typically requires experiences in addition to the perceptual task, such as feedback about performance. Recently, it has been shown that such experiences need not come from an external source or from a different perceptual modality. Rather, in some cases, a given perceptual modality can train itself. In this study, we sought to expand on the range of experiences in which this can occur for perception of the length of a wielded occluded object. Specifically, in two experiments, we investigated whether the act of perceiving the length of a wielded object from a given grasp position could recalibrate the perception of length from a different grasp position. In both experiments, three groups of participants perceived the lengths of wielded rods in a pretest, practice, and a posttest. The practice included either (a) experimenter feedback, (b) changing the grasp position on the object (and again attempting to perceive length), or (c) no additional experiences. In Experiment 1, participants changed their grasp position from the middle to the end of each rod, and in Experiment 2, they did so from the end to the middle of each rod. In both experiments, the results showed that perceiving length from a different grasp position can recalibrate (i.e., provide self-training for) the perception of length.

Estimating base rates of impairment in neuropsychological test batteries: a comparison of quantitative models.

Neuropsychologists frequently rely on a battery of neuropsychological tests which are normally distributed to determine impaired functioning. The statistical likelihood of Type I error in clinical decision-making is in part determined by the base rate of normative individuals obtaining atypical performance on neuropsychological tests. Base rates are most accurately obtained by co-normed measures, but this is rarely accomplished in neuropsychological testing. Several statistical methods have been proposed to estimate base rates for tests that are not co-normed. This study compared two statistical approaches (binomial and Monte Carlo models) used to estimate the base rates for flexible test batteries. The two approaches were compared against empirically derived base rates for a multitest co-normed battery of cognitive measures. Estimates were compared across a variety of conditions including age and different α levels (N =3,356). Monte Carlo R(2) estimates ranged from .980 to .997 across five different age groups, indicating a good fit. In contrast, the binomial model fit estimates ranged from 0.387 to 0.646. Results confirm that the binomial model is insufficient for estimating base rates because it does not take into account correlations among measures in a multitest battery. Although the Monte Carlo model produced more accurate results, minor biases occurred that are likely due to skewess and kurtosis of test variables. Implications for future research and applied practice are discussed.

Playing statistical ouija board with commonality analysis: good questions, wrong assumptions.

Commonality analysis, as used by Hale, Fiorello, and colleagues (e.g., Hale, J. B., Fiorello, C. A., Kavanagh, J. A., Holdnack, J. A., & Aloe, A. M. [2007]. Is the demise of IQ interpretation justified? A response to special issue authors. Applied Neuropsychology, 21, 37-51), often leads to unjustified conclusions. Analyses of simulated datasets show that the highest-order commonality is a poor operationalization of the general factor of intelligence (g). Under commonly observed conditions, it is plausible that lower-order commonalities and unique components in a commonality analysis can consist entirely of g. Other problems with Hale and colleagues' (2007) analyses and interpretations are also discussed.