Determining Associations Between Intervention Amount and Outcomes for Young Autistic Children: A Meta-Analysis.

Importance: Health professionals routinely recommend intensive interventions (ie, 20-40 hours per week) for autistic children. However, primary research backing this recommendation is sparse and plagued by methodological flaws. Objective: To examine whether different metrics of intervention amount are associated with intervention effects on any developmental domain for young autistic children. Data Sources: A large corpus of studies taken from a recent meta-analysis (with a search date of November 2021) of early interventions for autistic children. Study Selection: Studies were eligible if they reported a quasi-experimental or randomized clinical trial testing the effects of a nonpharmacological intervention on any outcome in participant samples comprising more than 50% autistic children 8 years or younger. Data Extraction and Synthesis: Data were independently extracted by multiple coders. Meta-regression models were constructed to determine whether each index of intervention amount was associated with effect sizes for each intervention type, while controlling for outcome domain, outcome proximity, age of participants, study design, and risk of detection bias. Data were analyzed from June 2023 to February 2024. This study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Main Outcomes and Measures: The primary predictor of interest was intervention amount, quantified using 3 different metrics (daily intensity, duration, and cumulative intensity). The primary outcomes of interest were gains in any developmental domain, quantified by Hedges g effect sizes. Results: A total of 144 studies including 9038 children (mean [SD] age, 49.3 [17.2] months; mean [SD] percent males, 82.6% [12.7%]) were included in this analysis. None of the meta-regression models evidenced a significant, positive association between any index of intervention amount and intervention effect size when considered within intervention type. Conclusions and Relevance: Findings of this meta-analysis do not support the assertion that intervention effects increase with increasing amounts of intervention. Health professionals recommending interventions should be advised that there is little robust evidence supporting the provision of intensive intervention.

The efficacy of combining cognitive training and noninvasive brain stimulation: A transdiagnostic systematic review and meta-analysis.

Over the past decade, an increasing number of studies investigated the innovative approach of supplementing cognitive training (CT) with noninvasive brain stimulation (NIBS) to increase the effects on outcomes. In this review, we aim to summarize the evidence for this treatment combination. We identified 72 published and unpublished studies (reporting 773 effect sizes), including 2,518 participants from healthy and clinical populations indexed in PubMed, MEDLINE, APA PsycInfo, ProQuest, Web of Science, and https://ClinicalTrials.gov (last search: August 9, 2022) that compared the effects of NIBS combined with CT on cognitive, symptoms, and everyday functioning to CT alone at postintervention and/or follow-up. We performed random-effects meta-analyses with robust variance estimation and assessed risk of bias with the Cochrane ROB tool. Only four studies had low risk of bias in all domains, and many studies lacked standard controls such as keeping the outcome assessor and trainer unaware of the treatment condition. Following sensitivity analyses, only learning/memory robustly improved significantly more when CT was combined with NIBS compared to CT only (g = 0.18, 95% CI [0.07, 0.29]) at postintervention, but not in the long term. The effect was small and limited by substantial heterogeneity. The other seven cognitive outcome domains, symptoms, and everyday functioning did not benefit from adding NIBS to CT. Given the methodological limitation of prior studies, more high-quality trials that focus on the potential of combining NIBS and CT to enhance benefits in everyday functioning in the short and long term are needed to evaluate whether combining NIBS and CT is relevant for clinical practice. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

High replicability of newly discovered social-behavioural findings is achievable.

Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigour-enhancing practices: confirmatory tests, large sample sizes, preregistration and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (P < 0.05) in 86% of attempts, slightly exceeding the maximum expected replicability based on observed effect sizes and sample sizes. When one lab attempted to replicate an effect discovered by another lab, the effect size in the replications was 97% that in the original study. This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries.

Between-case standardized mean differences: Flexible methods for single-case designs.

Single-case designs (SCDs) are a class of research methods for evaluating the effects of academic and behavioral interventions in educational and clinical settings. Although visual analysis is typically the first and main method for analysis of data from SCDs, quantitative methods are useful for synthesizing results and drawing systematic generalizations across bodies of single-case research. Researchers who are interested in synthesizing findings across SCDs and between-group designs might consider using the between-case standardized mean difference (BC-SMD) effect size, which aims to put results from both types of studies into a common metric. Currently available BC-SMD methods are limited to treatment reversal designs with replication across participants and across-participant multiple baseline designs, yet more complex designs are sometimes used in practice. In this study, we extend available BC-SMD methods to several variations of the multiple baseline design, including the replicated multiple baseline across behaviors or settings, the clustered multiple baseline design, and the multivariate multiple baseline across participants. For each variation, we describe methods for estimating BC-SMD effect sizes and illustrate our proposed approach by re-analyzing data from a published SCD study.

Systematic Review of Variables Related to Instruction in Augmentative and Alternative Communication Implementation: Group and Single-Case Design.

PURPOSE: This article provides a systematic review and analysis of group and single-case studies addressing augmentative and alternative communication (AAC) intervention with school-aged persons having autism spectrum disorder (ASD) and/or intellectual/developmental disabilities resulting in complex communication needs (CCNs). Specifically, we examined participant characteristics in group-design studies reporting AAC intervention outcomes and how these compared to those reported in single-case experimental designs (SCEDs). In addition, we compared the status of intervention features reported in group and SCED studies with respect to instructional strategies utilized. PARTICIPANTS: Participants included school-aged individuals with CCNs who also experienced ASD or ASD with an intellectual delay who utilized aided or unaided AAC. METHOD: A systematic review using descriptive statistics and effect sizes was implemented. RESULTS: Findings revealed that participant features such as race, ethnicity, and home language continue to be underreported in both SCED and group-design studies. Participants in SCED investigations more frequently used multiple communication modes when compared to participants in group studies. The status of pivotal skills such as imitation was sparsely reported in both types of studies. With respect to instructional features, group-design studies were more apt to utilize clinical rather than educational or home settings when compared with SCED studies. In addition, SCED studies were more apt to utilize instructional methods that closely adhered to instructional features more typically characterized as being associated with behavioral approaches. CONCLUSION: The authors discuss future research needs, practice implications, and a more detailed specification of treatment intensity parameters for future research.

Comparing random effects models, ordinary least squares, or fixed effects with cluster robust standard errors for cross-classified data.

Cross-classified random effects modeling (CCREM) is a common approach for analyzing cross-classified data in psychology, education research, and other fields. However, when the focus of a study is on the regression coefficients at Level 1 rather than on the random effects, ordinary least squares regression with cluster robust variance estimators (OLS-CRVE) or fixed effects regression with CRVE (FE-CRVE) could be appropriate approaches. These alternative methods are potentially advantageous because they rely on weaker assumptions than those required by CCREM. We conducted a Monte Carlo Simulation study to compare the performance of CCREM, OLS-CRVE, and FE-CRVE in models, including conditions where homoscedasticity assumptions and exogeneity assumptions held and conditions where they were violated, as well as conditions with unmodeled random slopes. We found that CCREM out-performed the alternative approaches when its assumptions are all met. However, when homoscedasticity assumptions are violated, OLS-CRVE and FE-CRVE provided similar or better performance than CCREM. When the exogeneity assumption is violated, only FE-CRVE provided adequate performance. Further, OLS-CRVE and FE-CRVE provided more accurate inferences than CCREM in the presence of unmodeled random slopes. Thus, we recommend two-way FE-CRVE as a good alternative to CCREM, particularly if the homoscedasticity or exogeneity assumptions of the CCREM might be in doubt. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

An Examination of Measurement Procedures and Characteristics of Baseline Outcome Data in Single-Case Research.

There has been growing interest in using statistical methods to analyze data and estimate effect size indices from studies that use single-case designs (SCDs), as a complement to traditional visual inspection methods. The validity of a statistical method rests on whether its assumptions are plausible representations of the process by which the data were collected, yet there is evidence that some assumptions-particularly regarding normality of error distributions-may be inappropriate for single-case data. To develop more appropriate modeling assumptions and statistical methods, researchers must attend to the features of real SCD data. In this study, we examine several features of SCDs with behavioral outcome measures in order to inform development of statistical methods. Drawing on a corpus of over 300 studies, including approximately 1,800 cases, from seven systematic reviews that cover a range of interventions and outcome constructs, we report the distribution of study designs, distribution of outcome measurement procedures, and features of baseline outcome data distributions for the most common types of measurements used in single-case research. We discuss implications for the development of more realistic assumptions regarding outcome distributions in SCD studies, as well as the design of Monte Carlo simulation studies evaluating the performance of statistical analysis techniques for SCD data.

Participant characteristics predicting communication outcomes in AAC implementation for individuals with ASD and IDD: a systematic review and meta-analysis.

This meta-analysis examined communication outcomes in single-case design studies of augmentative and alternative communication (AAC) interventions and their relationship to participant characteristics. Variables addressed included chronological age, pre-intervention communication mode, productive repertoire, and pre-intervention imitation skills. Investigators identified 114 single-case design studies that implemented AAC interventions with school-aged individuals with autism spectrum disorder and/or intellectual disability. Two complementary effect size indices, Tau(AB) and the log response ratio, were applied to synthesize findings. Both indices showed positive effects on average, but also exhibited a high degree of heterogeneity. Moderator analyses detected few differences in effectiveness when comparing across diagnoses, age, the number and type of communication modes, participant's productive repertoires, and imitation skills to intervention. A PRISMA-compliant abstract is available: https://bit.ly/30BzbLv.

The Relative Effects of Instruction Linking Word Reading and Word Meaning Compared to Word Reading Instruction Alone on the Accuracy, Fluency, and Word Meaning Knowledge of 4th-5th Grade Students With Dyslexia.

This within-subjects experimental study investigated the relative effects of word reading and word meaning instruction (WR+WM) compared to word-reading instruction alone (WR) on the accuracy, fluency, and word meaning knowledge of 4th-5th graders with dyslexia. We matched word lists on syllables, phonemes, frequency, number of definitions, and concreteness. We assigned half the words to WR and half to WR+WM. Word reading accuracy, word reading fluency, and word meaning knowledge were measured at pretest, immediately following each intervention session, and at posttest, administered immediately following the 12, 45-minute, daily instructional sessions. Compared to WR instruction alone, WR+WM significantly improved accuracy (d = 0.65), fluency (d = 0.43), and word meaning knowledge (d = 1.92) immediately following intervention, and significantly improved accuracy (d = 0.74), fluency (d = 0.84), and word meaning knowledge (d = 1.03) at posttest. Findings support the premise that word meaning knowledge facilitates accurate and fluent word reading, and that instruction explicitly integrating word reading and word meaning may be an effective support for upper elementary students with dyslexia.

Investigating Narrative Performance in Children With Developmental Language Disorder: A Systematic Review and Meta-Analysis.

PURPOSE: Narrative assessment is one potentially underutilized and inconsistent method speech-language pathologists may use when considering a diagnosis of developmental language disorder (DLD). However, narration research encompasses many varied methodologies. This systematic review and meta-analysis aimed to (a) investigate how various narrative assessment types (e.g., macrostructure, microstructure, and internal state language) differentiate children with typical development (TD) from children with DLD, (b) identify specific narrative assessment measures that result in greater group differences, and (c) evaluate participant and sample characteristics that may influence performance differences. METHOD: Electronic databases (PsycINFO, ERIC, and PubMed) and ASHAWire were searched on July 30, 2019, to locate studies that reported oral narrative language measures for both DLD and TD groups between ages 4 and 12 years; studies focusing on written narration or other developmental disorders only were excluded. We extracted data related to sample participants, narrative task(s) and assessment measures, and research design. Group differences were quantified using standardized mean differences. Analyses used mixed-effects meta-regression with robust variance estimation to account for effect size dependencies. RESULTS: Searches identified 37 eligible studies published between 1987 and 2019, including 382 effect sizes. Overall meta-analysis showed that children with DLD had decreased narrative performance relative to TD peers, with an overall average effect of -0.82 SD, 95% confidence interval [-0.99, -0.66]. Effect sizes showed significant heterogeneity both between and within studies, even after accounting for effect size-, sample-, and study-level predictors. Across model specifications, grammatical accuracy (microstructure) and story grammar (macrostructure) yielded the most consistent evidence of TD-DLD group differences. CONCLUSIONS: Present findings suggest some narrative assessment measures yield significantly different performance between children with and without DLD. However, researchers need to improve consistency of inclusionary criteria, descriptions of sample characteristics, and reporting of correlations between measures to determine which assessment measures reliably distinguish between groups. SUPPLEMENTAL MATERIAL: https://doi.org/10.23641/asha.21200380.

Multilevel meta-analysis of single-case experimental designs using robust variance estimation.

Single-case experimental designs (SCEDs) are used to study the effects of interventions on the behavior of individual cases, by making comparisons between repeated measurements of an outcome under different conditions. In research areas where SCEDs are prevalent, there is a need for methods to synthesize results across multiple studies. One approach to synthesis uses a multilevel meta-analysis (MLMA) model to describe the distribution of effect sizes across studies and across cases within studies. However, MLMA relies on having accurate sampling variances of effect size estimates for each case, which may not be possible due to auto-correlation in the raw data series. One possible solution is to combine MLMA with robust variance estimation (RVE), which provides valid assessments of uncertainty even if the sampling variances of effect size estimates are inaccurate. Another possible solution is to forgo MLMA and use simpler, ordinary least squares (OLS) methods with RVE. This study evaluates the performance of effect size estimators and methods of synthesizing SCEDs in the presence of auto-correlation, for several different effect size metrics, via a Monte Carlo simulation designed to emulate the features of real data series. Results demonstrate that the MLMA model with RVE performs properly in terms of bias, accuracy, and confidence interval coverage for estimating overall average log response ratios. The OLS estimator corrected with RVE performs the best in estimating overall average Tau effect sizes. None of the available methods perform adequately for meta-analysis of within-case standardized mean differences. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Cluster wild bootstrapping to handle dependent effect sizes in meta-analysis with a small number of studies.

The most common and well-known meta-regression models work under the assumption that there is only one effect size estimate per study and that the estimates are independent. However, meta-analytic reviews of social science research often include multiple effect size estimates per primary study, leading to dependence in the estimates. Some meta-analyses also include multiple studies conducted by the same lab or investigator, creating another potential source of dependence. An increasingly popular method to handle dependence is robust variance estimation (RVE), but this method can result in inflated Type I error rates when the number of studies is small. Small-sample correction methods for RVE have been shown to control Type I error rates adequately but may be overly conservative, especially for tests of multiple-contrast hypotheses. We evaluated an alternative method for handling dependence, cluster wild bootstrapping, which has been examined in the econometrics literature but not in the context of meta-analysis. Results from two simulation studies indicate that cluster wild bootstrapping maintains adequate Type I error rates and provides more power than extant small-sample correction methods, particularly for multiple-contrast hypothesis tests. We recommend using cluster wild bootstrapping to conduct hypothesis tests for meta-analyses with a small number of studies. We have also created an R package that implements such tests.

Meta-analysis with Robust Variance Estimation: Expanding the Range of Working Models.

In prevention science and related fields, large meta-analyses are common, and these analyses often involve dependent effect size estimates. Robust variance estimation (RVE) methods provide a way to include all dependent effect sizes in a single meta-regression model, even when the exact form of the dependence is unknown. RVE uses a working model of the dependence structure, but the two currently available working models are limited to each describing a single type of dependence. Drawing on flexible tools from multilevel and multivariate meta-analysis, this paper describes an expanded range of working models, along with accompanying estimation methods, which offer potential benefits in terms of better capturing the types of data structures that occur in practice and, under some circumstances, improving the efficiency of meta-regression estimates. We describe how the methods can be implemented using existing software (the "metafor" and "clubSandwich" packages for R), illustrate the proposed approach in a meta-analysis of randomized trials on the effects of brief alcohol interventions for adolescents and young adults, and report findings from a simulation study evaluating the performance of the new methods.

A systematic review and meta-analysis of effects of psychosocial interventions on spiritual well-being in adults with cancer.

Objective: Spiritual well-being (SpWb) is an important dimension of health-related quality of life for many cancer patients. Accordingly, an increasing number of psychosocial intervention studies have included SpWb as a study endpoint, and may improve SpWb even if not designed explicitly to do so. This meta-analysis of randomized controlled trials (RCTs) evaluated effects of psychosocial interventions on SpWb in adults with cancer and tested potential moderators of intervention effects. Methods: Six literature databases were systematically searched to identify RCTs of psychosocial interventions in which SpWb was an outcome. Doctoral-level rater pairs extracted data using Covidence following Preferred Reporting Items for Systematic reviews and Meta-Analyses guidelines. Standard meta-analytic techniques were applied, including meta-regression with robust variance estimation and risk-of-bias sensitivity analysis. Results: Forty-one RCTs were identified, encompassing 88 treatment effects among 3883 survivors. Interventions were associated with significant improvements in SpWb (g = 0.22, 95% CI [0.14, 0.29], p < 0.0001). Studies assessing the FACIT-Sp demonstrated larger effect sizes than did those using other measures of SpWb (g = 0.25, 95% CI [0.17, 0.34], vs. g = 0.10, 95% CI [-0.02, 0.23], p = 0.03]. No other intervention, clinical, or demographic characteristics significantly moderated effect size. Conclusions: Psychosocial interventions are associated with small-to-medium-sized effects on SpWb among cancer survivors. Future research should focus on conceptually coherent interventions explicitly targeting SpWb and evaluate interventions in samples that are diverse with respect to race and ethnicity, sex and cancer type.

Evaluating meta-analytic methods to detect selective reporting in the presence of dependent effect sizes.

Selective reporting of results based on their statistical significance threatens the validity of meta-analytic findings. A variety of techniques for detecting selective reporting, publication bias, or small-study effects are available and are routinely used in research syntheses. Most such techniques are univariate, in that they assume that each study contributes a single, independent effect size estimate to the meta-analysis. In practice, however, studies often contribute multiple, statistically dependent effect size estimates, such as for multiple measures of a common outcome construct. Many methods are available for meta-analyzing dependent effect sizes, but methods for investigating selective reporting while also handling effect size dependencies require further investigation. Using Monte Carlo simulations, we evaluate three available univariate tests for small-study effects or selective reporting, including the trim and fill test, Egger's regression test, and a likelihood ratio test from a three-parameter selection model (3PSM), when dependence is ignored or handled using ad hoc techniques. We also examine two variants of Egger's regression test that incorporate robust variance estimation (RVE) or multilevel meta-analysis (MLMA) to handle dependence. Simulation results demonstrate that ignoring dependence inflates Type I error rates for all univariate tests. Variants of Egger's regression maintain Type I error rates when dependent effect sizes are sampled or handled using RVE or MLMA. The 3PSM likelihood ratio test does not fully control Type I error rates. With the exception of the 3PSM, all methods have limited power to detect selection bias except under strong selection for statistically significant effects. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Psychosocial interventions for cancer survivors: A meta-analysis of effects on positive affect.

PURPOSE: Positive affect has demonstrated unique benefits in the context of health-related stress and is emerging as an important target for psychosocial interventions. The primary objective of this meta-analysis was to determine whether psychosocial interventions increase positive affect in cancer survivors. METHODS: We coded 28 randomized controlled trials of psychosocial interventions assessing 2082 cancer survivors from six electronic databases. We calculated 76 effect sizes for positive affect and conducted synthesis using random effects models with robust variance estimation. Tests for moderation included demographic, clinical, and intervention characteristics. RESULTS: Interventions had a modest effect on positive affect (g = 0.35, 95% CI [0.16, 0.54]) with substantial heterogeneity of effects across studies ([Formula: see text]; I2 = 78%). Three significant moderators were identified: in-person interventions outperformed remote interventions (P = .046), effects were larger when evaluated against standard of care or wait list control conditions versus attentional, educational, or component controls (P = .009), and trials with survivors of early-stage cancer diagnoses yielded larger effects than those with advanced-stage diagnoses (P = .046). We did not detect differential benefits of psychosocial interventions across samples varying in sex, age, on-treatment versus off-treatment status, or cancer type. Although no conclusive evidence suggested outcome reporting biases (P = .370), effects were smaller in studies with lower risk of bias. CONCLUSIONS: In-person interventions with survivors of early-stage cancers hold promise for enhancing positive affect, but more methodological rigor is needed. IMPLICATIONS FOR CANCER SURVIVORS: Positive affect strategies can be an explicit target in evidence-based medicine and have a role in patient-centered survivorship care, providing tools to uniquely mobilize human strengths.

Interventions to enhance self-efficacy in cancer patients: A meta-analysis of randomized controlled trials.

OBJECTIVE: Self-efficacy expectations are associated with improvements in problematic outcomes widely considered clinically significant (ie, emotional distress, fatigue, and pain), related to positive health behaviors, and as a type of personal agency, inherently valuable. Self-efficacy expectancies, estimates of confidence to execute behaviors, are important in that changes in self-efficacy expectations are positively related to future behaviors that promote health and well-being. The current meta-analysis investigated the impact of psychological interventions on self-efficacy expectations for a variety of health behaviors among cancer patients. METHODS: Ovid Medline, PsycINFO, CINAHL, EMBASE, Cochrane Library, and Web of Science were searched with specific search terms for identifying randomized controlled trials (RCTs) that focused on psychologically based interventions. Included studies had (a) an adult cancer sample, (b) a self-efficacy expectation measure of specific behaviors, and (c) an RCT design. Standard screening and reliability procedures were used for selecting and coding studies. Coding included theoretically informed moderator variables. RESULTS: Across 79 RCTs, 223 effect sizes, and 8678 participants, the weighted average effect of self-efficacy expectations was estimated as g = 0.274 (P < .001). Consistent with the self-efficacy theory, the average effect for in-person intervention delivery (g = 0.329) was significantly greater than for all other formats (g = 0.154, P = .023; eg, audiovisual, print, telephone, and Web/internet). CONCLUSIONS: The results establish the impact of psychological interventions on self-efficacy expectations as comparable in effect size with commonly reported outcomes (distress, fatigue, pain). Additionally, the result that in-person interventions achieved the largest effect is supported by the social learning theory and could inform research related to the development and evaluation of interventions.

Effects of psychosocial interventions on meaning and purpose in adults with cancer: A systematic review and meta-analysis.

Meaning and purpose in life are associated with the mental and physical health of patients with cancer and survivors and also constitute highly valued outcomes in themselves. Because meaning and purpose are often threatened by a cancer diagnosis and treatment, interventions have been developed to promote meaning and purpose. The present meta-analysis of randomized controlled trials (RCTs) evaluated effects of psychosocial interventions on meaning/purpose in adults with cancer and tested potential moderators of intervention effects. Six literature databases were systematically searched to identify RCTs of psychosocial interventions in which meaning or purpose was an outcome. Using Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, rater pairs extracted and evaluated data for quality. Findings were synthesized across studies with standard meta-analytic methods, including meta-regression with robust variance estimation and risk-of-bias sensitivity analysis. Twenty-nine RCTs were identified, and they encompassed 82 treatment effects among 2305 patients/survivors. Psychosocial interventions were associated with significant improvements in meaning/purpose (g = 0.37; 95% CI, 0.22-0.52; P < .0001). Interventions designed to enhance meaning/purpose (g = 0.42; 95% CI, 0.24-0.60) demonstrated significantly higher effect sizes than those targeting other primary outcomes (g = 0.18; 95% CI, 0.09-0.27; P = .009). Few other intervention, clinical, or demographic characteristics tested were significant moderators. In conclusion, the results suggest that psychosocial interventions are associated with small to medium effects in enhancing meaning/purpose among patients with cancer, and the benefits are comparable to those of interventions designed to reduce depression, pain, and fatigue in patients with cancer. Methodological concerns include small samples and ambiguity regarding allocation concealment. Future research should focus on explicitly meaning-centered interventions and identify optimal treatment or survivorship phases for implementation.

Current practices in meta-regression in psychology, education, and medicine.

Having surveyed the history and methods of meta-regression in a previous paper, in this paper, we review which and how meta-regression methods are applied in recent research syntheses. To do so, we reviewed studies published in 2016 across four leading research synthesis journals: Psychological Bulletin, the Journal of Applied Psychology, Review of Educational Research, and the Cochrane Library. We find that the best practices defined in the previous review are rarely carried out in practice. In light of the identified discrepancies, we consider how to move forward, first by identifying areas where further methods development is needed to address persistent problems in the field and second by discussing how to more effectively disseminate points of methodological consensus.

Procedural sensitivities of effect sizes for single-case designs with directly observed behavioral outcome measures.

A wide variety of effect size indices have been proposed for quantifying the magnitude of treatment effects in single-case designs. Commonly used measures include parametric indices such as the standardized mean difference as well as nonoverlap measures such as the percentage of nonoverlapping data, improvement rate difference, and nonoverlap of all pairs. Currently, little is known about the properties of these indices when applied to behavioral data collected by systematic direct observation, even though systematic direct observation is the most common method for outcome measurement in single-case research. This study uses Monte Carlo simulation to investigate the properties of several widely used single-case effect size measures when applied to systematic direct observation data. Results indicate that the magnitude of the nonoverlap measures and of the standardized mean difference can be strongly influenced by procedural details of the study's design, which is a significant limitation to using these indices as effect sizes for meta-analysis of single-case designs. A less widely used parametric index, the log response ratio, has the advantage of being insensitive to sample size and observation session length, although its magnitude is influenced by the use of partial interval recording. (PsycINFO Database Record (c) 2019 APA, all rights reserved).