The Permutation Distancing Test for dependent single-case observational AB-phase design data: A Monte Carlo simulation study.

The Permutation Distancing Test (PDT) is a nonparametric test for evaluating treatment effects in dependent single-case observational design (SCOD) AB-phase data without linear trends. Monte Carlo methods were used to estimate the PDT power and type I error rate, and to compare them to those of the Single-Case Randomization Test (SCRT) assuming a randomly determined intervention point and the traditional permutation test assuming full exchangeability. Data were simulated without linear trends for five treatment effect levels (- 2, - 1, 0, 1, 2), five autocorrelation levels (0, .15, .30, .45, .60), and four observation number levels (30, 60, 90, 120). The power was calculated multiple times for all combinations of factor levels each generating 1000 replications. With 30 observations, the PDT showed sufficient power (≥ 80%) to detect medium treatment effects up to autocorrelation ≤ .45. Using 60 observations, the PDT showed sufficient power to detect medium treatment effects regardless of autocorrelation. With ≥ 90 observations, the PDT could also detect small treatment effects up to autocorrelation ≤ .30. With 30 observations, the type I error rate was 5-7%. With 60 observations and more, the type I error rate was ≤ 5% with autocorrelation < .60. The PDT outperformed the SCRT regarding power, particularly with a small number of observations. The PDT outperformed the traditional permutation test regarding type I error rate control, especially when autocorrelation increased. In conclusion, the PDT is a useful and promising nonparametric test to evaluate treatment effects in dependent SCOD AB-phase data without linear trends.

Testing delayed, gradual, and temporary treatment effects in randomized single-case experiments: A general response function framework.

Randomization tests represent a class of significance tests to assess the statistical significance of treatment effects in randomized single-case experiments. Most applications of single-case randomization tests concern simple treatment effects: immediate, abrupt, and permanent changes in the level of the outcome variable. However, researchers are confronted with delayed, gradual, and temporary treatment effects; in general, with "response functions" that are markedly different from single-step functions. We here introduce a general framework that allows specifying a test statistic for a randomization test based on predicted response functions that is sensitive to a wide variety of data patterns beyond immediate and sustained changes in level: different latencies (degrees of delay) of effect, abrupt versus gradual effects, and different durations of the effect (permanent or temporary). There may be reasonable expectations regarding the kind of effect (abrupt or gradual), entailing a different focal data feature (e.g., level or slope). However, the exact amount of latency and the exact duration of a temporary effect may not be known a priori, justifying an exploratory approach studying the effect of specifying different latencies or delayed effects and different durations for temporary effects. We provide illustrations of the proposal with real data, and we present a user-friendly freely available web application implementing it.

It is probably a pattern: does spontaneous focusing on regularities in preschool predict reasoning about randomness four years later?

The many studies with coin-tossing tasks in literature show that the concept of randomness is challenging for adults as well as children. Systematic errors observed in coin-tossing tasks are often related to the representativeness heuristic, which refers to a mental shortcut that is used to judge randomness by evaluating how well a set of random events represents the typical example for random events we hold in our mind. Representative thinking is explained by our tendency to seek for patterns in our surroundings. In the present study, predictions of coin-tosses of 302 third-graders were explored. Findings suggest that in third grade of elementary school, children make correct as well as different types of erroneous predictions and individual differences exist. Moreover, erroneous predictions that were in line with representative thinking were positively associated with an early spontaneous focus on regularities, which was assessed when they were in second year of preschool. We concluded that previous studies might have underestimated children's reasoning about randomness in coin-tossing contexts and that representative thinking is indeed associated with pattern-based thinking tendencies.

Out of proportion or out of context? Comparing 8- to 9-year-olds' proportional reasoning abilities across fair-sharing, mixtures, and probability contexts.

Findings on children's proportional reasoning abilities strongly vary across studies. This might be due to the different contexts that can be used in proportional problems: fair-sharing, mixtures, and probability. A review of the scientific literature suggests that the context of proportional problems may not only impact the difficulty of the problem, but that it also plays an important role in how children approach the problems. In other words, different contexts might elicit different (erroneous) thinking strategies. The aim of the present study was to investigate the role of context in third graders' (n = 305) proportional reasoning abilities. Results showed that children performed significantly better in a fair-sharing context compared to a mixture and a probability context. No evidence was found for a difference in performance on the mixture and the probability context. However, the kind of erroneous answers that were given in the mixture and probability context differed slightly, with more additive answers in the mixture context and more one-dimensional answers in the probability context. These findings suggest that the type of answers elicited by proportional problems might depend on the specific context in which the problem is presented.

The randomized marker method for single-case randomization tests: Handling data missing at random and data missing not at random.

Single-case experiments are frequently plagued by missing data problems. In a recent study, the randomized marker method was found to be valid and powerful for single-case randomization tests when the missing data were missing completely at random. However, in real-life experiments, it is difficult for researchers to ascertain the missing data mechanism. For analyzing such experiments, it is essential that the missing data handling method is valid and powerful for various missing data mechanisms. Hence, we examined the performance of the randomized marker method for data that are missing at random and data that are missing not at random. In addition, we compared the randomized marker method with multiple imputation, because the latter is often considered the gold standard among imputation techniques. To compare and evaluate these two methods under various simulation conditions, we calculated the type I error rate and statistical power in single-case randomization tests using these two methods of handling missing data and compared them to the type I error rate and statistical power using complete datasets. The results indicate that while multiple imputation presents an advantage in the presence of strongly correlated covariate data, the randomized marker method remains valid and results in sufficient statistical power for most of the missing data conditions simulated in this study.

A systematic review of applied single-case research published between 2016 and 2018: Study designs, randomization, data aspects, and data analysis.

Single-case experimental designs (SCEDs) have become a popular research methodology in educational science, psychology, and beyond. The growing popularity has been accompanied by the development of specific guidelines for the conduct and analysis of SCEDs. In this paper, we examine recent practices in the conduct and analysis of SCEDs by systematically reviewing applied SCEDs published over a period of three years (2016-2018). Specifically, we were interested in which designs are most frequently used and how common randomization in the study design is, which data aspects applied single-case researchers analyze, and which analytical methods are used. The systematic review of 423 studies suggests that the multiple baseline design continues to be the most widely used design and that the difference in central tendency level is by far most popular in SCED effect evaluation. Visual analysis paired with descriptive statistics is the most frequently used method of data analysis. However, inferential statistical methods and the inclusion of randomization in the study design are not uncommon. We discuss these results in light of the findings of earlier systematic reviews and suggest future directions for the development of SCED methodology.

Estimating outcome-specific effects in meta-analyses of multiple outcomes: A simulation study.

In meta-analysis, primary studies often include multiple, dependent effect sizes. Several methods address this dependency, such as the multivariate approach, three-level models, and the robust variance estimation (RVE) method. As for today, most simulation studies that explore the performance of these methods have focused on the estimation of the overall effect size. However, researchers are sometimes interested in obtaining separate effect size estimates for different types of outcomes. A recent simulation study (Park & Beretvas, 2019) has compared the performance of the three-level approach and the RVE method in estimating outcome-specific effects when several effect sizes are reported for different types of outcomes within studies. The goal of this paper is to extend that study by incorporating additional simulation conditions and by exploring the performance of additional models, such as the multivariate model, a three-level model that specifies different study-effects for each type of outcome, a three-level model that specifies a common study-effect for all outcomes, and separate three-level models for each type of outcome. Additionally, we also tested whether the a posteriori application of the RV correction improves the standard error estimates and the 95% confidence intervals. Results show that the application of separate three-level models for each type of outcome is the only approach that consistently gives adequate standard error estimates. Also, the a posteriori application of the RV correction results in correct 95% confidence intervals in all models, even if they are misspecified, meaning that Type I error rate is adequate when the RV correction is implemented.

Family-centered practices in home-based support for families with children with an intellectual disability: Judgments of parents and professionals.

BACKGROUND: The realization of the family-centered approach (FCA) in home-based support (HBS) for families with children with an intellectual disability (ID) in Flanders was investigated, and parents' and family workers' perspectives were compared. The relation between parents' educational level, the family worker's education, and his/her experience in HBS; and parents' and family workers' judgments on the realization of the FCA was considered. METHOD: Parents (N = 58 families) and family workers (N = 46) completed the helpgiving practices scale and the enabling practices scale. RESULTS: The FCA was largely present, parents rated its realization higher than family workers. Considering family workers' answers, parents' educational level appeared an important factor for parental autonomy. CONCLUSIONS: The study confirms recent research on the realization of the FCA. Including different perspectives, a nuanced view on the realization of the FCA was obtained. Further research on the concrete meaning, interpretation, and elaboration of the FCA is needed.

Handling missing data in randomization tests for single-case experiments: A simulation study.

Single-case experiments have become increasingly popular in psychological and educational research. However, the analysis of single-case data is often complicated by the frequent occurrence of missing or incomplete data. If missingness or incompleteness cannot be avoided, it becomes important to know which strategies are optimal, because the presence of missing data or inadequate data handling strategies may lead to experiments no longer "meeting standards" set by, for example, the What Works Clearinghouse. For the examination and comparison of strategies to handle missing data, we simulated complete datasets for ABAB phase designs, randomized block designs, and multiple-baseline designs. We introduced different levels of missingness in the simulated datasets by randomly deleting 10%, 30%, and 50% of the data. We evaluated the type I error rate and statistical power of a randomization test for the null hypothesis that there was no treatment effect under these different levels of missingness, using different strategies for handling missing data: (1) randomizing a missing-data marker and calculating all reference statistics only for the available data points, (2) estimating the missing data points by single imputation using the state space representation of a time series model, and (3) multiple imputation based on regressing the available data points on preceding and succeeding data points. The results are conclusive for the conditions simulated: The randomized-marker method outperforms the other two methods in terms of statistical power in a randomization test, while keeping the type I error rate under control.

A randomization test wrapper for synthesizing single-case experiments using multilevel models: A Monte Carlo simulation study.

Multilevel models (MLMs) have been proposed in single-case research, to synthesize data from a group of cases in a multiple-baseline design (MBD). A limitation of this approach is that MLMs require several statistical assumptions that are often violated in single-case research. In this article we propose a solution to this limitation by presenting a randomization test (RT) wrapper for MLMs that offers a nonparametric way to evaluate treatment effects, without making distributional assumptions or an assumption of random sampling. We present the rationale underlying the proposed technique and validate its performance (with respect to Type I error rate and power) as compared to parametric statistical inference in MLMs, in the context of evaluating the average treatment effect across cases in an MBD. We performed a simulation study that manipulated the numbers of cases and of observations per case in a dataset, the data variability between cases, the distributional characteristics of the data, the level of autocorrelation, and the size of the treatment effect in the data. The results showed that the power of the RT wrapper is superior to the power of parametric tests based on F distributions for MBDs with fewer than five cases, and that the Type I error rate of the RT wrapper is controlled for bimodal data, whereas this is not the case for traditional MLMs.

The application of meta-analytic (multi-level) models with multiple random effects: A systematic review.

In meta-analysis, study participants are nested within studies, leading to a multilevel data structure. The traditional random effects model can be considered as a model with a random study effect, but additional random effects can be added in order to account for dependent effects sizes within or across studies. The goal of this systematic review is three-fold. First, we will describe how multilevel models with multiple random effects (i.e., hierarchical three-, four-, five-level models and cross-classified random effects models) are applied in meta-analysis. Second, we will illustrate how in some specific three-level meta-analyses, a more sophisticated model could have been used to deal with additional dependencies in the data. Third and last, we will describe the distribution of the characteristics of multilevel meta-analyses (e.g., distribution of the number of outcomes across studies or which dependencies are typically modeled) so that future simulation studies can simulate more realistic conditions. Results showed that four- or five-level or cross-classified random effects models are not often used although they might account better for the meta-analytic data structure of the analyzed datasets. Also, we found that the simulation studies done on multilevel meta-analysis with multiple random factors could have used more realistic simulation factor conditions. The implications of these results are discussed, and further suggestions are given.

Cognitive Behavioral Therapy for Treatment-Related Fatigue in Chronic Myeloid Leukemia Patients on Tyrosine Kinase Inhibitors: A Mixed-Method Study.

Treatment-related fatigue significantly limits quality of life among chronic myeloid leukemia (CML) patients receiving tyrosine kinase inhibitors (TKIs), yet no interventions to reduce this symptom have been studied. We examined preliminary feasibility and efficacy of cognitive behavioral therapy for TKI treatment-related fatigue in patients with CML. We used a mixed methods convergent design and collected quantitative data through randomized single-case experiments. We included CML patients receiving TKIs and reporting severe fatigue. Within each participant, we compared CBT to a no-treatment baseline period. Fatigue severity was measured weekly with the Checklist Individual Strength. Fatigue scores were subjected to visual analyses and randomization tests for single-case experimental designs. We conducted qualitative interviews after study participation and focused on feasibility and efficacy of CBT. Visual inspection of line graphs indicated downward trends in the expected direction for fatigue in two of the four participants. The test statistics showed a decrease in fatigue severity for all participants but randomization tests did not reach statistical significance (overall p = 0.18). Participants reported qualitative improvements in level of functioning and coping with fatigue. CBT was considered feasible and acceptable for severely fatigued CML patients. Our study provided preliminary evidence for the feasibility and acceptability of CBT for severely fatigued CML patients receiving targeted therapy. We recommend further efficacy testing of this promising intervention in a pilot randomized controlled trial.

Nonparametric meta-analysis for single-case research: Confidence intervals for combined effect sizes.

In this article we present a nonparametric technique for meta-analyzing randomized single-case experiments by using inverted randomization tests to calculate nonparametric confidence intervals for combined effect sizes (CICES). Over the years, several proposals for single-case meta-analysis have been made, but most of these proposals assume either specific population characteristics (e.g., heterogeneity of variances or normality) or independent observations. However, such assumptions are seldom plausible in single-case research. The CICES technique does not require such assumptions, but only assumes that the combined effect size of multiple randomized single-case experiments can be modeled as a constant difference in the phase means. CICES can be used to synthesize the results from various single-case alternation designs, single-case phase designs, or a combination of the two. Furthermore, the technique can be used with different standardized or unstandardized effect size measures. In this article, we explain the rationale behind the CICES technique and provide illustrations with empirical as well as hypothetical datasets. In addition, we discuss the strengths and weaknesses of this technique and offer some possibilities for future research. We have implemented the CICES technique for single-case meta-analysis in a freely available R function.

Randomized single-case AB phase designs: Prospects and pitfalls.

Single-case experimental designs (SCEDs) are increasingly used in fields such as clinical psychology and educational psychology for the evaluation of treatments and interventions in individual participants. The AB phase design, also known as the interrupted time series design, is one of the most basic SCEDs used in practice. Randomization can be included in this design by randomly determining the start point of the intervention. In this article, we first introduce this randomized AB phase design and review its advantages and disadvantages. Second, we present some data-analytical possibilities and pitfalls related to this design and show how the use of randomization tests can mitigate or remedy some of these pitfalls. Third, we demonstrate that the Type I error of randomization tests in randomized AB phase designs is under control in the presence of unexpected linear trends in the data. Fourth, we report the results of a simulation study investigating the effect of unexpected linear trends on the power of the randomization test in randomized AB phase designs. The implications of these results for the analysis of randomized AB phase designs are discussed. We conclude that randomized AB phase designs are experimentally valid, but that the power of these designs is sufficient only for large treatment effects and large sample sizes. For small treatment effects and small sample sizes, researchers should turn to more complex phase designs, such as randomized ABAB phase designs or randomized multiple-baseline designs.

A demonstration and evaluation of the use of cross-classified random-effects models for meta-analysis.

It is common for the primary studies in meta-analyses to report multiple effect sizes, generating dependence among them. Hierarchical three-level models have been proposed as a means to deal with this dependency. Sometimes, however, dependency may be due to multiple random factors, and random factors are not necessarily nested, but rather may be crossed. For instance, effect sizes may belong to different studies, and, at the same time, effect sizes might represent the effects on different outcomes. Cross-classified random-effects models (CCREMs) can be used to model this nonhierarchical dependent structure. In this article, we explore by means of a simulation study the performance of CCREMs in comparison with the use of other meta-analytic models and estimation procedures, including the use of three- and two-level models and robust variance estimation. We also evaluated the performance of CCREMs when the underlying data were generated using a multivariate model. The results indicated that, whereas the quality of fixed-effect estimates is unaffected by any misspecification in the model, the standard error estimates of the mean effect size and of the moderator variables' effects, as well as the variance component estimates, are biased under some conditions. Applying CCREMs led to unbiased fixed-effect and variance component estimates, outperforming the other models. Even when a CCREM was not used to generate the data, applying the CCREM yielded sound parameter estimates and inferences.

Concealed correlations meta-analysis: A new method for synthesizing standardized regression coefficients.

The synthesis of standardized regression coefficients is still a controversial issue in the field of meta-analysis. The difficulty lies in the fact that the standardized regression coefficients belonging to regression models that include different sets of covariates do not represent the same parameter, and thus their direct combination is meaningless. In the present study, a new approach called concealed correlations meta-analysis is proposed that allows for using the common information that standardized regression coefficients from different regression models contain to improve the precision of a combined focal standardized regression coefficient estimate. The performance of this new approach was compared with that of two other approaches: (1) carrying out separate meta-analyses for standardized regression coefficients from studies that used the same regression model, and (2) performing a meta-regression on the focal standardized regression coefficients while including an indicator variable as a moderator indicating the regression model to which each standardized regression coefficient belongs. The comparison was done through a simulation study. The results showed that, as expected, the proposed approach led to more accurate estimates of the combined standardized regression coefficients under both random- and fixed-effect models.

The conditional power of randomization tests for single-case effect sizes in designs with randomized treatment order: A Monte Carlo simulation study.

The conditional power (CP) of the randomization test (RT) was investigated in a simulation study in which three different single-case effect size (ES) measures were used as the test statistics: the mean difference (MD), the percentage of nonoverlapping data (PND), and the nonoverlap of all pairs (NAP). Furthermore, we studied the effect of the experimental design on the RT's CP for three different single-case designs with rapid treatment alternation: the completely randomized design (CRD), the randomized block design (RBD), and the restricted randomized alternation design (RRAD). As a third goal, we evaluated the CP of the RT for three types of simulated data: data generated from a standard normal distribution, data generated from a uniform distribution, and data generated from a first-order autoregressive Gaussian process. The results showed that the MD and NAP perform very similarly in terms of CP, whereas the PND performs substantially worse. Furthermore, the RRAD yielded marginally higher power in the RT, followed by the CRD and then the RBD. Finally, the power of the RT was almost unaffected by the type of the simulated data. On the basis of the results of the simulation study, we recommend at least 20 measurement occasions for single-case designs with a randomized treatment order that are to be evaluated with an RT using a 5% significance level. Furthermore, we do not recommend use of the PND, because of its low power in the RT.

Why Humans Fail in Solving the Monty Hall Dilemma: A Systematic Review.

The Monty Hall dilemma (MHD) is a difficult brain teaser. We present a systematic review of literature published between January 2000 and February 2018 addressing why humans systematically fail to react optimally to the MHD or fail to understand it. Based on a sequential analysis of the phases in the MHD, we first review causes in each of these phases that may prohibit humans to react optimally and to fully understand the problem. Next, we address the question whether humans' performance, in terms of choice behaviour and (probability) understanding, can be improved. Finally, we discuss individual differences related to people's suboptimal performance. This review provides novel insights by means of its holistic approach of the MHD: At each phase, there are reasons to expect that people respond suboptimally. Given that the occurrence of only one cause is sufficient, it is not surprising that suboptimal responses are so widespread and people rarely understand the MHD.

Mood disturbances during combined oral contraceptive use and the effect of androgen supplementation. Results of a double-blind, placebo-controlled, single-case alternation design pilot study.

OBJECTIVES: To evaluate the effect of androgen supplementation in healthy combined oral contraceptive (COC) users who experience mood disturbances during COC-use only. METHODS: Six women with mood disturbances during COC-use only, received COC with co-treatment of 50 mg dehydroepiandrosterone (DHEA) during three cycles and placebo during another three cycles in an individualized random order. Daily mood rating was measured by a single item: 'In what kind of mood have you been in the past 24 h?' The results were analysed using a randomisation test for single-case experimental designs. RESULTS: The p values for the alternation design randomisation tests on the raw data of the six healthy individuals varied between 0.21 and 1, indicating that the average daily mood ratings of the active treatment DHEA are not statistically significantly larger than the average daily mood ratings of placebo. The combined p value of the subjects using a DRSP-containing pill was 0.97, and of the subjects using an LNG-containing pill was 0.65, indicating no statistically significant treatment effect for any of the pill types. CONCLUSIONS: In this single-case alternation design study, concomitant treatment with DHEA for intermittent periods of 4 weeks did not result in improvement of mood disturbances related to COC-use, but had also no side-effects.

Confidence intervals for single-case effect size measures based on randomization test inversion.

In the current paper, we present a method to construct nonparametric confidence intervals (CIs) for single-case effect size measures in the context of various single-case designs. We use the relationship between a two-sided statistical hypothesis test at significance level α and a 100 (1 - α) % two-sided CI to construct CIs for any effect size measure θ that contain all point null hypothesis θ values that cannot be rejected by the hypothesis test at significance level α. This method of hypothesis test inversion (HTI) can be employed using a randomization test as the statistical hypothesis test in order to construct a nonparametric CI for θ. We will refer to this procedure as randomization test inversion (RTI). We illustrate RTI in a situation in which θ is the unstandardized and the standardized difference in means between two treatments in a completely randomized single-case design. Additionally, we demonstrate how RTI can be extended to other types of single-case designs. Finally, we discuss a few challenges for RTI as well as possibilities when using the method with other effect size measures, such as rank-based nonoverlap indices. Supplementary to this paper, we provide easy-to-use R code, which allows the user to construct nonparametric CIs according to the proposed method.