Multilevel Meta-Analysis of Individual Participant Data of Single-Case Experimental Designs: One-Stage versus Two-Stage Methods.

To conduct a multilevel meta-analysis of multiple single-case experimental design (SCED) studies, the individual participant data (IPD) can be analyzed in one or two stages. In the one-stage approach, a multilevel model is estimated based on the raw data. In the two-stage approach, an effect size is calculated for each participant and these effect sizes and their sampling variances are subsequently combined to estimate a meta-analytic multilevel model. The multilevel model in the two-stage approach has fewer parameters to estimate, in exchange for the reduction of information of the raw data to effect sizes. In this paper we explore how the one-stage and two-stage IPD approaches can be applied in the context of meta-analysis of single-case designs. Both approaches are compared for several single-case designs of increasing complexity. Through a simulation study we show that the two-stage approach obtains better convergence rates for more complex models, but that model estimation does not necessarily converge at a faster speed. The point estimates of the fixed effects are unbiased for both approaches across all models, as such confirming results from methodological research on IPD meta-analysis of group-comparison designs. In light of these results, we discuss the implementation of both methods in R.

Estimating outcome-specific effects in meta-analyses of multiple outcomes: A simulation study.

In meta-analysis, primary studies often include multiple, dependent effect sizes. Several methods address this dependency, such as the multivariate approach, three-level models, and the robust variance estimation (RVE) method. As for today, most simulation studies that explore the performance of these methods have focused on the estimation of the overall effect size. However, researchers are sometimes interested in obtaining separate effect size estimates for different types of outcomes. A recent simulation study (Park & Beretvas, 2019) has compared the performance of the three-level approach and the RVE method in estimating outcome-specific effects when several effect sizes are reported for different types of outcomes within studies. The goal of this paper is to extend that study by incorporating additional simulation conditions and by exploring the performance of additional models, such as the multivariate model, a three-level model that specifies different study-effects for each type of outcome, a three-level model that specifies a common study-effect for all outcomes, and separate three-level models for each type of outcome. Additionally, we also tested whether the a posteriori application of the RV correction improves the standard error estimates and the 95% confidence intervals. Results show that the application of separate three-level models for each type of outcome is the only approach that consistently gives adequate standard error estimates. Also, the a posteriori application of the RV correction results in correct 95% confidence intervals in all models, even if they are misspecified, meaning that Type I error rate is adequate when the RV correction is implemented.

MultiSCED: A tool for (meta-)analyzing single-case experimental data with multilevel modeling.

The MultiSCED web application has been developed to assist applied researchers in behavioral sciences to apply multilevel modeling to quantitatively summarize single-case experimental design (SCED) studies through a user-friendly point-and-click interface embedded within R. In this paper, we offer a brief introduction to the application, explaining how to define and estimate the relevant multilevel models and how to interpret the results numerically and graphically. The use of the application is illustrated through a re-analysis of an existing meta-analytic dataset. By guiding applied researchers through MultiSCED, we aim to make use of the multilevel modeling technique for combining SCED data across cases and across studies more comprehensible and accessible.

The application of meta-analytic (multi-level) models with multiple random effects: A systematic review.

In meta-analysis, study participants are nested within studies, leading to a multilevel data structure. The traditional random effects model can be considered as a model with a random study effect, but additional random effects can be added in order to account for dependent effects sizes within or across studies. The goal of this systematic review is three-fold. First, we will describe how multilevel models with multiple random effects (i.e., hierarchical three-, four-, five-level models and cross-classified random effects models) are applied in meta-analysis. Second, we will illustrate how in some specific three-level meta-analyses, a more sophisticated model could have been used to deal with additional dependencies in the data. Third and last, we will describe the distribution of the characteristics of multilevel meta-analyses (e.g., distribution of the number of outcomes across studies or which dependencies are typically modeled) so that future simulation studies can simulate more realistic conditions. Results showed that four- or five-level or cross-classified random effects models are not often used although they might account better for the meta-analytic data structure of the analyzed datasets. Also, we found that the simulation studies done on multilevel meta-analysis with multiple random factors could have used more realistic simulation factor conditions. The implications of these results are discussed, and further suggestions are given.

Multilevel meta-analysis of multiple regression coefficients from single-case experimental studies.

The focus of the current study is on handling the dependence among multiple regression coefficients representing the treatment effects when meta-analyzing data from single-case experimental studies. We compare the results when applying three different multilevel meta-analytic models (i.e., a univariate multilevel model avoiding the dependence, a multivariate multilevel model ignoring covariance at higher levels, and a multivariate multilevel model modeling the existing covariance) to deal with the dependent effect sizes. The results indicate better estimates of the overall treatment effects and variance components when a multivariate multilevel model is applied, independent of modeling or ignoring the existing covariance. These findings confirm the robustness of multilevel modeling to misspecifying the existing covariance at the case and study level in terms of estimating the overall treatment effects and variance components. The results also show that the overall treatment effect estimates are unbiased regardless of the underlying model, but the between-case and between-study variance components are biased in certain conditions. In addition, the between-study variance estimates are particularly biased when the number of studies is smaller than 40 (i.e., 10 or 20) and the true value of the between-case variance is relatively large (i.e., 8). The observed bias is larger for the between-case variance estimates compared to the between-study variance estimates when the true between-case variance is relatively small (i.e., 0.5).

Concealed correlations meta-analysis: A new method for synthesizing standardized regression coefficients.

The synthesis of standardized regression coefficients is still a controversial issue in the field of meta-analysis. The difficulty lies in the fact that the standardized regression coefficients belonging to regression models that include different sets of covariates do not represent the same parameter, and thus their direct combination is meaningless. In the present study, a new approach called concealed correlations meta-analysis is proposed that allows for using the common information that standardized regression coefficients from different regression models contain to improve the precision of a combined focal standardized regression coefficient estimate. The performance of this new approach was compared with that of two other approaches: (1) carrying out separate meta-analyses for standardized regression coefficients from studies that used the same regression model, and (2) performing a meta-regression on the focal standardized regression coefficients while including an indicator variable as a moderator indicating the regression model to which each standardized regression coefficient belongs. The comparison was done through a simulation study. The results showed that, as expected, the proposed approach led to more accurate estimates of the combined standardized regression coefficients under both random- and fixed-effect models.

A demonstration and evaluation of the use of cross-classified random-effects models for meta-analysis.

It is common for the primary studies in meta-analyses to report multiple effect sizes, generating dependence among them. Hierarchical three-level models have been proposed as a means to deal with this dependency. Sometimes, however, dependency may be due to multiple random factors, and random factors are not necessarily nested, but rather may be crossed. For instance, effect sizes may belong to different studies, and, at the same time, effect sizes might represent the effects on different outcomes. Cross-classified random-effects models (CCREMs) can be used to model this nonhierarchical dependent structure. In this article, we explore by means of a simulation study the performance of CCREMs in comparison with the use of other meta-analytic models and estimation procedures, including the use of three- and two-level models and robust variance estimation. We also evaluated the performance of CCREMs when the underlying data were generated using a multivariate model. The results indicated that, whereas the quality of fixed-effect estimates is unaffected by any misspecification in the model, the standard error estimates of the mean effect size and of the moderator variables' effects, as well as the variance component estimates, are biased under some conditions. Applying CCREMs led to unbiased fixed-effect and variance component estimates, outperforming the other models. Even when a CCREM was not used to generate the data, applying the CCREM yielded sound parameter estimates and inferences.

Analysis of single-case experimental count data using the linear mixed effects model: A simulation study.

When (meta-)analyzing single-case experimental design (SCED) studies by means of hierarchical or multilevel modeling, applied researchers almost exclusively rely on the linear mixed model (LMM). This type of model assumes that the residuals are normally distributed. However, very often SCED studies consider outcomes of a discrete rather than a continuous nature, like counts, percentages or rates. In those cases the normality assumption does not hold. The LMM can be extended into a generalized linear mixed model (GLMM), which can account for the discrete nature of SCED count data. In this simulation study, we look at the effects of misspecifying an LMM for SCED count data simulated according to a GLMM. We compare the performance of a misspecified LMM and of a GLMM in terms of goodness of fit, fixed effect parameter recovery, type I error rate, and power. Because the LMM and the GLMM do not estimate identical fixed effects, we provide a transformation to compare the fixed effect parameter recovery. The results show that, compared to the GLMM, the LMM has worse performance in terms of goodness of fit and power. Performance in terms of fixed effect parameter recovery is equally good for both models, and in terms of type I error rate the LMM performs better than the GLMM. Finally, we provide some guidelines for applied researchers about aspects to consider when using an LMM for analyzing SCED count data.

Avoid or engage? Outcomes of graded exposure in youth with chronic pain using a sequential replicated single-case randomized design.

Pain-related fear is typically associated with avoidance behavior and pain-related disability in youth with chronic pain. Youth with elevated pain-related fear have attenuated treatment responses; thus, targeted treatment is highly warranted. Evidence supporting graded in vivo exposure treatment (GET) for adults with chronic pain is considerable, but just emerging for youth. The current investigation represents the first sequential replicated and randomized single-case experimental phase design with multiple measures evaluating GET for youth with chronic pain, entitled GET Living. A cohort of 27 youth (81% female) with mixed chronic pain completed GET Living. For each participant, a no-treatment randomized baseline period was compared with GET Living and 3- and 6-month follow-ups. Daily changes in primary outcomes fear and avoidance and secondary outcomes pain catastrophizing, pain intensity, and pain acceptance were assessed using electronic diaries and subjected to descriptive and model-based inference analyses. Based on individual effect size calculations, a third of participants significantly improved by the end of treatment on fear, avoidance, and pain acceptance. By follow-up, over 80% of participants had improved across all primary and secondary outcomes. Model-based inference analysis results to examine the series of replicated cases were generally consistent. Improvements during GET Living was superior to the no-treatment randomized baseline period for avoidance, pain acceptance, and pain intensity, whereas fear and pain catastrophizing did not improve. All 5 outcomes emerged as significantly improved at 3- and 6-month follow-ups. The results of this replicated single-case experimental phase design support the effectiveness of graded exposure for youth with chronic pain and elevated pain-related fear avoidance.

Papular cutaneous lesions in a cat associated with feline infectious peritonitis.

A 7-month-old-intact male domestic shorthair cat was presented with fever, anterior uveitis in the right eye and respiratory distress when handled. These signs along with mild changes in serum protein levels and the exclusion of other potential causes were suggestive of feline infectious peritonitis (FIP). As the disease progressed, more clinical signs consistent with FIP, including renal involvement and later pleural effusion, became evident. Non-pruritic cutaneous lesions, characterized by slightly raised intradermal papules over the dorsal neck and over both lateral thoracic walls, were recognized at the end stage of the disease. The identification of papules in well-haired skin was difficult, and clipping of the fur facilitated their detection. Definitive diagnosis of FIP was made by histopathology and by immunohistochemical demonstration of coronavirus antigen in macrophages within kidney and skin lesions. The case was classified as a mixed form of FIP. Recognition of associated cutaneous lesions may facilitate a diagnosis of FIP in suspicious cases.

Methodological quality of meta-analyses of single-case experimental studies.

BACKGROUND: Methodological rigor is a fundamental factor in the validity and credibility of the results of a meta-analysis. AIM: Following an increasing interest in single-case experimental design (SCED) meta-analyses, the current study investigates the methodological quality of SCED meta-analyses. METHODS AND PROCEDURES: We assessed the methodological quality of 178 SCED meta-analyses published between 1985 and 2015 through the modified Revised-Assessment of Multiple Systematic Reviews (R-AMSTAR) checklist. OUTCOMES AND RESULTS: The main finding of the current review is that the methodological quality of the SCED meta-analyses has increased over time, but is still low according to the R-AMSTAR checklist. A remarkable percentage of the studies (93.80% of the included SCED meta-analyses) did not even reach the midpoint score (22, on a scale of 0-44). The mean and median methodological quality scores were 15.57 and 16, respectively. Relatively high scores were observed for "providing the characteristics of the included studies" and "doing comprehensive literature search". The key areas of deficiency were "reporting an assessment of the likelihood of publication bias" and "using the methods appropriately to combine the findings of studies". CONCLUSIONS AND IMPLICATIONS: Although the results of the current review reveal that the methodological quality of the SCED meta-analyses has increased over time, still more efforts are needed to improve their methodological quality.