Comparing methods for handling missing covariates in meta-regression.

Meta-analysts often encounter missing covariate values when estimating meta-regression models. In practice, ad hoc approaches involving data deletion have been widely used. The current study investigates the performance of different methods for handling missing covariates in meta-regression, including complete-case analysis (CCA), shifting-case analysis (SCA), multiple imputation (MI), and full information maximum likelihood (FIML), assuming missing at random mechanism. According to the simulation results, we advocate the use of MI and FIML than CCA and SCA approaches in practice. In addition, we cautiously note the challenges and potential advantages of using MI in the meta-analysis context.

Missingness mechanism messiness in meta-analysis: A response to "The importance of plausible data-generating mechanisms in simulation studies".

The current paper responds to the commentary on the article (doi:10.1002/jrsm.1605). We discuss our perspectives on the missing data mechanisms and models assumed and used in our simulation study while acknowledging the inherent generalizability limitations of any (simulation) study. The plausibility of the exact missing data mechanism is challenging to definitively identify in any applied dataset. We describe and justify our assumed scenario in meta-regression that we investigated. We also revisit the performance of the deletion method and how it is tied into the assumed missingness model. Lastly, we reiterate the importance of conducting sensitivity analyses assessing different ways of handling missing data given different assumptions and offer this study as a starting point for future research.

Cluster wild bootstrapping to handle dependent effect sizes in meta-analysis with a small number of studies.

The most common and well-known meta-regression models work under the assumption that there is only one effect size estimate per study and that the estimates are independent. However, meta-analytic reviews of social science research often include multiple effect size estimates per primary study, leading to dependence in the estimates. Some meta-analyses also include multiple studies conducted by the same lab or investigator, creating another potential source of dependence. An increasingly popular method to handle dependence is robust variance estimation (RVE), but this method can result in inflated Type I error rates when the number of studies is small. Small-sample correction methods for RVE have been shown to control Type I error rates adequately but may be overly conservative, especially for tests of multiple-contrast hypotheses. We evaluated an alternative method for handling dependence, cluster wild bootstrapping, which has been examined in the econometrics literature but not in the context of meta-analysis. Results from two simulation studies indicate that cluster wild bootstrapping maintains adequate Type I error rates and provides more power than extant small-sample correction methods, particularly for multiple-contrast hypothesis tests. We recommend using cluster wild bootstrapping to conduct hypothesis tests for meta-analyses with a small number of studies. We have also created an R package that implements such tests.

Multilevel Meta-Analysis of Individual Participant Data of Single-Case Experimental Designs: One-Stage versus Two-Stage Methods.

To conduct a multilevel meta-analysis of multiple single-case experimental design (SCED) studies, the individual participant data (IPD) can be analyzed in one or two stages. In the one-stage approach, a multilevel model is estimated based on the raw data. In the two-stage approach, an effect size is calculated for each participant and these effect sizes and their sampling variances are subsequently combined to estimate a meta-analytic multilevel model. The multilevel model in the two-stage approach has fewer parameters to estimate, in exchange for the reduction of information of the raw data to effect sizes. In this paper we explore how the one-stage and two-stage IPD approaches can be applied in the context of meta-analysis of single-case designs. Both approaches are compared for several single-case designs of increasing complexity. Through a simulation study we show that the two-stage approach obtains better convergence rates for more complex models, but that model estimation does not necessarily converge at a faster speed. The point estimates of the fixed effects are unbiased for both approaches across all models, as such confirming results from methodological research on IPD meta-analysis of group-comparison designs. In light of these results, we discuss the implementation of both methods in R.

Estimating outcome-specific effects in meta-analyses of multiple outcomes: A simulation study.

In meta-analysis, primary studies often include multiple, dependent effect sizes. Several methods address this dependency, such as the multivariate approach, three-level models, and the robust variance estimation (RVE) method. As for today, most simulation studies that explore the performance of these methods have focused on the estimation of the overall effect size. However, researchers are sometimes interested in obtaining separate effect size estimates for different types of outcomes. A recent simulation study (Park & Beretvas, 2019) has compared the performance of the three-level approach and the RVE method in estimating outcome-specific effects when several effect sizes are reported for different types of outcomes within studies. The goal of this paper is to extend that study by incorporating additional simulation conditions and by exploring the performance of additional models, such as the multivariate model, a three-level model that specifies different study-effects for each type of outcome, a three-level model that specifies a common study-effect for all outcomes, and separate three-level models for each type of outcome. Additionally, we also tested whether the a posteriori application of the RV correction improves the standard error estimates and the 95% confidence intervals. Results show that the application of separate three-level models for each type of outcome is the only approach that consistently gives adequate standard error estimates. Also, the a posteriori application of the RV correction results in correct 95% confidence intervals in all models, even if they are misspecified, meaning that Type I error rate is adequate when the RV correction is implemented.

The application of meta-analytic (multi-level) models with multiple random effects: A systematic review.

In meta-analysis, study participants are nested within studies, leading to a multilevel data structure. The traditional random effects model can be considered as a model with a random study effect, but additional random effects can be added in order to account for dependent effects sizes within or across studies. The goal of this systematic review is three-fold. First, we will describe how multilevel models with multiple random effects (i.e., hierarchical three-, four-, five-level models and cross-classified random effects models) are applied in meta-analysis. Second, we will illustrate how in some specific three-level meta-analyses, a more sophisticated model could have been used to deal with additional dependencies in the data. Third and last, we will describe the distribution of the characteristics of multilevel meta-analyses (e.g., distribution of the number of outcomes across studies or which dependencies are typically modeled) so that future simulation studies can simulate more realistic conditions. Results showed that four- or five-level or cross-classified random effects models are not often used although they might account better for the meta-analytic data structure of the analyzed datasets. Also, we found that the simulation studies done on multilevel meta-analysis with multiple random factors could have used more realistic simulation factor conditions. The implications of these results are discussed, and further suggestions are given.

Multilevel meta-analysis of multiple regression coefficients from single-case experimental studies.

The focus of the current study is on handling the dependence among multiple regression coefficients representing the treatment effects when meta-analyzing data from single-case experimental studies. We compare the results when applying three different multilevel meta-analytic models (i.e., a univariate multilevel model avoiding the dependence, a multivariate multilevel model ignoring covariance at higher levels, and a multivariate multilevel model modeling the existing covariance) to deal with the dependent effect sizes. The results indicate better estimates of the overall treatment effects and variance components when a multivariate multilevel model is applied, independent of modeling or ignoring the existing covariance. These findings confirm the robustness of multilevel modeling to misspecifying the existing covariance at the case and study level in terms of estimating the overall treatment effects and variance components. The results also show that the overall treatment effect estimates are unbiased regardless of the underlying model, but the between-case and between-study variance components are biased in certain conditions. In addition, the between-study variance estimates are particularly biased when the number of studies is smaller than 40 (i.e., 10 or 20) and the true value of the between-case variance is relatively large (i.e., 8). The observed bias is larger for the between-case variance estimates compared to the between-study variance estimates when the true between-case variance is relatively small (i.e., 0.5).

MultiSCED: A tool for (meta-)analyzing single-case experimental data with multilevel modeling.

The MultiSCED web application has been developed to assist applied researchers in behavioral sciences to apply multilevel modeling to quantitatively summarize single-case experimental design (SCED) studies through a user-friendly point-and-click interface embedded within R. In this paper, we offer a brief introduction to the application, explaining how to define and estimate the relevant multilevel models and how to interpret the results numerically and graphically. The use of the application is illustrated through a re-analysis of an existing meta-analytic dataset. By guiding applied researchers through MultiSCED, we aim to make use of the multilevel modeling technique for combining SCED data across cases and across studies more comprehensible and accessible.

The multivariate multiple-membership random-effect model: An introduction and evaluation.

In the present study, we focused on models that handle several data structure complexities simultaneously. We introduced and evaluated the multivariate multiple-membership random-effect model (MV-MMREM) for handling multiple-membership data in scenarios with multiple, related outcomes. Although a recent study introduced the idea of the MV-MMREM, no research has directly assessed its estimation nor demonstrated its use with real data. Therefore, we used real multiple-membership datasets that included multiple, related outcomes to demonstrate interpretation of the MV-MMREM parameters. In addition, a simulation study was conducted to assess estimation of the MV-MMREM under a number of design conditions. Also, the robustness of the results was assessed for multivariate multiple-membership data when they were analyzed using a multivariate hierarchical linear model that ignores the multiple-membership structure (MV-HLM), as well as when using multiple univariate MMREMs. The results showed that the MV-MMREM works well in comparison with both MV-HLM and univariate MMREMs when the data structure had missingness outcomes, multivariate outcomes, and multiple membership clusters. Finally, we discuss limitations of the MV-MMREM and areas for future research.

Analysis of single-case experimental count data using the linear mixed effects model: A simulation study.

When (meta-)analyzing single-case experimental design (SCED) studies by means of hierarchical or multilevel modeling, applied researchers almost exclusively rely on the linear mixed model (LMM). This type of model assumes that the residuals are normally distributed. However, very often SCED studies consider outcomes of a discrete rather than a continuous nature, like counts, percentages or rates. In those cases the normality assumption does not hold. The LMM can be extended into a generalized linear mixed model (GLMM), which can account for the discrete nature of SCED count data. In this simulation study, we look at the effects of misspecifying an LMM for SCED count data simulated according to a GLMM. We compare the performance of a misspecified LMM and of a GLMM in terms of goodness of fit, fixed effect parameter recovery, type I error rate, and power. Because the LMM and the GLMM do not estimate identical fixed effects, we provide a transformation to compare the fixed effect parameter recovery. The results show that, compared to the GLMM, the LMM has worse performance in terms of goodness of fit and power. Performance in terms of fixed effect parameter recovery is equally good for both models, and in terms of type I error rate the LMM performs better than the GLMM. Finally, we provide some guidelines for applied researchers about aspects to consider when using an LMM for analyzing SCED count data.

Synthesizing effects for multiple outcomes per study using robust variance estimation versus the three-level model.

Primary studies increasingly report information that can be used to provide multiple effect sizes. Of interest in this study, primary studies might compare a treatment and a control group on multiple related outcomes that result in multiple dependent effect sizes to be synthesized. There are a number of ways to handle the resulting within-study "multiple-outcome" dependency. The present study focuses on use of the multilevel meta-analysis model (Van den Noortgate, López-López, Marín-Martínez, & Sánchez-Meca, 2013) and robust variance estimation (Hedges, Tipton, & Johnson, 2010) for handling this dependency, as well as for estimating outcome-specific mean effect sizes. We assessed these two approaches under various conditions that differed from each other in within-study sample size; the number of effect sizes per outcome; the number of outcomes per study; the number of studies per meta-analysis; the ratio of variances at Levels 1, 2, and 3; and the true correlation between pairs of effect sizes at the between-study level. Limitations and directions for future research are discussed.

A demonstration and evaluation of the use of cross-classified random-effects models for meta-analysis.

It is common for the primary studies in meta-analyses to report multiple effect sizes, generating dependence among them. Hierarchical three-level models have been proposed as a means to deal with this dependency. Sometimes, however, dependency may be due to multiple random factors, and random factors are not necessarily nested, but rather may be crossed. For instance, effect sizes may belong to different studies, and, at the same time, effect sizes might represent the effects on different outcomes. Cross-classified random-effects models (CCREMs) can be used to model this nonhierarchical dependent structure. In this article, we explore by means of a simulation study the performance of CCREMs in comparison with the use of other meta-analytic models and estimation procedures, including the use of three- and two-level models and robust variance estimation. We also evaluated the performance of CCREMs when the underlying data were generated using a multivariate model. The results indicated that, whereas the quality of fixed-effect estimates is unaffected by any misspecification in the model, the standard error estimates of the mean effect size and of the moderator variables' effects, as well as the variance component estimates, are biased under some conditions. Applying CCREMs led to unbiased fixed-effect and variance component estimates, outperforming the other models. Even when a CCREM was not used to generate the data, applying the CCREM yielded sound parameter estimates and inferences.

Quantifying the association between acute leukemia and serum zinc, copper, and selenium: a meta-analysis.

Acute leukemia is a clonal malignant disorder that occurs when immature blast cells accumulate in bone marrow. Zinc (Zn) and copper (Cu) are related to normal lymphocyte maturation and immune function regulation. Selenium (Se) is protective against oxidative damage. The aim of this meta-analysis is to statistically synthesize results from studies that have investigated the levels of Zn, Cu, and Se in acute leukemia patients. The effect size, delta, was used to standardize the raw data. The robust variance estimation (RVE) method was performed to measure the pooled effect size and variance. Results suggest significant negative differences for levels of serum Zn (p < .05, delta = -1.21; 95% CI, -2.13--0.28) and Se (p < .05, delta = -1.84; 95% CI, -3.39--0.29) and significantly positive differences between serum Cu levels (p < .01, delta = 1.94; 95% CI, 1.02-2.87) in acute leukemia, as compared to the controls.

Zinc, Copper, and Iron in Oral Submucous Fibrosis: A Meta-Analysis.

Oral submucous fibrosis (OSF) is a potentially malignant disorder which causes fibrosis and inflammation of the oral mucosa. Studies have reported altered levels of trace elements in oral submucous fibrosis subjects, but findings have been inconsistent. The objective of this research is to perform a meta-analysis to summarize studies that report zinc (Zn), copper (Cu), and iron (Fe) in patients, with and without OSF. A literature search of Embase, PubMed, Cochrane Library, and Web of Science electronic databases was conducted for studies up to January 2017. A total of 34 reports met the inclusion criteria. The standardized mean difference was utilized as the effect size. The robust variance estimation method was chosen to handle dependency of multiple related outcomes in meta-analysis. There was a significant increase in the levels of Cu (effect size = 1.17, p value < 0.05, 95% confidence interval (CI): 0.164-2.171) and a significant decrease in levels of Zn (effect size = -1.95, p value < 0.05, 95% CI: -3.524 to -0.367) and Fe (effect size = -2.77, p value < 0.01, 95% CI: -4.126 to -1.406) in OSF patients. The estimation of Zn, Cu, and Fe levels may serve as additional biomarkers in the diagnosis and prognosis of OSF along with the clinical features.

Trace element status in type 2 diabetes: A meta-analysis.

INTRODUCTION: Type 2 diabetes is a chronic metabolic disorder that has been associated with alterations in the status of trace elements, including zinc, copper, iron and manganese. However, clinical studies reporting statuses of these trace elements in type 2 diabetes patients compared to controls have shown conflicting results. OBJECTIVE: This meta-analysis aimed to summarize the existing literature on the statuses of zinc, copper, iron, and manganese in Type 2 diabetes mellitus patients. METHODS: A literature search of Embase, PubMed, EBSCOHost, ScienceDirect, Scopus, Cochrane library and Web of Science electronic databases was conducted to find studies published from 1970 to November 2016 that compared the trace elements of interest between type 2 diabetic patients and healthy controls. Keywords used were type 2 diabetes, diabetes, hyperglycemia, insulin, glucose, HbA1c, trace elements, micronutrients, zinc, manganese, copper, ceruloplasmin, iron and ferritin. The bias corrected Hedges' g, was utilized as the effect sizes. Due to the biological interaction between trace elements, it is important to collectively evaluate the statuses of these minerals in type 2 diabetes. Thus, the robust variance estimation method was chosen to handle dependency between multiple outcomes. RESULTS: A total of 52 studies met the inclusion criteria, amounting to 98 effect sizes. Diabetic patients (n=20183) had significantly lower zinc status when compared to controls (effect size = -1.73, p<0.01); whereas copper (effect size = 1.10, p<0.05) and ferritin levels (effect size = 1.05, p<0.01) were significantly higher. Although not significant, ceruloplasmin (effect size = 1.85, p=0.06) and iron (effect size = 1.42, p=0.06) levels were higher, and manganese (effect size = 0.27, p=0.34) was lower in patients. CONCLUSION: Results from this meta-analysis indicate lower zinc status accompanied by increased copper and ferritin levels in patients with type 2 diabetes when compared to controls.

Estimation of a Latent Variable Regression Growth Curve Model for Individuals Cross-Classified by Clusters.

The cross-classified multiple membership latent variable regression (CCMM-LVR) model is a recent extension to the three-level latent variable regression (HM3-LVR) model which can be utilized for longitudinal data that contains individuals who changed clusters over time (for instance, student mobility across schools). The HM3-LVR model can include the initial status on growth effect as varying across those clusters and allows testing of more flexible hypotheses about the influence of initial status on growth and of factors that might impact that relationship, but only in the presence of pure clustering of participants within higher-level units. This Monte Carlo study was conducted to evaluate model estimation under a variety of conditions and to measure the impact of ignoring cross-classified data when estimating the incorrectly specified HM3-LVR model in a scenario in which true values for parameters are known. Furthermore, results from a real-data analysis were used to inform the design of the simulation. Overall, it would be recommended for researchers to utilize the CCMM-LVR model over the HM3-LVR model when individuals are cross-classified, and to use a bare minimum of more than 100 clustering units in order to avoid overestimation of the level-3 variance component estimates.

The impact of response-guided baseline phase extensions on treatment effect estimates.

BACKGROUND: When developmental disabilities researchers use multiple-baseline designs they are encouraged to delay the start of an intervention until the baseline stabilizes or until preceding cases have responded to intervention. Using ongoing visual analyses to guide the timing of the start of the intervention can help to resolve potential ambiguities in the graphical display; however, these forms of response-guided experimentation have been criticized as a potential source of bias in treatment effect estimation and inference. AIMS AND METHODS: Monte Carlo simulations were used to examine the bias and precision of average treatment effect estimates obtained from multilevel models of four-case multiple-baseline studies with series lengths that varied from 19 to 49 observations per case. We varied the size of the average treatment effect, the factors used to guide intervention decisions (baseline stability, response to intervention, both, or neither), and whether the ongoing analysis was masked or not. RESULTS: None of the methods of responding to the data led to appreciable bias in the treatment effect estimates. Furthermore, as timing-of-intervention decisions became responsive to more factors, baselines became longer and treatment effect estimates became more precise. CONCLUSIONS: Although the study was conducted under limited conditions, the response-guided practices did not lead to substantial bias. By extending baseline phases they reduced estimation error and thus improved the treatment effect estimates obtained from multilevel models.

Methodological quality of meta-analyses of single-case experimental studies.

BACKGROUND: Methodological rigor is a fundamental factor in the validity and credibility of the results of a meta-analysis. AIM: Following an increasing interest in single-case experimental design (SCED) meta-analyses, the current study investigates the methodological quality of SCED meta-analyses. METHODS AND PROCEDURES: We assessed the methodological quality of 178 SCED meta-analyses published between 1985 and 2015 through the modified Revised-Assessment of Multiple Systematic Reviews (R-AMSTAR) checklist. OUTCOMES AND RESULTS: The main finding of the current review is that the methodological quality of the SCED meta-analyses has increased over time, but is still low according to the R-AMSTAR checklist. A remarkable percentage of the studies (93.80% of the included SCED meta-analyses) did not even reach the midpoint score (22, on a scale of 0-44). The mean and median methodological quality scores were 15.57 and 16, respectively. Relatively high scores were observed for "providing the characteristics of the included studies" and "doing comprehensive literature search". The key areas of deficiency were "reporting an assessment of the likelihood of publication bias" and "using the methods appropriately to combine the findings of studies". CONCLUSIONS AND IMPLICATIONS: Although the results of the current review reveal that the methodological quality of the SCED meta-analyses has increased over time, still more efforts are needed to improve their methodological quality.