What do meta-analysts need in primary studies? Guidelines and the SEMI checklist for facilitating cumulative knowledge.

Meta-analysis is often recognized as the highest level of evidence due to its notable advantages. Therefore, ensuring the precision of its findings is of utmost importance. Insufficient reporting in primary studies poses challenges for meta-analysts, hindering study identification, effect size estimation, and meta-regression analyses. This manuscript provides concise guidelines for the comprehensive reporting of qualitative and quantitative aspects in primary studies. Adhering to these guidelines may help researchers enhance the quality of their studies and increase their eligibility for inclusion in future research syntheses, thereby enhancing research synthesis quality. Recommendations include incorporating relevant terms in titles and abstracts to facilitate study retrieval and reporting sufficient data for effect size calculation. Additionally, a new checklist is introduced to help applied researchers thoroughly report various aspects of their studies.

A meta-analysis of cognitive interventions for patients with recent onset psychosis: are they effective for improving functioning?

The increasing popularity of cognitive interventions for patients with psychosis calls for further exploration on how these interventions may benefit functional outcomes. We conducted a meta-analysis of randomized controlled trials (RCTs) to examine the effectiveness of cognitive interventions (i.e. Cognitive Remediation, Cognitive Training, Social Cognition, and their combination) on functioning of patients with recent onset psychosis, established as the period within the first five years from the first episode. The following databases were searched: Proquest, PUBMED/MEDLINE, PsycINFO, WOS, Scopus for research published until January 2022. In total, 12 studies were eligible. The total number of participants was 759, of which 32.2% in the intervention and 30.8% in the control group were female. We extracted data to calculate the standardized mean change from pre-test to post-test comparing the intervention with the control conditions. Overall, there was no effect of any of the cognitive intervention types on functioning. None of the examined factors (intervention type, length, and modality; control condition, follow-up time; cognitive functions; medication; symptoms) seemed to moderate these findings. Our results indicate that cognitive interventions as standalone interventions do not appear to improve functioning in patients with recent onset psychosis. Given the small number of eligible studies, further RCTs with larger and more refined samples are needed to test whether these interventions should be applied as single interventions with these patients.

Efficacy of technology-based interventions in psychosis: a systematic review and network meta-analysis.

BACKGROUND: Technology-based interventions (TBIs) are a useful approach when attempting to provide therapy to more patients with psychosis. METHODS: Randomized controlled trials of outcomes of TBIs v. face-to-face interventions in psychosis were identified in a systematic search conducted in PubMed/Ovid MEDLINE. Data were extracted independently by two researchers, and standardized mean changes were pooled using a three-level model and network meta-analysis. RESULTS: Fifty-eight studies were included. TBIs complementing treatment as usual (TAU) were generally superior to face-to-face interventions (g = 0.16, p ≤ 0.0001) and to specific outcomes, namely, neurocognition (g = 0.13, p ≤ 0.0001), functioning (g = 0.25, p = 0.006), and social cognition (g = 0.32, p ≤ 0.05). Based on the network meta-analysis, the effect of two TBIs differed significantly from zero; these were the TBIs cognitive training for the neurocognitive outcome [g = 0.16; 95% confidence interval (CI) 0.09-0.23] and cognitive behavioral therapy for quality of life (g = 1.27; 95% CI 0.46-2.08). The variables educational level, type of medication, frequency of the intervention, and contact during the intervention moderated the effectiveness of TBIs over face-to-face interventions in neurocognition and symptomatology. CONCLUSIONS: TBIs are effective for the management of neurocognition, symptomatology, functioning, social cognition, and quality of life outcomes in patients with psychosis. The results of the network meta-analysis showed the efficacy of some TBIs for neurocognition, symptomatology, and quality of life. Therefore, TBIs should be considered a complement to TAU in patients with psychosis.

Network meta-analysis in psychology and educational sciences: A systematic review of their characteristics.

Network meta-analysis (NMA) allows the combination of evidence on the effectiveness of several interventions. NMA has mainly been applied in the medical science field, whereas in the domain of psychology and educational sciences its use is less frequent. Consequently, systematic reviews that describe the characteristics of published NMAs are limited to the field of medicine, and nothing is known about the characteristics of NMAs published in the psychology and educational sciences field. However, this information is still relevant for the design of future simulation studies and for detecting good and bad research practices. Thus, this study describes the features of the meta-analytic datasets of NMAs published in the field of psychology and educational sciences, as well as their methodological characteristics, and compares them to those observed in the medical domain. Results show that the number of studies included is larger in NMAs from psychology and educational sciences, the most commonly used effect size is the standardized mean difference (unlike the odds ratio in medicine), the sample size is smaller, more intervention groups are included, and inconsistent effects are observed more often. These results can be used in future simulation studies to generate realistic datasets. Finally, we warn about the poor quality of reporting of some technical aspects of the NMA, such as the statistical model used.

Multilevel Meta-Analysis of Individual Participant Data of Single-Case Experimental Designs: One-Stage versus Two-Stage Methods.

To conduct a multilevel meta-analysis of multiple single-case experimental design (SCED) studies, the individual participant data (IPD) can be analyzed in one or two stages. In the one-stage approach, a multilevel model is estimated based on the raw data. In the two-stage approach, an effect size is calculated for each participant and these effect sizes and their sampling variances are subsequently combined to estimate a meta-analytic multilevel model. The multilevel model in the two-stage approach has fewer parameters to estimate, in exchange for the reduction of information of the raw data to effect sizes. In this paper we explore how the one-stage and two-stage IPD approaches can be applied in the context of meta-analysis of single-case designs. Both approaches are compared for several single-case designs of increasing complexity. Through a simulation study we show that the two-stage approach obtains better convergence rates for more complex models, but that model estimation does not necessarily converge at a faster speed. The point estimates of the fixed effects are unbiased for both approaches across all models, as such confirming results from methodological research on IPD meta-analysis of group-comparison designs. In light of these results, we discuss the implementation of both methods in R.

Estimating outcome-specific effects in meta-analyses of multiple outcomes: A simulation study.

In meta-analysis, primary studies often include multiple, dependent effect sizes. Several methods address this dependency, such as the multivariate approach, three-level models, and the robust variance estimation (RVE) method. As for today, most simulation studies that explore the performance of these methods have focused on the estimation of the overall effect size. However, researchers are sometimes interested in obtaining separate effect size estimates for different types of outcomes. A recent simulation study (Park & Beretvas, 2019) has compared the performance of the three-level approach and the RVE method in estimating outcome-specific effects when several effect sizes are reported for different types of outcomes within studies. The goal of this paper is to extend that study by incorporating additional simulation conditions and by exploring the performance of additional models, such as the multivariate model, a three-level model that specifies different study-effects for each type of outcome, a three-level model that specifies a common study-effect for all outcomes, and separate three-level models for each type of outcome. Additionally, we also tested whether the a posteriori application of the RV correction improves the standard error estimates and the 95% confidence intervals. Results show that the application of separate three-level models for each type of outcome is the only approach that consistently gives adequate standard error estimates. Also, the a posteriori application of the RV correction results in correct 95% confidence intervals in all models, even if they are misspecified, meaning that Type I error rate is adequate when the RV correction is implemented.

The application of meta-analytic (multi-level) models with multiple random effects: A systematic review.

In meta-analysis, study participants are nested within studies, leading to a multilevel data structure. The traditional random effects model can be considered as a model with a random study effect, but additional random effects can be added in order to account for dependent effects sizes within or across studies. The goal of this systematic review is three-fold. First, we will describe how multilevel models with multiple random effects (i.e., hierarchical three-, four-, five-level models and cross-classified random effects models) are applied in meta-analysis. Second, we will illustrate how in some specific three-level meta-analyses, a more sophisticated model could have been used to deal with additional dependencies in the data. Third and last, we will describe the distribution of the characteristics of multilevel meta-analyses (e.g., distribution of the number of outcomes across studies or which dependencies are typically modeled) so that future simulation studies can simulate more realistic conditions. Results showed that four- or five-level or cross-classified random effects models are not often used although they might account better for the meta-analytic data structure of the analyzed datasets. Also, we found that the simulation studies done on multilevel meta-analysis with multiple random factors could have used more realistic simulation factor conditions. The implications of these results are discussed, and further suggestions are given.

Multilevel meta-analysis of multiple regression coefficients from single-case experimental studies.

The focus of the current study is on handling the dependence among multiple regression coefficients representing the treatment effects when meta-analyzing data from single-case experimental studies. We compare the results when applying three different multilevel meta-analytic models (i.e., a univariate multilevel model avoiding the dependence, a multivariate multilevel model ignoring covariance at higher levels, and a multivariate multilevel model modeling the existing covariance) to deal with the dependent effect sizes. The results indicate better estimates of the overall treatment effects and variance components when a multivariate multilevel model is applied, independent of modeling or ignoring the existing covariance. These findings confirm the robustness of multilevel modeling to misspecifying the existing covariance at the case and study level in terms of estimating the overall treatment effects and variance components. The results also show that the overall treatment effect estimates are unbiased regardless of the underlying model, but the between-case and between-study variance components are biased in certain conditions. In addition, the between-study variance estimates are particularly biased when the number of studies is smaller than 40 (i.e., 10 or 20) and the true value of the between-case variance is relatively large (i.e., 8). The observed bias is larger for the between-case variance estimates compared to the between-study variance estimates when the true between-case variance is relatively small (i.e., 0.5).

A demonstration and evaluation of the use of cross-classified random-effects models for meta-analysis.

It is common for the primary studies in meta-analyses to report multiple effect sizes, generating dependence among them. Hierarchical three-level models have been proposed as a means to deal with this dependency. Sometimes, however, dependency may be due to multiple random factors, and random factors are not necessarily nested, but rather may be crossed. For instance, effect sizes may belong to different studies, and, at the same time, effect sizes might represent the effects on different outcomes. Cross-classified random-effects models (CCREMs) can be used to model this nonhierarchical dependent structure. In this article, we explore by means of a simulation study the performance of CCREMs in comparison with the use of other meta-analytic models and estimation procedures, including the use of three- and two-level models and robust variance estimation. We also evaluated the performance of CCREMs when the underlying data were generated using a multivariate model. The results indicated that, whereas the quality of fixed-effect estimates is unaffected by any misspecification in the model, the standard error estimates of the mean effect size and of the moderator variables' effects, as well as the variance component estimates, are biased under some conditions. Applying CCREMs led to unbiased fixed-effect and variance component estimates, outperforming the other models. Even when a CCREM was not used to generate the data, applying the CCREM yielded sound parameter estimates and inferences.

Analysis of single-case experimental count data using the linear mixed effects model: A simulation study.

When (meta-)analyzing single-case experimental design (SCED) studies by means of hierarchical or multilevel modeling, applied researchers almost exclusively rely on the linear mixed model (LMM). This type of model assumes that the residuals are normally distributed. However, very often SCED studies consider outcomes of a discrete rather than a continuous nature, like counts, percentages or rates. In those cases the normality assumption does not hold. The LMM can be extended into a generalized linear mixed model (GLMM), which can account for the discrete nature of SCED count data. In this simulation study, we look at the effects of misspecifying an LMM for SCED count data simulated according to a GLMM. We compare the performance of a misspecified LMM and of a GLMM in terms of goodness of fit, fixed effect parameter recovery, type I error rate, and power. Because the LMM and the GLMM do not estimate identical fixed effects, we provide a transformation to compare the fixed effect parameter recovery. The results show that, compared to the GLMM, the LMM has worse performance in terms of goodness of fit and power. Performance in terms of fixed effect parameter recovery is equally good for both models, and in terms of type I error rate the LMM performs better than the GLMM. Finally, we provide some guidelines for applied researchers about aspects to consider when using an LMM for analyzing SCED count data.

Concealed correlations meta-analysis: A new method for synthesizing standardized regression coefficients.

The synthesis of standardized regression coefficients is still a controversial issue in the field of meta-analysis. The difficulty lies in the fact that the standardized regression coefficients belonging to regression models that include different sets of covariates do not represent the same parameter, and thus their direct combination is meaningless. In the present study, a new approach called concealed correlations meta-analysis is proposed that allows for using the common information that standardized regression coefficients from different regression models contain to improve the precision of a combined focal standardized regression coefficient estimate. The performance of this new approach was compared with that of two other approaches: (1) carrying out separate meta-analyses for standardized regression coefficients from studies that used the same regression model, and (2) performing a meta-regression on the focal standardized regression coefficients while including an indicator variable as a moderator indicating the regression model to which each standardized regression coefficient belongs. The comparison was done through a simulation study. The results showed that, as expected, the proposed approach led to more accurate estimates of the combined standardized regression coefficients under both random- and fixed-effect models.

A proposal for the assessment of replication of effects in single-case experimental designs.

In science in general and in the context of single-case experimental designs, replication of the effects of the intervention within and/or across participants or experiments is crucial for establishing causality and for assessing the generality of the intervention effect. Specific developments and proposals for assessing whether an effect has been replicated or not (or to what extent) are scarce, in the general context of behavioral sciences, and practically null in the single-case experimental designs context. We propose an extension of the modified Brinley plot for assessing how many of the effects replicate. To make this assessment possible, a definition of replication is suggested, on the basis of expert judgment, rather than on statistical criteria. The definition of replication and its graphical representation are justified, presenting their strengths and limitations, and illustrated with real data. A user-friendly software is made available for obtaining automatically the graphical representation.

Network Meta-Analysis for Single-Case Design Studies: An Illustration.

Single-case designs (SCDs) are used to evaluate the effects of interventions on individual participants. By repeatedly measuring participants under different conditions, SCD studies focus on individual effects rather than on group summaries. The main limitation of SCDs remains its generalisability to wider populations, reducing the relevance of their findings for practice and policy making. With this limitation in mind, methodological developments for synthesising SCD data from different studies that investigate the same research question have intensified in the past decades (e.g. multilevel modelling). However, these techniques are restricted to comparing two interventions at a time and can only incorporate evidence from studies that directly compare the two treatments of interest. These limitations could be addressed by using network meta-analysis that incorporates both direct and indirect evidence to simultaneously compare multiple interventions. Despite its potential, network meta-analytical techniques have yet to be applied to SCD data. Thus, in this paper, we argue that network meta-analysis can be a valuable tool to synthesise SCD data. We demonstrate the use of network meta-analysis in SCD data using a real dataset, and we conclude by reflecting on the challenges that SCD researchers might face when applying network meta-analysis methods to their data.

Heart rate and skin conductance associations with physical aggression, psychopathy, antisocial personality disorder and conduct disorder: An updated meta-analysis.

The associations between physiological measures (i.e., heart rate and skin conductance) of autonomic nervous system (ANS) activity and severe antisocial spectrum behavior (AB) were meta-analyzed. We used an exhaustive partitioning of variables relevant to the ANS-AB association and investigated four highly relevant questions (on declining effect sizes, psychopathy subscales, moderators, and ANS measures) that are thought to be transformative for future research on AB. We investigated a broad spectrum of physiological measures (e.g., heart rate (variability), pre-ejection period) in relation to AB. The search date for the current meta-analysis was on January 1st, 2020, includes 101 studies and 769 effect sizes. Results indicate that effect sizes are heterogeneous and bidirectional. The careful partitioning of variables sheds light on the complex associations that were obscured in previous meta-analyses. Effects are largest for the most violent offenders and for psychopathy and are dependent on the experimental tasks used, parameters calculated, and analyses run. Understanding the specificity of physiological reactions may be expedient for differentiating between (and within) types of AB.

Moral foundations and political orientation: Systematic review and meta-analysis.

We investigate the relationship of morality and political orientation by focusing on the influential results showing that liberals and conservatives rely on different moral foundations. We conducted a comprehensive literature search from major databases and other sources for primary studies that used the Moral Foundations Questionnaire and a typical measure of political orientation, a political self-placement item. We used a predefined process for independent extraction of effect sizes by two authors and ran both study-level and individual-level analyses. With 89 samples, 605 effect sizes, and 33,804 independent participants, in addition to 192,870 participants from the widely used YourMorals.org website, the basic differences about conservatives and liberals are supported. Yet, heterogeneity is moderate, and the results may be less generalizable across samples and political cultures than previously thought. The effect sizes obtained from the YourMorals.org data appear inflated compared with independent samples, which is partly related to political interest and may be because of self-selection. The association of moral foundations to political orientation varies culturally (between regions and countries) and subculturally (between White and Black respondents and in response to political interest). The associations also differ depending on the choice of the social or economic dimension and its labeling, supporting both the bidimensional model of political orientation and the findings that the dimensions are often strongly correlated. Our findings have implications for interpreting published studies, as well as designing new ones where the political aspect of morality is relevant. The results are primarily limited by the validity of the measures and the homogeneity of the included studies in terms of sample origins. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Cannabis use and nonuse in patients with first-episode psychosis: A systematic review and meta-analysis of studies comparing neurocognitive functioning.

BACKGROUND: The implications of cannabis use in the onset of early psychosis and the severity of psychotic symptoms have resulted in a proliferation of studies on this issue. However, few have examined the effects of cannabis use on the cognitive symptoms of psychosis (i.e., neurocognitive functioning) in patients with first-episode psychosis (FEP). This systematic review and meta-analysis aim to assess the neurocognitive functioning of cannabis users (CU) and nonusers (NU) with FEP. METHODS: Of the 110 studies identified through the systematic review of 6 databases, 7 met the inclusion criteria, resulting in 14 independent samples and 78 effect sizes. The total sample included 304 CU with FEP and 369 NU with FEP. The moderator variables were age at first use, duration of use, percentage of males, and age. RESULTS: Effect sizes were not significantly different from zero in any neurocognitive domain when users and NU were compared. Part of the variability in effect sizes was explained by the inclusion of the following moderator variables: (1) frequency of cannabis use (ß = 0.013, F = 7.56, p = 0.017); (2) first-generation antipsychotics (ß = 0.019, F = 34.46, p ≤ 0.001); and (3) country where the study was carried out (ß = 0.266, t = 2.06, p = 0.043). CONCLUSIONS: This meta-analysis indicates that cannabis use is not generally associated with neurocognitive functioning in patients with FEP. However, it highlights the deleterious effect of low doses of cannabis in some patients. It also stresses the importance of the type of antipsychotic prescription and cannabis dose as moderator variables in the neurocognitive functioning of CU with FEP.

Loneliness and social anxiety across childhood and adolescence: Multilevel meta-analyses of cross-sectional and longitudinal associations.

Social relationships are of vital importance for children's and adolescents' development, and disruptions in these relationships can have serious implications. Such disruptions play a central role in both loneliness and social anxiety. Although both phenomena are closely related, they have largely been studied separately, and important questions have remained unanswered concerning how both go together within and across time. Multilevel meta-analyses were performed on 102 cross-sectional studies, published between 1981 and 2016, including 41,776 participants (39% males) with a mean age of 15.59 years. Longitudinal associations were examined in 10 studies, including 3,995 participants (46% males), using a novel technique that enables the examination of such associations even when these were not reported in the original empirical studies. Results indicated a strong, positive cross-sectional association between loneliness and social anxiety symptoms. This associations did not systematically differ in strength across childhood and adolescence. Moreover, results showed that loneliness and social anxiety symptoms were reciprocally associated over time. To conclude, loneliness and social anxiety symptoms are positively associated both within and across time, and across childhood and adolescence. Breaking this vicious cycle is of great importance, as both phenomena may be associated with profound problems in multiple domains of youth development. Moreover, failing to pay attention to both loneliness and social anxiety symptoms might substantially reduce the effectiveness of intervention programs focusing on either of the two. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Methodological quality of meta-analyses of single-case experimental studies.

BACKGROUND: Methodological rigor is a fundamental factor in the validity and credibility of the results of a meta-analysis. AIM: Following an increasing interest in single-case experimental design (SCED) meta-analyses, the current study investigates the methodological quality of SCED meta-analyses. METHODS AND PROCEDURES: We assessed the methodological quality of 178 SCED meta-analyses published between 1985 and 2015 through the modified Revised-Assessment of Multiple Systematic Reviews (R-AMSTAR) checklist. OUTCOMES AND RESULTS: The main finding of the current review is that the methodological quality of the SCED meta-analyses has increased over time, but is still low according to the R-AMSTAR checklist. A remarkable percentage of the studies (93.80% of the included SCED meta-analyses) did not even reach the midpoint score (22, on a scale of 0-44). The mean and median methodological quality scores were 15.57 and 16, respectively. Relatively high scores were observed for "providing the characteristics of the included studies" and "doing comprehensive literature search". The key areas of deficiency were "reporting an assessment of the likelihood of publication bias" and "using the methods appropriately to combine the findings of studies". CONCLUSIONS AND IMPLICATIONS: Although the results of the current review reveal that the methodological quality of the SCED meta-analyses has increased over time, still more efforts are needed to improve their methodological quality.