An Exact Bayesian Model for Meta-Analysis of the Standardized Mean Difference with Its Simultaneous Credible Intervals.

While Bayesian methodology is increasingly favored in behavioral research for its clear probabilistic inference and model structure, its widespread acceptance as a standard meta-analysis approach remains limited. Although some conventional Bayesian hierarchical models are frequently used for analysis, their performance has not been thoroughly examined. This study evaluates two commonly used Bayesian models for meta-analysis of standardized mean difference and identifies significant issues with these models. In response, we introduce a new Bayesian model equipped with novel features that address existing model concerns and a broader limitation of the current Bayesian meta-analysis. Furthermore, we introduce a simple computational approach to construct simultaneous credible intervals for the summary effect and between-study heterogeneity, based on their joint posterior samples. This fully captures the joint uncertainty in these parameters, a task that is challenging or impractical with frequentist models. Through simulation studies rooted in a joint Bayesian/frequentist paradigm, we compare our model's performance against existing ones under conditions that mirror realistic research scenarios. The results reveal that our new model outperforms others and shows enhanced statistical properties. We also demonstrate the practicality of our models using real-world examples, highlighting how our approach strengthens the robustness of inferences regarding the summary effect.

New Recommendations on the Use of R-Squared Differences in Multilevel Model Comparisons.

When comparing multilevel models (MLMs) differing in fixed and/or random effects, researchers have had continuing interest in using R-squared differences to communicate effect size and importance of included terms. However, there has been longstanding confusion regarding which R-squared difference measures should be used for which kind of MLM comparisons. Furthermore, several limitations of recent studies on R-squared differences in MLM have led to misleading or incomplete recommendations for practice. These limitations include computing measures that are by definition incapable of detecting a particular type of added term, considering only a subset of the broader class of available R-squared difference measures, and incorrectly defining what a given R-squared difference measure quantifies. The purpose of this paper is to elucidate and resolve these issues. To do so, we define a more general set of total, within-cluster, and between-cluster R-squared difference measures than previously considered in MLM comparisons and give researchers concrete step-by-step procedures for identifying which measure is relevant to which model comparison. We supply simulated and analytic demonstrations of limitations of previous MLM studies on R-squared differences and show how application of our step-by-step procedures and general set of measures overcomes each. Additionally, we provide and illustrate graphical tools and software allowing researchers to automatically compute and visualize our set of measures in an integrated manner. We conclude with recommendations, as well as extensions involving (a) how our framework relates to and can be used to obtain pseudo-R-squareds, and (b) how our framework can accommodate both simultaneous and hierarchical model-building approaches.

Analysis of single-case experimental count data using the linear mixed effects model: A simulation study.

When (meta-)analyzing single-case experimental design (SCED) studies by means of hierarchical or multilevel modeling, applied researchers almost exclusively rely on the linear mixed model (LMM). This type of model assumes that the residuals are normally distributed. However, very often SCED studies consider outcomes of a discrete rather than a continuous nature, like counts, percentages or rates. In those cases the normality assumption does not hold. The LMM can be extended into a generalized linear mixed model (GLMM), which can account for the discrete nature of SCED count data. In this simulation study, we look at the effects of misspecifying an LMM for SCED count data simulated according to a GLMM. We compare the performance of a misspecified LMM and of a GLMM in terms of goodness of fit, fixed effect parameter recovery, type I error rate, and power. Because the LMM and the GLMM do not estimate identical fixed effects, we provide a transformation to compare the fixed effect parameter recovery. The results show that, compared to the GLMM, the LMM has worse performance in terms of goodness of fit and power. Performance in terms of fixed effect parameter recovery is equally good for both models, and in terms of type I error rate the LMM performs better than the GLMM. Finally, we provide some guidelines for applied researchers about aspects to consider when using an LMM for analyzing SCED count data.

Application of the dyadic data analysis in behavioral medicine research: marital satisfaction and anxiety in infertile couples.

BACKGROUND: Dyadic data analysis (DDA) is increasingly being used to better understand, analyze and model intra- and inter-personal mechanisms of health in various types of dyads such as husband-wife, caregiver-patient, doctor-patient, and parent-child. A key strength of the DDA is its flexibility to take the nonindependence available in the dyads into account. In this article, we illustrate the value of using DDA to examine how anxiety is associated with marital satisfaction in infertile couples. METHODS: This cross-sectional study included 141 infertile couples from a referral infertility clinic in Tehran, Iran between February and May 2017. Anxiety and marital satisfaction were measured by the anxiety subscale of the Hospital Anxiety and Depression Scale and 10-Item ENRICH Marital Satisfaction Scale, respectively. We apply and compare tree different dyadic models to explore the effect of anxiety on marital satisfaction, including the Actor-Partner Interdependence Model (APIM), Mutual Influence Model (MIM), and Common Fate Model (CFM). RESULTS: This study demonstrated a practical application of the dyadic models. These dyadic models provide results that appear to give different interpretations of the data. The APIM analysis revealed that both men's and women's anxiety excreted an actor effect on their own marital satisfaction. In addition, women's anxiety exerted a significant partner effect on their husbands' marital satisfaction. In MIM analysis, in addition to significant actor effects of anxiety on marital satisfaction, women's reports of marital satisfaction significantly predicted men's marital satisfaction. The CFM analysis revealed that higher couple anxiety scores predicted lower couple marital satisfaction scores. CONCLUSION: In sum, the study highlights the usefulness of DDA to explore and test the phenomena with inherently dyadic nature. With regard to our empirical data, the findings confirmed that marital satisfaction was influenced by anxiety in infertile couples at both individual and dyadic level; thus, interventions to improve marital satisfaction should include both men and women. In addition, future studies should consider using DDA when dyadic data are available.

Participation in HIV Behavioral Research: Unanticipated Benefits and Burdens.

HIV behavioral research has provided an invaluable knowledge base for effective approaches to behavioral challenges along the HIV care cascade. Little attention has been paid to tracking unanticipated effects of research participation, whether negative or positive. We used qualitative methods to elicit impressions of unanticipated effects of participation in behavioral research. An instrument was developed and piloted to assess positive (emotional gains, practical gains, HIV prevention knowledge and skills gains) and negative (emotional stress, discomfort with research) unanticipated effects. Participants (N = 25) from five projects, including men who have sex with men, adults who use substances, and youth, reported multiple positive unanticipated effects (sexual and drug risk reduction, goal setting, improvements in self-esteem and mood, relationship gains, health care behavior gains, knowledge and introspection gains) and rare unanticipated negative effects. Developing a systematic tool of unanticipated positive and negative effects of participation in behavioral research is a crucial next step.

The Aarhus statement on cancer diagnostic research: turning recommendations into new survey instruments.

BACKGROUND: Over recent years there has been a growth in cancer early diagnosis (ED) research, which requires valid measurement of routes to diagnosis and diagnostic intervals. The Aarhus Statement, published in 2012, provided methodological guidance to generate valid data on these key pre-diagnostic measures. However, there is still a wide variety of measuring instruments of varying quality in published research. In this paper we test comprehension of self-completion ED questionnaire items, based on Aarhus Statement guidance, and seek input from patients, GPs and ED researchers to refine these questions. METHODS: We used personal interviews and consensus approaches to generate draft ED questionnaire items, then a combination of focus groups and telephone interviews to test comprehension and obtain feedback. A framework analysis approach was used, to identify themes and potential refinements to the items. RESULTS: We found that many of the questionnaire items still prompted uncertainty in respondents, in both routes to diagnosis and diagnostic interval measurement. Uncertainty was greatest in the context of multiple or vague symptoms, and potentially ambiguous time-points (such as 'date of referral'). CONCLUSIONS: There are limits on the validity of self-completion questionnaire responses, and refinements to the wording of questions may not be able to completely overcome these limitations. It's important that ED researchers use the best identifiable measuring instruments, but accommodate inevitable uncertainty in the interpretation of their results. Every effort should be made to increase clarity of questions and responses, and use of two or more data sources should be considered.

Case conceptualization research in cognitive behavior therapy: A state of the science review.

OBJECTIVE: Prominent models of cognitive behavior therapy (CBT) assert that case conceptualization is crucial for tailoring interventions to adequately address the needs of the individual client. We aimed to review the research on case conceptualization in CBT. METHOD: We conducted a systematic search of PsychINFO, MEDLINE, Psychology and Behavioral Science Collection, and CINAHL databases to February 2016. RESULTS: A total of 24 studies that met inclusion criteria were identified. It was notable that studies (a) focused on the assessment function of case conceptualization, (b) employed diverse methodologies, and, overall, (c) there remains a paucity of studies examining the in-session process of using case conceptualization or examining relations with outcome. CONCLUSION: Results from the existing studies suggest that experienced therapists can reliably construct some elements of case conceptualizations, but importance for the efficacy of case conceptualization in CBT has yet to be demonstrated. Research that involves direct observation of therapist competence in case conceptualization as a predictor of CBT outcomes is recommended as a focus for future hypothesis testing.

Measuring and testing the agreement of matrices.

The problem of comparing the agreement of two n × n matrices has a variety of applications in experimental psychology. A well-known index of agreement is based on the sum of the element-wise products of the matrices. Although less familiar to many researchers, measures of agreement based on within-row and/or within-column gradients can also be useful. We provide a suite of MATLAB programs for computing agreement indices and performing matrix permutation tests of those indices. Programs for computing exact p-values are available for small matrices, whereas resampling programs for approximate p-values are provided for larger matrices.

Testing moderator hypotheses in meta-analytic structural equation modeling using subgroup analysis.

Meta-analytic structural equation modeling (MASEM) is a statistical technique to pool correlation matrices and test structural equation models on the pooled correlation matrix. In Stage 1 of MASEM, correlation matrices from independent studies are combined to obtain a pooled correlation matrix, using fixed- or random-effects analysis. In Stage 2, a structural model is fitted to the pooled correlation matrix. Researchers applying MASEM may have hypotheses about how certain model parameters will differ across subgroups of studies. These moderator hypotheses are often addressed using suboptimal methods. The aim of the current article is to provide guidance and examples on how to test hypotheses about group differences in specific model parameters in MASEM. We illustrate the procedure using both fixed- and random-effects subgroup analysis with two real datasets. In addition, we present a small simulation study to evaluate the effect of the number of studies per subgroup on convergence problems. All data and the R-scripts for the examples are provided online.

On the ambiguity of interaction and nonlinear main effects in a regime of dependent covariates.

The analysis of large experimental datasets frequently reveals significant interactions that are difficult to interpret within the theoretical framework guiding the research. Some of these interactions actually arise from the presence of unspecified nonlinear main effects and statistically dependent covariates in the statistical model. Importantly, such nonlinear main effects may be compatible (or, at least, not incompatible) with the current theoretical framework. In the present literature, this issue has only been studied in terms of correlated (linearly dependent) covariates. Here we generalize to nonlinear main effects (i.e., main effects of arbitrary shape) and dependent covariates. We propose a novel nonparametric method to test for ambiguous interactions where present parametric methods fail. We illustrate the method with a set of simulations and with reanalyses (a) of effects of parental education on their children's educational expectations and (b) of effects of word properties on fixation locations during reading of natural sentences, specifically of effects of length and morphological complexity of the word to be fixated next. The resolution of such ambiguities facilitates theoretical progress.

Analyzing individual growth with clustered longitudinal data: A comparison between model-based and design-based multilevel approaches.

To prevent biased estimates of intraindividual growth and interindividual variability when working with clustered longitudinal data (e.g., repeated measures nested within students; students nested within schools), individual dependency should be considered. A Monte Carlo study was conducted to examine to what extent two model-based approaches (multilevel latent growth curve model - MLGCM, and maximum model - MM) and one design-based approach (design-based latent growth curve model - D-LGCM) could produce unbiased and efficient parameter estimates of intraindividual growth and interindividual variability given clustered longitudinal data. The solutions of a single-level latent growth curve model (SLGCM) were also provided to demonstrate the consequences of ignoring individual dependency. Design factors considered in the present simulation study were as follows: number of clusters (NC = 10, 30, 50, 100, 150, 200, and 500) and cluster size (CS = 5, 10, and 20). According to our results, when intraindividual growth is of interest, researchers are free to implement MLGCM, MM, or D-LGCM. With regard to interindividual variability, MLGCM and MM were capable of producing accurate parameter estimates and SEs. However, when D-LGCM and SLGCM were applied, parameter estimates of interindividual variability were not comprised exclusively of the variability in individual (e.g., students) growth but instead were the combined variability of individual and cluster (e.g., school) growth, which cannot be interpreted. The take-home message is that D-LGCM does not qualify as an alternative approach to analyzing clustered longitudinal data if interindividual variability is of interest.

Retrieving relevant factors with exploratory SEM and principal-covariate regression: A comparison.

Behavioral researchers often linearly regress a criterion on multiple predictors, aiming to gain insight into the relations between the criterion and predictors. Obtaining this insight from the ordinary least squares (OLS) regression solution may be troublesome, because OLS regression weights show only the effect of a predictor on top of the effects of other predictors. Moreover, when the number of predictors grows larger, it becomes likely that the predictors will be highly collinear, which makes the regression weights' estimates unstable (i.e., the "bouncing beta" problem). Among other procedures, dimension-reduction-based methods have been proposed for dealing with these problems. These methods yield insight into the data by reducing the predictors to a smaller number of summarizing variables and regressing the criterion on these summarizing variables. Two promising methods are principal-covariate regression (PCovR) and exploratory structural equation modeling (ESEM). Both simultaneously optimize reduction and prediction, but they are based on different frameworks. The resulting solutions have not yet been compared; it is thus unclear what the strengths and weaknesses are of both methods. In this article, we focus on the extents to which PCovR and ESEM are able to extract the factors that truly underlie the predictor scores and can predict a single criterion. The results of two simulation studies showed that for a typical behavioral dataset, ESEM (using the BIC for model selection) in this regard is successful more often than PCovR. Yet, in 93% of the datasets PCovR performed equally well, and in the case of 48 predictors, 100 observations, and large differences in the strengths of the factors, PCovR even outperformed ESEM.

Univariate and multivariate skewness and kurtosis for measuring nonnormality: Prevalence, influence and estimation.

Nonnormality of univariate data has been extensively examined previously (Blanca et al., Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 9(2), 78-84, 2013; Miceeri, Psychological Bulletin, 105(1), 156, 1989). However, less is known of the potential nonnormality of multivariate data although multivariate analysis is commonly used in psychological and educational research. Using univariate and multivariate skewness and kurtosis as measures of nonnormality, this study examined 1,567 univariate distriubtions and 254 multivariate distributions collected from authors of articles published in Psychological Science and the American Education Research Journal. We found that 74 % of univariate distributions and 68 % multivariate distributions deviated from normal distributions. In a simulation study using typical values of skewness and kurtosis that we collected, we found that the resulting type I error rates were 17 % in a t-test and 30 % in a factor analysis under some conditions. Hence, we argue that it is time to routinely report skewness and kurtosis along with other summary statistics such as means and variances. To facilitate future report of skewness and kurtosis, we provide a tutorial on how to compute univariate and multivariate skewness and kurtosis by SAS, SPSS, R and a newly developed Web application.

A comparative evaluation of off-the-shelf distributed semantic representations for modelling behavioural data.

In this paper we carry out an extensive comparison of many off-the-shelf distributed semantic vectors representations of words, for the purpose of making predictions about behavioural results or human annotations of data. In doing this comparison we also provide a guide for how vector similarity computations can be used to make such predictions, and introduce many resources available both in terms of datasets and of vector representations. Finally, we discuss the shortcomings of this approach and future research directions that might address them.

The prevalence of statistical reporting errors in psychology (1985-2013).

This study documents reporting errors in a sample of over 250,000 p-values reported in eight major psychology journals from 1985 until 2013, using the new R package "statcheck." statcheck retrieved null-hypothesis significance testing (NHST) results from over half of the articles from this period. In line with earlier research, we found that half of all published psychology papers that use NHST contained at least one p-value that was inconsistent with its test statistic and degrees of freedom. One in eight papers contained a grossly inconsistent p-value that may have affected the statistical conclusion. In contrast to earlier findings, we found that the average prevalence of inconsistent p-values has been stable over the years or has declined. The prevalence of gross inconsistencies was higher in p-values reported as significant than in p-values reported as nonsignificant. This could indicate a systematic bias in favor of significant results. Possible solutions for the high prevalence of reporting inconsistencies could be to encourage sharing data, to let co-authors check results in a so-called "co-pilot model," and to use statcheck to flag possible inconsistencies in one's own manuscript or during the review process.

Effect size measures in a two-independent-samples case with nonnormal and nonhomogeneous data.

In psychological science, the "new statistics" refer to the new statistical practices that focus on effect size (ES) evaluation instead of conventional null-hypothesis significance testing (Cumming, Psychological Science, 25, 7-29, 2014). In a two-independent-samples scenario, Cohen's (1988) standardized mean difference (d) is the most popular ES, but its accuracy relies on two assumptions: normality and homogeneity of variances. Five other ESs-the unscaled robust d (d r* ; Hogarty & Kromrey, 2001), scaled robust d (d r ; Algina, Keselman, & Penfield, Psychological Methods, 10, 317-328, 2005), point-biserial correlation (r pb ; McGrath & Meyer, Psychological Methods, 11, 386-401, 2006), common-language ES (CL; Cliff, Psychological Bulletin, 114, 494-509, 1993), and nonparametric estimator for CL (A w ; Ruscio, Psychological Methods, 13, 19-30, 2008)-may be robust to violations of these assumptions, but no study has systematically evaluated their performance. Thus, in this simulation study the performance of these six ESs was examined across five factors: data distribution, sample, base rate, variance ratio, and sample size. The results showed that A w and d r were generally robust to these violations, and A w slightly outperformed d r . Implications for the use of A w and d r in real-world research are discussed.

Systematic review shows only few reliable studies of physical activity intervention in adolescents.

Introduction. Several studies have pointed to the high prevalence of low levels of physical activity in adolescents, suggesting the need for more effective interventions for this group. The aim of this study was to present evidence of intervention programs for efficacy of physical activity for adolescents. Methods. Surveys in PubMed, SportDiscus, LiLacs, and SciELO databases were conducted using keywords to identify population, intervention, and outcome, as well as DeCS and MeSH terms in English, Portuguese, and Spanish, whenever appropriate. The review included observational studies with minimal intervention of six months, minimum sample size of 100 adolescents, written in any language, and those who have reached STROBE score greater than 70%. Results. Only seven studies met all inclusion criteria. Of these, five were pre- and postintervention and two had n > 2000 participants. Interventions were of several types, durations, and strategies for physical activity implementation. Behavior change was assessed in 43% of studies and three reported success in some way. Conclusion. Due to heterogeneity in their contents and methodologies, as well as the lack of jobs that accompany adolescents after the intervention period, one cannot draw conclusions about the actual effects of the intervention programs of physical activity on the behavior of young people.

When ab ≠ c - c': published errors in the reports of single-mediator models.

Accurate reports of mediation analyses are critical to the assessment of inferences related to causality, since these inferences are consequential for both the evaluation of previous research (e.g., meta-analyses) and the progression of future research. However, upon reexamination, approximately 15% of published articles in psychology contain at least one incorrect statistical conclusion (Bakker & Wicherts, Behavior research methods, 43, 666-678 2011), disparities that beget the question of inaccuracy in mediation reports. To quantify this question of inaccuracy, articles reporting standard use of single-mediator models in three high-impact journals in personality and social psychology during 2011 were examined. More than 24% of the 156 models coded failed an equivalence test (i.e., ab = c - c'), suggesting that one or more regression coefficients in mediation analyses are frequently misreported. The authors cite common sources of errors, provide recommendations for enhanced accuracy in reports of single-mediator models, and discuss implications for alternative methods.

Bootstrap analysis of the single subject with event related potentials.

Neural correlates of cognitive states in event-related potentials (ERPs) serve as markers for related cerebral processes. Although these are usually evaluated in subject groups, the ability to evaluate such markers statistically in single subjects is essential for case studies in neuropsychology. Here we investigated the use of a simple test based on nonparametric bootstrap confidence intervals for this purpose, by evaluating three different ERP phenomena: the face-selectivity of the N170, error-related negativity, and the P3 component in a Posner cueing paradigm. In each case, we compare single-subject analysis with statistical significance determined using bootstrap to conventional group analysis using analysis of variance (ANOVA). We found that the proportion of subjects who show a significant effect at the individual level based on bootstrap varied, being greatest for the N170 and least for the P3. Furthermore, it correlated with significance at the group level. We conclude that the bootstrap methodology can be a viable option for interpreting single-case ERP amplitude effects in the right setting, probably with well-defined stereotyped peaks that show robust differences at the group level, which may be more characteristic of early sensory components than late cognitive effects.