Multilevel multivariate meta-analysis made easy: An introduction to MLMVmeta.

The basic random effects meta-analytic model is overwhelmingly dominant in psychological research. Indeed, it is typically employed even when more complex multilevel multivariate meta-analytic models are warranted. In this paper, we aim to help overcome challenges so that multilevel multivariate meta-analytic models will be more often employed in practice. We do so by introducing MLMVmeta-an easy-to-use web application that implements multilevel multivariate meta-analytic methodology that is both specially tailored to contemporary psychological research and easily estimable, interpretable, and parsimonious-and illustrating it across three case studies. The three case studies demonstrate the more accurate and extensive results that can be obtained via multilevel multivariate meta-analytic models. Further, they sequentially build in complexity featuring increasing numbers of experimental factors and conditions, dependent variables, and levels; this in turn necessitates increasingly complex model specifications that also sequentially build upon one another.

Qualitative Development of a Discrete Choice Experiment for Physical Activity Interventions to Improve Knee Osteoarthritis.

OBJECTIVE: To describe the qualitative process used to develop attributes and attribute levels for inclusion in a discrete choice experiments (DCE) for older adult physical activity interventions. DESIGN: Five focus groups (n=41) were conducted, grounded in the Health Action Process Approach framework. Discussion emphasized identification and prioritization attributes for a DCE on physical activity. Semi-structured interviews (n=6) investigated attribute levels and lay-language for the DCE. A focus group with physical activity researchers and health care providers was the final stakeholder group used to establish a comprehensive approach for the generation of attributes and levels. A DCE pilot test (n=8) was then conducted with individuals of the target patient population. All transcripts were analyzed using a constant comparative approach. SETTING: General community and university-based research setting. PARTICIPANTS: Volunteers (N=55) aged >45 years with knee pain, aches, or stiffness for at least 1 month over the previous 12 months. INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: Interview guides, attributes, attribute levels, and discrete choice experiment. RESULTS: The most influential identified attributes for physical activity were time, effort, cost, convenience, enjoyment, and health benefits. Each attribute had 3 levels that were understandable in the pilot test of the DCE. CONCLUSIONS: The identification of 6 physical activity attributes that are most salient to adults with knee osteoarthritis resulted from a systematic qualitative process, including attribute-ranking exercises. A DCE will provide insight into the relative importance of these attributes for participating in physical activity, which can guide intervention development.

The multidimensional randomized response design: Estimating different aspects of the same sensitive behavior.

The conventional randomized response design is unidimensional in the sense that it measures a single dimension of a sensitive attribute, like its prevalence, frequency, magnitude, or duration. This paper introduces a multidimensional design characterized by categorical questions that each measure a different aspect of the same sensitive attribute. The benefits of the multidimensional design are (i) a substantial gain in power and efficiency, and the potential to (ii) evaluate the goodness-of-fit of the model, and (iii) test hypotheses about evasive response biases in case of a misfit. The method is illustrated for a two-dimensional design measuring both the prevalence and the magnitude of social security fraud.

Intermittent faking of personality profiles in high-stakes assessments: A grade of membership analysis.

In high stakes assessments of personality and similar attributes, test takers may engage in impression management (aka faking). This article proposes to consider responses of every test taker as a potential mixture of "real" (or retrieved) answers to questions, and "ideal" answers intended to create a desired impression, with each type of response characterized by its own distribution and factor structure. Depending on the particular mix of response types in the test taker profile, grades of membership in the "real" and "ideal" profiles are defined. This approach overcomes the limitation of existing psychometric models that assume faking behavior to be consistent across test items. To estimate the proposed faking-as-grade-of-membership (F-GoM) model, two-level factor mixture analysis is used, with two latent classes at the response (within) level, allowing grade of membership in "real" and "ideal" profiles, each underpinned by its own factor structure, at the person (between) level. For collected data, units of analysis can be item or scale scores, with the latter enabling analysis of questionnaires with many measured scales. The performance of the F-GoM model is evaluated in a simulation study, and compared against existing methods for statistical control of faking in an empirical application using archival recruitment data, which supported the validity of latent factors and classes assumed by the model using multiple control variables. The proposed approach is particularly useful for high-stakes assessment data and can be implemented with standard software packages. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Better Information From Survey Data: Filtering Out State Dependence Using Eye-Tracking Data.

Ideally, survey respondents read and understand survey instructions, questions, and response scales, and provide answers that carefully reflect their beliefs, attitudes, or knowledge. However, respondents may also arrive at their responses using cues or heuristics that facilitate the production of a response, but diminish the targeted information content. We use eye-tracking data as covariates in a Bayesian switching-mixture model to identify different response behaviors at the item-respondent level. The model distinguishes response behaviors that are predominantly influenced either positively or negatively by the previous response, and responses that reflect respondents' preexisting knowledge and experiences of interest. We find that controlling for multiple types of adaptive response behaviors allows for a more informative analysis of survey data and respondents.

Assessing item-feature effects with item response tree models.

Recent applications of item response tree models demonstrate that this model class is well suited to detect midpoint and extremity response style effects in both attitudinal and personality measurements. This paper proposes an extension of this approach that goes beyond measuring response styles and allows us to examine item-feature effects. In a reanalysis of three published data sets, it is shown that the proposed extension captures item-feature effects across affirmative and reverse-worded items in a psychological test. These effects are found to affect directional responses but not midpoint and extremity preferences. Moreover, accounting for item-feature effects substantially improves model fit and interpretation of the construct measurement. The proposed extension can be implemented readily with current software programs that facilitate maximum likelihood estimation of item response models with missing data.

Contextual Responses to Affirmative and/or Reversed-Worded Items.

This paper presents a systematic investigation of how affirmative and polar-opposite items presented either jointly or separately affect yea-saying tendencies. We measure these yea-saying tendencies with item response models that estimate a respondent's tendency to give a "yea"-response that may be unrelated to the target trait. In a re-analysis of the Zhang et al. (PLoS ONE, 11:1-15, 2016) data, we find that yea-saying tendencies depend on whether items are presented as part of a scale that contains affirmative and/or polar-opposite items. Yea-saying tendencies are stronger for affirmative than for polar-opposite items. Moreover, presenting polar-opposite items together with affirmative items creates lower yea-saying tendencies for polar-opposite items than when presented in isolation. IRT models that do not account for these yea-saying effects arrive at a two-dimensional representation of the target trait. These findings demonstrate that the contextual information provided by an item scale can serve as a determinant of differential item functioning.

Multilevel Multivariate Meta-analysis with Application to Choice Overload.

We introduce multilevel multivariate meta-analysis methodology designed to account for the complexity of contemporary psychological research data. Our methodology directly models the observations from a set of studies in a manner that accounts for the variation and covariation induced by the facts that observations differ in their dependent measures and moderators and are nested within, for example, papers, studies, groups of subjects, and study conditions. Our methodology is motivated by data from papers and studies of the choice overload hypothesis. It more fully accounts for the complexity of choice overload data relative to two prior meta-analyses and thus provides richer insight. In particular, it shows that choice overload varies substantially as a function of the six dependent measures and four moderators examined in the domain and that there are potentially interesting and theoretically important interactions among them. It also shows that the various dependent measures have differing levels of variation and that levels up to and including the highest (i.e., the fifth, or paper, level) are necessary to capture the variation and covariation induced by the nesting structure. Our results have substantial implications for future studies of choice overload.

Measuring response styles in Likert items.

The recently proposed class of item response tree models provides a flexible framework for modeling multiple response processes. This feature is particularly attractive for understanding how response styles may affect answers to attitudinal questions. Facilitating the disassociation of response styles and attitudinal traits, item response tree models can provide powerful process tests of how different response formats may affect the measurement of substantive traits. In an empirical study, 3 response formats were used to measure the 2-dimensional Personal Need for Structure traits. Different item response tree models are proposed to capture the response styles for each of the response formats. These models show that the response formats give rise to similar trait measures but different response-style effects. (PsycINFO Database Record

Response style analysis with threshold and multi-process IRT models: A review and tutorial.

Two different item response theory model frameworks have been proposed for the assessment and control of response styles in rating data. According to one framework, response styles can be assessed by analysing threshold parameters in Rasch models for ordinal data and in mixture-distribution extensions of such models. A different framework is provided by multi-process item response tree models, which can be used to disentangle response processes that are related to the substantive traits and response tendencies elicited by the response scale. In this tutorial, the two approaches are reviewed, illustrated with an empirical data set of the two-dimensional 'Personal Need for Structure' construct, and compared in terms of multiple criteria. Mplus is used as a software framework for (mixed) polytomous Rasch models and item response tree models as well as for demonstrating how parsimonious model variants can be specified to test assumptions on the structure of response styles and attitude strength. Although both frameworks are shown to account for response styles, they differ on the quantitative criteria of model selection, practical aspects of model estimation, and conceptual issues of representing response styles as continuous and multidimensional sources of individual differences in psychological assessment.

Planning sample sizes when effect sizes are uncertain: The power-calibrated effect size approach.

Statistical power and thus the sample size required to achieve some desired level of power depend on the size of the effect of interest. However, effect sizes are seldom known exactly in psychological research. Instead, researchers often possess an estimate of an effect size as well as a measure of its uncertainty (e.g., a standard error or confidence interval). Previous proposals for planning sample sizes either ignore this uncertainty thereby resulting in sample sizes that are too small and thus power that is lower than the desired level or overstate the impact of this uncertainty thereby resulting in sample sizes that are too large and thus power that is higher than the desired level. We propose a power-calibrated effect size (PCES) approach to sample size planning that accounts for the uncertainty associated with an effect size estimate in a properly calibrated manner: sample sizes determined on the basis of the PCES are neither too small nor too large and thus provide the desired level of power. We derive the PCES for comparisons of independent and dependent means, comparisons of independent and dependent proportions, and tests of correlation coefficients. We also provide a tutorial on setting sample sizes for a replication study using data from prior studies and discuss an easy-to-use website and code that implement our PCES approach to sample size planning.

Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes.

We review and evaluate selection methods, a prominent class of techniques first proposed by Hedges (1984) that assess and adjust for publication bias in meta-analysis, via an extensive simulation study. Our simulation covers both restrictive settings as well as more realistic settings and proceeds across multiple metrics that assess different aspects of model performance. This evaluation is timely in light of two recently proposed approaches, the so-called p-curve and p-uniform approaches, that can be viewed as alternative implementations of the original Hedges selection method approach. We find that the p-curve and p-uniform approaches perform reasonably well but not as well as the original Hedges approach in the restrictive setting for which all three were designed. We also find they perform poorly in more realistic settings, whereas variants of the Hedges approach perform well. We conclude by urging caution in the application of selection methods: Given the idealistic model assumptions underlying selection methods and the sensitivity of population average effect size estimates to them, we advocate that selection methods should be used less for obtaining a single estimate that purports to adjust for publication bias ex post and more for sensitivity analysis-that is, exploring the range of estimates that result from assuming different forms of and severity of publication bias.

A latent markov model for the analysis of longitudinal data collected in continuous time: states, durations, and transitions.

Markov models provide a general framework for analyzing and interpreting time dependencies in psychological applications. Recent work extended Markov models to the case of latent states because frequently psychological states are not directly observable and subject to measurement error. This article presents a further generalization of latent Markov models to allow for the analysis of rating data that are collected at arbitrary points in time. This extension offers new ways of investigating change processes by focusing explicitly on the durations that are spent in latent states. In an experience sampling application the author shows that such duration analyses can provide valuable insights about chronometric features of emotions.

Structural equation modeling of paired-comparison and ranking data.

L. L. Thurstone's (1927) model provides a powerful framework for modeling individual differences in choice behavior. An overview of Thurstonian models for comparative data is provided, including the classical Case V and Case III models as well as more general choice models with unrestricted and factor-analytic covariance structures. A flow chart summarizes the model selection process. The authors show how to embed these models within a more familiar structural equation modeling (SEM) framework. The different special cases of Thurstone's model can be estimated with a popular SEM statistical package, including factor analysis models for paired comparisons and rankings. Only minor modifications are needed to accommodate both types of data. As a result, complex models for comparative judgments can be both estimated and tested efficiently.

Comparative judgments as an alternative to ratings: identifying the scale origin.

Although comparative judgment methods have a number of distinct advantages over ratings, they share one common problem: On the basis of comparative judgments, it is not possible to recover the origin of item evaluations. One item may be judged more positively than another, but this result does not allow any conclusions about whether either of the items are attractive or unattractive. This article discusses the implications of this limitation for the interpretation of individual differences in comparative judgments. It also presents 3 different methods that may allow determination of the scale origin using a nested model comparison approach. An application illustrates the proposed approach as well as the benefits of determining the scale origin in understanding value judgments.

The structure of self-reported emotional experiences: a mixed-effects Poisson factor model.

Multivariate count data are commonly analysed by using Poisson distributions with varying intensity parameters, resulting in a random-effects model. In the analysis of a data set on the frequency of different emotion experiences we find that a Poisson model with a single random effect does not yield an adequate fit. An alternative model that requires as many random effects as emotion categories requires high-dimensional integration and the estimation of a large number of parameters. As a solution to these computational problems, we propose a factor-analytic Poisson model and show that a two-dimensional factor model fits the reported data very well. Moreover, it yields a substantively satisfactory solution: one factor describing the degree of pleasantness and unpleasantness of emotions and the other factor describing the activation levels of the emotions. We discuss the incorporation of covariates to facilitate rigorous tests of the random-effects structure. Marginal maximum likelihood methods lead to straight-forward estimation of the model, for which goodness-of-fit tests are also presented.

Modeling motivated misreports to sensitive survey questions.

Asking sensitive or personal questions in surveys or experimental studies can both lower response rates and increase item non-response and misreports. Although non-response is easily diagnosed, misreports are not. However, misreports cannot be ignored because they give rise to systematic bias. The purpose of this paper is to present a modeling approach that identifies misreports and corrects for them. Misreports are conceptualized as a motivated process under which respondents edit their answers before they report them. For example, systematic bias introduced by overreports of socially desirable behaviors or underreports of less socially desirable ones can be modeled, leading to more-valid inferences. The proposed approach is applied to a large-scale experimental study and shows that respondents who feel powerful tend to overclaim their knowledge.

You Cannot Step Into the Same River Twice: When Power Analyses Are Optimistic.

Statistical power depends on the size of the effect of interest. However, effect sizes are rarely fixed in psychological research: Study design choices, such as the operationalization of the dependent variable or the treatment manipulation, the social context, the subject pool, or the time of day, typically cause systematic variation in the effect size. Ignoring this between-study variation, as standard power formulae do, results in assessments of power that are too optimistic. Consequently, when researchers attempting replication set sample sizes using these formulae, their studies will be underpowered and will thus fail at a greater than expected rate. We illustrate this with both hypothetical examples and data on several well-studied phenomena in psychology. We provide formulae that account for between-study variation and suggest that researchers set sample sizes with respect to our generally more conservative formulae. Our formulae generalize to settings in which there are multiple effects of interest. We also introduce an easy-to-use website that implements our approach to setting sample sizes. Finally, we conclude with recommendations for quantifying between-study variation.

Modeling multiple response processes in judgment and choice.

In this article, I show how item response models can be used to capture multiple response processes in psychological applications. Intuitive and analytical responses, agree-disagree answers, response refusals, socially desirable responding, differential item functioning, and choices among multiple options are considered. In each of these cases, I show that the response processes can be measured via pseudoitems derived from the observed responses. The estimation of these models via standard software programs that allow for missing data is also discussed. The article concludes with two detailed applications that illustrate the prevalence of multiple response processes.

Choice by value encoding and value construction: processes of loss aversion.

Loss aversion and reference dependence are 2 keystones of behavioral theories of choice, but little is known about their underlying cognitive processes. We suggest an additional account for loss aversion that supplements the current account of the value encoding of attributes as gains or losses relative to a reference point, introducing a value construction account. Value construction suggests that loss aversion results from biased evaluations during information search and comparison processes. We develop hypotheses that identify the influence of both accounts and examine process-tracing data for evidence. Our data suggest that loss aversion is the result of the initial direct encoding of losses that leads to the subsequent process of directional comparisons distorting attribute valuations and the final choice.