Detecting Heterogeneity of Intervention Effects in Comparative Judgments.

Comparative measures such as paired comparisons and rankings are frequently used to evaluate health states and quality of life. The present article introduces log-linear Bradley-Terry (LLBT) models to evaluate intervention effectiveness when outcomes are measured as paired comparisons or rankings and presents a combination of the LLBT model and model-based recursive partitioning (MOB) to detect treatment effect heterogeneity. The MOB LLBT approach enables researchers to identify subgroups that differ in the preference order and in the effect an intervention has on choice behavior. Applicability of MOB LLBT models is demonstrated using an artificial data example with known data-generating mechanism and a real-world data example focusing on drug-harm perception among music festival visitors. In the artificial data example, the MOB LLBT model is able to adequately recover the "true" (population) model. In the real-world data example, the standard LLBT model confirms the existence of a situational willingness among festival visitors to trivialize drug harm when peer consumption behavior is made cognitively accessible. In addition, MOB LLBT results suggest that this trivialization effect is highly context-dependent and most pronounced for participants with low-to-moderate alcohol intoxication who also proactively contacted a substance counselor at the festival venue. Both data examples suggest that MOB LLBT models allow for more nuanced statements about the effectiveness of interventions. We provide R code examples to implement MOB LLBT models for paired comparisons, rankings, and rating (Likert-type) data.

Before/after Bayes: A comparison of frequentist and Bayesian mixed-effects models in applied psychological research.

Bayesian methods are becoming increasingly used in applied psychological research. Previous researchers have thoroughly written about much of the details already, including the philosophy underlying Bayesian methods, computational issues associated with Bayesian model estimation, Bayesian model development and summary, and the role of Bayesian methods in the so-called replication crisis. In this paper, we seek to provide case studies comparing the use of frequentist methods to the use of Bayesian methods in applied psychological research. These case studies are intended to 'illustrate by example' the ways that Bayesian modelling differs from frequentist modelling and the differing conclusions that one may arrive at using the two methods. The intended audience is applied psychological researchers who have been trained in the traditional frequentist framework, who are familiar with mixed-effects models and who are curious about how statistical results might look in a Bayesian context. Along with our case studies, we provide general opinions and guidance on the use of Bayesian methods in applied psychological research.

Score-based measurement invariance checks for Bayesian maximum-a-posteriori estimates in item response theory.

A family of score-based tests has been proposed in recent years for assessing the invariance of model parameters in several models of item response theory (IRT). These tests were originally developed in a maximum likelihood framework. This study discusses analogous tests for Bayesian maximum-a-posteriori estimates and multiple-group IRT models. We propose two families of statistical tests, which are based on an approximation using a pooled variance method, or on a simulation approach based on asymptotic results. The resulting tests were evaluated by a simulation study, which investigated their sensitivity against differential item functioning with respect to a categorical or continuous person covariate in the two- and three-parametric logistic models. Whereas the method based on pooled variance was found to be useful in practice with maximum likelihood as well as maximum-a-posteriori estimates, the simulation-based approach was found to require large sample sizes to lead to satisfactory results.

Computation and application of generalized linear mixed model derivatives using lme4.

Maximum likelihood estimation of generalized linear mixed models (GLMMs) is difficult due to marginalization of the random effects. Derivative computations of a fitted GLMM's likelihood are also difficult, especially because the derivatives are not by-products of popular estimation algorithms. In this paper, we first describe theoretical results related to GLMM derivatives along with a quadrature method to efficiently compute the derivatives, focusing on fitted lme4 models with a single clustering variable. We describe how psychometric results related to item response models are helpful for obtaining the derivatives, as well as for verifying the derivatives' accuracies. We then provide a tutorial on the many possible uses of these derivatives, including robust standard errors, score tests of fixed effect parameters, and likelihood ratio tests of non-nested models. The derivative computation methods and applications described in the paper are all available in easily obtained R packages.

Seeing the impossible: Visualizing latent variable models with flexplavaan.

Latent variable models (LVMs) are incredibly flexible tools that allow users to address research questions they might otherwise never be able to answer (McDonald, 2013). However, one major limitation of LVMs is evaluating model fit. There is no universal consensus about how to evaluate model fit, either globally or locally. Part of the reason evaluating these models is difficult is because fit is typically reduced to a handful of statistics that may or may not reflect the model's adequacy and/or assumptions. In this article we argue that proper evaluation of model fit must include visualizing both the raw data and the model-implied fit. Visuals reveal, at a glance, the fit of the model and whether the model's assumptions have been met. Unfortunately, tools for visualizing LVMs have historically been limited. In this article, we introduce new plots and reframe existing plots that provide necessary resources for evaluating LVMs. These plots are available in a new open-source R package called flexplavaan, which combines the model plotting capabilities of flexplot with the latent variable modeling capabilities of lavaan. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

A note on identification constraints and information criteria in Bayesian latent variable models.

It is well known that, in traditional SEM applications, a scale must be set for each latent variable: typically, either the latent variance or a factor loading is fixed to one. While this has no impact on the fit metrics in ML estimation, it can potentially lead to varying Bayesian model comparison metrics due to the use of different prior distributions under each parameterization. This is a problem, because a researcher could artificially improve one's preferred model simply by changing the identification constraint. Using a single-factor CFA as motivation for study, we first show that Bayesian model comparison metrics can systematically change depending on constraints used. We then study principled methods for setting the scale of the latent variable that stabilize the model comparison metrics. These methods involve (i) the placement of priors on ratios of factor loadings, as opposed to individual loadings; and (ii) use of effect coding. We illustrate the methods via simulation and application.

Score-based tests for detecting heterogeneity in linear mixed models.

Cross-level interactions among fixed effects in linear mixed models (also known as multilevel models) can be complicated by heterogeneity stemming from random effects and residuals. When heterogeneity is present, tests of fixed effects (including cross-level interaction terms) are subject to inflated type I or type II error. While the impact of variance change/heterogeneity has been noticed in the literature, few methods have been proposed to detect this heterogeneity in a simple, systematic way. In addition, when heterogeneity among clusters is detected, researchers often wish to know which clusters' variances differed from the others. In this study, we utilize a recently proposed family of score-based tests to distinguish between cross-level interactions and heterogeneity in variance components, also providing information about specific clusters that exhibit heterogeneity. These score-based tests only require estimation of the null model (when variance homogeneity is assumed to hold), and they have been previously applied to psychometric models to detect measurement invariance. In this paper, we extend the tests to linear mixed models, allowing us to use the tests to differentiate between interaction and heterogeneity. We detail the tests' implementation and performance via simulation, provide an empirical example of the tests' use in practice, and provide code illustrating the tests' general application.

Event-Level Predictors of Alcohol-Impaired Driving Intentions.

OBJECTIVE: Alcohol-impaired driving is a significant public safety concern and is highly prevalent among young adults. Considerable research has examined between-person predictors of alcohol-impaired driving, but there has been little research on factors that predict alcohol-impaired driving at the event level. This pilot/feasibility study was designed to identify within-person, event-level predictors of alcohol-impaired driving intentions in the natural environment using an ecological momentary assessment (EMA) design. METHOD: Thirty-six young adult, moderate drinkers (M age = 22.9 years; 72.2% female; M drinks per occasion = 3.2) were recruited from a university area to complete 2 weeks of EMA. They reported on their subjective levels of intoxication, perceived dangerousness of driving, and driving intentions during real-world drinking episodes. Breath alcohol concentrations were collected with a portable breath alcohol analyzer. RESULTS: Event-level perceived danger and subjective intoxication most strongly predicted intentions to drive after drinking, such that higher perceived danger and intoxication predicted lower willingness to drive, after adjusting for baseline alcohol-impaired driving attitudes (ps < .001). When we accounted for perceived danger during drinking episodes at the event and person level, baseline attitudes were no longer predictive of willingness to drive. Higher event-level breath alcohol concentration also predicted lower willingness to drive (p = .003). CONCLUSIONS: This study is the first to demonstrate that event-level risks of alcohol-impaired driving can be collected during drinking episodes in the natural environment. Findings indicate that subjective perceptions of intoxication and risk more strongly predict alcohol-impaired driving intentions than objective intoxication. Findings also suggest that event-level perceptions of intoxication and driving risk may be fruitful targets for interventions to reduce alcohol-impaired driving.

Repetition attenuates the influence of recency on recognition memory: Behavioral and electrophysiological evidence.

Studies of recognition memory often demonstrate a recency effect on behavioral performance, whereby response times (RTs) are faster for stimuli that were previously presented recently as opposed to more remotely in the past. One account of this relationship between performance and presentation lag posits that memories are accessed by serially searching backward in time, such that RT indicates the self-terminating moment of such a process. Here, we investigated the conditions under which this serial search gives way to more efficient means of retrieving memories. Event-related potentials (ERPs) were recorded during a continuous recognition task, in which subjects made binary old/new judgments to stimuli that were each presented up to four times across a range of lags. Stimulus repetition and shorter presentation lag both gave rise to speeded RTs, consistent with previous findings, and we novelly extend these effects to a robust latency measure of the left parietal ERP correlate of retrieval success. Importantly, the relationship between repetition and recency was further elucidated, such that repetition attenuated lag-related differences that were initially present in both the behavioral and neural latency data. These findings are consistent with the idea that an effortful search through recent memory can quickly be abandoned in favor of relying on more efficient "time-independent" cognitive processes or neural signals.

Model Selection of Nested and Non-Nested Item Response Models Using Vuong Tests.

In this paper, we apply Vuong's general approach of model selection to the comparison of nested and non-nested unidimensional and multidimensional item response theory (IRT) models. Vuong's approach of model selection is useful because it allows for formal statistical tests of both nested and non-nested models. However, only the test of non-nested models has been applied in the context of IRT models to date. After summarizing the statistical theory underlying the tests, we investigate the performance of all three distinct Vuong tests in the context of IRT models using simulation studies and real data. In the non-nested case we observed that the tests can reliably distinguish between the graded response model and the generalized partial credit model. In the nested case, we observed that the tests typically perform as well as or sometimes better than the traditional likelihood ratio test. Based on these results, we argue that Vuong's approach provides a useful set of tools for researchers and practitioners to effectively compare competing nested and non-nested IRT models.

Differential item functioning analysis of the CUDIT and relations with alcohol and tobacco use among men across five ethnic groups: The HELIUS study.

Cannabis use has been rising despite recognition of the negative consequences associated with heavy use. The severity of these consequences has been shown to differ across racial/ethnic groups, even when controlling for consumption levels. The present study conducted an item response theory (IRT) analysis of the Cannabis Use Disorders Identification Test (CUDIT) to better understand the patterns of problematic cannabis use and their relation with other substance use across ethnic groups in the Healthy Life in an Urban Setting (HELIUS) study. CUDIT responses from 1,960 cannabis-using African Surinamese, South-Asian Surinamese, Dutch, Moroccan, and Turkish ethnic origin participants were used to test for differential item functioning (DIF) within an IRT framework. After restricting the sample to men because of low frequency of use among women, several instances of uniform DIF were identified. Multiple-group IRT analysis yielded a harmonized cannabis use phenotype that was used to estimate ethnic group differences in problematic cannabis use and its relation to alcohol and tobacco co-use. These analyses suggested that cannabis users from certain ethnic minority groups experienced higher rates of problematic use than the majority group despite lower rates of cannabis use. Further, cannabis and tobacco use were positively related across groups, whereas only ethnic minority groups showed a positive relation between cannabis and alcohol use. These results demonstrate the importance of accounting for DIF when examining group differences in problematic cannabis use, and support prior evidence suggesting that certain ethnic minority groups may be more likely to experience problematic cannabis use and alcohol co-use relative to the majority group. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Bayesian Comparison of Latent Variable Models: Conditional Versus Marginal Likelihoods.

Typical Bayesian methods for models with latent variables (or random effects) involve directly sampling the latent variables along with the model parameters. In high-level software code for model definitions (using, e.g., BUGS, JAGS, Stan), the likelihood is therefore specified as conditional on the latent variables. This can lead researchers to perform model comparisons via conditional likelihoods, where the latent variables are considered model parameters. In other settings, however, typical model comparisons involve marginal likelihoods where the latent variables are integrated out. This distinction is often overlooked despite the fact that it can have a large impact on the comparisons of interest. In this paper, we clarify and illustrate these issues, focusing on the comparison of conditional and marginal Deviance Information Criteria (DICs) and Watanabe-Akaike Information Criteria (WAICs) in psychometric modeling. The conditional/marginal distinction corresponds to whether the model should be predictive for the clusters that are in the data or for new clusters (where "clusters" typically correspond to higher-level units like people or schools). Correspondingly, we show that marginal WAIC corresponds to leave-one-cluster out cross-validation, whereas conditional WAIC corresponds to leave-one-unit out. These results lead to recommendations on the general application of the criteria to models with latent variables.

Bayesian latent variable models for the analysis of experimental psychology data.

In this paper, we address the use of Bayesian factor analysis and structural equation models to draw inferences from experimental psychology data. While such application is non-standard, the models are generally useful for the unified analysis of multivariate data that stem from, e.g., subjects' responses to multiple experimental stimuli. We first review the models and the parameter identification issues inherent in the models. We then provide details on model estimation via JAGS and on Bayes factor estimation. Finally, we use the models to re-analyze experimental data on risky choice, comparing the approach to simpler, alternative methods.

Direction of dependence in measurement error models.

Methods to determine the direction of a regression line, that is, to determine the direction of dependence in reversible linear regression models (e.g., x→y vs. y→x), have experienced rapid development within the last decade. However, previous research largely rested on the assumption that the true predictor is measured without measurement error. The present paper extends the direction dependence principle to measurement error models. First, we discuss asymmetric representations of the reliability coefficient in terms of higher moments of variables and the attenuation of skewness and excess kurtosis due to measurement error. Second, we identify conditions where direction dependence decisions are biased due to measurement error and suggest method of moments (MOM) estimation as a remedy. Third, we address data situations in which the true outcome exhibits both regression and measurement error, and propose a sensitivity analysis approach to determining the robustness of direction dependence decisions against unreliably measured outcomes. Monte Carlo simulations were performed to assess the performance of MOM-based direction dependence measures and their robustness to violated measurement error assumptions (i.e., non-independence and non-normality). An empirical example from subjective well-being research is presented. The plausibility of model assumptions and links to modern causal inference methods for observational data are discussed.

Using trial-level data and multilevel modeling to investigate within-task change in event-related potentials.

EEG data, and specifically the ERP, provide psychologists with the power to examine quickly occurring cognitive processes at the native temporal resolution at which they occur. Despite the advantages conferred by ERPs to examine processes at different points in time, ERP researchers commonly ignore the trial-to-trial temporal dimension by collapsing across trials of similar types (i.e., the signal averaging approach) because of constraints imposed by repeated measures ANOVA. Here, we present the advantages of using multilevel modeling (MLM) to examine trial-level data to investigate change in neurocognitive processes across the course of an experiment. Two examples are presented to illustrate the usefulness of this technique. The first demonstrates decreasing differentiation in N170 amplitude to faces of different races across the course of a race categorization task. The second demonstrates attenuation of the ERN as participants commit more errors within a task designed to measure implicit racial bias. Although the examples presented here are within the realm of social psychology, the use of MLM to analyze trial-level EEG data has the potential to contribute to a number of different theoretical domains within psychology.

Score-Based Tests of Differential Item Functioning via Pairwise Maximum Likelihood Estimation.

Measurement invariance is a fundamental assumption in item response theory models, where the relationship between a latent construct (ability) and observed item responses is of interest. Violation of this assumption would render the scale misinterpreted or cause systematic bias against certain groups of persons. While a number of methods have been proposed to detect measurement invariance violations, they typically require advance definition of problematic item parameters and respondent grouping information. However, these pieces of information are typically unknown in practice. As an alternative, this paper focuses on a family of recently proposed tests based on stochastic processes of casewise derivatives of the likelihood function (i.e., scores). These score-based tests only require estimation of the null model (when measurement invariance is assumed to hold), and they have been previously applied in factor-analytic, continuous data contexts as well as in models of the Rasch family. In this paper, we aim to extend these tests to two-parameter item response models, with strong emphasis on pairwise maximum likelihood. The tests' theoretical background and implementation are detailed, and the tests' abilities to identify problematic item parameters are studied via simulation. An empirical example illustrating the tests' use in practice is also provided.

The iterative nature of person construal: Evidence from event-related potentials.

Recently, a dynamic-interactive model of person construal (DI model) has been proposed, whereby the social categories a person represents are determined on the basis of an iterative integration of bottom-up and top-down influences. The current study sought to test this model by leveraging the high temporal resolution of event-related brain potentials (ERPs) as 65 participants viewed male faces that varied by race (White vs Black), fixating either between the eyes or on the forehead. Within face presentations, the effect of fixation, meant to vary bottom-up visual input, initially was large but decreased across early latency neural responses identified by a principal components analysis (PCA). In contrast, the effect of race, reflecting a combination of top-down and bottom-up factors, initially was small but increased across early latency principal components. These patterns support the DI model prediction that bottom-up and top-down processes are iteratively integrated to arrive at a stable construal within 230 ms. Additionally, exploratory multilevel modeling of single trial ERP responses representing a component linked to outgroup categorization (the P2) suggests change in effects of the manipulations over the course of the experiment. Implications of the findings for the DI model are considered.

Factor analysis linking functions for simultaneously modeling neural and behavioral data.

A growing number of researchers have advocated for the advancement of cognitive neuroscience by blending cognitive models with neurophysiology. The recently proposed joint modeling framework is one way to bridge the gap between the abstractions assumed by cognitive models and the neurophysiology obtained by modern methods in neuroscience. Despite this advancement, the current method for linking the two domains is hindered by the dimensionality of the neural data. In this article, we present a new linking function based on factor analysis that allows joint models to grow linearly in complexity with increases in the number of neural features. The new linking function is then evaluated in two simulation studies. The first simulation study shows how the model parameters can be accurately recovered when there are many neural features, that mimics real-world applications. The second simulation shows how the new linking function can (1) properly recover a representation of the data generating model, even in the case of model misspecification, and (2) outperform the previous linking function in a cross-validation test. We close by applying a model equipped with the new linking function to real-world data from a perceptual decision making task. The model allows us to understand how differences in the model parameters emerge as a function of differences in brain function across speed and accuracy instruction.

Testing nonnested structural equation models.

In this article, we apply Vuong's (1989) likelihood ratio tests of nonnested models to the comparison of nonnested structural equation models (SEMs). Similar tests have been previously applied in SEM contexts (especially to mixture models), though the nonstandard output required to conduct the tests has limited their use and study. We review the theory underlying the tests and show how they can be used to construct interval estimates for differences in nonnested information criteria. Through both simulation and application, we then study the tests' performance in nonmixture SEMs and describe their general implementation via free R packages. The tests offer researchers a useful tool for nonnested SEM comparison, with barriers to test implementation now removed. (PsycINFO Database Record

Informed Decision Making: Assessment of the Quality of Physician Communication about Prostate Cancer Diagnosis and Treatment.

INTRODUCTION: Little is known about how physicians present diagnosis and treatment planning in routine practice in preference-sensitive treatment decisions. We evaluated completeness and quality of informed decision making in localized prostate cancer post biopsy encounters. METHODS: We analyzed audio-recorded office visits of 252 men with presumed localized prostate cancer (Gleason 6 and Gleason 7 scores) who were seeing 45 physicians at 4 Veterans Affairs Medical Centers. Data were collected between September 2008 and May 2012 in a trial of 2 decision aids (DAs). Braddock's previously validated Informed Decision Making (IDM) system was used to measure quality. Latent variable models for ordinal data examined the relationship of IDM score to treatment received. RESULTS: Mean IDM score showed modest quality (7.61±2.45 out of 18) and high variability. Treatment choice and risks and benefits were discussed in approximately 95% of encounters. However, in more than one-third of encounters, physicians provided a partial set of treatment options and omitted surveillance as a choice. Informing quality was greater in patients treated with surveillance (ß = 1.1, p = .04). Gleason score (7 vs 6) and lower age were often cited as reasons to exclude surveillance. Patient preferences were elicited in the majority of cases, but not used to guide treatment planning. Encounter time was modestly correlated with IDM score (r = 0.237, p = .01). DA type was not associated with IDM score. DISCUSSION: Physicians informed patients of options and risks and benefits, but infrequently engaged patients in core shared decision-making processes. Despite patients having received DAs, physicians rarely provided an opportunity for preference-driven decision making. More attention to the underused patient decision-making and engagement elements could result in improved shared decision making.