Chasing consistency: On the measurement error in self-reported affect in experiments.

How feelings change over time is a central topic in emotion research. To study these affective fluctuations, researchers often ask participants to repeatedly indicate how they feel on a self-report rating scale. Despite widespread recognition that this kind of data is subject to measurement error, the extent of this error remains an open question. Complementing many daily-life studies, this study aimed to investigate this question in an experimental setting. In such a setting, multiple trials follow each other at a fast pace, forcing experimenters to use a limited number of questions to measure affect during each trial. A total of 1398 participants completed a probabilistic reward task in which they were unknowingly presented with the same string of outcomes multiple times throughout the study. This allowed us to assess the test-retest consistency of their affective responses to the rating scales under investigation. We then compared these consistencies across different types of rating scales in hopes of finding out whether a given type of scale led to a greater consistency of affective measurements. Overall, we found moderate to good consistency of the affective measurements. Surprisingly, however, we found no differences in consistency across rating scales, which suggests that the specific rating scale that is used does not influence the measurement consistency.

Testing a computational model of subjective well-being: a preregistered replication of Rutledge et al. (2014).

Subjective well-being changes over time. While the causes of these changes have been investigated extensively, few attempts have been made to capture these changes through computational modelling. One notable exception is the study by Rutledge et al. [Rutledge, R. B., Skandali, N., Dayan, P., & Dolan, R. J. (2014). A computational and neural model of momentary subjective well-being. Proceedings of the National Academy of Sciences, 111(33), 12252-12257. https://doi.org/10.1073/pnas.1407535111], in which a model that captures momentary changes in subjective well-being was proposed. The model incorporates how an individual processes rewards and punishments in a decision context. Using this model, the authors were able to successfully explain fluctuations in subjective well-being observed in a gambling paradigm. Although Rutledge et al. reported an in-paper replication, a successful independent replication would further increase the credibility of their results. In this paper, we report a preregistered close replication of the behavioural experiment and analyses by Rutledge et al. The results of Rutledge et al. were mostly confirmed, providing further evidence for the role of rewards and punishments in subjective well-being fluctuations. Additionally, the association between personality traits and the way people process rewards and punishments was examined. No evidence for such associations was found, leaving this an open question for future research.

No harmful effect of propranolol administered prior to fear memory extinction in rats and humans.

Exposure therapy is an evidence-based treatment option for anxiety-related disorders. Many patients also take medication that could, in principle, affect exposure therapy efficacy. Clinical and laboratory evidence indeed suggests that benzodiazepines may have detrimental effects. Large clinical trials with propranolol, a common beta-blocker, are currently lacking, but several preclinical studies do indicate impaired establishment of safety memories. Here, we investigated the effects of propranolol given prior to extinction training in 9 rat studies (N = 215) and one human study (N = 72). A Bayesian meta-analysis of our rat studies provided strong evidence against propranolol-induced extinction memory impairment during a drug-free test, and the human study found no significant difference with placebo. Two of the rat studies actually suggested a small beneficial effect of propranolol. Lastly, two rat studies with a benzodiazepine (midazolam) group provided some evidence for a harmful effect on extinction memory, i.e., impaired extinction retention. In conclusion, our midazolam findings are in line with prior literature (i.e., an extinction retention impairment), but this is not the case for the 10 studies with propranolol. Our data thus support caution regarding the use of benzodiazepines during exposure therapy, but argue against a harmful effect of propranolol on extinction learning.

Subjective evidence evaluation survey for many-analysts studies.

Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

Idealness and similarity in goal-derived categories: a computational examination.

The finding that the typicality gradient in goal-derived categories is mainly driven by ideals rather than by exemplar similarity has stood uncontested for nearly three decades. Due to the rather rigid earlier implementations of similarity, a key question has remained--that is, whether a more flexible approach to similarity would alter the conclusions. In the present study, we evaluated whether a similarity-based approach that allows for dimensional weighting could account for findings in goal-derived categories. To this end, we compared a computational model of exemplar similarity (the generalized context model; Nosofsky, Journal of Experimental Psychology. General 115:39-57, 1986) and a computational model of ideal representation (the ideal-dimension model; Voorspoels, Vanpaemel, & Storms, Psychonomic Bulletin & Review 18:1006-114, 2011) in their accounts of exemplar typicality in ten goal-derived categories. In terms of both goodness-of-fit and generalizability, we found strong evidence for an ideal approach in nearly all categories. We conclude that focusing on a limited set of features is necessary but not sufficient to account for the observed typicality gradient. A second aspect of ideal representations--that is, that extreme rather than common, central-tendency values drive typicality--seems to be crucial.

Quantum models of cognition as Orwellian newspeak.

Faced with probabilistic relationships between causes and effects, quantum theory assumes that deterministic causes do not exist, and that only incomplete probabilistic expressions of knowledge are possible. As in its application to physics, this fundamental epistemological stance severely limits the ability of quantum theory to provide insight and understanding in human cognition.

Data sharing upon request and statistical consistency errors in psychology: A replication of Wicherts, Bakker and Molenaar (2011).

Sharing research data allows the scientific community to verify and build upon published work. However, data sharing is not common practice yet. The reasons for not sharing data are myriad: Some are practical, others are more fear-related. One particular fear is that a reanalysis may expose errors. For this explanation, it would be interesting to know whether authors that do not share data genuinely made more errors than authors who do share data. (Wicherts, Bakker and Molenaar 2011) examined errors that can be discovered based on the published manuscript only, because it is impossible to reanalyze unavailable data. They found a higher prevalence of such errors in papers for which the data were not shared. However, (Nuijten et al. 2017) did not find support for this finding in three large studies. To shed more light on this relation, we conducted a replication of the study by (Wicherts et al. 2011). Our study consisted of two parts. In the first part, we reproduced the analyses from (Wicherts et al. 2011) to verify the results, and we carried out several alternative analytical approaches to evaluate the robustness of the results against other analytical decisions. In the second part, we used a unique and larger data set that originated from (Vanpaemel et al. 2015) on data sharing upon request for reanalysis, to replicate the findings in (Wicherts et al. 2011). We applied statcheck for the detection of consistency errors in all included papers and manually corrected false positives. Finally, we again assessed the robustness of the replication results against other analytical decisions. Everything taken together, we found no robust empirical evidence for the claim that not sharing research data for reanalysis is associated with consistency errors.

A review of applications of the Bayes factor in psychological research.

The last 25 years have shown a steady increase in attention for the Bayes factor as a tool for hypothesis evaluation and model selection. The present review highlights the potential of the Bayes factor in psychological research. We discuss six types of applications: Bayesian evaluation of point null, interval, and informative hypotheses, Bayesian evidence synthesis, Bayesian variable selection and model averaging, and Bayesian evaluation of cognitive models. We elaborate what each application entails, give illustrative examples, and provide an overview of key references and software with links to other applications. The article is concluded with a discussion of the opportunities and pitfalls of Bayes factor applications and a sketch of corresponding future research lines. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Stimulus-Driven Affective Change: Evaluating Computational Models of Affect Dynamics in Conjunction with Input.

The way in which emotional experiences change over time can be studied through the use of computational models. An important question with regard to such models is which characteristics of the data a model should account for in order to adequately describe these data. Recently, attention has been drawn on the potential importance of nonlinearity as a characteristic of affect dynamics. However, this conclusion was reached through the use of experience sampling data in which no information was available about the context in which affect was measured. However, affective stimuli may induce some or all of the observed nonlinearity. This raises the question of whether computational models of affect dynamics should account for nonlinearity, or whether they just need to account for the affective stimuli a person encounters. To investigate this question, we used a probabilistic reward task in which participants either won or lost money at each trial. A number of plausible ways in which the experimental stimuli played a role were considered and applied to the nonlinear Affective Ising Model (AIM) and the linear Bounded Ornstein-Uhlenbeck (BOU) model. In order to reach a conclusion, the relative and absolute performance of these models were assessed. Results suggest that some of the observed nonlinearity could indeed be attributed to the experimental stimuli. However, not all nonlinearity was accounted for by these stimuli, suggesting that nonlinearity may present an inherent feature of affect dynamics. As such, nonlinearity should ideally be accounted for in the computational models of affect dynamics. Supplementary Information: The online version contains supplementary material available at 10.1007/s42761-022-00118-5.

A pre-registered, multi-lab non-replication of the action-sentence compatibility effect (ACE).

The Action-sentence Compatibility Effect (ACE) is a well-known demonstration of the role of motor activity in the comprehension of language. Participants are asked to make sensibility judgments on sentences by producing movements toward the body or away from the body. The ACE is the finding that movements are faster when the direction of the movement (e.g., toward) matches the direction of the action in the to-be-judged sentence (e.g., Art gave you the pen describes action toward you). We report on a pre-registered, multi-lab replication of one version of the ACE. The results show that none of the 18 labs involved in the study observed a reliable ACE, and that the meta-analytic estimate of the size of the ACE was essentially zero.

Prototype-based category learning in autism: A review.

Similarity-based categorization, as an important cognitive skill, can be performed by abstracting a categories' central tendency, the so-called prototype, or by memorizing individual exemplars of a category. The flexible selection of an appropriate strategy is crucial for effective cognitive functioning. The detail-focused cognitive style in individuals with autism spectrum disorders (ASD) has been hypothesized to specifically impair prototype-based categorization but to leave exemplar-based categorization unimpaired. We first give an overview of approaches to investigate prototype-based abstraction in the prototype-distortion task, with an emphasis on model-based approaches suitable to discern the two strategies on the individual level. The second part summarizes literature speaking to prototype-based categorization in ASD using that task. Despite considerable inconsistencies, most studies appear to confirm that autistic individuals have more difficulties to perform prototype-distortion tasks than non-autistic individuals. We highlight how inconsistencies in literature can be resolved by taking the differences in task designs into account. The current review illustrates the need for sensitive computational approaches, suitable to detect hidden individual differences and potential compensatory strategies.

Comparing dream to reality: an assessment of adherence of the first generation of preregistered studies.

Preregistration is a method to increase research transparency by documenting research decisions on a public, third-party repository prior to any influence by data. It is becoming increasingly popular in all subfields of psychology and beyond. Adherence to the preregistration plan may not always be feasible and even is not necessarily desirable, but without disclosure of deviations, readers who do not carefully consult the preregistration plan might get the incorrect impression that the study was exactly conducted and reported as planned. In this paper, we have investigated adherence and disclosure of deviations for all articles published with the Preregistered badge in Psychological Science between February 2015 and November 2017 and shared our findings with the corresponding authors for feedback. Two out of 27 preregistered studies contained no deviations from the preregistration plan. In one study, all deviations were disclosed. Nine studies disclosed none of the deviations. We mainly observed (un)disclosed deviations from the plan regarding the reported sample size, exclusion criteria and statistical analysis. This closer look at preregistrations of the first generation reveals possible hurdles for reporting preregistered studies and provides input for future reporting guidelines. We discuss the results and possible explanations, and provide recommendations for preregistered research.

The reproducibility of statistical results in psychological research: An investigation using unpublished raw data.

We investigated the reproducibility of the major statistical conclusions drawn in 46 articles published in 2012 in three APA journals. After having identified 232 key statistical claims, we tried to reproduce, for each claim, the test statistic, its degrees of freedom, and the corresponding p value, starting from the raw data that were provided by the authors and closely following the Method section in the article. Out of the 232 claims, we were able to successfully reproduce 163 (70%), 18 of which only by deviating from the article's analytical description. Thirteen (7%) of the 185 claims deemed significant by the authors are no longer so. The reproduction successes were often the result of cumbersome and time-consuming trial-and-error work, suggesting that APA style reporting in conjunction with raw data makes numerical verification at least hard, if not impossible. This article discusses the types of mistakes we could identify and the tediousness of our reproduction efforts in the light of a newly developed taxonomy for reproducibility. We then link our findings with other findings of empirical research on this topic, give practical recommendations on how to achieve reproducibility, and discuss the challenges of large-scale reproducibility checks as well as promising ideas that could considerably increase the reproducibility of psychological research. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Geometric and featural representations in semantic concepts.

We explore the adequacy of two types of similarity representation in the context of semantic concepts. To this end, we evaluate different categorization models, assuming either a geometric or a featural representation, using categorization decisions involving familiar and unfamiliar foods and animals. The study aims to assess the optimal stimulus representation as a function of the familiarity of the stimuli. For the unfamiliar stimuli, the geometric categorization models provide the best account of the categorization data, whereas for the familiar stimuli, the featural categorization models provide the best account. This pattern of results suggests that people rely on perceptual information to assign an unfamiliar stimulus to a category but rely on more elaborate conceptual knowledge when assigning a familiar stimulus.

Abstraction and model evaluation in category learning.

Thirty previously published data sets, from seminal category learning tasks, are reanalyzed using the varying abstraction model (VAM). Unlike a prototype-versus-exemplar analysis, which focuses on extreme levels of abstraction only, a VAM analysis also considers the possibility of partial abstraction. Whereas most data sets support no abstraction when only the extreme possibilities are considered, we show that evidence for abstraction can be provided using the broader view on abstraction provided by the VAM. The present results generalize earlier demonstrations of partial abstraction (Vanpaemel & Storms, 2008), in which only a small number of data sets was analyzed. Following the dominant modus operandi in category learning research, Vanpaemel and Storms evaluated the models on their best fit, a practice known to ignore the complexity of the models under consideration. In the present study, in contrast, model evaluation not only relies on the maximal likelihood, but also on the marginal likelihood, which is sensitive to model complexity. Finally, using a large recovery study, it is demonstrated that, across the 30 data sets, complexity differences between the models in the VAM family are small. This indicates that a (computationally challenging) complexity-sensitive model evaluation method is uncalled for, and that the use of a (computationally straightforward) complexity-insensitive model evaluation method is justified.

Strong theory testing using the prior predictive and the data prior.

In their seminal article, Roberts and Pashler (2000) highlighted that providing a good fit to empirical data does not necessarily provide strong support for a theory. For a good fit to be persuasive and for a theory to be strongly supported, the theory should have survived a strong test, in the sense that it is plausible that the theory might have failed the test. The most common way to accommodate the problem of the limited value of a good fit alone is to not only report a measure of goodness-of-fit, but also a measure of the complexity. A recent example of this line of reasoning is provided by Veksler, Myers, and Gluck (2015). In this article, I argue that whereas considering complexity provides useful information when theory testing, using complexity to gauge the severity of a test, or, equivalently, the persuasiveness of a good fit, is misguided. The reason is that complexity only provides information about the possibility of a bad fit, which does not guarantee a strong test. A condition for a test to be strong and a good fit to be persuasive is the demonstration of the plausibility of a bad fit. I provide a worked example of a more complete answer to assessing whether a good fit is persuasive. Providing a strong theory test requires the use of what can be called a data prior, which quantifies-before taking the empirical data into account-which outcomes are plausible. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

BayesGCM: software for Bayesian inference with the generalized context model.

This article describes and demonstrates the BayesGCM software package. The software is designed to perform Bayesian analysis with the generalized context model (GCM). It is intended to make the GCM easily accessible to a general public of experimental, social, and clinical psychologists interested in category learning, sensitivity, and attention. The software uses MATLAB and relies on WinBUGS to draw samples from the posterior distribution of the GCM's parameters. The returned output comprises the full set of posterior samples, summary descriptive statistics, and graphs of the posterior distribution for each parameter of interest.

Cross-cultural consistency and relativity in the enjoyment of thinking versus doing.

Which is more enjoyable: trying to think enjoyable thoughts or doing everyday solitary activities? Wilson et al. (2014) found that American participants much preferred solitary everyday activities, such as reading or watching TV, to thinking for pleasure. To see whether this preference generalized outside of the United States, we replicated the study with 2,557 participants from 12 sites in 11 countries. The results were consistent in every country: Participants randomly assigned to do something reported significantly greater enjoyment than did participants randomly assigned to think for pleasure. Although we found systematic differences by country in how much participants enjoyed thinking for pleasure, we used a series of nested structural equation models to show that these differences were fully accounted for by country-level variation in 5 individual differences, 4 of which were positively correlated with thinking for pleasure (need for cognition, openness to experience, meditation experience, and initial positive affect) and 1 of which was negatively correlated (reported phone usage). (PsycINFO Database Record (c) 2019 APA, all rights reserved).

In search of abstraction: the varying abstraction model of categorization.

A longstanding debate in the categorization literature concerns representational abstraction. Generally, when exemplar models, which assume no abstraction, have been contrasted with prototype models, which assume total abstraction, the former models have been found to be superior to the latter. Although these findings may rule out the idea that total abstraction takes place during category learning and instead suggest that no abstraction is involved, the idea of abstraction retains considerable intuitive appeal. In this article, we propose the varying abstraction model of categorization (VAM), which investigates the possibility that partial abstraction may play a role in category learning. We apply the VAM to four previously published data sets that have been used to argue that no abstraction is involved. Contrary to the previous findings, our results provide support for the idea that some form of partial abstraction can be used in people's category representations.

Exemplars and prototypes in natural language concepts: a typicality-based evaluation.

Are natural language categories represented by instances of the category or by a summary representation? We used an exemplar model and a prototype model, both derived within the framework of the generalized context model (Nosofsky, 1984, 1986), to predict typicality ratings for 12 superordinate natural language concepts. The models were fitted to typicality ratings averaged across participants and to the typicality judgments of individual participants. Both analyses yielded results in favor of the exemplar model. These results suggest that higher-level natural language concepts are represented by their subordinate members, rather than by a summary representation.