The impact of ChatGPT on human data collection: A case study involving typicality norming data.

Tools like ChatGPT, which allow people to unlock the potential of large language models (LLMs), have taken the world by storm. ChatGPT's ability to produce written output of remarkable quality has inspired, or forced, academics to consider its consequences for both research and education. In particular, the question of what constitutes authorship, and how to evaluate (scientific) contributions has received a lot of attention. However, its impact on (online) human data collection has mostly flown under the radar. The current paper examines how ChatGPT can be (mis)used in the context of generating norming data. We found that ChatGPT is able to produce sensible output, resembling that of human participants, for a typicality rating task. Moreover, the test-retest reliability of ChatGPT's ratings was similar to that of human participants tested 1 day apart. We discuss the relevance of these findings in the context of (online) human data collection, focusing both on opportunities (e.g., (risk-)free pilot data) and challenges (e.g., data fabrication).

The Optimal Learning Cocktail for Placebo Analgesia: A Randomized Controlled Trial Comparing Individual and Combined Techniques.

This study investigated for the first time the effects of individual and combined application of 3 learning techniques (verbal suggestions, classical conditioning, and observational learning) on placebo analgesia and extinction. Healthy participants (N = 206) were assigned to 8 different groups in which they were taught through either a verbal suggestion, a conditioning paradigm, a video observing someone, or any combination thereof that a placebo device (inactive transcutaneous electric nerve stimulation [TENS]) was capable of alleviating heat pain, whereas one group did not (control). Placebo analgesia was quantified as the within-group difference in experienced pain when the placebo device was (sham) 'activated' or 'inactivated' during equal pain stimuli, and compared between groups. Placebo analgesia was induced in groups with 2 or 3 learning techniques. Significantly stronger placebo analgesia was induced in the combination of all 3 learning techniques as compared to the individual learning techniques or control condition, underlining the additional contribution of 3 combined techniques. Extinction did not differ between groups. Furthermore, pain expectancies, but not state anxiety or trust, mediated placebo analgesia. Our findings emphasize the added value of combining 3 learning techniques to optimally shape expectancies that lead to placebo analgesia, which can be used in experimental and clinical settings. PERSPECTIVE: This unique experimental study compared the individual versus combined effects of 3 important ways of learning (verbal suggestions, classical conditioning, and observational learning) on expectation-based pain relief. The findings indicate that placebo effects occurring in clinical practice could be optimally strengthened if healthcare providers apply these techniques in combination.

Data sharing upon request and statistical consistency errors in psychology: A replication of Wicherts, Bakker and Molenaar (2011).

Sharing research data allows the scientific community to verify and build upon published work. However, data sharing is not common practice yet. The reasons for not sharing data are myriad: Some are practical, others are more fear-related. One particular fear is that a reanalysis may expose errors. For this explanation, it would be interesting to know whether authors that do not share data genuinely made more errors than authors who do share data. (Wicherts, Bakker and Molenaar 2011) examined errors that can be discovered based on the published manuscript only, because it is impossible to reanalyze unavailable data. They found a higher prevalence of such errors in papers for which the data were not shared. However, (Nuijten et al. 2017) did not find support for this finding in three large studies. To shed more light on this relation, we conducted a replication of the study by (Wicherts et al. 2011). Our study consisted of two parts. In the first part, we reproduced the analyses from (Wicherts et al. 2011) to verify the results, and we carried out several alternative analytical approaches to evaluate the robustness of the results against other analytical decisions. In the second part, we used a unique and larger data set that originated from (Vanpaemel et al. 2015) on data sharing upon request for reanalysis, to replicate the findings in (Wicherts et al. 2011). We applied statcheck for the detection of consistency errors in all included papers and manually corrected false positives. Finally, we again assessed the robustness of the replication results against other analytical decisions. Everything taken together, we found no robust empirical evidence for the claim that not sharing research data for reanalysis is associated with consistency errors.

The process of replication target selection in psychology: what to consider?

Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target. The resulting checklist can be used for transparently communicating the rationale for selecting studies for replication.

Putting the Self in Self-Correction: Findings From the Loss-of-Confidence Project.

Science is often perceived to be a self-correcting enterprise. In principle, the assessment of scientific claims is supposed to proceed in a cumulative fashion, with the reigning theories of the day progressively approximating truth more accurately over time. In practice, however, cumulative self-correction tends to proceed less efficiently than one might naively suppose. Far from evaluating new evidence dispassionately and infallibly, individual scientists often cling stubbornly to prior findings. Here we explore the dynamics of scientific self-correction at an individual rather than collective level. In 13 written statements, researchers from diverse branches of psychology share why and how they have lost confidence in one of their own published findings. We qualitatively characterize these disclosures and explore their implications. A cross-disciplinary survey suggests that such loss-of-confidence sentiments are surprisingly common among members of the broader scientific population yet rarely become part of the public record. We argue that removing barriers to self-correction at the individual level is imperative if the scientific community as a whole is to achieve the ideal of efficient self-correction.

Correction notices in psychology: impactful or inconsequential?

Science is self-correcting, or so the adage goes, but to what extent is that indeed the case? Answering this question requires careful consideration of the various approaches to achieve the collective goal of self-correction. One of the most straightforward mechanisms is individual self-correction: researchers rectifying their own mistakes by publishing a correction notice. Although it offers an efficient route to correcting the scientific record, it has received little to no attention from a metascientific point of view. We aim to fill this void by analysing the content of correction notices published from 2010 until 2018 in the three psychology journals featuring the highest number of corrections over that timespan based on the Scopus database (i.e. Psychological Science with N = 58, Frontiers in Psychology with N = 99 and Journal of Affective Disorders with N = 57). More concretely, we examined which aspects of the original papers were affected (e.g. hypotheses, data-analyses, metadata such as author order, affiliations, funding information etc.) as well as the perceived implications for the papers' main findings. Our exploratory analyses showed that many corrections involved inconsequential errors. Furthermore, authors rarely revised their conclusions, even though several corrections concerned changes to the results. We conclude with a discussion of current policies, and suggest ways to improve upon the present situation by (i) preventing mistakes, and (ii) transparently rectifying those mistakes that do find their way into the literature.

Sound-Symbolism Effects in the Absence of Awareness: A Replication Study.

People have been shown to link particular sounds with particular shapes. For instance, the round-sounding nonword bouba tends to be associated with curved shapes, whereas the sharp-sounding nonword kiki is deemed to be related to angular shapes. People's tendency to associate sounds and shapes has been observed across different languages. In the present study, we reexamined the claim by Hung, Styles, and Hsieh (2017) that such sound-shape mappings can occur before an individual becomes aware of the visual stimuli. More precisely, we replicated their first experiment, in which congruent and incongruent stimuli (e.g., bouba presented in a round shape or an angular shape, respectively) were rendered invisible through continuous flash suppression. The results showed that congruent combinations, on average, broke suppression faster than incongruent combinations, thus providing converging evidence for Hung and colleagues' assertions. Collectively, these findings now provide a solid basis from which to explore the boundary conditions of the effect.

Can prediction-based distributional semantic models predict typicality?

Recent advances in the field of computational linguistics have led to the development of various prediction-based models of semantics. These models seek to infer word representations from large text collections by predicting target words from neighbouring words (or vice versa). The resulting representations are vectors in a continuous space, collectively called word embeddings. Although psychological plausibility was not a primary concern for the developers of predictive models, it has been the topic of several recent studies in the field of psycholinguistics. That is, word embeddings have been linked to similarity ratings, word associations, semantic priming, word recognition latencies, and so on. Here, we build on this work by investigating category structure. Throughout seven experiments, we sought to predict human typicality judgements from two languages, Dutch and English, using different semantic spaces. More specifically, we extracted a number of predictor variables, and evaluated how well they could capture the typicality gradient of common categories (e.g., birds, fruit, vehicles, etc.). Overall, the performance of predictive models was rather modest and did not compare favourably with that of an older count-based model. These results are somewhat disappointing given the enthusiasm surrounding predictive models. Possible explanations and future directions are discussed.

Is jellyfish more of a fish in English than in Dutch? The effect of informative labels.

Some words are lexically suggestive about the taxonomic position of their referent (e.g., jellyfish in English), and this information can vary across languages (e.g., in Dutch the equivalent of jellyfish holds no taxonomic information: kwal). To evaluate the role of such lexical suggestions, we conducted a cross-linguistic study in which similarity judgements from two language groups (Dutch and English speakers) were compared. We paired asymmetrically informative items with items that are considered to be typical members of the referenced category (e.g., jellyfish-salmon). Our analyses revealed that items were deemed more similar by speakers of a language in which the lexical information was present (e.g., English speakers tended to give relatively higher ratings for jellyfish-salmon than Dutch participants did for the non-informative equivalent kwal-zalm). Results are discussed in light of theories of concept representation and compound processing.

The (un)reliability of item-level semantic priming effects.

Many researchers have tried to predict semantic priming effects using a myriad of variables (e.g., prime-target associative strength or co-occurrence frequency). The idea is that relatedness varies across prime-target pairs, which should be reflected in the size of the priming effect (e.g., cat should prime dog more than animal does). However, it is only insightful to predict item-level priming effects if they can be measured reliably. Thus, in the present study we examined the split-half and test-retest reliabilities of item-level priming effects under conditions that should discourage the use of strategies. The resulting priming effects proved extremely unreliable, and reanalyses of three published priming datasets revealed similar cases of low reliability. These results imply that previous attempts to predict semantic priming were unlikely to be successful. However, one study with an unusually large sample size yielded more favorable reliability estimates, suggesting that big data, in terms of items and participants, should be the future for semantic priming research.

Relationship between extensions and intensions in categorization: A match made in heaven?

The present study investigated the relationship between category extension and intension for 11 different semantic categories. It is often tacitly assumed that there is a (strong) extension-intension link. However, a recent study by Hampton and Passanisi (2016) examining the patterns of stable individual differences in concepts across participants called this hypothesis into question. To conceptually replicate their findings, two studies were conducted. We employed a category judgment task to measure category extensions, whereas a property generation (in Study 1) and property judgment task (Study 2) were used to measure intensions. Using their method, that is, correlating extension and intension similarity matrices, we found nonsignificant correlations in both studies, supporting their conclusion that similarity between individuals for extensional judgments does not map onto similarity between individuals for intensional judgments. However, multilevel logistic regression analyses showed that the properties a person generated (Study 1) or endorsed (Study 2) better predicted her own category judgments compared to other people's category judgments. This result provides evidence in favor of a link between extension and intension at the subject level. The conflicting findings, resulting from two different approaches, and their theoretical repercussions are discussed. (PsycINFO Database Record

Feature taxonomy: What type of features do children associate with categories and how do they fare in predicting category judgments?

The present study investigates category intension in school-aged children and adults at two different levels of abstraction (i.e., superordinate and basic level) for two category types (i.e., artefacts and natural kinds). We addressed two critical questions: what kind of features do children and adults generate to define semantic categories and which features predict category membership judgment best at each abstraction level? Overall, participants generated relatively more entity features for natural kinds categories, compared to artefact categories, as well as for basic level categories, compared to superordinate categories. Furthermore, the results showed that older children and adults generated relatively more entity features than younger children. Finally, situation features play the most important role in the prediction of category judgments at both levels of abstraction. Theoretical implications and comparable results from previous studies are described in detail.

Is semantic priming (ir)rational? Insights from the speeded word fragment completion task.

Semantic priming, the phenomenon that a target is recognized faster if it is preceded by a semantically related prime, is a well-established effect. However, the mechanisms producing semantic priming are subject of debate. Several theories assume that the underlying processes are controllable and tuned to prime utility. In contrast, purely automatic processes, like automatic spreading activation, should be independent of the prime's usefulness. The present study sought to disentangle both accounts by creating a situation where prime processing is actually detrimental. Specifically, participants were asked to quickly complete word fragments with either the letter a or e (e.g., sh_ve to be completed as shave). Critical fragments were preceded by a prime that was either related (e.g., push) or unrelated (write) to a prohibited completion of the target (e.g., shove). In 2 experiments, we found a significant inhibitory priming effect, which is inconsistent with purely "rational" explanations of semantic priming. (PsycINFO Database Record

Uncovering underlying processes of semantic priming by correlating item-level effects.

The current study examines the underlying processes of semantic priming using the largest priming database available (i.e., Semantic Priming Project, Hutchison et al. Behavior Research Methods, 45(4), 1099-1114, 2013). Specifically, it compares priming effects in two tasks: lexical decision and pronunciation. Task similarities were assessed at two different stimulus onset asynchronies (SOAs) (i.e., 200 and 1,200 ms) and for both primary and other associates. To evaluate how consistent priming is across these two tasks, item-level priming effects obtained in each task were correlated for each condition separately. The results revealed significant correlations at the short SOA for both primary and other associates. The correlations at the long SOA were significantly smaller and only reached significance when z-transformed response times were used. Furthermore, this pattern remained essentially the same when only asymmetric forward associates (e.g., panda-bear) were considered, suggesting that the cross-task stability at the short SOA was not merely caused by retrospective processes such as semantic matching. Instead, these findings provide evidence for a rapidly operating, item-based, relational characteristic such as spreading activation.

Filling the gaps: A speeded word fragment completion megastudy.

In the speeded word fragment completion task, participants have to complete fragments such as tom_to as quickly and accurately as possible. Previous work has shown that this paradigm can successfully capture subtle priming effects (Heyman, De Deyne, Hutchison, & Storms Behavior Research Methods, 47, 580-606, 2015). In addition, it has several advantages over the widely used lexical decision task. That is, the speeded word fragment completion task is more efficient, more engaging, and easier. Given its potential, we conducted a study to gather speeded word fragment completion norms. The goal of this megastudy was twofold. On the one hand, it provides a rich database of over 8,000 stimuli, which can, for instance, be used in future research to equate stimuli on baseline response times. On the other hand, the aim was to gain insight into the underlying processes of the speeded word fragment completion task. To this end, item-level regression and mixed-effects analyses were performed on the response latencies using 23 predictor variables. Since all items were selected from the Dutch Lexicon Project (Keuleers, Diependaele, & Brysbaert Frontiers in Psychology, 1, 174, 2010), we ran the same analyses on lexical decision latencies to compare the two tasks. Overall, the results revealed many similarities, but also some remarkable differences, which are discussed. We propose that both tasks are complementary when examining visual word recognition. The article ends with a discussion of potential process models of the speeded word fragment completion task.

Using the speeded word fragment completion task to examine semantic priming.

The present research investigates semantic priming with an adapted version of the word fragment completion task. In this task, which we refer to as the speeded word fragment completion task, participants need to complete words such as lett_ce (lettuce), from which one letter was omitted, as quickly as possible. This paradigm has some interesting qualities in comparison with the traditionally used lexical decision task. That is, it requires no pseudowords, it is more engaging for participants, and most importantly, it allows for a more fine-grained investigation of semantic activation. In two studies, we found that words were completed faster when the preceding trial comprised a semantically related fragment such as tom_to (tomato) than when it comprised an unrelated fragment such as guit_r (guitar). A third experiment involved a lexical decision task, to compare both paradigms. The results showed that the magnitude of the priming effect was similar, but item-level priming effects were inconsistent over tasks. Crucially, the speeded word fragment completion task obtained strong priming effects for highly frequent, central words, such as work, money, and warm, whereas the lexical decision task did not. In a final experiment featuring only short, highly frequent words, the lexical decision task failed to find a priming effect, whereas the fragment completion task did obtain a robust effect. Taken together, these results suggest that the speeded word fragment completion task may prove a viable alternative for examining semantic priming.

The influence of working memory load on semantic priming.

The present research examines the nature of the different processes that have been proposed to underlie semantic priming. Specifically, it has been argued that priming arises as a result of automatic target activation and/or the use of strategies like prospective expectancy generation and retrospective semantic matching. This article investigates the extent that these processes rely on cognitive resources by experimentally manipulating working memory load. To disentangle prospective and retrospective processes, prime-target pairs were selected such that they were symmetrically associated (e.g., answer-question; SYM) or asymmetrically associated in either the forward direction (e.g., panda-bear; FA) or the backward direction (e.g., ball-catch; BA). The results showed that priming for FA pairs completely evaporated under a high working memory load but that it remained stable for BA and SYM pairs. This was taken to mean that prospective processes, which are assumed to cause FA priming, require cognitive resources, whereas retrospective processes, which lead to BA priming, are relatively effortless.

Some Differences in Some: Examining Variability in the Interpretation of Scalars Using Latent Class Analysis.

The present study investigated people's understanding of underinformative sentences like 'Some oaks are trees'. Specifically, the scalar term 'some' can be interpreted pragmatically, Not all oaks are trees, or logically, some and possibly all oaks are trees. The aim of this study was to capture the interindividual variability in the interpretation of such sentences. In two experiments, participants provided truth value judgments for 20 underinformative sentences on which a latent class analysis was performed. The results revealed three latent classes: a consistent pragmatic group, a consistent logical group and an inconsistent group. Furthermore, we examined whether this interindividual variability could be explained by text characteristics, response times, cognitive abilities and personality traits. The results showed that only participants' response times to the underinformative sentences could predict class membership. Specifically, the slower participants responded, the more likely they were to interpret underinformative sentences consistently pragmatic or inconsistent instead of consistently logical.