The process of replication target selection in psychology: what to consider?

Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target. The resulting checklist can be used for transparently communicating the rationale for selecting studies for replication.

Linguistic and non-linguistic cues in motion event endpoint description: The selection between English to and towards.

The current study aims to explore the factors that could affect people's description of a motion event endpoint. The study conducted by Liao, Dijkstra, and Zwaan (2021, Language and Cognition, 13[2], 161-190) found that two non-linguistic factors (i.e., the actor's goal and the interlocutor's social status) affect people's choice between two Dutch directional prepositions (i.e., naar and richting) during event description tasks. The current study aims to extend these findings by examining the choice between a similar pair of directional prepositions in English (i.e., to and towards). Moreover, we aim to study whether grammatical aspect (i.e., the English simple present and the English progressive aspect) affects the sensitivity to the two non-linguistic factors and consequently also affects how people describe a motion event endpoint. In Experiment 1, we used the English simple present for all sentence stimuli (e.g., he walks (?) the trash bin). We found a significant effect of Interlocutor (the interlocutor's social status) on preposition choice, but no significant effect of Intention (the actor's goal). In Experiment 2, we replaced the English simple present with the English progressive aspect (e.g., he is walking (?) the trash bin). We found significant main effects of both Interlocutor and Intention on preposition choice. These findings extend those reported in Liao et al. (2021) Language, Cognition and Neuroscience, 35(4), 498-520 in that protagonist intention and interlocutor status were found to indeed affect motion event endpoint description. The current findings furthermore show that grammatical aspect affects people's sensitivity to these factors, thus also affecting how a motion event endpoint is described.

A longitudinal analysis of conspiracy beliefs and Covid-19 health responses.

BACKGROUND: Little is known about how conspiracy beliefs and health responses are interrelated over time during the course of the coronavirus disease 2019 (Covid-19) pandemic. This longitudinal study tested two contrasting, but not mutually exclusive, hypotheses through cross-lagged modeling. First, based on the consequential nature of conspiracy beliefs, we hypothesize that conspiracy beliefs predict an increase in detrimental health responses over time. Second, as people may rationalize their behavior through conspiracy beliefs, we hypothesize that detrimental health responses predict increased conspiracy beliefs over time. METHODS: We measured conspiracy beliefs and several health-related responses (i.e. physical distancing, support for lockdown policy, and the perception of the coronavirus as dangerous) at three phases of the pandemic in the Netherlands (N = 4913): During the first lockdown (Wave 1: April 2020), after the first lockdown (Wave 2: June 2020), and during the second lockdown (Wave 3: December 2020). RESULTS: For physical distancing and perceived danger, the overall cross-lagged effects supported both hypotheses, although the standardized effects were larger for the effects of conspiracy beliefs on these health responses than vice versa. The within-person change results only supported an effect of conspiracy beliefs on these health responses, depending on the phase of the pandemic. Furthermore, an overall cross-lagged effect of conspiracy beliefs on reduced support for lockdown policy emerged from Wave 2 to 3. CONCLUSIONS: The results provide stronger support for the hypothesis that conspiracy beliefs predict health responses over time than for the hypothesis that health responses predict conspiracy beliefs over time.

Conspiracy thinking as situation model construction.

Conspiracy thinking can be viewed as a form of narrative comprehension. We routinely infer actors' goals and plans when someone tells us a story. However, comprehenders extend this practice when considering real-world events, which are not orchestrated by a narrator. Comprehenders routinely favor information that is consistent with their perspective, but conspiracy thinkers likely do this to a greater extent, due to the low levels of cognitive reflection they exhibit. Comprehenders supplement incoming information with background knowledge. Conspiracy thinkers do this as well, but their knowledge base deviates from that of the mainstream, as a result of exposure to large amounts of misinformation.

The use of direct and indirect speech across psychological distance.

The current study investigated how psychological distance affects people's preference for direct and indirect speech in a narrative task. In three experiments, participants were instructed to first watch a video and then retell what happened in the video to an imagined/anticipated listener. We manipulated social distance (Experiment 1), temporal distance (Experiment 2), and spatial distance (Experiment 3) between participants and the listener. We compared the proportions of direct speech in the narrations from psychologically proximal versus distal conditions. Experiments 1 and 2 showed that social and temporal proximity increased the rates of direct speech. Social and temporal distance, conversely, increased the rates of indirect speech. Experiment 3 did not yield a significant difference in the use of direct and indirect speech between spatially proximal and distal conditions. Taken together, our results indicate that different psychological dimensions might have discrepant effects on people's choices between direct and indirect speech. Possible explanations for the discrepancy among different psychological distance dimensions are discussed.

A pre-registered, multi-lab non-replication of the action-sentence compatibility effect (ACE).

The Action-sentence Compatibility Effect (ACE) is a well-known demonstration of the role of motor activity in the comprehension of language. Participants are asked to make sensibility judgments on sentences by producing movements toward the body or away from the body. The ACE is the finding that movements are faster when the direction of the movement (e.g., toward) matches the direction of the action in the to-be-judged sentence (e.g., Art gave you the pen describes action toward you). We report on a pre-registered, multi-lab replication of one version of the ACE. The results show that none of the 18 labs involved in the study observed a reliable ACE, and that the meta-analytic estimate of the size of the ACE was essentially zero.

Gathering, processing, and interpreting information about COVID-19.

Does cognitive motivation influence how people gather and interpret information about COVID-19 and their adherence to measures? To address these questions, we conducted a longitudinal survey among European and American respondents. Wave 1 (N = 501) was conducted on March 27, 2020 and Wave 2 (N = 326) on July 1, 2020. We assessed COVID-19 knowledge, endorsement of COVID-19 conspiracy theories, media use, Need for Cognition (NC), Need for Cognitive Closure (NCC), and self-reported adherence to governmental measures taken. Results showed that nearly three-quarters of our respondents actively searched for information about COVID-19. Most at least once a day. Information seeking behaviour was not influenced by cognitive motivation (i.e., NC and NCC). However, cognitive motivation was related to (1) knowledge about COVID-19, (2) conspiracy rejection, and (3) change in knowledge over time. Respondents with more knowledge on COVID-19 also indicated to adhere more often to measures taken by their government. Self-reported adherence to measures was not influenced by cognitive motivation. Implications of these findings will be discussed.

Two Challenges to "Embodied Cognition" Research And How to Overcome Them.

From the time the notion "embodied cognition" has entered the field, researchers have been concerned about its meaning. Does the term refer to a coherent theoretical framework? Despite these concerns, use of the term "embodied cognition" has increased over the years to plateau in recent years. I will argue that the best way forward is not to search for evidence for or against some vague label but rather to systematically, in large-scale projects, address a series of questions that focus on well-defined cognitive tasks. Such projects ought involve preregistration, replication, and open materials, code, and data. For this enterprise to take off, it is important that incentives in the field be aligned with the goal to increase the reliability and validity of our research. There is reason to be optimistic that such an alignment will occur in the near future.

Is color continuously activated in mental simulations across a broader discourse context?

Previous studies have provided contradictory information regarding the activation of perceptual information in a changing discourse context. The current study examines the continued activation of color in mental simulations across one (Experiment 1), two (Experiment 2), and five sentences (Experiment 3), using a sentence-picture verification paradigm. In Experiment 1, the sentence either contained a reference to a color (e.g., a red bicycle) or no reference to a color (e.g., bicycle). In Experiments 2 and 3, either the first or the final sentence contained a reference to a color. Participants responded to pictures either matching the color mentioned in the sentence, or shown in grayscale. The results illustrated that color was activated in mental simulations when the final sentence contained a reference to color. When the target object (e.g., bicycle) was mentioned in all sentences (i.e., in Experiment 2), color remained activated in the mental simulation, even when only the first sentence made a reference to a color. When the focus of the story was shifted elsewhere and the target object was not present across all sentences (i.e., in Experiment 3), color was no longer activated in the mental simulation. These findings suggest that color remains active in mental simulations so long as the target object is present in every sentence. As soon as the focus of the story shifts to another event, this perceptual information is deactivated in the mental simulation. As such, there is no continued activation of color across a broader discourse context.

Object combination in mental simulations.

Studies on the presence of mental simulations during language comprehension have typically focused only on single object properties. This study investigates whether two objects are combined in mental simulations, and whether this is influenced by task instructions. In both experiments, participants read sentences describing animals using a tool in some way. After each sentence, they saw an image of a cartoon animal holding a tool, and they indicated whether the animal (Experiment 1) or the tool (Experiment 2) was mentioned in the previous sentence or not. The shown image completely matched, partially matched, partially mismatched, or completely mismatched the preceding sentence. In total, 90 Dutch psychology students took part in Experiment 1, and 92 students took part in Experiment 2, both experiments were pre-registered. The results suggest that mental simulations indeed combine multiple objects during language comprehension and that this is not influenced by task instructions. Regardless of the instruction type, participants always responded quickest in the complete match condition compared to the partial match condition, suggesting that language comprehension leads to the creation of a complete mental simulation.

Does Object Size Matter With Regard to the Mental Simulation of Object Orientation?

Language comprehenders have been arguing to mentally represent the implied orientation of objects. However, compared to the effects of shape, size, and color, the effect of orientation is rather small. We examined a potential explanation for the relatively low magnitude of the orientation effect: Object size moderates the orientation effect. Theoretical considerations led us to predict a smaller orientation effect for small objects than for large objects in a sentence-picture verification task. We furthermore investigated whether this pattern generalizes across languages (Chinese, Dutch, and English) and tasks (picture-naming task). The results of the verification task show an orientation effect overall, which is not moderated by object size (contrary to our hypothesis) and language (consistent with our hypothesis). Meanwhile, the preregistered picture-picture verification task showed the predicted interaction between object size and orientation effect. We conducted exploratory analyses to address additional questions.

Co-thought gesturing supports more complex problem solving in subjects with lower visual working-memory capacity.

During silent problem solving, hand gestures arise that have no communicative intent. The role of such co-thought gestures in cognition has been understudied in cognitive research as compared to co-speech gestures. We investigated whether gesticulation during silent problem solving supported subsequent performance in a Tower of Hanoi problem-solving task, in relation to visual working-memory capacity and task complexity. Seventy-six participants were assigned to either an instructed gesture condition or a condition that allowed them to gesture, but without explicit instructions to do so. This resulted in three gesture groups: (1) non-gesturing; (2) spontaneous gesturing; (3) instructed gesturing. In line with the embedded/extended cognition perspective on gesture, gesturing benefited complex problem-solving performance for participants with a lower visual working-memory capacity, but not for participants with a lower spatial working-memory capacity.

The influence of state change on object representations in language comprehension.

To understand language people form mental representations of described situations. Linguistic cues are known to influence these representations. In the present study, participants were asked to verify whether the object presented in a picture was mentioned in the preceding words. Crucially, the picture either showed an intact original state or a modified state of an object. Our results showed that the end state of the target object influenced verification responses. When no linguistic context was provided, participants responded faster to the original state of the object compared to the changed state (Experiment 1). However, when linguistic context was provided, participants responded faster to the modified state when it matched, rather than mismatched, the expected outcome of the described event (Experiment 2 and Experiment 3). Interestingly, as for the original state, the match/mismatch effects were only revealed after reading the past tense (Experiment 2) sentences but not the future-tense sentences (Experiment 3). Our findings highlight the need to take account of the dynamics of event representation in language comprehension that captures the interplay between general semantic knowledge about objects and the episodic knowledge introduced by the sentential context.

How are mental simulations updated across sentences?

We examined how grounded mental simulations are updated when there is an implied change of shape, over the course of two (Experiment 1) and four (Experiment 2) sentences. In each preregistered experiment, 84 psychology students completed a sentence-picture verification task in which they judged as quickly and accurately as possible whether the pictured object was mentioned in the previous sentence. Participants had significantly higher accuracy scores and significantly shorter response times when pictures matched the shape implied by the previous sentence than when pictures mismatched the implied shape. These findings suggest that during language comprehension, mental simulations can be actively updated to reflect new incoming information.

Improving social and behavioral science by making replication mainstream: A response to commentaries.

The commentaries on our target article are insightful and constructive. There were some critical notes, but many commentaries agreed with, or even amplified our message. The first section of our response addresses comments pertaining to specific parts of the target article. The second section provides a response to the commentaries' suggestions to make replication mainstream. The final section contains concluding remarks.

Participant Nonnaiveté and the reproducibility of cognitive psychology.

Many argue that there is a reproducibility crisis in psychology. We investigated nine well-known effects from the cognitive psychology literature-three each from the domains of perception/action, memory, and language, respectively-and found that they are highly reproducible. Not only can they be reproduced in online environments, but they also can be reproduced with nonnaïve participants with no reduction of effect size. Apparently, some cognitive tasks are so constraining that they encapsulate behavior from external influences, such as testing situation and prior recent experience with the experiment to yield highly robust effects.

Making replication mainstream.

Many philosophers of science and methodologists have argued that the ability to repeat studies and obtain similar results is an essential component of science. A finding is elevated from single observation to scientific evidence when the procedures that were used to obtain it can be reproduced and the finding itself can be replicated. Recent replication attempts show that some high profile results - most notably in psychology, but in many other disciplines as well - cannot be replicated consistently. These replication attempts have generated a considerable amount of controversy, and the issue of whether direct replications have value has, in particular, proven to be contentious. However, much of this discussion has occurred in published commentaries and social media outlets, resulting in a fragmented discourse. To address the need for an integrative summary, we review various types of replication studies and then discuss the most commonly voiced concerns about direct replication. We provide detailed responses to these concerns and consider different statistical ways to evaluate replications. We conclude there are no theoretical or statistical obstacles to making direct replication a routine aspect of psychological science.

Is color an integral part of a rich mental simulation?

Research suggests that language comprehenders simulate visual features such as shape during language comprehension. In sentence-picture verification tasks, whenever pictures match the shape or orientation implied by the previous sentence, responses are faster than when the pictures mismatch implied visual aspects. However, mixed results have been demonstrated when the sentence-picture paradigm was applied to color (Connell, Cognition, 102(3), 476-485, 2007; Zwaan & Pecher, PLOS ONE, 7(12), e51382, 2012). One of the aims of the current investigation was to resolve this issue. This was accomplished by conceptually replicating the original study on color, using the same paradigm but a different stimulus set. The second goal of this study was to assess how much perceptual information is included in a mental simulation. We examined this by reducing color saturation, a manipulation that does not sacrifice object identifiability. If reduction of one aspect of color does not alter the match effect, it would suggest that not all perceptual information is relevant for a mental simulation. Our results did not support this: We found a match advantage when objects were shown at normal levels of saturation, but this match advantage disappeared when saturation was reduced, yet still aided in object recognition compared to when color was entirely removed. Taken together, these results clearly show a strong match effect for color, and the perceptual richness of mental simulations during language comprehension.

Implied motion language can influence visual spatial memory.

How do language and vision interact? Specifically, what impact can language have on visual processing, especially related to spatial memory? What are typically considered errors in visual processing, such as remembering the location of an object to be farther along its motion trajectory than it actually is, can be explained as perceptual achievements that are driven by our ability to anticipate future events. In two experiments, we tested whether the prior presentation of motion language influences visual spatial memory in ways that afford greater perceptual prediction. Experiment 1 showed that motion language influenced judgments for the spatial memory of an object beyond the known effects of implied motion present in the image itself. Experiment 2 replicated this finding. Our findings support a theory of perception as prediction.

Are gesture and speech mismatches produced by an integrated gesture-speech system? A more dynamically embodied perspective is needed for understanding gesture-related learning.

We observe a tension in the target article as it stresses an integrated gesture-speech system that can nevertheless consist of contradictory representational states, which are reflected by mismatches in gesture and speech or sign. Beyond problems of coherence, this prevents furthering our understanding of gesture-related learning. As a possible antidote, we invite a more dynamically embodied perspective to the stage.