Modal and amodal cognition: an overarching principle in various domains of psychology.

Accounting for how the human mind represents the internal and external world is a crucial feature of many theories of human cognition. Central to this question is the distinction between modal as opposed to amodal representational formats. It has often been assumed that one but not both of these two types of representations underlie processing in specific domains of cognition (e.g., perception, mental imagery, and language). However, in this paper, we suggest that both formats play a major role in most cognitive domains. We believe that a comprehensive theory of cognition requires a solid understanding of these representational formats and their functional roles within and across different domains of cognition, the developmental trajectory of these representational formats, and their role in dysfunctional behavior. Here we sketch such an overarching perspective that brings together research from diverse subdisciplines of psychology on modal and amodal representational formats so as to unravel their functional principles and their interactions.

Experiential traces first: Does holding a location in visuospatial working memory affect the processing of space-associated words?

This study aimed to systematically examine whether actively maintaining a visual location in working memory can influence the processing of spatially related words. In five experiments, we asked participants to maintain either the location or the shape of a visually presented stimulus in working memory so that it could later be compared with a test stimulus concerning the relevant target features. In between, we presented participants with words that refer to objects typically encountered in the upper or lower vertical space (roof vs. root, respectively). The task participants performed as a response to these words differed between experiments. In Experiments 1-3, participants performed a lexical decision task, in Experiment 4 they performed a semantic task (deciding whether the word refers to an occupation), and in Experiment 5 they performed a spatial task (deciding whether the word refers to something in the upper or lower visual field.) Only in Experiment 5 did we observe an interaction between the position of the visual stimulus held in working memory (up vs. down) and the meaning of the spatial words (associated with up vs. down). Our results therefore suggest that actively maintaining a stimulus location in working memory does not automatically affect the processing of spatially related words, but does so if the relevant spatial dimension is made highly salient by the task. The results are thus in line with studies showing a strong context-dependency of embodiment effects and thus allow the conclusion that language processing proper is not operating on a sensorimotor representational format.

Mental association of time and valence.

Five experiments investigated the association between time and valence. In the first experiment, participants classified temporal expressions (e.g., past, future) and positively or negatively connotated words (e.g., glorious, nasty) based on temporal reference or valence. They responded slower and made more errors in the mismatched condition (positive/past mapped to one hand, negative/future to the other) compared with the matched condition (positive/future to one hand, negative/past to the other hand). Experiment 2 confirmed the generalization of the match effect to nonspatial responses, while Experiment 3 found no reversal of this effect for left-handers. Overall, the results of the three experiments indicate a robust match effect, associating the past with negative valence and the future with positive valence. Experiment 4 involved rating the valence of time-related words, showing higher ratings for future-related words. Additionally, Experiment 5 employed latent semantic analysis and revealed that linguistic experiences are unlikely to be the source of this time-valence association. An interactive activation model offers a quantitative explanation of the match effect, potentially arising from a favorable perception of the future over the past.

Practice makes perfect: Restrained eaters' heightened control for food images.

OBJECTIVE: Restrained eaters (RE) show behaviourally unregulated food intake, which is often explained by a deficit in inhibitory control. Despite evidence for general inhibitory deficits in RE, it remains unclear how the variety of (food) cues in our environment can influence cognitive control. METHOD: In this re-analysis, we explored the inhibitory capacity of RE and unrestrained eaters (URE) on a stop-signal task with modal (pictures) and amodal (word) food and non-food stimuli. RESULTS: Although we did not find the expected inhibitory deficits in RE compared to URE, we found a significant Group × Modality × Stimulus Type interaction. This indicated that RE have relatively good inhibitory control for food, compared to non-food modal cues, and that this relationship is reversed for amodal cues. CONCLUSIONS: Hence, we showed differential processing of information based on food-specificity and presentation format in RE. The format of food cues is thus an important new avenue to understand how the food environment impedes those struggling with regulating their eating behaviour.

The experiential basis of compatibility effects in reading-by-rotating paradigms.

The current study originates from inconsistent findings within the framework of embodied language processing, specifically in the reading-by-rotating literature: whereas some studies report a match advantage (e.g., Zwaan and Taylor (J Exp Psychol 135:1, 2006)), i.e., shorter reading times when the direction of a linguistically conveyed manual rotation matched rather than mismatched the direction of an actually to be performed manual rotation Claus (Acta Psychol 156:104-113, 2015) found a mismatch advantage. The current study addresses two explanations that were previously discussed as potentially responsible for this inconsistency: on the one hand, differences in the knob devices employed; on the other hand, differences in the perspectives adopted by the readers depending on the number of characters involved in the narratives. Concurrently, the study exploits individual differences in motoric experience to explore the experiential basis of action-sentence compatibility effects. The results are inconclusive with respect to the two explanations. However, in their overall picture, they contribute suggestive considerations for the ongoing debate on action-simulation effects by pointing to the potential role of interindividual variation in motoric experience.

Replacing vertical actions by mouse movements: a web-suited paradigm for investigating vertical spatial associations.

The number of web-based studies in experimental psychology has been growing tremendously throughout the last few years. However, a straightforward web-based implementation does not exist for all types of experimental paradigms. In the current paper, we focus on how vertical response movements-which play a crucial role in spatial cognition and language research-can be translated into a web-based setup. Specifically, we introduce a web-suited counterpart of the vertical Stroop task (e.g., Fox & Shor, in Bull Psychon Soc 7:187-189, 1976; Lachmair et al., in Psychon Bull Rev 18:1180-1188, 2011; Thornton et al., in J Exp Psychol Hum Percept Perform 39:964-973, 2013). We employed nouns referring to entities typically located in lower or upper vertical space (e.g., "worm" and "bird", respectively) in Experiments 1 and 2, and emotional valence words associated with a crouched or an upward bodily posture (e.g., "sadness" and "excitement", respectively) in Experiment 3. Depending on the font color, our participants used their mouse to drag the words to the lower or upper screen location. Across all experiments, we consistently observed congruency effects analogous to those obtained with the lab paradigm using actual vertical arm movements. Consequently, we conclude that our web-suited paradigm establishes a reliable approach to examining vertical spatial associations.

Activation levels of plausible alternatives in conversational negation.

Negation is often used to contradict or correct (e.g. There is no dog here.). While rejecting some state of affairs that is presumed to hold for the recipient (e.g. There is a dog here.), the speaker might implicitly suggest a set of plausible alternatives (e.g. There is a wolf instead.). Prior work indicates that alternatives are highly relevant to the comprehension of sentences involving focus: in priming studies, listeners infer plausible alternatives to focused items even when they are not contextually available. So far it is unclear whether negation similarly activates an automatic search for plausible alternatives. The current study was designed to investigate this question, by looking at the activation levels of nouns after negative and affirmative sentences. In a series of priming experiments, subjects were presented with negative and affirmative sentences (e.g. There is an/no apple.), followed by a lexical decision task with targets including plausible alternatives (e.g. pear), as well as semantically related but implausible alternatives (e.g. seed). An interaction of Sentence Polarity and Prime-Target Relation was expected, with negation facilitating responses to plausible alternatives. Results of the first experiment were numerically in line with the hypothesis but the interaction just missed significance level. A post hoc analysis revealed the expected significant interaction. Possible roles of sentential context and goodness of alternatives are discussed. A further experiment confirms that the goodness of alternatives is in fact critical in modulating the effect.

Sentence-based mental simulations: Evidence from behavioral experiments using garden-path sentences.

Language comprehenders activate mental representations of sensorimotor experiences related to the content of utterances they process. However, it is still unclear whether these sensorimotor simulations are driven by associations with words or by a more complex process of meaning composition into larger linguistic expressions, such as sentences. In two experiments, we investigated whether comprehenders indeed create sentence-based simulations. Materials were constructed such that simulation effects could only emerge from sentence meaning and not from word-based associations alone. We additionally asked when during sentence processing these simulations are constructed, using a garden-path paradigm. Participants read either a garden-path sentence (e.g., "As Mary ate the egg was in the fridge") or a corresponding unambiguous control with the same meaning and words (e.g., "The egg was in the fridge as Mary ate"). Participants then judged whether a depicted entity was mentioned in the sentence or not. In both experiments, picture response times were faster when the picture was compatible (vs. incompatible) with the sentence-based interpretation of the target entity (e.g., both for garden-path and control sentence: an unpeeled egg), suggesting that participants created simulations based on the sentence content and only operating over the sentence as a whole.

The limits of automatic sensorimotor processing during word processing: investigations with repeated linguistic experience, memory consolidation during sleep, and rich linguistic learning contexts.

While a number of studies have repeatedly demonstrated an automatic activation of sensorimotor experience during language processing in the form of action-congruency effects, as predicted by theories of grounded cognition, more recent research has not found these effects for words that were just learned from linguistic input alone, without sensorimotor experience with their referents. In the present study, we investigate whether this absence of effects can be attributed to a lack of repeated experience and consolidation of the associations between words and sensorimotor experience in memory. To address these issues, we conducted four experiments in which (1 and 2) participants engaged in two separate learning phases in which they learned novel words from language alone, with an intervening period of memory-consolidating sleep, and (3 and 4) we employed familiar words whose referents speakers have no direct experience with (such as plankton). However, we again did not observe action-congruency effects in subsequent test phases in any of the experiments. This indicates that direct sensorimotor experience with word referents is a necessary requirement for automatic sensorimotor activation during word processing.

Contrary to Expectations: Does Context Influence the Processing Cost Associated with Negation?

Negative sentences are hard to process when they are presented out of context. When embedded in a context of plausible denial their processing difficulty decreases or is completely eliminated. We investigated in six behavioral experiments whether the processing of negation is eased in a denial context triggered by discourse markers (e.g. Contrary to expectations, John has/hasn't eaten the soup). In order to investigate the necessary conditions for a context of plausible denial to reduce the processing cost of negation, we contrasted the processing of affirmative and negative sentences in minimal and extended denial and non-denial contexts (represented by either no context or a different type of context). We expected significantly longer response times (RTs) for negative sentences in comparison with affirmation in non-denial contexts and similar RTs for affirmative and negative sentences in denial contexts. The results from a sensibility judgement task (Experiment 1 and 2) and from a self-paced reading paradigm (Experiment 3 and 4) showed two robust main effects of context and polarity but no significant interaction between the two factors, suggesting that the processing of negative sentences was not facilitated in a context of minimal denial triggered solely by discourse markers. However, when the discourse markers were replaced with the explicit mention of the expectation to be denied and longer narratives used, the processing difficulty was eliminated specifically in the denial contexts (Experiment 5). Furthermore, when the discourse markers were used in longer narratives, a facilitation effect was also found (Experiment 6). All in all, the present findings suggest that, although negative sentences are felicitous in a context of plausible denial, the interplay of pragmatic factors like relevance or informativity is decisive in easing their processing difficulty.

The Processing of Negation and Polarity: An Overview.

Negation is a universal component of human language; polarity sensitivity (i.e., lexical distributional constraints in relation to negation) is arguably so while being pervasive across languages. Negation has long been a field of inquiry in psychological theories and experiments of reasoning, which inspired many follow-up studies of negation and negation-related phenomena in psycholinguistics. In generative theoretical linguistics, negation and polarity sensitivity have been extensively studied, as the related phenomena are situated at the interfaces of syntax, semantics and pragmatics, and are thus extremely revealing about the architecture of grammar. With the now long tradition of research on negation and polarity in psychology and psycholinguistics, and the emerging field of experimental semantics and pragmatics, a multitude of interests and experimental paradigms have emerged which call for re-evaluations and further development and integration. This special issue contains a collection of 16 research articles on the processing of negation and negation-related phenomena including polarity items, questions, conditionals, and irony, using a combination of behavioral (e.g., rating, reading, eye-tracking and sentence completion) and neuroimaging techniques (e.g., EEG). They showcase the processing of negation and polarity with or without context, in various languages and across different populations (adults, typically developing and ADHD children). The integration of multiple theoretical and empirical perspectives in this collection provides new insights, methodological advances and directions for future research.

Negative clauses imply affirmative topics and affirmative antecedents.

We propose that negative clauses are generally interpreted as if the affirmative portion of the clause is under discussion, a likely topic. This predicts a preference for affirmative (topical) antecedents over negative antecedents of a following missing verb phrase (VP). Three experiments tested the predictions of this hypothesis in sentences containing negation in the first clause followed by an ambiguous as-clause as in Don't cross on red as a stupid person would and its counterpart with smart replacing stupid. In Experiment 1 sentences containing an undesirable attribute adjective such as stupid were rated as more natural, and read faster, than their desirable attribute counterparts (smart), with or without a comma preceding as. The second experiment indicated that the interpretation of the missing VP reflected the attribute adjective's desirability, with processing difficulty presumably reflecting reanalysis from the initial affirmative antecedent (cross on red) to include negation when the initial interpretation violated plausibility. A third experiment generalized the effect beyond sentences with an initial contracted don't.

Negation Processing in Children with ADHD: The Generic Problem of Using Negation in Instructions.

Recent studies have suggested that negation comprehension falls back onto inhibitory brain systems that are also crucial for impulse control and other non-linguistic control domains (Beltran et al., 2018, 2019; de Vega et al., 2016; Liu et al., 2020). Against this backdrop, the present pilot study investigated the use of negation within directional instructions (i.e., "not left", "now left", "not right", "now right") in children with ADHD and a control group. The results indicate that children in general have a long response delay following negative compared to affirmative instructions. Additionally, there was a tendency for this effect to be more pronounced in the ADHD group. Together, these results suggest that negation processing might indeed demand inhibitory control processes, which are differently available across different subgroups. Thus, the current study provides evidence that using negation in imperatives or instructions is generally rather critical and should be avoided if possible, but that negation use is probably even more problematic in specific clinical populations. Potential implications of these results will be discussed.

The Role of Predictability During Negation Processing in Truth-Value Judgment Tasks.

In experiments investigating the processing of true and false negative sentences, it is often reported that polarity interacts with truth-value, in the sense that true sentences lead to faster reaction times than false sentences in affirmative conditions whereas the same does not hold for negative sentences. Various reasons for this difference between affirmative and negative sentences have been discussed in the literature (e.g., lexical associations, predictability, ease of comparing sentence and world). In the present study, we excluded lexical associations as a potential influencing factor. Participants saw artificial visual worlds (e.g., a white square and a black circle) and corresponding sentences (i.e., "The square/circle is (not) white"). The results showed a clear effect of truth-value for affirmative sentences (true faster than false) but not for negative sentences. This result implies that the well-known truth-value-by-polarity interaction cannot solely be due to long-term lexical associations. Additional predictability manipulations allowed us to also rule out an explanatory account that attributes the missing truth-value effect for negative sentences to low predictability. We also discuss the viability of an informativeness account.

Green as a cbemcuru: modal as well as amodal color cues can help to solve anagrams.

Embodied cognition theories have been getting much support in recent years from studies showing that multimodal experiential traces are activated during language comprehension. However, there are almost no studies examining this influence in the opposite direction. Here, we investigated the influence of modal (physical color patch) and amodal (color word) cues on anagram solving times. We manipulated the association between the color cue and the solution word's referent color (e.g., finding the solution word "cucumber" for the anagram "cmrbucue" should be facilitated by the word "green" or a green color patch). In a third experiment, both cues were combined: a color word was presented inside a color patch before the anagram appeared. We indeed observed priming effects: anagrams were solved faster when the preceding color patch or color word matched the solution word's referent compared to a mismatching color patch or color word. When combining these cues, a priming effect only was found when both color word and color patch matched the solution word's referent. These results further strengthen the notion that multimodal experiential traces play an important role in language comprehension and expand upon the results of earlier studies on anagram solution tasks.

When words are upside down: Language-space associations in children and adults.

The nonlinguistic sensorimotor experiences and contexts that accompany language learning are often assumed to play an integral role in meaning representation. However, despite embodied models of language comprehension being well established in the literature, evidence is mainly derived from adult studies. For example, it has been shown that for adult comprehenders there is a close link between language and space, resulting in automatic reactivation of a referent's spatial location during word processing. In the current study, we investigated whether this link can also be found in young children (4;8-7;5 years;months). In a Stroop-like paradigm, our participants responded to a colored circle on the screen with an upward or downward arm movement after auditorily perceiving a task-irrelevant noun. We chose noun's with referents that are typically located either in upper or lower space (e.g., "sun" vs. "shoe"). Although it was not necessary to process the meaning of the nouns in order to fulfill the task, we found a spatial compatibility effect that was similar to language-space associations previously reported for adult participants. We conclude that sensorimotor experiences in the spatial domain seem to play a role during meaning processing from early childhood onward, a finding crucial for embodied models of language comprehension.

Do I need to have my hands free to understand hand-related language? Investigating the functional relevance of experiential simulations.

Theories of embodiment state that people mentally simulate the described situations and events during language comprehension. While several studies have provided evidence that these simulations exist, it is still unclear whether they are functionally relevant for comprehension. To investigate this question, we studied the effects of a secondary task on the processing of hand- and foot-related nouns. The secondary task occupied either the hand or the foot system, thereby impeding hand- or foot-related simulations, respectively. Participants performed a lexical decision task by responding to the presented nouns with their left hand or foot, depending on the color of the words, while withholding their response to pseudowords. In half of the experimental blocks, participants performed a simultaneous tapping task with their right hand (Experiment 1) or foot (Experiment 2). If simulations are functionally relevant for comprehension, the secondary task should affect the processing of hand words to a larger degree than the processing foot words in Experiment 1 and vice versa in Experiment 2. In both experiments, hand responses were faster for hand words than foot words, whereas the opposite was true for foot responses. This finding indicates that participants indeed simulated the words' meanings. Importantly, there was no difference between the influence of the hand tapping and the foot tapping task on lexical decision times to hand and foot words, indicating that experiential simulation might just be an optional by-product of language processing.

Reading sentences describing high- or low-pitched auditory events: only pianists show evidence for a horizontal space-pitch association.

This study explored differences between pianists and non-musicians during reading of sentences describing high- or low-pitched auditory events. Based on the embodied model of language comprehension, it was hypothesized that the experience of playing the piano encourages a corresponding association between high-pitched sounds and the right and low-pitched sounds and the left. This pitch-space association is assumed to become elicited during understanding of sentences describing either a high- or low-pitched auditory event. In this study, pianists and non-musicians were tested based on the hypothesis that only pianists show a compatibility effect between implied pitch height and horizontal space, because only pianists have the corresponding experience with the piano keyboard. Participants read pitch-related sentences (e.g., the bear growls deeply, the soprano singer sings an aria) and judged whether the sentence was sensible or not by pressing either a left or right response key. The results indicated that only the pianists showed the predicted compatibility effect between implied pitch height and response location. Based on the results, it can be inferred that the experience of playing the piano led to an association between horizontal space and pitch height in pianists, while no such spatial association was elicited in non-musicians.

The Role of Simple Semantics in the Process of Artificial Grammar Learning.

This study investigated the effect of semantic information on artificial grammar learning (AGL). Recursive grammars of different complexity levels (regular language, mirror language, copy language) were investigated in a series of AGL experiments. In the with-semantics condition, participants acquired semantic information prior to the AGL experiment; in the without-semantics control condition, participants did not receive semantic information. It was hypothesized that semantics would generally facilitate grammar acquisition and that the learning benefit in the with-semantics conditions would increase with increasing grammar complexity. Experiment 1 showed learning effects for all grammars but no performance difference between conditions. Experiment 2 replicated the absence of a semantic benefit for all grammars even though semantic information was more prominent during grammar acquisition as compared to Experiment 1. Thus, we did not find evidence for the idea that semantics facilitates grammar acquisition, which seems to support the view of an independent syntactic processing component.

The Sounds of Sentences: Differentiating the Influence of Physical Sound, Sound Imagery, and Linguistically Implied Sounds on Physical Sound Processing.

Both the imagery literature and grounded models of language comprehension emphasize the tight coupling of high-level cognitive processes, such as forming a mental image of something or language understanding, and low-level sensorimotor processes in the brain. In an electrophysiological study, imagery and language processes were directly compared and the sensory associations of processing linguistically implied sounds or imagined sounds were investigated. Participants read sentences describing auditory events (e.g., "The dog barks"), heard a physical (environmental) sound, or had to imagine such a sound. We examined the influence of the 3 sound conditions (linguistic, physical, imagery) on subsequent physical sound processing. Event-related potential (ERP) difference waveforms indicated that in all 3 conditions, prime compatibility influenced physical sound processing. The earliest compatibility effect was observed in the physical condition, starting in the 80-110 ms time interval with a negative maximum over occipital electrode sites. In contrast, the linguistic and the imagery condition elicited compatibility effects starting in the 180-220 ms time window with a maximum over central electrode sites. In line with the ERPs, the analysis of the oscillatory activity showed that compatibility influenced early theta and alpha band power changes in the physical, but not in the linguistic and imagery, condition. These dissociations were further confirmed by dipole localization results showing a clear separation between the source of the compatibility effect in the physical sound condition (superior temporal area) and the source of the compatibility effect triggered by the linguistically implied sounds or the imagined sounds (inferior temporal area). Implications for grounded models of language understanding are discussed.