Mental simulation of colour properties during language comprehension: influence of context and comprehension stages.

Many studies have shown that mental simulation may occur during language comprehension. Supporting evidence is derived from the matching effects in the sentence-picture verification (SPV) task often used to assess mental simulations of object properties, such as size, orientation, and shape. However, mixed results have been obtained regarding object colour, with researchers reporting matching or mismatching effects. This study investigated the impact of colour information clarity within sentences on the process of mental simulation during language comprehension. Employing the SPV task and using novel objects, we examined whether there is a mental simulation of colour after excluding typical/atypical colour bias and how varying levels of colour information clarity in sentences influence the emergence of matching effects at different stages of comprehension. To address these issues, we conducted two experiments. In Experiment 1, the participants read normal sentences and subsequently engaged in picture verification with a novel object after a 500 ms delay. In Experiment 2, the participants encountered sentences containing both clear and unclear colour information and, after either a 0 ms or 1500 ms interval, completed picture verification tasks with a novel object. Null effects were found in the 500 ms condition for normal sentences and the 0 ms condition for unclear colour information sentences. A mismatching effect appeared in the 0 ms condition after clear colour information sentences, and a matching effect appeared in the 1500 ms condition for all sentences. The results indicated that after excluding colour bias, the participants still formed mental simulations of colour during language comprehension. Our results also indicated that ongoing colour simulation with time pressure impacted the participant responses. The participants ignored unclear colour information under time pressure, but without time pressure, they constructed simulations that were as detailed as possible, regardless of whether the implicit colour information in the sentence was clear.

Automatic mental simulation in native and non-native speakers.

Pictures of objects are verified faster when they match the implied orientation, shape, and color in a sentence-picture verification task, suggesting that people mentally simulate these features during language comprehension. Previous studies had an unintended correlation between match status and the required response, which may have influenced participants' responses by eliciting strategic use of this correlation. We removed this correlation by including color-matching filler trials and investigated if the color-match effect was still obtained. In both a native sample (Experiment 1) and a non-native sample (Experiment 2), we found strong evidence for a color-match advantage on median reaction time and error rates. Our results are consistent with the view that color is automatically simulated during language comprehension as predicted by the grounded cognition framework.

Task Demands and Sentence Reading Comprehension among Healthy Older Adults: The Complementary Roles of Cognitive Reserve and Working Memory.

Ageing entails different functional brain changes. Education, reading experience, and leisure activities, among others, might contribute to the maintenance of cognitive performance among older adults and are conceptualised as proxies for cognitive reserve. However, ageing also conveys a depletion of working memory capacity, which adversely impacts language comprehension. This study investigated how cognitive reserve proxies and working memory jointly predict the performance of healthy older adults in a sentence reading comprehension task, and how their predictive value changes depending on sentence structure and task demands. Cognitively healthy older adults (n = 120) completed a sentence-picture verification task under two conditions: concurrent viewing of the sentence and picture or their sequential presentation, thereby imposing greater demands on working memory. They also completed a questionnaire on cognitive reserve proxies as well as a verbal working memory test. The sentence structure was manipulated by altering the canonical word order and modifying the amount of propositional information. While the cognitive reserve was the main predictor in the concurrent condition, the predictive role of working memory increased under the sequential presentation, particularly for complex sentences. These findings highlight the complementary roles played by cognitive reserve and working memory in the reading comprehension of older adults.

Development is in the details: Event-related theta oscillations reveal children and adults verify multiplication facts differently.

When verifying the correctness of single-digit multiplication problems, children and adults show a robust ERP correctness effect thought to reflect similar cognitive processes across groups. Recent studies suggest that this effect is instead a modulation of the negative-going N400 component in children, reflecting access to semantic memory, and the positive-going P300 component in adults, reflecting stimulus categorization. However, the relative difference in ERP amplitude is the same for both components, more positive for correct than incorrect solutions, presenting a challenge to ascertaining the appropriate interpretation. Time-frequency analysis (TFA) of the N400/P300 window provides an objective approach to dissociating these effects. TFA measured from solution onset during single-digit multiplication verification revealed significant modulations of event-related as theta power (3-6 Hz) in both groups. Correct trials elicit less power in children (9-12 years) and more power in adults relative to incorrect trials. These findings are consistent with modulations of the N400 and P300, respectively, where opposite effects were predicted for spectral power. The ERP results further support a reinterpretation of the multiplication correctness effect. In contrast, TFA of the N400 effect elicited to a word-picture verification task revealed the same event-related theta effect in both groups, with increased power for mismatched than matched pictures. Together, these findings provide evidence for a developmental shift in cognitive processing specific to the multiplication task. Models of arithmetic should account for this overlooked difference in cognitive processing between children and adults.

Using word-picture verification to inform language impairment locus in chronic post-stroke aphasia.

Word-picture verification, a task that requires a yes/no response to whether a word and a picture match, has been used for both receptive and expressive language; however, there is limited systematic investigation on the linguistic subprocesses targeted by the task. Verification may help to identify linguistic strengths and weaknesses to ultimately provide more targeted, individualized lexical retrieval intervention. The current study assessed the association of semantic and phonological skills with verification performance to demonstrate early efficacy of the paradigm as an aphasia assessment. Sixteen adults with chronic post-stroke aphasia completed a battery of language assessments in addition to reading and auditory verification tasks. Verification scores were positively correlated with auditory and reading comprehension. Accuracy of semantic and phonological verification were positively correlated with accuracy on respective receptive language tasks. More semantic errors were made during verification than naming. The relationship of phonological errors between naming and verification varied by modality (reading or listening). Semantic and phonological performance significantly predicted verification response accuracy and latency. In sum, we propose that verification tasks are particularly useful because they inform semantics pre-lemma selection and phonological decoding, helping to localize individual linguistic strengths and weaknesses, especially in the presence of significant motor speech impairment that can obscure expressive language abilities.

Does Object Size Matter With Regard to the Mental Simulation of Object Orientation?

Language comprehenders have been arguing to mentally represent the implied orientation of objects. However, compared to the effects of shape, size, and color, the effect of orientation is rather small. We examined a potential explanation for the relatively low magnitude of the orientation effect: Object size moderates the orientation effect. Theoretical considerations led us to predict a smaller orientation effect for small objects than for large objects in a sentence-picture verification task. We furthermore investigated whether this pattern generalizes across languages (Chinese, Dutch, and English) and tasks (picture-naming task). The results of the verification task show an orientation effect overall, which is not moderated by object size (contrary to our hypothesis) and language (consistent with our hypothesis). Meanwhile, the preregistered picture-picture verification task showed the predicted interaction between object size and orientation effect. We conducted exploratory analyses to address additional questions.

The influence of state change on object representations in language comprehension.

To understand language people form mental representations of described situations. Linguistic cues are known to influence these representations. In the present study, participants were asked to verify whether the object presented in a picture was mentioned in the preceding words. Crucially, the picture either showed an intact original state or a modified state of an object. Our results showed that the end state of the target object influenced verification responses. When no linguistic context was provided, participants responded faster to the original state of the object compared to the changed state (Experiment 1). However, when linguistic context was provided, participants responded faster to the modified state when it matched, rather than mismatched, the expected outcome of the described event (Experiment 2 and Experiment 3). Interestingly, as for the original state, the match/mismatch effects were only revealed after reading the past tense (Experiment 2) sentences but not the future-tense sentences (Experiment 3). Our findings highlight the need to take account of the dynamics of event representation in language comprehension that captures the interplay between general semantic knowledge about objects and the episodic knowledge introduced by the sentential context.

Are visual processes causally involved in "perceptual simulation" effects in the sentence-picture verification task?

Many studies have shown that sentences implying an object to have a certain shape produce a robust reaction time advantage for shape-matching pictures in the sentence-picture verification task. Typically, this finding has been interpreted as evidence for perceptual simulation, i.e., that access to implicit shape information involves the activation of modality-specific visual processes. It follows from this proposal that disrupting visual processing during sentence comprehension should interfere with perceptual simulation and obliterate the match effect. Here we directly test this hypothesis. Participants listened to sentences while seeing either visual noise that was previously shown to strongly interfere with basic visual processing or a blank screen. Experiments 1 and 2 replicated the match effect but crucially visual noise did not modulate it. When an interference technique was used that targeted high-level semantic processing (Experiment 3) however the match effect vanished. Visual noise specifically targeting high-level visual processes (Experiment 4) only had a minimal effect on the match effect. We conclude that the shape match effect in the sentence-picture verification paradigm is unlikely to rely on perceptual simulation.

Different mechanisms for role relations versus verb-action congruence effects: evidence from ERPs in picture-sentence verification.

Extant accounts of visually situated language processing do make general predictions about visual context effects on incremental sentence comprehension; these, however, are not sufficiently detailed to accommodate potentially different visual context effects (such as a scene-sentence mismatch based on actions versus thematic role relations, e.g., (Altmann & Kamide, 2007; Knoeferle & Crocker, 2007; Taylor & Zwaan, 2008; Zwaan & Radvansky, 1998)). To provide additional data for theory testing and development, we collected event-related brain potentials (ERPs) as participants read a subject-verb-object sentence (500 ms SOA in Experiment 1 and 300 ms SOA in Experiment 2), and post-sentence verification times indicating whether or not the verb and/or the thematic role relations matched a preceding picture (depicting two participants engaged in an action). Though incrementally processed, these two types of mismatch yielded different ERP effects. Role-relation mismatch effects emerged at the subject noun as anterior negativities to the mismatching noun, preceding action mismatch effects manifest as centro-parietal N400s greater to the mismatching verb, regardless of SOAs. These two types of mismatch manipulations also yielded different effects post-verbally, correlated differently with a participant's mean accuracy, verbal working memory and visual-spatial scores, and differed in their interactions with SOA. Taken together these results clearly implicate more than a single mismatch mechanism for extant accounts of picture-sentence processing to accommodate.