The Impact of Linguistic Prediction Violations on Downstream Recognition Memory and Sentence Recall.

Predicting upcoming words during language comprehension not only affects processing in the moment but also has consequences for memory, although the source of these memory effects (e.g., whether driven by lingering pre-activations, re-analysis following prediction violations, or other mechanisms) remains underspecified. Here, we investigated downstream impacts of prediction on memory in two experiments. First, we recorded EEG as participants read strongly and weakly constraining sentences with expected, unexpected but plausible, or semantically anomalous endings ("He made a holster for his gun / father / train") and were tested on their recognition memory for the sentence endings. Participants showed similar rates of false alarms for predicted but never presented sentence endings whether the prediction violation was plausible or anomalous, suggesting that these arise from pre-activation of the expected words during reading. During sentence reading, especially in strongly constraining sentences, plausible prediction violations elicited an anterior positivity; anomalous endings instead elicited a posterior positivity, whose amplitude was predictive of later memory for those anomalous words. ERP patterns at the time of recognition differentiated plausible and anomalous sentence endings: Words that had been plausible prediction violations elicited enhanced late positive complex amplitudes, suggesting greater episodic recollection, whereas anomalous sentence endings elicited greater N1 amplitudes, suggesting attentional tagging. In a follow-up behavioral study, a separate group of participants read the same sentence stimuli and were tested for sentence-level recall. We found that recall of full sentences was impaired when sentences ended with a prediction violation. Taken together, the results suggest that prediction violations draw attention and affect encoding of the violating word, in a manner that depends on plausibility, and that this, in turn, may impair future memory of the gist of the sentence.

The Brain's Sensitivity to Real-world Statistical Regularity Does Not Require Full Attention.

Predictive coding accounts of perception state that the brain generates perceptual predictions in the service of processing incoming sensory data. These predictions are hypothesized to be afforded by the brain's ability to internalize useful patterns, that is, statistical regularities, from the environment. We have previously argued that the N300 ERP component serves as an index of the brain's use of representations of (real-world) statistical regularities. However, we do not yet know whether overt attention is necessary in order for this process to engage. We addressed this question by presenting stimuli of either high or low real-world statistical regularity in terms of their representativeness (good/bad exemplars of natural scene categories) to participants who either fully attended the stimuli or were distracted by another task (attended/distracted conditions). Replicating past work, N300 responses were larger to bad than to good scene exemplars, and furthermore, we demonstrate minimal impacts of distraction on N300 effects. Thus, it seems that overtly focused attention is not required to maintain the brain's sensitivity to real-world statistical regularity. Furthermore, in an exploratory analysis, we showed that providing additional, artificial regularities, formed by altering the proportions of good and bad exemplars within blocks, further enhanced the N300 effect in both attended and distracted conditions, shedding light on the relationship between statistical regularities learned in the real world and those learned within the context of an experiment.

Multiple mechanisms of visual prediction as revealed by the timecourse of scene-object facilitation.

Not only semantic, but also recently learned arbitrary associations have the potential to facilitate visual processing in everyday life-for example, knowledge of a (moveable) object's location at a specific time may facilitate visual processing of that object. In our prior work, we showed that previewing a scene can facilitate processing of recently associated objects at the level of visual analysis (Smith and Federmeier in Journal of Cognitive Neuroscience, 32(5):783-803, 2020). In the current study, we assess how rapidly this facilitation unfolds by manipulating scene preview duration. We then compare our results to studies using well-learned object-scene associations in a first-pass assessment of whether systems consolidation might speed up high-level visual prediction. In two ERP experiments (N = 60), we had participants study categorically organized novel object-scene pairs in an explicit paired associate learning task. At test, we varied contextual pre-exposure duration, both between (200 vs. 2500 ms) and within subjects (0-2500 ms). We examined the N300, an event-related potential component linked to high-level visual processing of objects and scenes and found that N300 effects of scene congruity increase with longer scene previews, up to approximately 1-2 s. Similar results were obtained for response times and in a separate component-neutral ERP analysis of visual template matching. Our findings contrast with prior evidence that scenes can rapidly facilitate visual processing of commonly associated objects. This raises the possibility that systems consolidation might mediate different kinds of predictive processing with different temporal profiles.

Read carefully, because this is important! How value-driven strategies impact sentence memory.

Little is understood about how people strategically process and remember important but complex information, such as sentences. In the current study, we asked whether people can effectively prioritize memory for sentences as a function of their relative importance (operationalized as a reward point value) and whether they do so, in part, by changing their sentence processing strategies when value information is available in advance. We adapted the value-directed remembering paradigm (Castel, Psychol Learn Motiv 48:225-270, 2007) for sentences that varied in constraint and predictability. Each sentence was associated with a high or low value for subsequent free recall (whole sentence) and recognition (sentence-final words) tests. Value information appeared after or before each sentence as a between-subject manipulation. Regardless of condition, we observed that high-value sentences were recalled more often than low-value sentences, showing that people can strategically prioritize their encoding of sentences. However, memory patterns differed depending on when value information was available. Recall for high-value sentences that ended unexpectedly (and therefore violated one's predictions) was reduced in the Before compared to the After condition. Before condition participants also showed a greater tendency to false alarm to lures (words that were the predicted - but not obtained - ending) from strongly constraining sentences. These observations suggest that when people try to prioritize sentence-level information that they know is valuable, the reading strategies they employ may paradoxically lead to worse memory.

Emotional dissociations in temporal associations: opposing effects of arousal on memory for details surrounding unpleasant events.

Research targeting emotion's impact on relational episodic memory has largely focused on spatial aspects, but less is known about emotion's impact on memory for an event's temporal associations. The present research investigated this topic. Participants viewed a series of interspersed negative and neutral images with instructions to create stories linking successive images. Later, participants performed a surprise memory test, which measured temporal associations between pairs of consecutive pictures where one picture was negative and one was neutral. Analyses focused on how the order of negative and neutral images during encoding influenced retrieval accuracy. Converging results from a discovery study (N = 72) and pre-registered replication study (N = 150) revealed a "forward-favouring" effect of emotion in temporal memory encoding: Participants encoded associations between negative stimuli and subsequent neutral stimuli more strongly than associations between negative stimuli and preceding neutral stimuli. This finding may reflect a novel trade-off regarding emotion's effects on memory and is relevant for understanding affective disorders, as key clinical symptoms can be conceptualised as maladaptive memory retrieval of temporal details.

Older Adults Show Intraindividual Variation in the Use of Predictive Processing.

BACKGROUND: The use of prediction can aid language comprehension through preactivation of relevant word features. However, predictions can be wrong, and it has been proposed that resolving the mismatch between the predicted and presented item requires cognitive resources. Older adults tend not to predict and instead rely more on passive comprehension. Here, we tested, using an intraindividual approach, whether older adults consistently use this less demanding processing strategy while reading or whether they attempt to predict on some trials. METHODS: We used a cross-task conflict paradigm. Younger and older participants self-paced to read sentences that ended with either an expected or unexpected word. Each sentence was then followed by a flanker stimulus that could be congruent or incongruent. We examined responses within and across the two tasks. RESULTS: Unexpected words were in general read as quickly as expected words, indicating that typical processing of these words was similar. However, for both younger and older adults, there was a greater proportion of very slow trials for unexpected words, revealing different processing on a subset of trials. Critically, in older adults, these slowly read unexpected words engaged control, as seen in speeded responses to incongruent flanker stimuli. CONCLUSION: Using a cross-task conflict paradigm, we showed that older adults are able to predict and engage cognitive resources to cope with prediction violations, but do not opt to use these processes consistently.

Representational Pattern Similarity of Electrical Brain Activity Reveals Rapid and Specific Prediction during Language Comprehension.

Predicting upcoming events is a critical function of the brain, and language provides a fertile testing ground for studying prediction, as comprehenders use context to predict features of upcoming words. Many aspects of the mechanisms of prediction remain elusive, partly due to a lack of methodological tools to probe prediction formation in the moment. To elucidate what features are neurally preactivated and when, we used representational similarity analysis on previously collected sentence reading data. We compared EEG activity patterns elicited by expected and unexpected sentence final words to patterns from the preceding words of the sentence, in both strongly and weakly constraining sentences. Pattern similarity with the final word was increased in an early time window following the presentation of the pre-final word, and this increase was modulated by both expectancy and constraint. This was not seen at earlier words, suggesting that predictions were precisely timed. Additionally, pre-final word activity-the predicted representation-had negative similarity with later final word activity, but only for strongly expected words. These findings shed light on the mechanisms of prediction in the brain: rapid preactivation occurs following certain cues, but the predicted features may receive reduced processing upon confirmation.

Neural Signatures of Learning Novel Object-Scene Associations.

Objects are perceived within rich visual contexts, and statistical associations may be exploited to facilitate their rapid recognition. Recent work using natural scene-object associations suggests that scenes can prime the visual form of associated objects, but it remains unknown whether this relies on an extended learning process. We asked participants to learn categorically structured associations between novel objects and scenes in a paired associate memory task while ERPs were recorded. In the test phase, scenes were first presented (2500 msec), followed by objects that matched or mismatched the scene; degree of contextual mismatch was manipulated along visual and categorical dimensions. Matching objects elicited a reduced N300 response, suggesting visuostructural priming based on recently formed associations. Amplitude of an extended positivity (onset ∼200 msec) was sensitive to visual distance between the presented object and the contextually associated target object, most likely indexing visual template matching. Results suggest recent associative memories may be rapidly recruited to facilitate object recognition in a top-down fashion, with clinical implications for populations with impairments in hippocampal-dependent memory and executive function.

Lingering expectations: A pseudo-repetition effect for words previously expected but not presented.

Prediction can help support rapid language processing. However, it is unclear whether prediction has downstream consequences, beyond processing in the moment. In particular, when a prediction is disconfirmed, does it linger, or is it suppressed? This study manipulated whether words were actually seen or were only expected, and probed their fate in memory by presenting the words (again) a few sentences later. If disconfirmed predictions linger, subsequent processing of the previously expected (but never presented) word should be similar to actual word repetition. At initial presentation, electrophysiological signatures of prediction disconfirmation demonstrated that participants had formed expectations. Further downstream, relative to unseen words, repeated words elicited a strong N400 decrease, an enhanced late positive complex (LPC), and late alpha band power decreases. Critically, like repeated words, words previously expected but not presented also attenuated the N400. This "pseudo-repetition effect" suggests that disconfirmed predictions can linger at some stages of processing, and demonstrates that prediction has downstream consequences beyond rapid on-line processing.

Sensory and semantic activations evoked by action attributes of manipulable objects: Evidence from ERPs.

"Two route" theories of object-related action processing posit different temporal activation profiles of grasp-to-move actions (rapidly evoked based on object structure) versus skilled use actions (more slowly activated based on semantic knowledge). We capitalized on the exquisite temporal resolution and multidimensionality of Event-Related Potentials (ERPs) to directly test this hypothesis. Participants viewed manipulable objects (e.g., calculator) preceded by objects sharing either "grasp", "use", or no action attributes (e.g., bar of soap, keyboard, earring, respectively), as well as by action-unrelated but taxonomically-related objects (e.g., abacus); participants judged whether the two objects were related. The results showed more positive responses to "grasp-to-move" primed objects than "skilled use" primed objects or unprimed objects starting in the P1 (0-150 ms) time window and continuing onto the subsequent N1 and P2 components (150-300 ms), suggesting that only "grasp-to-move", but not "skilled use", actions may facilitate visual attention to object attributes. Furthermore, reliably reduced N400s (300-500 ms), an index of semantic processing, were observed to taxonomically primed and "skilled use" primed objects relative to unprimed objects, suggesting that "skilled use" action attributes are a component of distributed, multimodal semantic representations of objects. Together, our findings provide evidence supporting two-route theories by demonstrating that "grasp-to-move" and "skilled use" actions impact different aspects of object processing and highlight the relationship of "skilled use" information to other aspects of semantic memory.

Pace Yourself: Intraindividual Variability in Context Use Revealed by Self-paced Event-related Brain Potentials.

Event-related brain potentials (ERPs) have revealed multiple mechanisms by which contextual constraints impact language processing. At the same time, little work has examined the trial-to-trial dynamics of context use in the brain. In the current study, we probed intraindividual variability in behavioral and neural indices of context processing during reading. In a concurrent self-paced reading and ERP paradigm, participants read sentences that were either strongly or weakly constraining completed with an expected or unexpected target word. Our findings revealed substantial within-subject variability in behavioral and neural responses to contextual constraints. First, context-based amplitude reductions of the N400, a component linked to semantic memory access, were largest among trials eliciting the slowest RTs. Second, the RT distribution of unexpected words in strongly constraining contexts was positively skewed, reflecting an increased proportion of very slow RTs to trials that violated semantic predictions. Among those prediction-violating trials eliciting faster RTs, a late sustained anterior positivity was observed. However, among trials producing the differentially slowed RTs to prediction violations, we observed a markedly earlier effect of constraint in the form of an anterior N2, a component linked to conflict resolution and the cognitive control of behavior. The current study provides the first neurophysiological evidence for the direct role of cognitive control functions in the volitional control of reading. Collectively, our findings suggest that context use varies substantially within individual participants and that coregistering behavioral and neural indices of online sentence processing offers a window into these single-item dynamics.

Evidence for similar patterns of neural activity elicited by picture- and word-based representations of natural scenes.

A long-standing core question in cognitive science is whether different modalities and representation types (pictures, words, sounds, etc.) access a common store of semantic information. Although different input types have been shown to activate a shared network of brain regions, this does not necessitate that there is a common representation, as the neurons in these regions could still differentially process the different modalities. However, multi-voxel pattern analysis can be used to assess whether, e.g., pictures and words evoke a similar pattern of activity, such that the patterns that separate categories in one modality transfer to the other. Prior work using this method has found support for a common code, but has two limitations: they have either only examined disparate categories (e.g. animals vs. tools) that are known to activate different brain regions, raising the possibility that the pattern separation and inferred similarity reflects only large scale differences between the categories or they have been limited to individual object representations. By using natural scene categories, we not only extend the current literature on cross-modal representations beyond objects, but also, because natural scene categories activate a common set of brain regions, we identify a more fine-grained (i.e. higher spatial resolution) common representation. Specifically, we studied picture- and word-based representations of natural scene stimuli from four different categories: beaches, cities, highways, and mountains. Participants passively viewed blocks of either phrases (e.g. "sandy beach") describing scenes or photographs from those same scene categories. To determine whether the phrases and pictures evoke a common code, we asked whether a classifier trained on one stimulus type (e.g. phrase stimuli) would transfer (i.e. cross-decode) to the other stimulus type (e.g. picture stimuli). The analysis revealed cross-decoding in the occipitotemporal, posterior parietal and frontal cortices. This similarity of neural activity patterns across the two input types, for categories that co-activate local brain regions, provides strong evidence of a common semantic code for pictures and words in the brain.

Getting ahead of yourself: Parafoveal word expectancy modulates the N400 during sentence reading.

An important question in the reading literature regards the nature of the semantic information readers can extract from the parafovea (i.e., the next word in a sentence). Recent eye-tracking findings have found a semantic parafoveal preview benefit under many circumstances, and findings from event-related brain potentials (ERPs) also suggest that readers can at least detect semantic anomalies parafoveally (Barber, Van der Meij, & Kutas, Psychophysiology, 50(1), 48-59, 2013). We use ERPs to ask whether fine-grained aspects of semantic expectancy can affect the N400 elicited by a word appearing in the parafovea. In an RSVP-with-flankers paradigm, sentences were presented word by word, flanked 2° bilaterally by the previous and upcoming words. Stimuli consisted of high constraint sentences that were identical up to the target word, which could be expected, unexpected but plausible, or anomalous, as well as low constraint sentences that were always completed with the most expected ending. Findings revealed an N400 effect to the target word when it appeared in the parafovea, which was graded with respect to the target's expectancy and congruency within the sentence context. Furthermore, when targets appeared at central fixation, this graded congruency effect was mitigated, suggesting that the semantic information gleaned from parafoveal vision functionally changes the semantic processing of those words when foveated.

It's All in the Family: Brain Asymmetry and Syntactic Processing of Word Class.

Although left-hemisphere (LH) specialization for language is often viewed as a key example of functional lateralization, there is increasing evidence that the right hemisphere (RH) can also extract meaning from words and sentences. However, the right hemisphere's ability to appreciate syntactic aspects of language remains poorly understood. In the current study, we used separable, functionally well-characterized electrophysiological indices of lexico-semantic and syntactic processes to demonstrate RH sensitivity to syntactic violations among right-handers with a strong manual preference. Critically, however, the nature of this RH sensitivity to structural information was modulated by a genetically determined factor--familial sinistrality. The right hemisphere in right-handers without left-handed family members processed syntactic violations via the words' accompanying lexico-semantic unexpectedness. In contrast, the right hemisphere in right-handers with left-handed family members could process syntactic information in a manner qualitatively similar to that of the left hemisphere.

The association between aerobic fitness and language processing in children: implications for academic achievement.

Event-related brain potentials (ERPs) have been instrumental for discerning the relationship between children's aerobic fitness and aspects of cognition, yet language processing remains unexplored. ERPs linked to the processing of semantic information (the N400) and the analysis of language structure (the P600) were recorded from higher and lower aerobically fit children as they read normal sentences and those containing semantic or syntactic violations. Results revealed that higher fit children exhibited greater N400 amplitude and shorter latency across all sentence types, and a larger P600 effect for syntactic violations. Such findings suggest that higher fitness may be associated with a richer network of words and their meanings, and a greater ability to detect and/or repair syntactic errors. The current findings extend previous ERP research explicating the cognitive benefits associated with greater aerobic fitness in children and may have important implications for learning and academic performance.

Spontaneous revisitation during visual exploration as a link among strategic behavior, learning, and the hippocampus.

Effective exploratory behaviors involve continuous updating of sensory sampling to optimize the efficacy of information gathering. Despite some work on this issue in animals, little information exists regarding the cognitive or neural mechanisms for this sort of behavioral optimization in humans. Here we examined a visual exploration phenomenon that occurred when human subjects studying an array of objects spontaneously looked "backward" in their scanning paths to view recently seen objects again. This "spontaneous revisitation" of recently viewed objects was associated with enhanced hippocampal activity and superior subsequent memory performance in healthy participants, but occurred only rarely in amnesic patients with severe damage to the hippocampus. These findings demonstrate the necessity of the hippocampus not just in the aspects of long-term memory with which it has been associated previously, but also in the short-term adaptive control of behavior. Functional neuroimaging showed hippocampal engagement occurring in conjunction with frontocerebellar circuits, thereby revealing some of the larger brain circuitry essential for the strategic deployment of information-seeking behaviors that optimize learning.

Graded and ungraded expectation patterns: Prediction dynamics during active comprehension.

Language comprehension can be facilitated by the accurate prediction of upcoming words, but prediction effects are not ubiquitous, and comprehenders likely use predictive processing to varying degrees depending on task demands. To ascertain the processing consequences of prioritizing prediction, we here compared ERPs elicited when young adult participants simply read for comprehension with those collected in a subsequent block that required active prediction. We were particularly interested in frontally-distributed post-N400 effects for expected and unexpected words in strongly constraining contexts, which have previously been documented as two distinct patterns: an enhanced positivity ("anterior positivity") observed for prediction violations compared to words that are merely unpredictable (because they occur in weakly constraining sentences) and a distinction between expected endings in more constraining contexts and those same weakly constrained words ("frontal negativity" to the strongly predicted words). We found that the size of the anterior positivity effect was unchanged between passive comprehension and active prediction, suggesting that some processes related to prediction may engage state-like networks. On the other hand, the frontal negativity showed graded patterns from the interaction of task and sentence type. These differing patterns support the hypothesis that there are two separate effects with frontal scalp distributions that occur after the N400 and further suggest that the impact of violating predictions (as long as prediction is engaged at all) is largely stable across varying levels of effort/attention directed toward prediction, whereas other comprehension processes can be modulated by task demands.

Early parafoveal semantic integration in natural reading.

Humans can read and comprehend text rapidly, implying that readers might process multiple words per fixation. However, the extent to which parafoveal words are previewed and integrated into the evolving sentence context remains disputed. We investigated parafoveal processing during natural reading by recording brain activity and eye movements using MEG and an eye tracker while participants silently read one-line sentences. The sentences contained an unpredictable target word that was either congruent or incongruent with the sentence context. To measure parafoveal processing, we flickered the target words at 60 Hz and measured the resulting brain responses (i.e. Rapid Invisible Frequency Tagging, RIFT) during fixations on the pre-target words. Our results revealed a significantly weaker tagging response for target words that were incongruent with the previous context compared to congruent ones, even within 100ms of fixating the word immediately preceding the target. This reduction in the RIFT response was also found to be predictive of individual reading speed. We conclude that semantic information is not only extracted from the parafovea but can also be integrated with the previous context before the word is fixated. This early and extensive parafoveal processing supports the rapid word processing required for natural reading. Our study suggests that theoretical frameworks of natural reading should incorporate the concept of deep parafoveal processing.

Highly memorable images are more readily perceived.

Image memorability, the likelihood that a person will remember a particular image, has been shown to be an intrinsic property of the image that is distinct from many other visual and cognitive features. Research thus far has not identified particular visual features that can sufficiently explain this intrinsic memorability, but one possibility is that more and less memorable images differ in their statistical regularity (i.e., how prototypical or distinctive they are). Statistical regularity is known to affect detection time for images, such that stimuli with higher statistical regularity can be detected with shorter presentation directions. Therefore, in the present study, we probed whether memorability affects how quickly an image can be detected. High- and low-memorability images were presented in an intact/scrambled task wherein participants were asked to indicate whether they saw an intact image or noise, and we estimated the presentation duration necessary for participants to reach 70.7% accuracy. Across two experiments using different stimulus materials, we observed and then replicated that more memorable images are associated with shorter detection thresholds than those for less memorable images. The results support the idea that memorable stimuli may better match stored templates used for image perception and/or recognition. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Animal cognition: Dogs build semantic expectations between spoken words and objects.

A recent study has used scalp-recorded electroencephalography to obtain evidence of semantic processing of human speech and objects by domesticated dogs. The results suggest that dogs do comprehend the meaning of familiar spoken words, in that a word can evoke the mental representation of the object to which it refers.