Hippocampal-neocortical interactions sharpen over time for predictive actions.

When an action is familiar, we are able to anticipate how it will change the state of the world. These expectations can result from retrieval of action-outcome associations in the hippocampus and the reinstatement of anticipated outcomes in visual cortex. How does this role for the hippocampus in action-based prediction change over time? We use high-resolution fMRI and a dual-training behavioral paradigm to examine how the hippocampus interacts with visual cortex during predictive and nonpredictive actions learned either three days earlier or immediately before the scan. Just-learned associations led to comparable background connectivity between the hippocampus and V1/V2, regardless of whether actions predicted outcomes. However, three-day-old associations led to stronger background connectivity and greater differentiation between neural patterns for predictive vs. nonpredictive actions. Hippocampal prediction may initially reflect indiscriminate binding of co-occurring events, with action information pruning weaker associations and leading to more selective and accurate predictions over time.

Linking pattern completion in the hippocampus to predictive coding in visual cortex.

Models of predictive coding frame perception as a generative process in which expectations constrain sensory representations. These models account for expectations about how a stimulus will move or change from moment to moment, but do not address expectations about what other, distinct stimuli are likely to appear based on prior experience. We show that such memory-based expectations in human visual cortex are related to the hippocampal mechanism of pattern completion.

Action-Based Learning of Multistate Objects in the Medial Temporal Lobe.

Actions constrain perception by changing the appearance of objects in the environment. As such, they provide an interactive basis for learning the structure of visual input. If an action systematically transforms one stimulus into another, then these stimuli are more likely to reflect different states of the same persisting object over time. Here we show that such multistate objects are represented in the human medial temporal lobe--the result of a mechanism in which actions influence associative learning of how objects transition between states. We further demonstrate that greater recruitment of these action-based representations during object perception is accompanied by attenuated activity in stimulus-selective visual cortex. In this way, our interactions with the environment help build visual knowledge that predictively facilitates perceptual processing.

Competition between Mutually Exclusive Object States in Event Comprehension.

Successful language comprehension requires one to correctly match symbols in an utterance to referents in the world, but the rampant ambiguity present in that mapping poses a challenge. Sometimes the ambiguity lies in which of two (or more) types of things in the world are under discussion (i.e., lexical ambiguity); however, even a word with a single sense can have an ambiguous referent. This ambiguity occurs when an object can exist in multiple states. Here, we consider two cases in which the presence of multiple object states may render a single-sense word ambiguous. In the first case, one must disambiguate between two states of a single object token in a short discourse. In the second case, the discourse establishes two different tokens of the object category. Both cases involve multiple object states: These states are mutually exclusive in the first case, whereas in the second case, these states can logically exist at the same time. We use fMRI to contrast same-token and different-token discourses, using responses in left posterior ventrolateral prefrontal cortex (pVLPFC) as an indicator of conflict. Because the left pVLPFC is sensitive to competition between multiple, incompatible representations, we predicted that state ambiguity should engender conflict only when those states are mutually exclusive. Indeed, we find evidence of conflict in same-token, but not different-token, discourses. Our data support a theory of left pVLPFC function in which general conflict resolution mechanisms are engaged to select between multiple incompatible representations that arise in many kinds of ambiguity present in language.

A cortical network for the encoding of object change.

Understanding events often requires recognizing unique stimuli as alternative, mutually exclusive states of the same persisting object. Using fMRI, we examined the neural mechanisms underlying the representation of object states and object-state changes. We found that subjective ratings of visual dissimilarity between a depicted object and an unseen alternative state of that object predicted the corresponding multivoxel pattern dissimilarity in early visual cortex during an imagery task, while late visual cortex patterns tracked dissimilarity among distinct objects. Early visual cortex pattern dissimilarity for object states in turn predicted the level of activation in an area of left posterior ventrolateral prefrontal cortex (pVLPFC) most responsive to conflict in a separate Stroop color-word interference task, and an area of left ventral posterior parietal cortex (vPPC) implicated in the relational binding of semantic features. We suggest that when visualizing object states, representational content instantiated across early and late visual cortex is modulated by processes in left pVLPFC and left vPPC that support selection and binding, and ultimately event comprehension.

The effect of object state-changes on event processing: do objects compete with themselves?

When an object is described as changing state during an event, do the representations of those states compete? The distinct states they represent cannot coexist at any one moment in time, yet each representation must be retrievable at the cost of suppressing the other possible object states. We used functional magnetic resonance imaging of human participants to test whether such competition does occur, and whether this competition between object states recruits brain areas sensitive to other forms of conflict. In Experiment 1, the same object was changed either substantially or minimally by one of two actions. In Experiment 2, the same action either substantially or minimally changed one of two objects. On a subject-specific basis, we identified voxels most responsive to conflict in a Stroop color-word interference task. Voxels in left posterior ventrolateral prefrontal cortex most responsive to Stroop conflict were also responsive to our object state-change manipulation, and were not responsive to the imageability of the described action. In contrast, voxels in left middle frontal gyrus responsive to Stroop conflict were not responsive even to language, and voxels in left middle temporal gyrus that were responsive to language and imageability were not responsive to object state-change. Results suggest that, when representing object state-change, multiple incompatible representations of an object compete, and the greater the difference between the initial state and the end state of an object, the greater the conflict.

Computer-mouse tracking reveals TMS disruptions of prefrontal function during semantic retrieval.

Converging evidence from neuroimaging and neuropsychological studies is essential for understanding human frontal cortical function. We introduce a new method for studying the effects of transient disruptions of frontal activity during transcranial magnetic stimulation (TMS). Using a novel combination of TMS and computer-mouse tracking, through two experiments we tested process models of semantic competition in left ventrolateral prefrontal cortex (VLPFC). On TMS stimulation of left mid-VLPFC just after presentation of an ambiguous stimulus, participants' mouse-movement trajectories deviated more toward the incorrect target for weak associate trials than for any other trial type. This effect was extinguished when participants were simultaneously shown both target and cue stimuli. Results suggest that left mid-VLPFC is necessary to resolve semantic competition when a response is underdetermined by the stimulus and the interpretive context of the stimulus is ambiguous. Computer-mouse movements reveal the dynamics of competitive interactions as they resolve, making this technique ideally suited for studying cognitive control processes and a more sensitive index of TMS disruption than reaction time and accuracy alone.

Tracking the continuity of language comprehension: computer mouse trajectories suggest parallel syntactic processing.

Although several theories of online syntactic processing assume the parallel activation of multiple syntactic representations, evidence supporting simultaneous activation has been inconclusive. Here, the continuous and non-ballistic properties of computer mouse movements are exploited, by recording their streaming x, y coordinates to procure evidence regarding parallel versus serial processing. Participants heard structurally ambiguous sentences while viewing scenes with properties either supporting or not supporting the difficult modifier interpretation. The curvatures of the elicited trajectories revealed both an effect of visual context and graded competition between simultaneously active syntactic representations. The results are discussed in the context of 3 major groups of theories within the domain of sentence processing.