What Happened When? Cerebral Processing of Modified Structure and Content in Episodic Cueing.

Episodic memories are not static but can change on the basis of new experiences, potentially allowing us to make valid predictions in the face of an ever-changing environment. Recent research has identified prediction errors during memory retrieval as a possible trigger for such changes. In this study, we used modified episodic cues to investigate whether different types of mnemonic prediction errors modulate brain activity and subsequent memory performance. Participants encoded episodes that consisted of short toy stories. During a subsequent fMRI session, participants were presented videos showing the original episodes, or slightly modified versions thereof. In modified videos, either the order of two subsequent action steps was changed or an object was exchanged for another. Content modifications recruited parietal, temporo-occipital, and parahippocampal areas reflecting the processing of the new object information. In contrast, structure modifications elicited activation in right dorsal premotor, posterior temporal, and parietal areas, reflecting the processing of new sequence information. In a post-fMRI memory test, the participants' tendency to accept modified episodes as originally encoded increased significantly when they had been presented modified versions already during the fMRI session. After experiencing modifications, especially those of the episodes' structure, the recognition of originally encoded episodes was impaired as well. Our study sheds light onto the neural processing of different types of episodic prediction errors and their influence on subsequent memory recall.

Corrigendum: Seeing What I Did (Not): Cerebral and Behavioral Effects of Agency and Perspective on Episodic Memory Re-activation.

[This corrects the article DOI: 10.3389/fnbeh.2021.793115.].

Seeing What I Did (Not): Cerebral and Behavioral Effects of Agency and Perspective on Episodic Memory Re-activation.

Intuitively, we assume that we remember episodes better when we actively participated in them and were not mere observers. Independently of this, we can recall episodes from either the first-person perspective (1pp) or the third-person perspective (3pp). In this functional magnetic resonance imaging (fMRI) study, we tested whether agency and perspective modulate neural activity during memory retrieval and subsequently enhance memory performance. Subjects encoded a set of different episodes by either imitating or only observing videos that showed short toy stories. A week later, we conducted fMRI and cued episodic retrieval by presenting the original videos, or slightly modified versions thereof, from 1pp or from 3pp. The hippocampal formation was sensitive to self-performed vs. only observed actions only when there was an episodic mismatch. In a post-fMRI memory test a history of self-performance did not improve behavioral memory performance. However, modified videos were often (falsely) accepted as showing truly experienced episodes when: (i) they were already presented in this modified version during fMRI or (ii) they were presented in their original form during fMRI but from 3pp. While the overall effect of modification was strong, the effects of perspective and agency were more subtle. Together, our findings demonstrate that self-performance and self-perspective modulate the strength of a memory trace in different ways. Even when memory performance remains the same for different agentive states, the brain is capable of detecting mismatching information. Re-experiencing the latter impairs memory performance as well as retrieving encoded episodes from 3pp.

Using enriched semantic event chains to model human action prediction based on (minimal) spatial information.

Predicting other people's upcoming action is key to successful social interactions. Previous studies have started to disentangle the various sources of information that action observers exploit, including objects, movements, contextual cues and features regarding the acting person's identity. We here focus on the role of static and dynamic inter-object spatial relations that change during an action. We designed a virtual reality setup and tested recognition speed for ten different manipulation actions. Importantly, all objects had been abstracted by emulating them with cubes such that participants could not infer an action using object information. Instead, participants had to rely only on the limited information that comes from the changes in the spatial relations between the cubes. In spite of these constraints, participants were able to predict actions in, on average, less than 64% of the action's duration. Furthermore, we employed a computational model, the so-called enriched Semantic Event Chain (eSEC), which incorporates the information of different types of spatial relations: (a) objects' touching/untouching, (b) static spatial relations between objects and (c) dynamic spatial relations between objects during an action. Assuming the eSEC as an underlying model, we show, using information theoretical analysis, that humans mostly rely on a mixed-cue strategy when predicting actions. Machine-based action prediction is able to produce faster decisions based on individual cues. We argue that human strategy, though slower, may be particularly beneficial for prediction of natural and more complex actions with more variable or partial sources of information. Our findings contribute to the understanding of how individuals afford inferring observed actions' goals even before full goal accomplishment, and may open new avenues for building robots for conflict-free human-robot cooperation.

Impaired context-sensitive adjustment of behaviour in Parkinson's disease patients tested on and off medication: An fMRI study.

The brain's sensitivity to and accentuation of unpredicted over predicted sensory signals plays a fundamental role in learning. According to recent theoretical models of the predictive coding framework, dopamine is responsible for balancing the interplay between bottom-up input and top-down predictions by controlling the precision of surprise signals that guide learning. Using functional MRI, we investigated whether patients with Parkinson's disease (PD) show impaired learning from prediction errors requiring either adaptation or stabilisation of current predictions. Moreover, we were interested in whether deficits in learning over a specific time scale would be accompanied by altered surprise responses in dopamine-related brain structures. To this end, twenty-one PD patients tested on and off dopaminergic medication and twenty-one healthy controls performed a digit prediction paradigm. During the task, violations of sequence-based predictions either signalled the need to update or to stabilise the current prediction and, thus, to react to them or ignore them, respectively. To investigate contextual adaptation to prediction errors, the probability (or its inverse, surprise) of the violations fluctuated across the experiment. When the probability of prediction errors over a specific time scale increased, healthy controls but not PD patients off medication became more flexible, i.e., error rates at violations requiring a motor response decreased in controls but increased in patients. On the neural level, this learning deficit in patients was accompanied by reduced signalling in the substantia nigra and the caudate nucleus. In contrast, differences between the groups regarding the probabilistic modulation of behaviour and neural responses were much less pronounced at prediction errors requiring only stabilisation but no adaptation. Interestingly, dopaminergic medication could neither improve learning from prediction errors nor restore the physiological, neurotypical pattern. Our findings point to a pivotal role of dysfunctions of the substantia nigra and caudate nucleus in deficits in learning from flexibility-demanding prediction errors in PD. Moreover, the data witness poor effects of dopaminergic medication on learning in PD.

Predictive Impact of Contextual Objects during Action Observation: Evidence from Functional Magnetic Resonance Imaging.

The processing of congruent stimuli, such as an object or action in its typical location, is usually associated with reduced neural activity, probably due to facilitated recognition. However, in some situations, congruency increases neural activity-for example, when objects next to observed actions are likely versus unlikely to be involved in forthcoming action steps. Here, we investigated using fMRI whether the processing of contextual cues during action perception is driven by their (in)congruency and, thus, informative value to make sense of an observed scene. Specifically, we tested whether both highly congruent contextual objects (COs), which strongly indicate a future action step, and highly incongruent COs, which require updating predictions about possible forthcoming action steps, provide more anticipatory information about the action course than moderately congruent COs. In line with our hypothesis that especially the inferior frontal gyrus (IFG) subserves the integration of the additional information into the predictive model of the action, we found highly congruent and incongruent COs to increase bilateral activity in action observation nodes, that is, the IFG, the occipitotemporal cortex, and the intraparietal sulcus. Intriguingly, BA 47 was significantly stronger engaged for incongruent COs reflecting the updating of prediction in response to conflicting information. Our findings imply that the IFG reflects the informative impact of COs on observed actions by using contextual information to supply and update the currently operating predictive model. In the case of an incongruent CO, this model has to be reconsidered and extended toward a new overarching action goal.

Association of grey matter changes with stability and flexibility of prediction in akinetic-rigid Parkinson's disease.

Parkinson's disease (PD), which is caused by degeneration of dopaminergic neurons in the midbrain, results in a heterogeneous clinical picture including cognitive decline. Since the phasic signal of dopamine neurons is proposed to guide learning by signifying mismatches between subjects' expectations and external events, we here investigated whether akinetic-rigid PD patients without mild cognitive impairment exhibit difficulties in dealing with either relevant (requiring flexibility) or irrelevant (requiring stability) prediction errors. Following our previous study on flexibility and stability in prediction (Trempler et al. J Cogn Neurosci 29(2):298-309, 2017), we then assessed whether deficits would correspond with specific structural alterations in dopaminergic regions as well as in inferior frontal cortex, medial prefrontal cortex, and the hippocampus. Twenty-one healthy controls and twenty-one akinetic-rigid PD patients on and off medication performed a task which required to serially predict upcoming items. Switches between predictable sequences had to be indicated via button press, whereas sequence omissions had to be ignored. Independent of the disease, midbrain volume was related to a general response bias to unexpected events, whereas right putamen volume correlated with the ability to discriminate between relevant and irrelevant prediction errors. However, patients compared with healthy participants showed deficits in stabilisation against irrelevant prediction errors, associated with thickness of right inferior frontal gyrus and left medial prefrontal cortex. Flexible updating due to relevant prediction errors was also affected in patients compared with controls and associated with right hippocampus volume. Dopaminergic medication influenced behavioural performance across, but not within the patients. Our exploratory study warrants further research on deficient prediction error processing and its structural correlates as a core of cognitive symptoms occurring already in early stages of the disease.

Making sense of objects lying around: How contextual objects shape brain activity during action observation.

Action recognition involves not only the readout of body movements and involved objects but also the integration of contextual information, e.g. the environment in which an action takes place. Notably, inferring superordinate goals and generating predictions about forthcoming action steps should benefit from screening the actor's immediate environment, in particular objects located in the actor's peripersonal space and thus potentially used in following action steps. Critically, if such contextual objects (COs) afford actions that are semantically related to the observed action, they may trigger or facilitate the inference of goals and the prediction of following actions. This fMRI study investigated the neural mechanisms underlying the integration of COs in semantic and spatial relation to observed actions. Specifically, we tested the hypothesis that the inferior frontal gyrus (IFG) subserves this integration. Participants observed action videos in which COs and observed actions had common overarching goals or not (goal affinity) and varied in their location relative to the actor. High goal affinity increased bilateral activity in action observation network nodes, i.e. the occipitotemporal cortex and the intraparietal sulcus, but also in the precuneus and middle frontal gyri. This finding suggests that the semantic relation between COs and actions is considered during action observation and triggers (rather than facilitates) processes beyond those usually involved in action observation. Moreover, COs with high goal affinity located close to the actor's dominant hand additionally engaged bilateral IFG, corroborating the view that IFG is critically involved in the integration of action steps under a common overarching goal.

Frontostriatal Contribution to the Interplay of Flexibility and Stability in Serial Prediction.

Surprising events may be relevant or irrelevant for behavior, requiring either flexible adjustment or stabilization of our model of the world and according response strategies. Cognitive flexibility and stability in response to environmental demands have been described as separable cognitive states, associated with activity of striatal and lateral prefrontal regions, respectively. It so far remains unclear, however, whether these two states act in an antagonistic fashion and which neural mechanisms mediate the selection of respective responses, on the one hand, and a transition between these states, on the other. In this study, we tested whether the functional dichotomy between striatal and prefrontal activity applies for the separate functions of updating (in response to changes in the environment, i.e., switches) and shielding (in response to chance occurrences of events violating expectations, i.e., drifts) of current predictions. We measured brain activity using fMRI while 20 healthy participants performed a task that required to serially predict upcoming items. Switches between predictable sequences had to be indicated via button press while sequence omissions (drifts) had to be ignored. We further varied the probability of switches and drifts to assess the neural network supporting the transition between flexible and stable cognitive states as a function of recent performance history in response to environmental demands. Flexible switching between models was associated with activation in medial pFC (BA 9 and BA 10), whereas stable maintenance of the internal model corresponded to activation in the lateral pFC (BA 6 and inferior frontal gyrus). Our findings extend previous studies on the interplay of flexibility and stability, suggesting that different prefrontal regions are activated by different types of prediction errors, dependent on their behavioral requirements. Furthermore, we found that striatal activation in response to switches and drifts was modulated by participants' successful behavior toward these events, suggesting the striatum to be responsible for response selections following unpredicted stimuli. Finally, we observed that the dopaminergic midbrain modulates the transition between different cognitive states, thresholded by participants' individual performance history in response to temporal environmental demands.