Neural event segmentation of continuous experience in human infants.

How infants experience the world is fundamental to understanding their cognition and development. A key principle of adult experience is that, despite receiving continuous sensory input, we perceive this input as discrete events. Here we investigate such event segmentation in infants and how it differs from adults. Research on event cognition in infants often uses simplified tasks in which (adult) experimenters help solve the segmentation problem for infants by defining event boundaries or presenting discrete actions/vignettes. This presupposes which events are experienced by infants and leaves open questions about the principles governing infant segmentation. We take a different, data-driven approach by studying infant event segmentation of continuous input. We collected whole-brain functional MRI (fMRI) data from awake infants (and adults, for comparison) watching a cartoon and used a hidden Markov model to identify event states in the brain. We quantified the existence, timescale, and organization of multiple-event representations across brain regions. The adult brain exhibited a known hierarchical gradient of event timescales, from shorter events in early visual regions to longer events in later visual and associative regions. In contrast, the infant brain represented only longer events, even in early visual regions, with no timescale hierarchy. The boundaries defining these infant events only partially overlapped with boundaries defined from adult brain activity and behavioral judgments. These findings suggest that events are organized differently in infants, with longer timescales and more stable neural patterns, even in sensory regions. This may indicate greater temporal integration and reduced temporal precision during dynamic, naturalistic perception.

Consolidation Enhances Sequential Multistep Anticipation but Diminishes Access to Perceptual Features.

Many experiences unfold predictably over time. Memory for these temporal regularities enables anticipation of events multiple steps into the future. Because temporally predictable events repeat over days, weeks, and years, we must maintain-and potentially transform-memories of temporal structure to support adaptive behavior. We explored how individuals build durable models of temporal regularities to guide multistep anticipation. Healthy young adults (Experiment 1: N = 99, age range = 18-40 years; Experiment 2: N = 204, age range = 19-40 years) learned sequences of scene images that were predictable at the category level and contained incidental perceptual details. Individuals then anticipated upcoming scene categories multiple steps into the future, immediately and at a delay. Consolidation increased the efficiency of anticipation, particularly for events further in the future, but diminished access to perceptual features. Further, maintaining a link-based model of the sequence after consolidation improved anticipation accuracy. Consolidation may therefore promote efficient and durable models of temporal structure, thus facilitating anticipation of future events.

High-Order Areas and Auditory Cortex Both Represent the High-Level Event Structure of Music.

Recent fMRI studies of event segmentation have found that default mode regions represent high-level event structure during movie watching. In these regions, neural patterns are relatively stable during events and shift at event boundaries. Music, like narratives, contains hierarchical event structure (e.g., sections are composed of phrases). Here, we tested the hypothesis that brain activity patterns in default mode regions reflect the high-level event structure of music. We used fMRI to record brain activity from 25 participants (male and female) as they listened to a continuous playlist of 16 musical excerpts and additionally collected annotations for these excerpts by asking a separate group of participants to mark when meaningful changes occurred in each one. We then identified temporal boundaries between stable patterns of brain activity using a hidden Markov model and compared the location of the model boundaries to the location of the human annotations. We identified multiple brain regions with significant matches to the observer-identified boundaries, including auditory cortex, medial prefrontal cortex, parietal cortex, and angular gyrus. From these results, we conclude that both higher-order and sensory areas contain information relating to the high-level event structure of music. Moreover, the higher-order areas in this study overlap with areas found in previous studies of event perception in movies and audio narratives, including regions in the default mode network.

Brain kernel: A new spatial covariance function for fMRI data.

A key problem in functional magnetic resonance imaging (fMRI) is to estimate spatial activity patterns from noisy high-dimensional signals. Spatial smoothing provides one approach to regularizing such estimates. However, standard smoothing methods ignore the fact that correlations in neural activity may fall off at different rates in different brain areas, or exhibit discontinuities across anatomical or functional boundaries. Moreover, such methods do not exploit the fact that widely separated brain regions may exhibit strong correlations due to bilateral symmetry or the network organization of brain regions. To capture this non-stationary spatial correlation structure, we introduce the brain kernel, a continuous covariance function for whole-brain activity patterns. We define the brain kernel in terms of a continuous nonlinear mapping from 3D brain coordinates to a latent embedding space, parametrized with a Gaussian process (GP). The brain kernel specifies the prior covariance between voxels as a function of the distance between their locations in embedding space. The GP mapping warps the brain nonlinearly so that highly correlated voxels are close together in latent space, and uncorrelated voxels are far apart. We estimate the brain kernel using resting-state fMRI data, and we develop an exact, scalable inference method based on block coordinate descent to overcome the challenges of high dimensionality (10-100K voxels). Finally, we illustrate the brain kernel's usefulness with applications to brain decoding and factor analysis with multiple task-based fMRI datasets.

Rapid Memory Reactivation at Movie Event Boundaries Promotes Episodic Encoding.

Segmentation of continuous experience into discrete events is driven by rapid fluctuations in encoding stability at context shifts (i.e., event boundaries), yet the mechanisms underlying the online formation of event memories are poorly understood. We investigated the neural per-time point spatial similarity patterns of the scalp electrophysiological (EEG) activity of 30 human participants (male and female) watching a 50 min movie and found that event boundaries triggered the rapid reinstatement of the just-encoded movie event EEG patterns. We also found that the onset of memory reinstatement at boundary onset was accompanied by a left-lateralized anterior negative ERP effect, which likely reflects the detection of a shift in the narrative structure of the movie. A data-driven approach based on Hidden Markov modeling allowed us to detect event boundaries as shifts between stable patterns of brain EEG activity during encoding, and to identify their reactivation during a free recall task. These results provide the first neurophysiological underpinnings for how the memory systems segment a continuous long stream of experience into episodic events.SIGNIFICANCE OF STATEMENT Memory for specific episodic events are the building blocks of our autobiographical memory. However, it is still unclear how the memory systems structure the unfolding experience into discrete event units that can be understood and remembered at the long-term. Here, we show that the detection of context shifts, or event boundaries, during a 50 min movie viewing triggers the rapid memory reactivation of the just-encoded event to promote its successful encoding into long-term memory. By finding that memory reactivation, a neural mechanism critical for episodic memory formation and consolidation, takes place under these ecologically valid experimental circumstances, our results provide valuable insights into how the brain shapes the ongoing experience into episodic memories in the real-life.

Representation of Real-World Event Schemas during Narrative Perception.

Understanding movies and stories requires maintaining a high-level situation model that abstracts away from perceptual details to describe the location, characters, actions, and causal relationships of the currently unfolding event. These models are built not only from information present in the current narrative, but also from prior knowledge about schematic event scripts, which describe typical event sequences encountered throughout a lifetime. We analyzed fMRI data from 44 human subjects (male and female) presented with 16 three-minute stories, consisting of four schematic events drawn from two different scripts (eating at a restaurant or going through the airport). Aside from this shared script structure, the stories varied widely in terms of their characters and storylines, and were presented in two highly dissimilar formats (audiovisual clips or spoken narration). One group was presented with the stories in an intact temporal sequence, while a separate control group was presented with the same events in scrambled order. Regions including the posterior medial cortex, medial prefrontal cortex (mPFC), and superior frontal gyrus exhibited schematic event patterns that generalized across stories, subjects, and modalities. Patterns in mPFC were also sensitive to overall script structure, with temporally scrambled events evoking weaker schematic representations. Using a Hidden Markov Model, patterns in these regions predicted the script (restaurant vs airport) of unlabeled data with high accuracy and were used to temporally align multiple stories with a shared script. These results extend work on the perception of controlled, artificial schemas in human and animal experiments to naturalistic perception of complex narratives.SIGNIFICANCE STATEMENT In almost all situations we encounter in our daily lives, we are able to draw on our schematic knowledge about what typically happens in the world to better perceive and mentally represent our ongoing experiences. In contrast to previous studies that investigated schematic cognition using simple, artificial associations, we measured brain activity from subjects watching movies and listening to stories depicting restaurant or airport experiences. Our results reveal a network of brain regions that is sensitive to the shared temporal structure of these naturalistic situations. These regions abstract away from the particular details of each story, activating a representation of the general type of situation being perceived.

Mapping between fMRI responses to movies and their natural language annotations.

Several research groups have shown how to map fMRI responses to the meanings of presented stimuli. This paper presents new methods for doing so when only a natural language annotation is available as the description of the stimulus. We study fMRI data gathered from subjects watching an episode of BBCs Sherlock (Chen et al., 2017), and learn bidirectional mappings between fMRI responses and natural language representations. By leveraging data from multiple subjects watching the same movie, we were able to perform scene classification with 72% accuracy (random guessing would give 4%) and scene ranking with average rank in the top 4% (random guessing would give 50%). The key ingredients underlying this high level of performance are (a) the use of the Shared Response Model (SRM) and its variant SRM-ICA (Chen et al., 2015; Zhang et al., 2016) to aggregate fMRI data from multiple subjects, both of which are shown to be superior to standard PCA in producing low-dimensional representations for the tasks in this paper; (b) a sentence embedding technique adapted from the natural language processing (NLP) literature (Arora et al., 2017) that produces semantic vector representation of the annotations; (c) using previous timestep information in the featurization of the predictor data. These optimizations in how we featurize the fMRI data and text annotations provide a substantial improvement in classification performance, relative to standard approaches.

Human-Object Interactions Are More than the Sum of Their Parts.

Understanding human-object interactions is critical for extracting meaning from everyday visual scenes and requires integrating complex relationships between human pose and object identity into a new percept. To understand how the brain builds these representations, we conducted 2 fMRI experiments in which subjects viewed humans interacting with objects, noninteracting human-object pairs, and isolated humans and objects. A number of visual regions process features of human-object interactions, including object identity information in the lateral occipital complex (LOC) and parahippocampal place area (PPA), and human pose information in the extrastriate body area (EBA) and posterior superior temporal sulcus (pSTS). Representations of human-object interactions in some regions, such as the posterior PPA (retinotopic maps PHC1 and PHC2) are well predicted by a simple linear combination of the response to object and pose information. Other regions, however, especially pSTS, exhibit representations for human-object interaction categories that are not predicted by their individual components, indicating that they encode human-object interactions as more than the sum of their parts. These results reveal the distributed networks underlying the emergent representation of human-object interactions necessary for social perception.

Pinpointing the peripheral bias in neural scene-processing networks during natural viewing.

Peripherally presented stimuli evoke stronger activity in scene-processing regions than foveally presented stimuli, suggesting that scene understanding is driven largely by peripheral information. We used functional MRI to investigate whether functional connectivity evoked during natural perception of audiovisual movies reflects this peripheral bias. For each scene-sensitive region--the parahippocampal place area (PPA), retrosplenial cortex, and occipital place area--we computed two measures: the extent to which its activity could be predicted by V1 activity (connectivity strength) and the eccentricities within V1 to which it was most closely related (connectivity profile). Scene regions were most related to peripheral voxels in V1, but the detailed nature of this connectivity varied within and between these regions. The retrosplenial cortex showed the most consistent peripheral bias but was less predictable from V1 activity, while the occipital place area was related to a wider range of eccentricities and was strongly coupled to V1. We divided the PPA along its posterior-anterior axis into retinotopic maps PHC1, PHC2, and anterior PPA, and found that a peripheral bias was detectable throughout all subregions, though the anterior PPA showed a less consistent relationship to eccentricity and a substantially weaker overall relationship to V1. We also observed an opposite foveal bias in object-perception regions including the lateral occipital complex and fusiform face area. These results show a fine-scale relationship between eccentricity biases and functional correlation during natural perception, giving new insight into the structure of the scene-perception network.

The brain hierarchically represents the past and future during multistep anticipation.

Memory for temporal structure enables both planning of future events and retrospection of past events. We investigated how the brain flexibly represents extended temporal sequences into the past and future during anticipation. Participants learned sequences of environments in immersive virtual reality. Pairs of sequences had the same environments in a different order, enabling context-specific learning. During fMRI, participants anticipated upcoming environments multiple steps into the future in a given sequence. Temporal structure was represented in the hippocampus and across higher-order visual regions (1) bidirectionally, with graded representations into the past and future and (2) hierarchically, with further events into the past and future represented in successively more anterior brain regions. In hippocampus, these bidirectional representations were context-specific, and suppression of far-away environments predicted response time costs in anticipation. Together, this work sheds light on how we flexibly represent sequential structure to enable planning over multiple timescales.

Top-down attention shifts behavioral and neural event boundaries in narratives with overlapping event scripts.

Understanding and remembering the complex experiences of everyday life relies critically on prior schematic knowledge about how events in our world unfold over time. How does the brain construct event representations from a library of schematic scripts, and how does activating a specific script impact the way that events are segmented in time? We developed a novel set of 16 audio narratives, each of which combines one of four location-relevant event scripts (restaurant, airport, grocery store, and lecture hall) with one of four socially relevant event scripts (breakup, proposal, business deal, and meet cute), and presented them to participants in an fMRI study and a separate online study. Responses in the angular gyrus, parahippocampal gyrus, and subregions of the medial prefrontal cortex (mPFC) were driven by scripts related to both location and social information, showing that these regions can track schematic sequences from multiple domains. For some stories, participants were primed to attend to one of the two scripts by training them to listen for and remember specific script-relevant episodic details. Activating a location-related event script shifted the timing of subjective event boundaries to align with script-relevant changes in the narratives, and this behavioral shift was mirrored in the timing of neural responses, with mPFC event boundaries (identified using a hidden Markov model) aligning to location-relevant rather than socially relevant boundaries when participants were location primed. Our findings demonstrate that neural event dynamics are actively modulated by top-down goals and provide new insight into how narrative event representations are constructed through the activation of temporally structured prior knowledge.

Differential connectivity within the Parahippocampal Place Area.

The Parahippocampal Place Area (PPA) has traditionally been considered a homogeneous region of interest, but recent evidence from both human studies and animal models has suggested that PPA may be composed of functionally distinct subunits. To investigate this hypothesis, we utilize a functional connectivity measure for fMRI that can estimate connectivity differences at the voxel level. Applying this method to whole-brain data from two experiments, we provide the first direct evidence that anterior and posterior PPA exhibit distinct connectivity patterns, with anterior PPA more strongly connected to regions in the default mode network (including the parieto-medial temporal pathway) and posterior PPA more strongly connected to occipital visual regions. We show that object sensitivity in PPA also has an anterior-posterior gradient, with stronger responses to abstract objects in posterior PPA. These findings cast doubt on the traditional view of PPA as a single coherent region, and suggest that PPA is composed of one subregion specialized for the processing of low-level visual features and object shape, and a separate subregion more involved in memory and scene context.

Studying waves of prediction in the brain using narratives.

Narrative stimuli offer a unique opportunity for research in cognitive neuroscience because they evoke cognitive processes that are difficult or impossible to study with traditional paradigms. An especially compelling feature of narratives is their temporal structure, which allows for meaningful predictions about upcoming events. As we proceed through a narrative, we can maintain a complex set of short- and long-term guesses about the future and continually refine our predictions as the story unfolds. Experiments using narratives can allow researchers to probe the ways in which memory systems are flexibly used during perception, including the mechanisms by which continuous experiences are segmented into discrete events. Despite the challenges of using narratives and other naturalistic stimuli in experimental research, these approaches offer a new window into critical components of real-world cognition.

img2fmri: a python package for predicting group-level fMRI responses to visual stimuli using deep neural networks.

Here we introduce a new python package, img2fmri, to predict group-level fMRI responses to individual images. This prediction model uses an artificial deep neural network (DNN), as DNNs have been successful at predicting cortical responses in the human visual cortex when trained on real world visual categorization tasks. To validate our model, we predict fMRI responses to images our model has not previously seen from a new dataset. We then show how our frame-by-frame prediction model can be extended to a continuous visual stimulus by predicting an fMRI response to Pixar Animation Studio's short film Partly Cloudy. In analyzing the timepoint-timepoint similarity of our predicted fMRI response around human-annotated event boundaries in the movie, we find that our model outperforms the baseline model in describing the dynamics of the real fMRI response around these event boundaries, particularly in the timepoints just before and at an event. These analyses suggest that in visual areas of the brain, at least some of the temporal dynamics we see in the brain's processing of continuous, naturalistic stimuli can be explained by dynamics in the stimulus itself, since they can be predicted from our frame-by-frame model. All code, analyses, tutorials, and installation instructions can be found at https://github.com/dpmlab/img2fmri.

Schema-based predictive eye movements support sequential memory encoding.

When forming a memory of an experience that is unfolding over time, we can use our schematic knowledge about the world (constructed based on many prior episodes) to predict what will transpire. We developed a novel paradigm to study how the development of a complex schema influences predictive processes during perception and impacts sequential memory. Participants learned to play a novel board game ('four-in-a-row') across six training sessions and repeatedly performed a memory test in which they watched and recalled sequences of moves from the game. We found that participants gradually became better at remembering sequences from the game as their schema developed, driven by improved accuracy for schema-consistent moves. Eye tracking revealed that increased predictive eye movements during encoding, which were most prevalent in expert players, were associated with better memory. Our results identify prediction as a mechanism by which schematic knowledge can improve episodic memory.

The brain hierarchically represents the past and future during multistep anticipation.

Memory for temporal structure enables both planning of future events and retrospection of past events. We investigated how the brain flexibly represents extended temporal sequences into the past and future during anticipation. Participants learned sequences of environments in immersive virtual reality. Pairs of sequences had the same environments in a different order, enabling context-specific learning. During fMRI, participants anticipated upcoming environments multiple steps into the future in a given sequence. Temporal structure was represented in the hippocampus and across visual regions (1) bidirectionally, with graded representations into the past and future and (2) hierarchically, with further events into the past and future represented in successively more anterior brain regions. Further, context-specific predictions were prioritized in the forward but not backward direction. Together, this work sheds light on how we flexibly represent sequential structure to enable planning over multiple timescales.

Voxel-level functional connectivity using spatial regularization.

Discovering functional connectivity between and within brain regions is a key concern in neuroscience. Due to the noise inherent in fMRI data, it is challenging to characterize the properties of individual voxels, and current methods are unable to flexibly analyze voxel-level connectivity differences. We propose a new functional connectivity method which incorporates a spatial smoothness constraint using regularized optimization, enabling the discovery of voxel-level interactions between brain regions from the small datasets characteristic of fMRI experiments. We validate our method in two separate experiments, demonstrating that we can learn coherent connectivity maps that are consistent with known results. First, we examine the functional connectivity between early visual areas V1 and VP, confirming that this connectivity structure preserves retinotopic mapping. Then, we show that two category-selective regions in ventral cortex - the Parahippocampal Place Area (PPA) and the Fusiform Face Area (FFA) - exhibit an expected peripheral versus foveal bias in their connectivity with visual area hV4. These results show that our approach is powerful, widely applicable, and capable of uncovering complex connectivity patterns with only a small amount of input data.

Developmental changes in story-evoked responses in the neocortex and hippocampus.

How does the representation of naturalistic life events change with age? Here, we analyzed fMRI data from 414 children and adolescents (5-19 years) as they watched a narrative movie. In addition to changes in the degree of inter-subject correlation (ISC) with age in sensory and medial parietal regions, we used a novel measure (between-group ISC) to reveal age-related shifts in the responses across the majority of the neocortex. Over the course of development, brain responses became more discretized into stable and coherent events and shifted earlier in time to anticipate upcoming perceived event transitions, measured behaviorally in an age-matched sample. However, hippocampal responses to event boundaries actually decreased with age, suggesting a shifting division of labor between episodic encoding processes and schematic event representations between the ages of 5 and 19.

Schema representations in distinct brain networks support narrative memory during encoding and retrieval.

Schematic prior knowledge can scaffold the construction of event memories during perception and also provide structured cues to guide memory search during retrieval. We measured the activation of story-specific and schematic representations using fMRI while participants were presented with 16 stories and then recalled each of the narratives, and related these activations to memory for specific story details. We predicted that schema representations in medial prefrontal cortex (mPFC) would be correlated with successful recall of story details. In keeping with this prediction, an anterior mPFC region showed a significant correlation between activation of schema representations at encoding and subsequent behavioral recall performance; however, this mPFC region was not implicated in schema representation during retrieval. More generally, our analyses revealed largely distinct brain networks at encoding and retrieval in which schema activation was related to successful recall. These results provide new insight into when and where event knowledge can support narrative memory.

Our day-to-day experiences are incredibly complex, so how does the brain remember them? Cognitive scientists have shown that memories rely on knowledge of common events that we have experienced before. Think about going to a restaurant: you arrive, you find a table, you order food, and then you eat. This kind of predictable sequence is called a schema. When humans make memories, our brains use schemas like these as scaffolding. They take a basic pattern constructed from past experience and fill it in with the specific details of an event. When memories are recalled, our brains use schemas as step-by-step guides to remember the events in the right order. Most research so far on how the brain uses schemas for memory has involved showing participants pictures or words and then testing their memory by asking 'true or false' questions. This revealed that a brain area called the medial prefrontal cortex plays an important role in creating and retrieving memories for items related to a schema. But, studies have not yet assessed exactly how the brain uses schemas to understand and remember a long, realistic event that unfolds over several minutes. To answer this question, Masís-Obando et al. scanned people's brains while they watched or listened to clips of two familiar experiences: eating at a restaurant or catching a flight at an airport. Then, the participants were scanned while they tried to retell each story in their own words. The volunteers were graded based on how many details they recalled. The scans showed that when volunteers' medial prefrontal cortex kept track of the schema throughout the whole time that an event was happening, they were more likely to score well on the memory test. But it wasn't necessary for medial prefrontal cortex to hold the schema in mind when remembering the story. Instead, a different set of brain regions maintained schema information during successful remembering. This study reveals new information about how memories and schemas work that could help explain why people develop problems making or recalling memories in diseases such as Alzheimer's. The findings could also be used to help people make experiences or stories more memorable.

Anticipation of temporally structured events in the brain.

Learning about temporal structure is adaptive because it enables the generation of expectations. We examined how the brain uses experience in structured environments to anticipate upcoming events. During fMRI (functional magnetic resonance imaging), individuals watched a 90 s movie clip six times. Using a hidden Markov model applied to searchlights across the whole brain, we identified temporal shifts between activity patterns evoked by the first vs. repeated viewings of the movie clip. In many regions throughout the cortex, neural activity patterns for repeated viewings shifted to precede those of initial viewing by up to 15 s. This anticipation varied hierarchically in a posterior (less anticipation) to anterior (more anticipation) fashion. We also identified specific regions in which the timing of the brain's event boundaries was related to those of human-labeled event boundaries, with the timing of this relationship shifting on repeated viewings. With repeated viewing, the brain's event boundaries came to precede human-annotated boundaries by 1-4 s on average. Together, these results demonstrate a hierarchy of anticipatory signals in the human brain and link them to subjective experiences of events.

Anticipating future events is essential. It allows individuals to plan and prepare what they will do seconds, minutes, or hours in the future. But how the brain can predict future events in both the short-term and long-term is not yet clear. Researchers know that the brain processes images or other sensory information in stages. For example, visual features are processed from lines to shapes to objects, and eventually scenes. This staged approach allows the brain to create representations of many parts of the world simultaneously. A similar hierarchy may be at play in anticipation. Different parts of the brain may track what is happening now, and what could happen in the next few seconds and minutes. This would provide a way for the brain to forecast upcoming events in the immediate, near, and more distant future at the same time. Now, Lee et al. show that the regions in the back of the brain anticipate the immediate future, while longer-term predictions are made in brain regions near the front. In the experiments, study participants watched a 90-second clip of the movie 'The Grand Budapest Hotel' six times while undergoing functional magnetic resonance imaging (fMRI). Then, Lee et al. used computer modeling to compare the brain activity captured by fMRI during successive viewings. This allowed the researchers to watch participants' brain activity moment-by-moment. As the participants repeatedly watched the movie clip, their brains began to anticipate what was coming next. Regions near the back of the brain like the visual cortex anticipated events in the next 1 to 4 seconds. Areas in the middle of the brain anticipated 5 to 8 seconds in the future. The front of brain anticipated 8 to 15 seconds into the future. Lee et al. show that many parts of the brain work together to predict the near and more distant future. More research is needed to understand how this information translates into actions. Learning more may help scientists understand how diseases or injuries affect people's ability to plan and respond to future events.