Neural event segmentation of continuous experience in human infants.

How infants experience the world is fundamental to understanding their cognition and development. A key principle of adult experience is that, despite receiving continuous sensory input, we perceive this input as discrete events. Here we investigate such event segmentation in infants and how it differs from adults. Research on event cognition in infants often uses simplified tasks in which (adult) experimenters help solve the segmentation problem for infants by defining event boundaries or presenting discrete actions/vignettes. This presupposes which events are experienced by infants and leaves open questions about the principles governing infant segmentation. We take a different, data-driven approach by studying infant event segmentation of continuous input. We collected whole-brain functional MRI (fMRI) data from awake infants (and adults, for comparison) watching a cartoon and used a hidden Markov model to identify event states in the brain. We quantified the existence, timescale, and organization of multiple-event representations across brain regions. The adult brain exhibited a known hierarchical gradient of event timescales, from shorter events in early visual regions to longer events in later visual and associative regions. In contrast, the infant brain represented only longer events, even in early visual regions, with no timescale hierarchy. The boundaries defining these infant events only partially overlapped with boundaries defined from adult brain activity and behavioral judgments. These findings suggest that events are organized differently in infants, with longer timescales and more stable neural patterns, even in sensory regions. This may indicate greater temporal integration and reduced temporal precision during dynamic, naturalistic perception.

Integration of event experiences to build relational knowledge in the human brain.

We investigated how the human brain integrates experiences of specific events to build general knowledge about typical event structure. We examined an episodic memory area important for temporal relations, anterior-lateral entorhinal cortex, and a semantic memory area important for action concepts, middle temporal gyrus, to understand how and when these areas contribute to these processes. Participants underwent functional magnetic resonance imaging while learning and recalling temporal relations among novel events over two sessions 1 week apart. Across distinct contexts, individual temporal relations among events could either be consistent or inconsistent with each other. Within each context, during the recall phase, we measured associative coding as the difference of multivoxel correlations among related vs unrelated pairs of events. Neural regions that form integrative representations should exhibit stronger associative coding in the consistent than the inconsistent contexts. We found evidence of integrative representations that emerged quickly in anterior-lateral entorhinal cortex (at session 1), and only subsequently in middle temporal gyrus, which showed a significant change across sessions. A complementary pattern of findings was seen with signatures during learning. This suggests that integrative representations are established early in anterior-lateral entorhinal cortex and may be a pathway to the later emergence of semantic knowledge in middle temporal gyrus.

Distinctive features of experiential time: Duration, speed and event density.

William James's use of "time in passing" and "stream of thoughts" may be two sides of the same coin that emerge from the brain segmenting the continuous flow of information into discrete events. Herein, we investigated how the density of events affects two temporal experiences: the felt duration and speed of time. Using a temporal bisection task, participants classified seconds-long videos of naturalistic scenes as short or long (duration), or slow or fast (passage of time). Videos contained a varying number and type of events. We found that a large number of events lengthened subjective duration and accelerated the felt passage of time. Surprisingly, participants were also faster at estimating their felt passage of time compared to duration. The perception of duration scaled with duration and event density, whereas the felt passage of time scaled with the rate of change. Altogether, our results suggest that distinct mechanisms underlie these two experiential times.

People can reliably detect action changes and goal changes during naturalistic perception.

As a part of ongoing perception, the human cognitive system segments others' activities into discrete episodes (event segmentation). Although prior research has shown that this process is likely related to changes in an actor's actions and goals, it has not yet been determined whether untrained observers can reliably identify action and goal changes as naturalistic activities unfold, or whether the changes they identify are tied to visual features of the activity (e.g., the beginnings and ends of object interactions). This study addressed these questions by examining untrained participants' identification of action changes, goal changes, and event boundaries while watching videos of everyday activities that were presented in both first-person and third-person perspectives. We found that untrained observers can identify goal changes and action changes consistently, and these changes are not explained by visual change and the onsets or offsets of contact with objects. Moreover, the action and goal changes identified by untrained observers were associated with event boundaries, even after accounting for objective visual features of the videos. These findings suggest that people can identify action and goal changes consistently and with high agreement, that they do so by using sensory information flexibly, and that the action and goal changes they identify may contribute to event segmentation.

Evidence That Event Boundaries Are Access Points for Memory Retrieval.

When recalling memories, we often scan information-rich continuous episodes, for example, to find our keys. How does our brain access and search through those memories? We suggest that high-level structure, marked by event boundaries, guides us through this process: In our computational model, memory scanning is sped up by skipping ahead to the next event boundary upon reaching a decision threshold. In adult Mechanical Turk workers from the United States, we used a movie (normed for event boundaries; Study 1, N = 203) to prompt memory scanning of movie segments for answers (Study 2, N = 298) and mental simulation (Study 3, N = 100) of these segments. Confirming model predictions, we found that memory-scanning times varied as a function of the number of event boundaries within a segment and the distance of the search target to the previous boundary (the key diagnostic parameter). Mental simulation times were also described by a skipping process with a higher skipping threshold than memory scanning. These findings identify event boundaries as access points to memory.

Events and objects are similar cognitive entities.

Logico-semantic theories have long noted parallels between the linguistic representation of temporal entities (events) and spatial entities (objects): bounded (or telic) predicates such as fix a car resemble count nouns such as sandcastle because they are "atoms" that have well-defined boundaries, contain discrete minimal parts and cannot be divided arbitrarily. By contrast, unbounded (or atelic) phrases such as drive a car resemble mass nouns such as sand in that they are unspecified for atomic features. Here, we demonstrate for the first time the parallels in the perceptual-cognitive representation of events and objects even in entirely non-linguistic tasks. Specifically, after viewers form categories of bounded or unbounded events, they can extend the category to objects or substances respectively (Experiments 1 and 2). Furthermore, in a training study, people successfully learn event-to-object mappings that respect atomicity (i.e., grouping bounded events with objects and unbounded events with substances) but fail to acquire the opposite, atomicity-violating mappings (Experiment 3). Finally, viewers can spontaneously draw connections between events and objects without any prior training (Experiment 4). These striking similarities between the mental representation of events and objects have implications for current theories of event cognition, as well as the relationship between language and thought.

Setting boundaries: Development of neural and behavioral event cognition in early childhood.

The ongoing stream of sensory experience is so complex and ever-changing that we tend to parse this experience at "event boundaries," which structures and strengthens memory. Memory processes undergo profound change across early childhood. Whether young children also divide their ongoing processing along event boundaries, and if those boundaries relate to memory, could provide important insight into the development of memory systems. In Study 1, 4-7-year-old children and adults segmented a cartoon, and we tested their memory. Children's event boundaries were more variable than adults' and differed in location and consistency of agreement. Older children's event segmentation was more adult-like than younger children's, and children who segmented events more like adults had better memory for those events. In Study 2, we asked whether these developmental differences in event segmentation had their roots in distinct neural representations. A separate group of 4-8-year-old children watched the same cartoon while undergoing an fMRI scan. In the right hippocampus, greater pattern dissimilarity across event boundaries compared to within events was evident for both child and adult behavioral boundaries, suggesting children and adults share similar event cognition. However, the boundaries identified by a data-driven Hidden Markov Model found that a different brain region-the left and right angular gyrus-aligned only with event boundaries defined by children. Overall, these data suggest that children's event cognition is reasonably well-developed by age 4 but continues to become more adult-like across early childhood. RESEARCH HIGHLIGHTS: Adults naturally break their experience into events, which structures and strengthens memory, but less is known about children's event perception and memory. Study 1 had adults and children segment and remember events from an animated show, and Study 2 compared those segmentations to other children's fMRI data. Children show better recognition and temporal order memory and more adult-like event segmentation with age, and children who segment more like adults have better memory. Children's and adults' behavioral boundaries mapped onto pattern similarity differences in hippocampus, and children's behavioral boundaries matched a data-driven model's boundaries in angular gyrus.

Aging and the encoding of changes in events: The role of neural activity pattern reinstatement.

When encountering unexpected event changes, memories of relevant past experiences must be updated to form new representations. Current models of memory updating propose that people must first generate memory-based predictions to detect and register that features of the environment have changed, then encode the new event features and integrate them with relevant memories of past experiences to form configural memory representations. Each of these steps may be impaired in older adults. Using functional MRI, we investigated these mechanisms in healthy young and older adults. In the scanner, participants first watched a movie depicting everyday activities in a day of an actor's life. They next watched a second nearly identical movie in which some scenes ended differently. Crucially, before watching the last part of each activity, the second movie stopped, and participants were asked to mentally replay how the activity previously ended. Three days later, participants were asked to recall the activities. Neural activity pattern reinstatement in medial temporal lobe (MTL) during the replay phase of the second movie was associated with detecting changes and with better memory for the original activity features. Reinstatements in posterior medial cortex (PMC) additionally predicted better memory for changed features. Compared to young adults, older adults showed a reduced ability to detect and remember changes and weaker associations between reinstatement and memory performance. These findings suggest that PMC and MTL contribute to change processing by reinstating previous event features, and that older adults are less able to use reinstatement to update memory for changed features.

Measuring event segmentation: An investigation into the stability of event boundary agreement across groups.

People spontaneously divide everyday experience into smaller units (event segmentation). To measure event segmentation, studies typically ask participants to explicitly mark the boundaries between events as they watch a movie (segmentation task). Their data may then be used to infer how others are likely to segment the same movie. However, significant variability in performance across individuals could undermine the ability to generalize across groups, especially as more research moves online. To address this concern, we used several widely employed and novel measures to quantify segmentation agreement across different sized groups (n = 2-32) using data collected on different platforms and movie types (in-lab & commercial film vs. online & everyday activities). All measures captured nonrandom and video-specific boundaries, but with notable between-sample variability. Samples of 6-18 participants were required to reliably detect video-driven segmentation behavior within a single sample. As sample size increased, agreement values improved and eventually stabilized at comparable sample sizes for in-lab & commercial film data and online & everyday activities data. Stabilization occurred at smaller sample sizes when measures reflected (1) agreement between two groups versus agreement between an individual and group, and (2) boundary identification between small (fine-grained) rather than large (coarse-grained) events. These analyses inform the tailoring of sample sizes based on the comparison of interest, materials, and data collection platform. In addition to demonstrating the reliability of online and in-lab segmentation performance at moderate sample sizes, this study supports the use of segmentation data to infer when events are likely to be segmented.

The multi-angle extended three-dimensional activities (META) stimulus set: A tool for studying event cognition.

To study complex human activity and how it is perceived and remembered, it is valuable to have large-scale, well-characterized stimuli that are representative of such activity. We present the Multi-angle Extended Three-dimensional Activities (META) stimulus set, a structured and highly instrumented set of extended event sequences performed in naturalistic settings. Performances were captured with two color cameras and a Kinect v2 camera with color and depth sensors, allowing the extraction of three-dimensional skeletal joint positions. We tracked the positions and identities of objects for all chapters using a mixture of manual coding and an automated tracking pipeline, and hand-annotated the timings of high-level actions. We also performed an online experiment to collect normative event boundaries for all chapters at a coarse and fine grain of segmentation, which allowed us to quantify event durations and agreement across participants. We share these materials publicly to advance new discoveries in the study of complex naturalistic activity.

Melting Ice With Your Mind: Representational Momentum for Physical States.

When a log burns, it transforms from a block of wood into a pile of ash. Such state changes are among the most dramatic ways objects change, going beyond mere changes of position or orientation. How does the mind represent changes of state? A foundational result in visual cognition is that memory extrapolates the positions of moving objects-a distortion called representational momentum. Here, five experiments (N = 400 adults) exploited this phenomenon to investigate mental representations in state space. Participants who viewed objects undergoing state changes (e.g., ice melting, logs burning, or grapes shriveling) remembered them as more changed (e.g., more melted, burned, or shriveled) than they actually were. This pattern extended to several types of state changes, went beyond their low-level properties, and even adhered to their natural trajectories in state space. Thus, mental representations of objects actively incorporate how they change-not only in their relation to their environment, but also in their essential qualities.

Understanding Everyday Events: Predictive-Looking Errors Drive Memory Updating.

Memory-guided predictions can improve event comprehension by guiding attention and the eyes to the location where an actor is about to perform an action. But when events change, viewers may experience predictive-looking errors and need to update their memories. In two experiments (Ns = 38 and 98), we examined the consequences of mnemonic predictive-looking errors for comprehending and remembering event changes. University students watched movies of everyday activities with actions that were repeated exactly and actions that were repeated with changed features-for example, an actor reached for a paper towel on one occasion and a dish towel on the next. Memory guidance led to predictive-looking errors that were associated with better memory for subsequently changed event features. These results indicate that retrieving recent event features can guide predictions during unfolding events and that error signals derived from mismatches between mnemonic predictions and actual events contribute to new learning.

Narratives bridge the divide between distant events in episodic memory.

Many studies suggest that information about past experience, or episodic memory, is divided into discrete units called "events." Yet we can often remember experiences that span multiple events. Events that occur in close succession might simply be linked because of their proximity to one another, but we can also build links between events that occur farther apart in time. Intuitively, some kind of organizing principle should enable temporally distant events to become bridged in memory. We tested the hypothesis that episodic memory exhibits a narrative-level organization, enabling temporally distant events to be better remembered if they form a coherent narrative. Furthermore, we tested whether post-encoding memory consolidation is necessary to integrate temporally distant events. In three experiments, participants learned and subsequently recalled events from fictional stories, in which pairs of temporally distant events involving side characters ("sideplots") either formed one coherent narrative or two unrelated narratives. Across participants, we varied whether recall was assessed immediately after learning, or after a delay: 24 hours, 12 hours between morning and evening ("wake"), or 12 hours between evening and morning ("sleep"). Participants recalled more information about coherent than unrelated narrative events, in most delay conditions, including immediate recall and wake conditions, suggesting that post-encoding consolidation was not necessary to integrate temporally distant events into a larger narrative. Furthermore, post hoc modeling across experiments suggested that narrative coherence facilitated recall over and above any effects of sentence-level semantic similarity. This reliable memory benefit for coherent narrative events supports theoretical accounts which propose that narratives provide a high-level architecture for episodic memory.

The Paradox of Time in Dynamic Causal Systems.

Recent work has shown that people use temporal information including order, delay, and variability to infer causality between events. In this study, we build on this work by investigating the role of time in dynamic systems, where causes take continuous values and also continually influence their effects. Recent studies of learning in these systems explored short interactions in a setting with rapidly evolving dynamics and modeled people as relying on simpler, resource-limited strategies to grapple with the stream of information. A natural question that arises from such an account is whether interacting with systems that unfold more slowly might reduce the systematic errors that result from these strategies. Paradoxically, we find that slowing the task indeed reduced the frequency of one type of error, albeit at the cost of increasing the overall error rate. To explain these results we posit that human learners analyze continuous dynamics into discrete events and use the observed relationships between events to draw conclusions about causal structure. We formalize this intuition in terms of a novel Causal Event Abstraction model and show that this model indeed captures the observed pattern of errors. We comment on the implications these results have for causal cognition.

Mapping the organization and dynamics of the posterior medial network during movie watching.

Brain regions within a posterior medial network (PMN) are characterized by sensitivity to episodic tasks, and they also demonstrate strong functional connectivity as part of the default network. Despite its cohesive structure, delineating the intranetwork organization and functional diversity of the PMN is crucial for understanding its contributions to multidimensional event cognition. Here, we probed functional connectivity of the PMN during movie watching to identify its pattern of connections and subnetwork functions in a split-sample replication of 136 participants. Consistent with prior findings of default network fractionation, we identified distinct PMN subsystems: a Ventral PM subsystem (retrosplenial cortex, parahippocampal cortex, posterior angular gyrus) and a Dorsal PM subsystem (medial prefrontal cortex, hippocampus, precuneus, posterior cingulate cortex, anterior angular gyrus). Ventral and Dorsal PM subsystems were differentiated by functional connectivity with parahippocampal cortex and precuneus and integrated by retrosplenial cortex and posterior cingulate cortex, respectively. Finally, the distinction between PMN subsystems is functionally relevant: whereas both Dorsal and Ventral PM connectivity tracked the movie content, only Ventral PM connections increased in strength at event transitions and appeared sensitive to episodic memory. Overall, these findings reveal PMN functional pathways and the distinct functional roles of intranetwork subsystems during event cognition.

Brain-environment alignment during movie watching predicts fluid intelligence and affective function in adulthood.

BOLD fMRI studies have provided compelling evidence that the human brain demonstrates substantial moment-to-moment fluctuations in both activity and functional connectivity (FC) patterns. While the role of brain signal variability in fostering cognitive adaptation to ongoing environmental demands is well-documented, the relevance of moment-to-moment changes in FC patterns is still debated. Here, we adopt a graph theoretical approach in order to shed light on the cognitive-affective implications of FC variability and associated profiles of functional network communication in adulthood. Our goal is to identify brain communication pathways underlying FC reconfiguration at multiple timescales, thereby improving understanding of how faster perceptually bound versus slower conceptual processes shape neural tuning to the dynamics of the external world and, thus, indirectly, mold affective and cognitive responding to the environment. To this end, we used neuroimaging and behavioural data collected during movie watching by the Cambridge Center for Ageing and Neuroscience (N = 642, 326 women) and the Human Connectome Project (N = 176, 106 women). FC variability evoked by changes to both the concrete perceptual and the more abstract conceptual representation of an ongoing situation increased from young to older adulthood. However, coupling between variability in FC patterns and concrete environmental features was stronger at younger ages. FC variability (both moment-to-moment/concrete featural and abstract conceptual boundary-evoked) was associated with age-distinct profiles of network communication, specifically, greater functional integration of the default mode network in older adulthood, but greater informational flow across neural networks implicated in environmentally driven attention and control (cingulo-opercular, salience, ventral attention) in younger adulthood. Whole-brain communication pathways anchored in default mode regions relevant to episodic and semantic context creation (i.e., angular and middle temporal gyri) supported FC reconfiguration in response to changes in the conceptual representation of an ongoing situation (i.e., narrative event boundaries), as well as stronger coupling between moment-to-moment fluctuations in FC and concrete environmental features. Fluid intelligence/abstract reasoning was directly linked to levels of brain-environment alignment, but only indirectly associated with levels of FC variability. Specifically, stronger coupling between moment-to-moment FC variability and concrete environmental features predicted poorer fluid intelligence and greater affectively driven environmental vigilance. Complementarily, across the adult lifespan, higher fluid (but not crystallised) intelligence was related to stronger expression of the network communication profile underlying momentary and event boundary-based FC variability during youth. Our results indicate that the adaptiveness of dynamic FC reconfiguration during naturalistic information processing changes across the lifespan due to the associated network communication profiles. Moreover, our findings on brain-environment alignment complement the existing literature on the beneficial consequences of modulating brain signal variability in response to environmental complexity. Specifically, they imply that coupling between moment-to-moment FC variability and concrete environmental features may index a bias towards perceptually-bound, rather than conceptual processing, which hinders affective functioning and strategic cognitive engagement with the external environment.

Representing agents, patients, goals and instruments in causative events: A cross-linguistic investigation of early language and cognition.

Although it is widely assumed that the linguistic description of events is based on a structured representation of event components at the perceptual/conceptual level, little empirical work has tested this assumption directly. Here, we test the connection between language and perception/cognition cross-linguistically, focusing on the relative salience of causative event components in language and cognition. We draw on evidence from preschoolers speaking English or Turkish. In a picture description task, Turkish-speaking 3-5-year-olds mentioned Agents less than their English-speaking peers (Turkish allows subject drop); furthermore, both language groups mentioned Patients more frequently than Goals, and Instruments less frequently than either Patients or Goals. In a change blindness task, both language groups were equally accurate at detecting changes to Agents (despite surface differences in Agent mentions). The remaining components also behaved similarly: both language groups were less accurate in detecting changes to Instruments than either Patients or Goals (even though Turkish-speaking preschoolers were less accurate overall than their English-speaking peers). To our knowledge, this is the first study offering evidence for a strong-even though not strict-homology between linguistic and conceptual event roles in young learners cross-linguistically.

The role of prior-event retrieval in encoding changed event features.

When people experience everyday activities, their comprehension can be shaped by expectations that derive from similar recent experiences, which can affect the encoding of a new experience into memory. When a new experience includes changes-such as a driving route being blocked by construction-this can lead to interference in subsequent memory. One potential mechanism of effective encoding of event changes is the retrieval of related features from previous events. Another such mechanism is the generation of a prediction error when a predicted feature is contradicted. In two experiments, we tested for effects of these two mechanisms on memory for changed features in movies of everyday activities. Participants viewed movies of an actor performing everyday activities across two fictitious days. Some event features changed across the days, and some features violated viewers' predictions. Retrieval of previous event features while viewing the second movie was associated with better subsequent memory, providing evidence for the retrieval mechanism. Contrary to our hypotheses, there was no support for the error mechanism: Prediction error was not associated with better memory when it was observed correlationally (Experiment 1) or directly manipulated (Experiment 2). These results support a key role for episodic retrieval in the encoding of new events. They also indicate boundary conditions on the role of prediction errors in driving new learning. Both findings have clear implications for theories of event memory.

A novel study: long-lasting event memory.

The current study reports two experiments that assessed memory for complex sets of information (i.e., four novels) over the course of several years. This was done to explore several issues, including (a) the content and durability of memory, (b) causal connectivity among the described events, (c) serial position within the novel, (d) pattern of retention and forgetting, and (e) interest in the novel. This study revealed that people remember events read in a novel for long periods of time with a shallow rate of forgetting. We also found that they remembered information better when it was presented earlier on in the novel, was more causally connected to other events in the novel, and was transitional in the character or story plot. Consistent with prior research, reported level of interest was not related to later memory. Because we experience events through different learning modalities (e.g., novels, film, autobiographical experience) which are likely remembered similarly, this research broadens our understanding of how people remember the events they learn about through different means.

How does COVID-19 emergency cognition influence public pro-environmental behavioral intentions? An affective event perspective.

Social impacts and serious damages caused by the COVID-19 pandemic have resulted in public introspection on the issue of ecological environmental protection. However, whether the public cognition of COVID-19 can promote pro-environmental behavioral intentions (PEBI) has not yet been determined; this is crucial for studying the ecological significance of the pandemic. Based on the affective events theory (AET), this study investigated the mechanism by which COVID-19 emergency cognition influences public PEBI. Following an analysis of 873 public questionnaires, the results reveal that public cognition of COVID-19 emergency can significantly promote PEBI. Among them, the effect of emergency coping is stronger than that of emergency relevance. Besides, the positive and negative environmental affective reactions aroused by COVID-19 pandemic play a mediating role between the emergency cognition and PEBI. Moreover, the positive environmental affective reactions show a stronger positive effect on household-sphere PEBI. However, the negative environmental affective reactions are more prominent in promoting public-sphere PEBI. This research aims to bridge a research gap by establishing a link between COVID-19 pandemic and PEBI. The findings can provide useful recommendations for policymakers to find the opportunity behind the COVID-19 emergency to promote public PEBI.