Neural event segmentation of continuous experience in human infants.

How infants experience the world is fundamental to understanding their cognition and development. A key principle of adult experience is that, despite receiving continuous sensory input, we perceive this input as discrete events. Here we investigate such event segmentation in infants and how it differs from adults. Research on event cognition in infants often uses simplified tasks in which (adult) experimenters help solve the segmentation problem for infants by defining event boundaries or presenting discrete actions/vignettes. This presupposes which events are experienced by infants and leaves open questions about the principles governing infant segmentation. We take a different, data-driven approach by studying infant event segmentation of continuous input. We collected whole-brain functional MRI (fMRI) data from awake infants (and adults, for comparison) watching a cartoon and used a hidden Markov model to identify event states in the brain. We quantified the existence, timescale, and organization of multiple-event representations across brain regions. The adult brain exhibited a known hierarchical gradient of event timescales, from shorter events in early visual regions to longer events in later visual and associative regions. In contrast, the infant brain represented only longer events, even in early visual regions, with no timescale hierarchy. The boundaries defining these infant events only partially overlapped with boundaries defined from adult brain activity and behavioral judgments. These findings suggest that events are organized differently in infants, with longer timescales and more stable neural patterns, even in sensory regions. This may indicate greater temporal integration and reduced temporal precision during dynamic, naturalistic perception.

Hippocampal Mechanisms Support Cortisol-Induced Memory Enhancements.

Stress can powerfully influence episodic memory, often enhancing memory encoding for emotionally salient information. These stress-induced memory enhancements stand at odds with demonstrations that stress and the stress-related hormone cortisol can negatively affect the hippocampus, a brain region important for episodic memory encoding. To resolve this apparent conflict and determine whether and how the hippocampus supports memory encoding under cortisol, we combined behavioral assays of associative memory, high-resolution fMRI, and pharmacological manipulation of cortisol in a within-participant, double-blinded procedure (in both sexes). Behaviorally, hydrocortisone promoted the encoding of subjectively arousing, positive associative memories. Neurally, hydrocortisone led to enhanced functional connectivity between hippocampal subregions, which predicted subsequent memory enhancements for emotional associations. Cortisol also modified the relationship between hippocampal representations and associative memory: whereas hippocampal signatures of distinctiveness predicted memory under placebo, relative integration predicted memory under cortisol. Together, these data provide novel evidence that the human hippocampus contains the necessary machinery to support emotional associative memory enhancements under cortisol.SIGNIFICANCE STATEMENT Our daily lives are filled with stressful events, which powerfully shape the way we form episodic memories. For example, stress and stress-related hormones can enhance our memory for emotional events. However, the mechanisms underlying these memory benefits are unclear. In the current study, we combined functional neuroimaging, behavioral tests of memory, and double-blind, placebo-controlled hydrocortisone administration to uncover the effects of the stress-related hormone cortisol on the function of the human hippocampus, a brain region important for episodic memory. We identified novel ways in which cortisol can enhance hippocampal function to promote emotional memories, highlighting the adaptive role of cortisol in shaping memory formation.

Acute Stress Effects on Statistical Learning and Episodic Memory.

Stress is widely considered to negatively impact hippocampal function, thus impairing episodic memory. However, the hippocampus is not merely the seat of episodic memory. Rather, it also (via distinct circuitry) supports statistical learning. On the basis of rodent work suggesting that stress may impair the hippocampal pathway involved in episodic memory while sparing or enhancing the pathway involved in statistical learning, we developed a behavioral experiment to investigate the effects of acute stress on both episodic memory and statistical learning in humans. Participants were randomly assigned to one of three conditions: stress (socially evaluated cold pressor) immediately before learning, stress ∼15 min before learning, or no stress. In the learning task, participants viewed a series of trial-unique scenes (allowing for episodic encoding of each image) in which certain scene categories reliably followed one another (allowing for statistical learning of associations between paired categories). Memory was assessed 24 hr later to isolate stress effects on encoding/learning rather than retrieval. We found modest support for our hypothesis that acute stress can amplify statistical learning: Only participants stressed ∼15 min in advance exhibited reliable evidence of learning across multiple measures. Furthermore, stress-induced cortisol levels predicted statistical learning retention 24 hr later. In contrast, episodic memory did not differ by stress condition, although we did find preliminary evidence that acute stress promoted memory for statistically predictable information and attenuated competition between statistical and episodic encoding. Together, these findings provide initial insights into how stress may differentially modulate learning processes within the hippocampus.

Statistical learning in epilepsy: Behavioral and anatomical mechanisms in the human brain.

OBJECTIVE: Statistical learning, the fundamental cognitive ability of humans to extract regularities across experiences over time, engages the medial temporal lobe (MTL) in the healthy brain. This leads to the hypothesis that statistical learning (SL) may be impaired in patients with epilepsy (PWE) involving the temporal lobe, and that this impairment could contribute to their varied memory deficits. In turn, studies done in collaboration with PWE, that evaluate the necessity of MTL circuitry through disease and causal perturbations, provide an opportunity to advance basic understanding of SL. METHODS: We implemented behavioral testing, volumetric analysis of the MTL substructures, and direct electrical brain stimulation to examine SL across a cohort of 61 PWE and 28 healthy controls. RESULTS: We found that behavioral performance in an SL task was negatively associated with seizure frequency irrespective of seizure origin. The volume of hippocampal subfields CA1 and CA2/3 correlated with SL performance, suggesting a more specific role of the hippocampus. Transient direct electrical stimulation of the hippocampus disrupted SL. Furthermore, the relationship between SL and seizure frequency was selective, as behavioral performance in an episodic memory task was not impacted by seizure frequency. SIGNIFICANCE: Overall, these results suggest that SL may be hippocampally dependent and that the SL task could serve as a clinically useful behavioral assay of seizure frequency that may complement existing approaches such as seizure diaries. Simple and short SL tasks may thus provide patient-centered endpoints for evaluating the efficacy of novel treatments in epilepsy.

Neural signatures of second language proficiency in narrative processing.

Making sense of speech in a second language relies on multiple abilities. Differences in brain activity related to proficiency in language tasks have often been attributed to processing demands. However, during naturalistic narrative comprehension, listeners at different proficiency levels may form different representations of the same speech. We hypothesized that the intersubject synchronization of these representations could be used to measure second-language proficiency. Using a searchlight-shared response model, we found highly proficient participants showed synchronization in regions similar to those of native speakers, including in the default mode network and the lateral prefrontal cortex. In contrast, participants with low proficiency showed more synchronization in auditory cortex and word-level semantic processing areas in the temporal lobe. Moderate proficiency showed the greatest neural diversity, suggesting lower consistency in the source of this partial proficiency. Based on these synchronization differences, we were able to classify the proficiency level or predict behavioral performance on an independent English test in held-out participants, suggesting the identified neural systems represented proficiency-sensitive information that was generalizable to other individuals. These findings suggest higher second-language proficiency leads to more native-like neural processing of naturalistic language, including in systems beyond the cognitive control network or the core language network.

Functional networks in the infant brain during sleep and wake states.

Functional brain networks are assessed differently earlier versus later in development: infants are almost universally scanned asleep, whereas adults are typically scanned awake. Observed differences between infant and adult functional networks may thus reflect differing states of consciousness rather than or in addition to developmental changes. We explore this question by comparing functional networks in functional magnetic resonance imaging (fMRI) scans of infants during natural sleep and awake movie-watching. As a reference, we also scanned adults during awake rest and movie-watching. Whole-brain functional connectivity was more similar within the same state (sleep and movie in infants; rest and movie in adults) compared with across states. Indeed, a classifier trained on patterns of functional connectivity robustly decoded infant state and even generalized to adults; interestingly, a classifier trained on adult state did not generalize as well to infants. Moreover, overall similarity between infant and adult functional connectivity was modulated by adult state (stronger for movie than rest) but not infant state (same for sleep and movie). Nevertheless, the connections that drove this similarity, particularly in the frontoparietal control network, were modulated by infant state. In sum, infant functional connectivity differs between sleep and movie states, highlighting the value of awake fMRI for studying functional networks over development.

Attention recruits frontal cortex in human infants.

Young infants learn about the world by overtly shifting their attention to perceptually salient events. In adults, attention recruits several brain regions spanning the frontal and parietal lobes. However, it is unclear whether these regions are sufficiently mature in infancy to support attention and, more generally, how infant attention is supported by the brain. We used event-related functional magnetic resonance imaging (fMRI) in 24 sessions from 20 awake behaving infants 3 mo to 12 mo old while they performed a child-friendly attentional cuing task. A target was presented to either the left or right of the infant's fixation, and offline gaze coding was used to measure the latency with which they saccaded to the target. To manipulate attention, a brief cue was presented before the target in three conditions: on the same side as the upcoming target (valid), on the other side (invalid), or on both sides (neutral). All infants were faster to look at the target on valid versus invalid trials, with valid faster than neutral and invalid slower than neutral, indicating that the cues effectively captured attention. We then compared the fMRI activity evoked by these trial types. Regions of adult attention networks activated more strongly for invalid than valid trials, particularly frontal regions. Neither behavioral nor neural effects varied by infant age within the first year, suggesting that these regions may function early in development to support the orienting of attention. Together, this furthers our mechanistic understanding of how the infant brain controls the allocation of attention.

Temporal Dynamics of Competition between Statistical Learning and Episodic Memory in Intracranial Recordings of Human Visual Cortex.

The function of long-term memory is not just to reminisce about the past, but also to make predictions that help us behave appropriately and efficiently in the future. This predictive function of memory provides a new perspective on the classic question from memory research of why we remember some things but not others. If prediction is a key outcome of memory, then the extent to which an item generates a prediction signifies that this information already exists in memory and need not be encoded. We tested this principle using human intracranial EEG as a time-resolved method to quantify prediction in visual cortex during a statistical learning task and link the strength of these predictions to subsequent episodic memory behavior. Epilepsy patients of both sexes viewed rapid streams of scenes, some of which contained regularities that allowed the category of the next scene to be predicted. We verified that statistical learning occurred using neural frequency tagging and measured category prediction with multivariate pattern analysis. Although neural prediction was robust overall, this was driven entirely by predictive items that were subsequently forgotten. Such interference provides a mechanism by which prediction can regulate memory formation to prioritize encoding of information that could help learn new predictive relationships.SIGNIFICANCE STATEMENT When faced with a new experience, we are rarely at a loss for what to do. Rather, because many aspects of the world are stable over time, we rely on past experiences to generate expectations that guide behavior. Here we show that these expectations during a new experience come at the expense of memory for that experience. From intracranial recordings of visual cortex, we decoded what humans expected to see next in a series of photographs based on patterns of neural activity. Photographs that generated strong neural expectations were more likely to be forgotten in a later behavioral memory test. Prioritizing the storage of experiences that currently lead to weak expectations could help improve these expectations in future encounters.

Intracranial Entrainment Reveals Statistical Learning across Levels of Abstraction.

We encounter the same people, places, and objects in predictable sequences and configurations. Humans efficiently learn these regularities via statistical learning. Importantly, statistical learning creates knowledge not only of specific regularities but also of regularities that apply more generally across related experiences (i.e., across members of a category). Prior evidence for different levels of learning comes from post-exposure behavioral tests, leaving open the question of whether more abstract regularities are detected online during initial exposure. We address this question by measuring neural entrainment in intracranial recordings. Neurosurgical patients viewed a stream of photographs with regularities at one of two levels: In the exemplar-level structured condition, the same photographs appeared repeatedly in pairs. In the category-level structured condition, the photographs were trial-unique but their categories were paired across repetitions. In a baseline random condition, the same photographs repeated but in a scrambled order. We measured entrainment at the frequency of individual photographs, which was expected in all conditions, but critically also at half that frequency-the rate at which to-be-learned pairs appeared in the two structured (but not random) conditions. Entrainment to both exemplar and category pairs emerged within minutes throughout visual cortex and in frontal and temporal regions. Many electrode contacts were sensitive to only one level of structure, but a significant number encoded both levels. These findings suggest that the brain spontaneously uncovers category-level regularities during statistical learning, providing insight into the brain's unsupervised mechanisms for building flexible and robust knowledge that generalizes across input variation and conceptual hierarchies.

Perception and memory retrieval states are reflected in distributed patterns of background functional connectivity.

The same visual input can serve as the target of perception or as a trigger for memory retrieval depending on whether cognitive processing is externally oriented (perception) or internally oriented (memory retrieval). While numerous human neuroimaging studies have characterized how visual stimuli are differentially processed during perception versus memory retrieval, perception and memory retrieval may also be associated with distinct neural states that are independent of stimulus-evoked neural activity. Here, we combined human fMRI with full correlation matrix analysis (FCMA) to reveal potential differences in "background" functional connectivity across perception and memory retrieval states. We found that perception and retrieval states could be discriminated with high accuracy based on patterns of connectivity across (1) the control network, (2) the default mode network (DMN), and (3) retrosplenial cortex (RSC). In particular, clusters in the control network increased connectivity with each other during the perception state, whereas clusters in the DMN were more strongly coupled during the retrieval state. Interestingly, RSC switched its coupling between networks as the cognitive state shifted from retrieval to perception. Finally, we show that background connectivity (1) was fully independent from stimulus-related variance in the signal and, further, (2) captured distinct aspects of cognitive states compared to traditional classification of stimulus-evoked responses. Together, our results reveal that perception and memory retrieval are associated with sustained cognitive states that manifest as distinct patterns of connectivity among large-scale brain networks.

Associative memory retrieval modulates upcoming perceptual decisions.

Expectations can inform fast, accurate decisions. But what informs expectations? Here we test the hypothesis that expectations are set by dynamic inference from memory. Participants performed a cue-guided perceptual decision task with independently-varying memory and sensory evidence. Cues established expectations by reminding participants of past stimulus-stimulus pairings, which predicted the likely target in a subsequent noisy image stream. Participant's responses used both memory and sensory information, in accordance to their relative reliability. Formal model comparison showed that the sensory inference was best explained when its parameters were set dynamically at each trial by evidence sampled from memory. Supporting this model, neural pattern analysis revealed that responses to the probe were modulated by the specific content and fidelity of memory reinstatement that occurred before the probe appeared. Together, these results suggest that perceptual decisions arise from the continuous sampling of memory and sensory evidence.

Statistical prediction of the future impairs episodic encoding of the present.

Memory is typically thought of as enabling reminiscence about past experiences. However, memory also informs and guides processing of future experiences. These two functions of memory are often at odds: Remembering specific experiences from the past requires storing idiosyncratic properties that define particular moments in space and time, but by definition such properties will not be shared with similar situations in the future and thus may not be applicable to future situations. We discovered that, when faced with this conflict, the brain prioritizes prediction over encoding. Behavioral tests of recognition and source recall showed that items allowing for prediction of what will appear next based on learned regularities were less likely to be encoded into memory. Brain imaging revealed that the hippocampus was responsible for this interference between statistical learning and episodic memory. The more that the hippocampus predicted the category of an upcoming item, the worse the current item was encoded. This competition may serve an adaptive purpose, focusing encoding on experiences for which we do not yet have a predictive model.

Face processing in the infant brain after pandemic lockdown.

The role of visual experience in the development of face processing has long been debated. We present a new angle on this question through a serendipitous study that cannot easily be repeated. Infants viewed short blocks of faces during fMRI in a repetition suppression task. The same identity was presented multiple times in half of the blocks (repeat condition) and different identities were presented once each in the other half (novel condition). In adults, the fusiform face area (FFA) tends to show greater neural activity for novel versus repeat blocks in such designs, suggesting that it can distinguish same versus different face identities. As part of an ongoing study, we collected data before the COVID-19 pandemic and after an initial local lockdown was lifted. The resulting sample of 12 infants (9-24 months) divided equally into pre- and post-lockdown groups with matching ages and data quantity/quality. The groups had strikingly different FFA responses: pre-lockdown infants showed repetition suppression (novel > repeat), whereas post-lockdown infants showed the opposite (repeat > novel), often referred to as repetition enhancement. These findings provide speculative evidence that altered visual experience during the lockdown, or other correlated environmental changes, may have affected face processing in the infant brain.

Revisiting the Role of the Medial Temporal Lobe in Motor Learning.

Classic taxonomies of memory distinguish explicit and implicit memory systems, placing motor skills squarely in the latter branch. This assertion is in part a consequence of foundational discoveries showing significant motor learning in amnesics. Those findings suggest that declarative memory processes in the medial temporal lobe (MTL) do not contribute to motor learning. Here, we revisit this issue, testing an individual (L. S. J.) with severe MTL damage on four motor learning tasks and comparing her performance to age-matched controls. Consistent with previous findings in amnesics, we observed that L. S. J. could improve motor performance despite having significantly impaired declarative memory. However, she tended to perform poorly relative to age-matched controls, with deficits apparently related to flexible action selection. Further supporting an action selection deficit, L. S. J. fully failed to learn a task that required the acquisition of arbitrary action-outcome associations. We thus propose a modest revision to the classic taxonomic model: Although MTL-dependent memory processes are not necessary for some motor learning to occur, they play a significant role in the acquisition, implementation, and retrieval of action selection strategies. These findings have implications for our understanding of the neural correlates of motor learning, the psychological mechanisms of skill, and the theory of multiple memory systems.

RT-Cloud: A cloud-based software framework to simplify and standardize real-time fMRI.

Real-time fMRI (RT-fMRI) neurofeedback has been shown to be effective in treating neuropsychiatric disorders and holds tremendous promise for future breakthroughs, both with regard to basic science and clinical applications. However, the prevalence of its use has been hampered by computing hardware requirements, the complexity of setting up and running an experiment, and a lack of standards that would foster collaboration. To address these issues, we have developed RT-Cloud (https://github.com/brainiak/rt-cloud), a flexible, cloud-based, open-source Python software package for the execution of RT-fMRI experiments. RT-Cloud uses standardized data formats and adaptable processing streams to support and expand open science in RT-fMRI research and applications. Cloud computing is a key enabling technology for advancing RT-fMRI because it eliminates the need for on-premise technical expertise and high-performance computing; this allows installation, configuration, and maintenance to be automated and done remotely. Furthermore, the scalability of cloud computing makes it easier to deploy computationally-demanding multivariate analyses in real time. In this paper, we describe how RT-Cloud has been integrated with open standards, including the Brain Imaging Data Structure (BIDS) standard and the OpenNeuro database, how it has been applied thus far, and our plans for further development and deployment of RT-Cloud in the coming years.

Relating Visual Production and Recognition of Objects in Human Visual Cortex.

Drawing is a powerful tool that can be used to convey rich perceptual information about objects in the world. What are the neural mechanisms that enable us to produce a recognizable drawing of an object, and how does this visual production experience influence how this object is represented in the brain? Here we evaluate the hypothesis that producing and recognizing an object recruit a shared neural representation, such that repeatedly drawing the object can enhance its perceptual discriminability in the brain. We scanned human participants (N = 31; 11 male) using fMRI across three phases of a training study: during training, participants repeatedly drew two objects in an alternating sequence on an MR-compatible tablet; before and after training, they viewed these and two other control objects, allowing us to measure the neural representation of each object in visual cortex. We found that: (1) stimulus-evoked representations of objects in visual cortex are recruited during visually cued production of drawings of these objects, even throughout the period when the object cue is no longer present; (2) the object currently being drawn is prioritized in visual cortex during drawing production, while other repeatedly drawn objects are suppressed; and (3) patterns of connectivity between regions in occipital and parietal cortex supported enhanced decoding of the currently drawn object across the training phase, suggesting a potential neural substrate for learning how to transform perceptual representations into representational actions. Together, our study provides novel insight into the functional relationship between visual production and recognition in the brain.SIGNIFICANCE STATEMENT Humans can produce simple line drawings that capture rich information about their perceptual experiences. However, the mechanisms that support this behavior are not well understood. Here we investigate how regions in visual cortex participate in the recognition of an object and the production of a drawing of it. We find that these regions carry diagnostic information about an object in a similar format both during recognition and production, and that practice drawing an object enhances transmission of information about it to downstream regions. Together, our study provides novel insight into the functional relationship between visual production and recognition in the brain.

Behavioral and Neural Fusion of Expectation with Sensation.

Humans perceive expected stimuli faster and more accurately. However, the mechanism behind the integration of expectations with sensory information during perception remains unclear. We investigated the hypothesis that such integration depends on "fusion"-the weighted averaging of different cues informative about stimulus identity. We first trained participants to map a range of tones onto faces spanning a male-female continuum via associative learning. These two features served as expectation and sensory cues to sex, respectively. We then tested specific predictions about the consequences of fusion by manipulating the congruence of these cues in psychophysical and fMRI experiments. Behavioral judgments and patterns of neural activity in auditory association regions revealed fusion of sensory and expectation cues, providing evidence for a precise computational account of how expectations influence perception.

Emergence and organization of adult brain function throughout child development.

Adult cognitive neuroscience has guided the study of human brain development by identifying regions associated with cognitive functions at maturity. The activity, connectivity, and structure of a region can be compared across ages to characterize the developmental trajectory of the corresponding function. However, developmental differences may reflect both the maturation of the function and also its organization across the brain. That is, a function may be present in children but supported by different brain regions, leading its maturity to be underestimated. Here we test the presence, maturity, and localization of adult functions in children using shared response modeling, a machine learning approach for functional alignment. After learning a lower-dimensional feature space from fMRI activity as adults watched a movie, we translated these shared features into the anatomical brain space of children 3-12 years old. To evaluate functional maturity, we correlated this reconstructed activity with children's actual fMRI activity as they watched the same movie. We found reliable correlations throughout cortex, even in the youngest children. The strength of the correlation in the precuneus, inferior frontal gyrus, and lateral occipital cortex predicted chronological age. These age-related changes were driven by three types of developmental trajectories: emergence from absence to presence, consistency in anatomical expression, and reorganization from one anatomical region to another. We also found evidence that the processing of pain-related events in the movie underwent reorganization across childhood. This data-driven, naturalistic approach provides a new perspective on the development of functional neuroanatomy throughout childhood.

Brain kernel: A new spatial covariance function for fMRI data.

A key problem in functional magnetic resonance imaging (fMRI) is to estimate spatial activity patterns from noisy high-dimensional signals. Spatial smoothing provides one approach to regularizing such estimates. However, standard smoothing methods ignore the fact that correlations in neural activity may fall off at different rates in different brain areas, or exhibit discontinuities across anatomical or functional boundaries. Moreover, such methods do not exploit the fact that widely separated brain regions may exhibit strong correlations due to bilateral symmetry or the network organization of brain regions. To capture this non-stationary spatial correlation structure, we introduce the brain kernel, a continuous covariance function for whole-brain activity patterns. We define the brain kernel in terms of a continuous nonlinear mapping from 3D brain coordinates to a latent embedding space, parametrized with a Gaussian process (GP). The brain kernel specifies the prior covariance between voxels as a function of the distance between their locations in embedding space. The GP mapping warps the brain nonlinearly so that highly correlated voxels are close together in latent space, and uncorrelated voxels are far apart. We estimate the brain kernel using resting-state fMRI data, and we develop an exact, scalable inference method based on block coordinate descent to overcome the challenges of high dimensionality (10-100K voxels). Finally, we illustrate the brain kernel's usefulness with applications to brain decoding and factor analysis with multiple task-based fMRI datasets.

Searching through functional space reveals distributed visual, auditory, and semantic coding in the human brain.

The extent to which brain functions are localized or distributed is a foundational question in neuroscience. In the human brain, common fMRI methods such as cluster correction, atlas parcellation, and anatomical searchlight are biased by design toward finding localized representations. Here we introduce the functional searchlight approach as an alternative to anatomical searchlight analysis, the most commonly used exploratory multivariate fMRI technique. Functional searchlight removes any anatomical bias by grouping voxels based only on functional similarity and ignoring anatomical proximity. We report evidence that visual and auditory features from deep neural networks and semantic features from a natural language processing model, as well as object representations, are more widely distributed across the brain than previously acknowledged and that functional searchlight can improve model-based similarity and decoding accuracy. This approach provides a new way to evaluate and constrain computational models with brain activity and pushes our understanding of human brain function further along the spectrum from strict modularity toward distributed representation.