A model of autonomous interactions between hippocampus and neocortex driving sleep-dependent memory consolidation.

How do we build up our knowledge of the world over time? Many theories of memory formation and consolidation have posited that the hippocampus stores new information, then "teaches" this information to the neocortex over time, especially during sleep. But it is unclear, mechanistically, how this actually works-How are these systems able to interact during periods with virtually no environmental input to accomplish useful learning and shifts in representation? We provide a framework for thinking about this question, with neural network model simulations serving as demonstrations. The model is composed of hippocampus and neocortical areas, which replay memories and interact with one another completely autonomously during simulated sleep. Oscillations are leveraged to support error-driven learning that leads to useful changes in memory representation and behavior. The model has a non-rapid eye movement (NREM) sleep stage, where dynamics between the hippocampus and neocortex are tightly coupled, with the hippocampus helping neocortex to reinstate high-fidelity versions of new attractors, and a REM sleep stage, where neocortex is able to more freely explore existing attractors. We find that alternating between NREM and REM sleep stages, which alternately focuses the model's replay on recent and remote information, facilitates graceful continual learning. We thus provide an account of how the hippocampus and neocortex can interact without any external input during sleep to drive useful new cortical learning and to protect old knowledge as new information is integrated.

Improving Memory Search Through Model-Based Cue Selection.

We often use cues from our environment when we get stuck searching our memories, but prior research has failed to show benefits of cuing with other, randomly selected list items during memory search. What accounts for this discrepancy? We proposed that cues' content critically determines their effectiveness and sought to select the right cues by building a computational model of how cues affect memory search. Participants (N = 195 young adults from the United States) recalled significantly more items when receiving our model's best (vs. worst) cue. Our model provides an account of why some cues better aid recall: Effective cues activate contexts most similar to the remaining items' contexts, facilitating recall in an unsearched area of memory. We discuss our contributions in relation to prominent theories about the effect of external cues.

Associative memory retrieval modulates upcoming perceptual decisions.

Expectations can inform fast, accurate decisions. But what informs expectations? Here we test the hypothesis that expectations are set by dynamic inference from memory. Participants performed a cue-guided perceptual decision task with independently-varying memory and sensory evidence. Cues established expectations by reminding participants of past stimulus-stimulus pairings, which predicted the likely target in a subsequent noisy image stream. Participant's responses used both memory and sensory information, in accordance to their relative reliability. Formal model comparison showed that the sensory inference was best explained when its parameters were set dynamically at each trial by evidence sampled from memory. Supporting this model, neural pattern analysis revealed that responses to the probe were modulated by the specific content and fidelity of memory reinstatement that occurred before the probe appeared. Together, these results suggest that perceptual decisions arise from the continuous sampling of memory and sensory evidence.

Evidence That Event Boundaries Are Access Points for Memory Retrieval.

When recalling memories, we often scan information-rich continuous episodes, for example, to find our keys. How does our brain access and search through those memories? We suggest that high-level structure, marked by event boundaries, guides us through this process: In our computational model, memory scanning is sped up by skipping ahead to the next event boundary upon reaching a decision threshold. In adult Mechanical Turk workers from the United States, we used a movie (normed for event boundaries; Study 1, N = 203) to prompt memory scanning of movie segments for answers (Study 2, N = 298) and mental simulation (Study 3, N = 100) of these segments. Confirming model predictions, we found that memory-scanning times varied as a function of the number of event boundaries within a segment and the distance of the search target to the previous boundary (the key diagnostic parameter). Mental simulation times were also described by a skipping process with a higher skipping threshold than memory scanning. These findings identify event boundaries as access points to memory.

High-Order Areas and Auditory Cortex Both Represent the High-Level Event Structure of Music.

Recent fMRI studies of event segmentation have found that default mode regions represent high-level event structure during movie watching. In these regions, neural patterns are relatively stable during events and shift at event boundaries. Music, like narratives, contains hierarchical event structure (e.g., sections are composed of phrases). Here, we tested the hypothesis that brain activity patterns in default mode regions reflect the high-level event structure of music. We used fMRI to record brain activity from 25 participants (male and female) as they listened to a continuous playlist of 16 musical excerpts and additionally collected annotations for these excerpts by asking a separate group of participants to mark when meaningful changes occurred in each one. We then identified temporal boundaries between stable patterns of brain activity using a hidden Markov model and compared the location of the model boundaries to the location of the human annotations. We identified multiple brain regions with significant matches to the observer-identified boundaries, including auditory cortex, medial prefrontal cortex, parietal cortex, and angular gyrus. From these results, we conclude that both higher-order and sensory areas contain information relating to the high-level event structure of music. Moreover, the higher-order areas in this study overlap with areas found in previous studies of event perception in movies and audio narratives, including regions in the default mode network.

RT-Cloud: A cloud-based software framework to simplify and standardize real-time fMRI.

Real-time fMRI (RT-fMRI) neurofeedback has been shown to be effective in treating neuropsychiatric disorders and holds tremendous promise for future breakthroughs, both with regard to basic science and clinical applications. However, the prevalence of its use has been hampered by computing hardware requirements, the complexity of setting up and running an experiment, and a lack of standards that would foster collaboration. To address these issues, we have developed RT-Cloud (https://github.com/brainiak/rt-cloud), a flexible, cloud-based, open-source Python software package for the execution of RT-fMRI experiments. RT-Cloud uses standardized data formats and adaptable processing streams to support and expand open science in RT-fMRI research and applications. Cloud computing is a key enabling technology for advancing RT-fMRI because it eliminates the need for on-premise technical expertise and high-performance computing; this allows installation, configuration, and maintenance to be automated and done remotely. Furthermore, the scalability of cloud computing makes it easier to deploy computationally-demanding multivariate analyses in real time. In this paper, we describe how RT-Cloud has been integrated with open standards, including the Brain Imaging Data Structure (BIDS) standard and the OpenNeuro database, how it has been applied thus far, and our plans for further development and deployment of RT-Cloud in the coming years.

Relating Visual Production and Recognition of Objects in Human Visual Cortex.

Drawing is a powerful tool that can be used to convey rich perceptual information about objects in the world. What are the neural mechanisms that enable us to produce a recognizable drawing of an object, and how does this visual production experience influence how this object is represented in the brain? Here we evaluate the hypothesis that producing and recognizing an object recruit a shared neural representation, such that repeatedly drawing the object can enhance its perceptual discriminability in the brain. We scanned human participants (N = 31; 11 male) using fMRI across three phases of a training study: during training, participants repeatedly drew two objects in an alternating sequence on an MR-compatible tablet; before and after training, they viewed these and two other control objects, allowing us to measure the neural representation of each object in visual cortex. We found that: (1) stimulus-evoked representations of objects in visual cortex are recruited during visually cued production of drawings of these objects, even throughout the period when the object cue is no longer present; (2) the object currently being drawn is prioritized in visual cortex during drawing production, while other repeatedly drawn objects are suppressed; and (3) patterns of connectivity between regions in occipital and parietal cortex supported enhanced decoding of the currently drawn object across the training phase, suggesting a potential neural substrate for learning how to transform perceptual representations into representational actions. Together, our study provides novel insight into the functional relationship between visual production and recognition in the brain.SIGNIFICANCE STATEMENT Humans can produce simple line drawings that capture rich information about their perceptual experiences. However, the mechanisms that support this behavior are not well understood. Here we investigate how regions in visual cortex participate in the recognition of an object and the production of a drawing of it. We find that these regions carry diagnostic information about an object in a similar format both during recognition and production, and that practice drawing an object enhances transmission of information about it to downstream regions. Together, our study provides novel insight into the functional relationship between visual production and recognition in the brain.

Relating the Past with the Present: Information Integration and Segregation during Ongoing Narrative Processing.

This study examined how the brain dynamically updates event representations by integrating new information over multiple minutes while segregating irrelevant input. A professional writer custom-designed a narrative with two independent storylines, interleaving across minute-long segments (ABAB). In the last (C) part, characters from the two storylines meet and their shared history is revealed. Part C is designed to induce the spontaneous recall of past events, upon the recurrence of narrative motifs from A/B, and to shed new light on them. Our fMRI results showed storyline-specific neural patterns, which were reinstated (i.e., became more active) during storyline transitions. This effect increased along the processing timescale hierarchy, peaking in the default mode network. Similarly, the neural reinstatement of motifs was found during Part C. Furthermore, participants showing stronger motif reinstatement performed better in integrating A/B and C events, demonstrating the role of memory reactivation in information integration over intervening irrelevant events.

Brain kernel: A new spatial covariance function for fMRI data.

A key problem in functional magnetic resonance imaging (fMRI) is to estimate spatial activity patterns from noisy high-dimensional signals. Spatial smoothing provides one approach to regularizing such estimates. However, standard smoothing methods ignore the fact that correlations in neural activity may fall off at different rates in different brain areas, or exhibit discontinuities across anatomical or functional boundaries. Moreover, such methods do not exploit the fact that widely separated brain regions may exhibit strong correlations due to bilateral symmetry or the network organization of brain regions. To capture this non-stationary spatial correlation structure, we introduce the brain kernel, a continuous covariance function for whole-brain activity patterns. We define the brain kernel in terms of a continuous nonlinear mapping from 3D brain coordinates to a latent embedding space, parametrized with a Gaussian process (GP). The brain kernel specifies the prior covariance between voxels as a function of the distance between their locations in embedding space. The GP mapping warps the brain nonlinearly so that highly correlated voxels are close together in latent space, and uncorrelated voxels are far apart. We estimate the brain kernel using resting-state fMRI data, and we develop an exact, scalable inference method based on block coordinate descent to overcome the challenges of high dimensionality (10-100K voxels). Finally, we illustrate the brain kernel's usefulness with applications to brain decoding and factor analysis with multiple task-based fMRI datasets.

BrainIAK tutorials: User-friendly learning materials for advanced fMRI analysis.

Advanced brain imaging analysis methods, including multivariate pattern analysis (MVPA), functional connectivity, and functional alignment, have become powerful tools in cognitive neuroscience over the past decade. These tools are implemented in custom code and separate packages, often requiring different software and language proficiencies. Although usable by expert researchers, novice users face a steep learning curve. These difficulties stem from the use of new programming languages (e.g., Python), learning how to apply machine-learning methods to high-dimensional fMRI data, and minimal documentation and training materials. Furthermore, most standard fMRI analysis packages (e.g., AFNI, FSL, SPM) focus on preprocessing and univariate analyses, leaving a gap in how to integrate with advanced tools. To address these needs, we developed BrainIAK (brainiak.org), an open-source Python software package that seamlessly integrates several cutting-edge, computationally efficient techniques with other Python packages (e.g., Nilearn, Scikit-learn) for file handling, visualization, and machine learning. To disseminate these powerful tools, we developed user-friendly tutorials (in Jupyter format; https://brainiak.org/tutorials/) for learning BrainIAK and advanced fMRI analysis in Python more generally. These materials cover techniques including: MVPA (pattern classification and representational similarity analysis); parallelized searchlight analysis; background connectivity; full correlation matrix analysis; inter-subject correlation; inter-subject functional connectivity; shared response modeling; event segmentation using hidden Markov models; and real-time fMRI. For long-running jobs or large memory needs we provide detailed guidance on high-performance computing clusters. These notebooks were successfully tested at multiple sites, including as problem sets for courses at Yale and Princeton universities and at various workshops and hackathons. These materials are freely shared, with the hope that they become part of a pool of open-source software and educational materials for large-scale, reproducible fMRI analysis and accelerated discovery.

Targeted Memory Reactivation during Sleep Elicits Neural Signals Related to Learning Content.

Retrieval of learning-related neural activity patterns is thought to drive memory stabilization. However, finding reliable, noninvasive, content-specific indicators of memory retrieval remains a central challenge. Here, we attempted to decode the content of retrieved memories in the EEG during sleep. During encoding, male and female human subjects learned to associate spatial locations of visual objects with left- or right-hand movements, and each object was accompanied by an inherently related sound. During subsequent slow-wave sleep within an afternoon nap, we presented half of the sound cues that were associated (during wake) with left- and right-hand movements before bringing subjects back for a final postnap test. We trained a classifier on sleep EEG data (focusing on lateralized EEG features that discriminated left- vs right-sided trials during wake) to predict learning content when we cued the memories during sleep. Discrimination performance was significantly above chance and predicted subsequent memory, supporting the idea that retrieval leads to memory stabilization. Moreover, these lateralized signals increased with postcue sleep spindle power, demonstrating that retrieval has a strong relationship with spindles. These results show that lateralized activity related to individual memories can be decoded from sleep EEG, providing an effective indicator of offline retrieval.SIGNIFICANCE STATEMENT Memories are thought to be retrieved during sleep, leading to their long-term stabilization. However, there has been relatively little work in humans linking neural measures of retrieval of individual memories during sleep to subsequent memory performance. This work leverages the prominent electrophysiological signal triggered by lateralized movements to robustly demonstrate the retrieval of specific cued memories during sleep. Moreover, these signals predict subsequent memory and are correlated with sleep spindles, neural oscillations that have previously been implicated in memory stabilization. Together, these findings link memory retrieval to stabilization and provide a powerful tool for investigating memory in a wide range of learning contexts and human populations.

Leveraging shared connectivity to aggregate heterogeneous datasets into a common response space.

Connectivity hyperalignment can be used to estimate a single shared response space across disjoint datasets. We develop a connectivity-based shared response model that factorizes aggregated fMRI datasets into a single reduced-dimension shared connectivity space and subject-specific topographic transformations. These transformations resolve idiosyncratic functional topographies and can be used to project response time series into shared space. We evaluate this algorithm on a large collection of heterogeneous, naturalistic fMRI datasets acquired while subjects listened to spoken stories. Projecting subject data into shared space dramatically improves between-subject story time-segment classification and increases the dimensionality of shared information across subjects. This improvement generalizes to subjects and stories excluded when estimating the shared space. We demonstrate that estimating a simple semantic encoding model in shared space improves between-subject forward encoding and inverted encoding model performance. The shared space estimated across all datasets is distinct from the shared space derived from any particular constituent dataset; the algorithm leverages shared connectivity to yield a consensus shared space conjoining diverse story stimuli.

Temporal integration of narrative information in a hippocampal amnesic patient.

Default network regions appear to integrate information over time windows of 30 â€‹s or more during narrative listening. Does this long-timescale capability require the hippocampus? Amnesic behavior suggests that regions other than the hippocampus can independently support some online processing when input is continuous and semantically rich: amnesics can participate in conversations and tell stories spanning minutes, and when tested immediately on recently heard prose they are able to retain some information. We hypothesized that default network regions can integrate the semantically coherent information of a narrative across long time windows, even in the absence of an intact hippocampus. To test this prediction, we measured BOLD activity in the brain of a hippocampal amnesic patient (D.A.) and healthy control participants while they listened to a 7 min narrative. The narrative was played either in its intact form, or as a paragraph-scrambled version, which has been previously shown to interfere with the long-range temporal dependencies in default network activity. In the intact story condition, D.A.'s moment-by-moment BOLD activity spatial patterns were similar to those of controls in low-level auditory cortex as well as in some high-level default network regions (including lateral and medial posterior parietal cortex). Moreover, as in controls, D.A.'s response patterns in medial and lateral posterior parietal cortex were disrupted when paragraphs of the story were presented in a shuffled order, suggesting that activity in these areas did depend on information from 30 â€‹s or more in the past. Together, these results suggest that some default network cortical areas can integrate information across long timescales, even when the hippocampus is severely damaged.

Neural Overlap in Item Representations Across Episodes Impairs Context Memory.

We frequently encounter the same item in different contexts, and when that happens, memories of earlier encounters can get reactivated. We examined how existing memories are changed as a result of such reactivation. We hypothesized that when an item's initial and subsequent neural representations overlap, this allows the initial item to become associated with novel contextual information, interfering with later retrieval of the initial context. Specifically, we predicted a negative relationship between representational similarity across repeated experiences of an item and subsequent source memory for the initial context. We tested this hypothesis in an fMRI study, in which objects were presented multiple times during different tasks. We measured the similarity of the neural patterns in lateral occipital cortex that were elicited by the first and second presentations of objects, and related this neural overlap score to subsequent source memory. Consistent with our hypothesis, greater item-specific pattern similarity was linked to worse source memory for the initial task. In contrast, greater reactivation of the initial context was associated with better source memory. Our findings suggest that the influence of novel experiences on an existing context memory depends on how reliably a shared component (i.e., item) is represented across these episodes.

Representation of Real-World Event Schemas during Narrative Perception.

Understanding movies and stories requires maintaining a high-level situation model that abstracts away from perceptual details to describe the location, characters, actions, and causal relationships of the currently unfolding event. These models are built not only from information present in the current narrative, but also from prior knowledge about schematic event scripts, which describe typical event sequences encountered throughout a lifetime. We analyzed fMRI data from 44 human subjects (male and female) presented with 16 three-minute stories, consisting of four schematic events drawn from two different scripts (eating at a restaurant or going through the airport). Aside from this shared script structure, the stories varied widely in terms of their characters and storylines, and were presented in two highly dissimilar formats (audiovisual clips or spoken narration). One group was presented with the stories in an intact temporal sequence, while a separate control group was presented with the same events in scrambled order. Regions including the posterior medial cortex, medial prefrontal cortex (mPFC), and superior frontal gyrus exhibited schematic event patterns that generalized across stories, subjects, and modalities. Patterns in mPFC were also sensitive to overall script structure, with temporally scrambled events evoking weaker schematic representations. Using a Hidden Markov Model, patterns in these regions predicted the script (restaurant vs airport) of unlabeled data with high accuracy and were used to temporally align multiple stories with a shared script. These results extend work on the perception of controlled, artificial schemas in human and animal experiments to naturalistic perception of complex narratives.SIGNIFICANCE STATEMENT In almost all situations we encounter in our daily lives, we are able to draw on our schematic knowledge about what typically happens in the world to better perceive and mentally represent our ongoing experiences. In contrast to previous studies that investigated schematic cognition using simple, artificial associations, we measured brain activity from subjects watching movies and listening to stories depicting restaurant or airport experiences. Our results reveal a network of brain regions that is sensitive to the shared temporal structure of these naturalistic situations. These regions abstract away from the particular details of each story, activating a representation of the general type of situation being perceived.

Neurofeedback helps to reveal a relationship between context reinstatement and memory retrieval.

Theories of mental context and memory posit that successful mental context reinstatement enables better retrieval of memories from the same context, at the expense of memories from other contexts. To test this hypothesis, we had participants study lists of words, interleaved with task-irrelevant images from one category (e.g., scenes). Following encoding, participants were cued to mentally reinstate the context associated with a particular list, by thinking about the images that had appeared between the words. We measured context reinstatement by applying multivariate pattern classifiers to fMRI, and related this to performance on a free recall test that followed immediately afterwards. To increase sensitivity, we used a closed-loop neurofeedback procedure, whereby higher classifier evidence for the cued category elicited increased visibility of the images from the studied context onscreen. Our goal was to create a positive feedback loop that amplified small fluctuations in mental context reinstatement, making it easier to experimentally detect a relationship between context reinstatement and recall. As predicted, we found that greater amounts of classifier evidence were associated with better recall of words from the reinstated context, and worse recall of words from a different context. In a second experiment, we assessed the role of neurofeedback in identifying this brain-behavior relationship by presenting context images again and manipulating whether their visibility depended on classifier evidence. When neurofeedback was removed, the relationship between classifier evidence and memory retrieval disappeared. Together, these findings demonstrate a clear effect of context reinstatement on memory recall and suggest that neurofeedback can be a useful tool for characterizing brain-behavior relationships.

Refresh my memory: Episodic memory reinstatements intrude on working memory maintenance.

A fundamental question in memory research is how different forms of memory interact. Previous research has shown that people rely on working memory (WM) in short-term recognition tasks; a common view is that episodic memory (EM) only influences performance on these tasks when WM maintenance is disrupted. However, retrieval of memories from EM has been widely observed during brief periods of quiescence, raising the possibility that EM retrievals during maintenance-critically, before a response can be prepared-might affect short-term recognition memory performance even in the absence of distraction. We hypothesized that this influence would be mediated by the lingering presence of reactivated EM content in WM. We obtained support for this hypothesis in three experiments, showing that delay-period EM reactivation introduces incidentally associated information (context) into WM, and that these retrieved associations negatively impact subsequent recognition, leading to substitution errors (Experiment 1) and slowing of accurate responses (Experiment 2). FMRI pattern analysis showed that slowing is mediated by the content of EM reinstatement (Experiment 3). These results expose a previously hidden influence of EM on WM, raising new questions about the adaptive nature of their interaction.

Neural Differentiation of Incorrectly Predicted Memories.

When an item is predicted in a particular context but the prediction is violated, memory for that item is weakened (Kim et al., 2014). Here, we explore what happens when such previously mispredicted items are later reencountered. According to prior neural network simulations, this sequence of events-misprediction and subsequent restudy-should lead to differentiation of the item's neural representation from the previous context (on which the misprediction was based). Specifically, misprediction weakens connections in the representation to features shared with the previous context and restudy allows new features to be incorporated into the representation that are not shared with the previous context. This cycle of misprediction and restudy should have the net effect of moving the item's neural representation away from the neural representation of the previous context. We tested this hypothesis using human fMRI by tracking changes in item-specific BOLD activity patterns in the hippocampus, a key structure for representing memories and generating predictions. In left CA2/3/DG, we found greater neural differentiation for items that were repeatedly mispredicted and restudied compared with items from a control condition that was identical except without misprediction. We also measured prediction strength in a trial-by-trial fashion and found that greater misprediction for an item led to more differentiation, further supporting our hypothesis. Therefore, the consequences of prediction error go beyond memory weakening. If the mispredicted item is restudied, the brain adaptively differentiates its memory representation to improve the accuracy of subsequent predictions and to shield it from further weakening.SIGNIFICANCE STATEMENT Competition between overlapping memories leads to weakening of nontarget memories over time, making it easier to access target memories. However, a nontarget memory in one context might become a target memory in another context. How do such memories get restrengthened without increasing competition again? Computational models suggest that the brain handles this by reducing neural connections to the previous context and adding connections to new features that were not part of the previous context. The result is neural differentiation away from the previous context. Here, we provide support for this theory, using fMRI to track neural representations of individual memories in the hippocampus and how they change based on learning.

A probabilistic approach to discovering dynamic full-brain functional connectivity patterns.

Recent research shows that the covariance structure of functional magnetic resonance imaging (fMRI) data - commonly described as functional connectivity - can change as a function of the participant's cognitive state (for review see Turk-Browne, 2013). Here we present a Bayesian hierarchical matrix factorization model, termed hierarchical topographic factor analysis (HTFA), for efficiently discovering full-brain networks in large multi-subject neuroimaging datasets. HTFA approximates each subject's network by first re-representing each brain image in terms of the activities of a set of localized nodes, and then computing the covariance of the activity time series of these nodes. The number of nodes, along with their locations, sizes, and activities (over time) are learned from the data. Because the number of nodes is typically substantially smaller than the number of fMRI voxels, HTFA can be orders of magnitude more efficient than traditional voxel-based functional connectivity approaches. In one case study, we show that HTFA recovers the known connectivity patterns underlying a collection of synthetic datasets. In a second case study, we illustrate how HTFA may be used to discover dynamic full-brain activity and connectivity patterns in real fMRI data, collected as participants listened to a story. In a third case study, we carried out a similar series of analyses on fMRI data collected as participants viewed an episode of a television show. In these latter case studies, we found that the HTFA-derived activity and connectivity patterns can be used to reliably decode which moments in the story or show the participants were experiencing. Further, we found that these two classes of patterns contained partially non-overlapping information, such that decoders trained on combinations of activity-based and dynamic connectivity-based features performed better than decoders trained on activity or connectivity patterns alone. We replicated this latter result with two additional (previously developed) methods for efficiently characterizing full-brain activity and connectivity patterns.

Mapping between fMRI responses to movies and their natural language annotations.

Several research groups have shown how to map fMRI responses to the meanings of presented stimuli. This paper presents new methods for doing so when only a natural language annotation is available as the description of the stimulus. We study fMRI data gathered from subjects watching an episode of BBCs Sherlock (Chen et al., 2017), and learn bidirectional mappings between fMRI responses and natural language representations. By leveraging data from multiple subjects watching the same movie, we were able to perform scene classification with 72% accuracy (random guessing would give 4%) and scene ranking with average rank in the top 4% (random guessing would give 50%). The key ingredients underlying this high level of performance are (a) the use of the Shared Response Model (SRM) and its variant SRM-ICA (Chen et al., 2015; Zhang et al., 2016) to aggregate fMRI data from multiple subjects, both of which are shown to be superior to standard PCA in producing low-dimensional representations for the tasks in this paper; (b) a sentence embedding technique adapted from the natural language processing (NLP) literature (Arora et al., 2017) that produces semantic vector representation of the annotations; (c) using previous timestep information in the featurization of the predictor data. These optimizations in how we featurize the fMRI data and text annotations provide a substantial improvement in classification performance, relative to standard approaches.