Semantic context-dependent neural representations of odors in the human piriform cortex revealed by 7T MRI.

Olfactory perception depends not only on olfactory inputs but also on semantic context. Although multi-voxel activity patterns of the piriform cortex, a part of the primary olfactory cortex, have been shown to represent odor perception, it remains unclear whether semantic contexts modulate odor representation in this region. Here, we investigated whether multi-voxel activity patterns in the piriform cortex change when semantic context modulates odor perception and, if so, whether the modulated areas communicate with brain regions involved in semantic and memory processing beyond the piriform cortex. We also explored regional differences within the piriform cortex, which are influenced by olfactory input and semantic context. We used 2 × 2 combinations of word labels and odorants that were perceived as congruent and measured piriform activity with a 1-mm isotropic resolution using 7T MRI. We found that identical odorants labeled with different words were perceived differently. This labeling effect was observed in multi-voxel activity patterns in the piriform cortex, as the searchlight decoding analysis distinguished identical odors with different labels for half of the examined stimulus pairs. Significant functional connectivity was observed between parts of the piriform cortex that were modulated by labels and regions associated with semantic and memory processing. While the piriform multi-voxel patterns evoked by different olfactory inputs were also distinguishable, the decoding accuracy was significant for only one stimulus pair, preventing definitive conclusions regarding the locational differences between areas influenced by word labels and olfactory inputs. These results suggest that multi-voxel patterns of piriform activity can be modulated by semantic context, possibly due to communication between the piriform cortex and the semantic and memory regions.

Homogenization of word relationships in schizophrenia: Topological analysis of cortical semantic representations.

AIM: Patients with schizophrenia typically exhibit symptoms of disorganized thought and display concreteness and over-inclusion in verbal reports, depending on the level of abstraction. While concreteness and over-inclusion may appear contradictory, the underlying psychopathology that explains these symptoms remains unclear. In the current study, we used functional magnetic resonance imaging with an encoding modeling approach to examine how concepts of various words, represented as brain activity, are anomalously connected at different levels of abstraction in patients with schizophrenia. METHODS: Fourteen individuals diagnosed with schizophrenia and 17 healthy controls underwent functional magnetic resonance imaging to measure brain activity representing concepts of various words. We used a persistent homology (PH) method to analyze the topological structures of word representations in schizophrenia patients, healthy controls, and random data, across different levels of abstraction by varying dissimilarity scales in the representation space. RESULTS: The results revealed that patients with schizophrenia exhibited more homogeneous word relationships across different levels of abstraction compared with healthy controls. Additionally, topological structures exhibited a shift toward a random network structure in patients with schizophrenia compared with controls. The PH method successfully distinguished semantic representations of patients with schizophrenia from those of controls. CONCLUSIONS: The current results provide an explanation for the mechanisms underlying the deficits in abstraction ability observed in schizophrenia. The isotopic connection of individual concepts reflects both the reduction of contextual connections at a semantically fine-grained scale and the absence of clear boundaries between related concepts at a coarse scale, which lead to concreteness and over-inclusion, respectively.

Artificial neural network modelling of the neural population code underlying mathematical operations.

Mathematical operations have long been regarded as a sparse, symbolic process in neuroimaging studies. In contrast, advances in artificial neural networks (ANN) have enabled extracting distributed representations of mathematical operations. Recent neuroimaging studies have compared distributed representations of the visual, auditory and language domains in ANNs and biological neural networks (BNNs). However, such a relationship has not yet been examined in mathematics. Here we hypothesise that ANN-based distributed representations can explain brain activity patterns of symbolic mathematical operations. We used the fMRI data of a series of mathematical problems with nine different combinations of operators to construct voxel-wise encoding/decoding models using both sparse operator and latent ANN features. Representational similarity analysis demonstrated shared representations between ANN and BNN, an effect particularly evident in the intraparietal sulcus. Feature-brain similarity (FBS) analysis served to reconstruct a sparse representation of mathematical operations based on distributed ANN features in each cortical voxel. Such reconstruction was more efficient when using features from deeper ANN layers. Moreover, latent ANN features allowed the decoding of novel operators not used during model training from brain activity. The current study provides novel insights into the neural code underlying mathematical thought.

Quantitative modelling demonstrates format-invariant representations of mathematical problems in the brain.

Mathematical problems can be described in either symbolic form or natural language. Previous studies have reported that activation overlaps exist for these two types of mathematical problems, but it is unclear whether they are based on similar brain representations. Furthermore, quantitative modelling of mathematical problem solving has yet to be attempted. In the present study, subjects underwent 3 h of functional magnetic resonance experiments involving math word and math expression problems, and a read word condition without any calculations was used as a control. To evaluate the brain representations of mathematical problems quantitatively, we constructed voxel-wise encoding models. Both intra- and cross-format encoding modelling significantly predicted brain activity predominantly in the left intraparietal sulcus (IPS), even after subtraction of the control condition. Representational similarity analysis and principal component analysis revealed that mathematical problems with different formats had similar cortical organization in the IPS. These findings support the idea that mathematical problems are represented in the brain in a format-invariant manner.

Brain networks are decoupled from external stimuli during internal cognition.

Our cognition can be directed to external stimuli or to internal information. While there are many different forms of internal cognition (mind-wandering, recall, imagery etc.), their essential feature is independence from the immediate sensory input, conceptually referred to as perceptual decoupling. Perceptual decoupling is thought to be reflected in brain activity transitioning from a stimulus-processing to internally-processing mode, but a direct investigation of this remains outstanding. Here we present a conceptual and analysis framework that quantifies the extent to which brain networks reflect stimulus processing. We tested this framework by presenting subjects with an audiovisual stimulus and instructing them to either attend to the stimulus (external task) or engage in mental imagery, recall or arithmetic (internal tasks) while measuring the evoked brain activity using functional MRI. We found that stimulus responses were generally attenuated for the internal tasks, though they increased in a subset of tasks and brain networks. However, using our new framework, we showed that brain networks became less reflective of stimulus processing, even in the subset of tasks and brain networks in which stimulus responses increased. These results quantitatively demonstrate that during internal cognition brain networks become decoupled from the external stimuli, opening the door for a fundamental and quantitative understanding of internal cognition.

Behavioral correlates of cortical semantic representations modeled by word vectors.

The quantitative modeling of semantic representations in the brain plays a key role in understanding the neural basis of semantic processing. Previous studies have demonstrated that word vectors, which were originally developed for use in the field of natural language processing, provide a powerful tool for such quantitative modeling. However, whether semantic representations in the brain revealed by the word vector-based models actually capture our perception of semantic information remains unclear, as there has been no study explicitly examining the behavioral correlates of the modeled brain semantic representations. To address this issue, we compared the semantic structure of nouns and adjectives in the brain estimated from word vector-based brain models with that evaluated from human behavior. The brain models were constructed using voxelwise modeling to predict the functional magnetic resonance imaging (fMRI) response to natural movies from semantic contents in each movie scene through a word vector space. The semantic dissimilarity of brain word representations was then evaluated using the brain models. Meanwhile, data on human behavior reflecting the perception of semantic dissimilarity between words were collected in psychological experiments. We found a significant correlation between brain model- and behavior-derived semantic dissimilarities of words. This finding suggests that semantic representations in the brain modeled via word vectors appropriately capture our perception of word meanings.

Convergence of Modality Invariance and Attention Selectivity in the Cortical Semantic Circuit.

The human linguistic system is characterized by modality invariance and attention selectivity. Previous studies have examined these properties independently and reported perisylvian region involvement for both; however, their relationship and the linguistic information they harbor remain unknown. Participants were assessed by functional magnetic resonance imaging, while spoken narratives (auditory) and written texts (visual) were presented, either separately or simultaneously. Participants were asked to attend to one stimulus when both were presented. We extracted phonemic and semantic features from these auditory and visual modalities, to train multiple, voxel-wise encoding models. Cross-modal examinations of the trained models revealed that perisylvian regions were associated with modality-invariant semantic representations. Attentional selectivity was quantified by examining the modeling performance for attended and unattended conditions. We have determined that perisylvian regions exhibited attention selectivity. Both modality invariance and attention selectivity are both prominent in models that use semantic but not phonemic features. Modality invariance was significantly correlated with attention selectivity in some brain regions; however, we also identified cortical regions associated with only modality invariance or only attention selectivity. Thus, paying selective attention to a specific sensory input modality may regulate the semantic information that is partly processed in brain networks that are shared across modalities.

Distinct dimensions of emotion in the human brain and their representation on the cortical surface.

We experience a rich variety of emotions in daily life, and a fundamental goal of affective neuroscience is to determine how these emotions are represented in the brain. Recent psychological studies have used naturalistic stimuli (e.g., movies) to reveal high dimensional representational structures of diverse daily-life emotions. However, relatively little is known about how such diverse emotions are represented in the brain because most of the affective neuroscience studies have used only a small number of controlled stimuli. To reveal that, we measured functional MRI to obtain blood-oxygen-level-dependent (BOLD) responses from human subjects while they watched emotion-inducing audiovisual movies over a period of 3 hours. For each of the one-second movie scenes, we annotated the movies with respect to 80 emotions selected based on a wide range of previous emotion literature. By quantifying canonical correlations between the emotion ratings and the BOLD responses, the results suggest that around 25 distinct dimensions (ranging from 18 to 36 and being subject-dependent) of the emotion ratings contribute to emotion representations in the brain. For demonstrating how the 80 emotion categories were represented in the cortical surface, we visualized a continuous semantic space of the emotion representation and mapped it on the cortical surface. We found that the emotion categories were changed from unimodal to transmodal regions on the cortical surface. This study presents a cortical representation of a rich variety of emotion categories, which covers many of the emotional experiences of daily living.

Decoding naturalistic experiences from human brain activity via distributed representations of words.

Natural visual scenes induce rich perceptual experiences that are highly diverse from scene to scene and from person to person. Here, we propose a new framework for decoding such experiences using a distributed representation of words. We used functional magnetic resonance imaging (fMRI) to measure brain activity evoked by natural movie scenes. Then, we constructed a high-dimensional feature space of perceptual experiences using skip-gram, a state-of-the-art distributed word embedding model. We built a decoder that associates brain activity with perceptual experiences via the distributed word representation. The decoder successfully estimated perceptual contents consistent with the scene descriptions by multiple annotators. Our results illustrate three advantages of our decoding framework: (1) three types of perceptual contents could be decoded in the form of nouns (objects), verbs (actions), and adjectives (impressions) contained in 10,000 vocabulary words; (2) despite using such a large vocabulary, we could decode novel words that were absent in the datasets to train the decoder; and (3) the inter-individual variability of the decoded contents co-varied with that of the contents of scene descriptions. These findings suggest that our decoding framework can recover diverse aspects of perceptual experiences in naturalistic situations and could be useful in various scientific and practical applications.

Mapping stimulus feature selectivity in macaque V1 by two-photon Ca2+ imaging: Encoding-model analysis of fluorescence responses to natural movies.

In vivo calcium (Ca2+) imaging using two-photon microscopy allows activity to be monitored simultaneously from hundreds of individual neurons within a local population. While this allows us to gain important insights into how cortical neurons represent sensory information, factors such as photo-bleaching of the Ca2+ indicator limit imaging duration (and thus the numbers of stimuli that can be tested), which in turn hampers the full characterization of neuronal response properties. Here, we demonstrate that using an encoding model combined with presentation of natural movies results in detailed characterization of receptive field (RF) properties despite the relatively short time for data collection. During presentation of natural movie clips to macaque monkeys, we recorded fluorescence signals from primary visual cortex (V1) neurons that had been loaded with a Ca2+ indicator. For each recorded neuron, we constructed an encoding model that comprised an array of motion-energy filters that tiled over the RFs. We optimized the weight of each filter's output so that the linear sum of the outputs across the filters mimicked the neuron's Ca2+-signal responses. These models were able to predict the neural responses to a different set of natural movies with a significant degree of accuracy. Moreover, the orientation tunings of neurons simulated by the model were highly correlated with those experimentally obtained when grating stimuli were presented to the monkeys. The model predictions were also consistent with what is known about spatial frequency tunings, the structure of excitatory subfields of RFs (i.e., classical RFs), and functional maps for these RF properties in V1. Further analysis revealed a new aspect of V1 functional architecture; the extent and distribution of suppressive RF subfields varied among nearby neurons, while those for excitatory subfields were shared. Thus, applying our encoding-model analysis to two-photon Ca2+ imaging of neuronal responses to natural movies provides a reliable and efficient means of analyzing a wide range of RF properties in multiple neurons imaged in a local region.

Functional Subdomains within Scene-Selective Cortex: Parahippocampal Place Area, Retrosplenial Complex, and Occipital Place Area.

Functional MRI studies suggest that at least three brain regions in human visual cortex-the parahippocampal place area (PPA), retrosplenial complex (RSC), and occipital place area (OPA; often called the transverse occipital sulcus)-represent large-scale information in natural scenes. Tuning of voxels within each region is often assumed to be functionally homogeneous. To test this assumption, we recorded blood oxygenation level-dependent responses during passive viewing of complex natural movies. We then used a voxelwise modeling framework to estimate voxelwise category tuning profiles within each scene-selective region. In all three regions, cluster analysis of the voxelwise tuning profiles reveals two functional subdomains that differ primarily in their responses to animals, man-made objects, social communication, and movement. Thus, the conventional functional definitions of the PPA, RSC, and OPA appear to be too coarse. One attractive hypothesis is that this consistent functional subdivision of scene-selective regions is a reflection of an underlying anatomical organization into two separate processing streams, one selectively biased toward static stimuli and one biased toward dynamic stimuli. SIGNIFICANCE STATEMENT: Visual scene perception is a critical ability to survive in the real world. It is therefore reasonable to assume that the human brain contains neural circuitry selective for visual scenes. Here we show that responses in three scene-selective areas-identified in previous studies-carry information about many object and action categories encountered in daily life. We identify two subregions in each area: one that is selective for categories of man-made objects, and another that is selective for vehicles and locomotion-related action categories that appear in dynamic scenes. This consistent functional subdivision may reflect an anatomical organization into two processing streams, one biased toward static stimuli and one biased toward dynamic stimuli.

Eye movement-invariant representations in the human visual system.

During natural vision, humans make frequent eye movements but perceive a stable visual world. It is therefore likely that the human visual system contains representations of the visual world that are invariant to eye movements. Here we present an experiment designed to identify visual areas that might contain eye-movement-invariant representations. We used functional MRI to record brain activity from four human subjects who watched natural movies. In one condition subjects were required to fixate steadily, and in the other they were allowed to freely make voluntary eye movements. The movies used in each condition were identical. We reasoned that the brain activity recorded in a visual area that is invariant to eye movement should be similar under fixation and free viewing conditions. In contrast, activity in a visual area that is sensitive to eye movement should differ between fixation and free viewing. We therefore measured the similarity of brain activity across repeated presentations of the same movie within the fixation condition, and separately between the fixation and free viewing conditions. The ratio of these measures was used to determine which brain areas are most likely to contain eye movement-invariant representations. We found that voxels located in early visual areas are strongly affected by eye movements, while voxels in ventral temporal areas are only weakly affected by eye movements. These results suggest that the ventral temporal visual areas contain a stable representation of the visual world that is invariant to eye movements made during natural vision.

Early monocular defocus disrupts the normal development of receptive-field structure in V2 neurons of macaque monkeys.

Experiencing different quality images in the two eyes soon after birth can cause amblyopia, a developmental vision disorder. Amblyopic humans show the reduced capacity for judging the relative position of a visual target in reference to nearby stimulus elements (position uncertainty) and often experience visual image distortion. Although abnormal pooling of local stimulus information by neurons beyond striate cortex (V1) is often suggested as a neural basis of these deficits, extrastriate neurons in the amblyopic brain have rarely been studied using microelectrode recording methods. The receptive field (RF) of neurons in visual area V2 in normal monkeys is made up of multiple subfields that are thought to reflect V1 inputs and are capable of encoding the spatial relationship between local stimulus features. We created primate models of anisometropic amblyopia and analyzed the RF subfield maps for multiple nearby V2 neurons of anesthetized monkeys by using dynamic two-dimensional noise stimuli and reverse correlation methods. Unlike in normal monkeys, the subfield maps of V2 neurons in amblyopic monkeys were severely disorganized: subfield maps showed higher heterogeneity within each neuron as well as across nearby neurons. Amblyopic V2 neurons exhibited robust binocular suppression and the strength of the suppression was positively correlated with the degree of hereogeneity and the severity of amblyopia in individual monkeys. Our results suggest that the disorganized subfield maps and robust binocular suppression of amblyopic V2 neurons are likely to adversely affect the higher stages of cortical processing resulting in position uncertainty and image distortion.

Functional subdomains within human FFA.

The fusiform face area (FFA) is a well-studied human brain region that shows strong activation for faces. In functional MRI studies, FFA is often assumed to be a homogeneous collection of voxels with similar visual tuning. To test this assumption, we used natural movies and a quantitative voxelwise modeling and decoding framework to estimate category tuning profiles for individual voxels within FFA. We find that the responses in most FFA voxels are strongly enhanced by faces, as reported in previous studies. However, we also find that responses of individual voxels are selectively enhanced or suppressed by a wide variety of other categories and that these broader tuning profiles differ across FFA voxels. Cluster analysis of category tuning profiles across voxels reveals three spatially segregated functional subdomains within FFA. These subdomains differ primarily in their responses for nonface categories, such as animals, vehicles, and communication verbs. Furthermore, this segregation does not depend on the statistical threshold used to define FFA from responses to functional localizers. These results suggest that voxels within FFA represent more diverse information about object and action categories than generally assumed.

Mental image reconstruction from human brain activity: Neural decoding of mental imagery via deep neural network-based Bayesian estimation.

Visual images observed by humans can be reconstructed from their brain activity. However, the visualization (externalization) of mental imagery is challenging. Only a few studies have reported successful visualization of mental imagery, and their visualizable images have been limited to specific domains such as human faces or alphabetical letters. Therefore, visualizing mental imagery for arbitrary natural images stands as a significant milestone. In this study, we achieved this by enhancing a previous method. Specifically, we demonstrated that the visual image reconstruction method proposed in the seminal study by Shen et al. (2019) heavily relied on low-level visual information decoded from the brain and could not efficiently utilize the semantic information that would be recruited during mental imagery. To address this limitation, we extended the previous method to a Bayesian estimation framework and introduced the assistance of semantic information into it. Our proposed framework successfully reconstructed both seen images (i.e., those observed by the human eye) and imagined images from brain activity. Quantitative evaluation showed that our framework could identify seen and imagined images highly accurately compared to the chance accuracy (seen: 90.7%, imagery: 75.6%, chance accuracy: 50.0%). In contrast, the previous method could only identify seen images (seen: 64.3%, imagery: 50.4%). These results suggest that our framework would provide a unique tool for directly investigating the subjective contents of the brain such as illusions, hallucinations, and dreams.

Text and image generation from intracranial electroencephalography using an embedding space for text and images.

Objective.Invasive brain-computer interfaces (BCIs) are promising communication devices for severely paralyzed patients. Recent advances in intracranial electroencephalography (iEEG) coupled with natural language processing have enhanced communication speed and accuracy. It should be noted that such a speech BCI uses signals from the motor cortex. However, BCIs based on motor cortical activities may experience signal deterioration in users with motor cortical degenerative diseases such as amyotrophic lateral sclerosis. An alternative approach to using iEEG of the motor cortex is necessary to support patients with such conditions.Approach. In this study, a multimodal embedding of text and images was used to decode visual semantic information from iEEG signals of the visual cortex to generate text and images. We used contrastive language-image pretraining (CLIP) embedding to represent images presented to 17 patients implanted with electrodes in the occipital and temporal cortices. A CLIP image vector was inferred from the high-γpower of the iEEG signals recorded while viewing the images.Main results.Text was generated by CLIPCAP from the inferred CLIP vector with better-than-chance accuracy. Then, an image was created from the generated text using StableDiffusion with significant accuracy.Significance.The text and images generated from iEEG through the CLIP embedding vector can be used for improved communication.

Disorganization of Semantic Brain Networks in Schizophrenia Revealed by fMRI.

OBJECTIVES: Schizophrenia is a mental illness that presents with thought disorders including delusions and disorganized speech. Thought disorders have been regarded as a consequence of the loosening of associations between semantic concepts since the term "schizophrenia" was first coined by Bleuler. However, a mechanistic account of this cardinal disturbance in terms of functional dysconnection has been lacking. To evaluate how aberrant semantic connections are expressed through brain activity, we characterized large-scale network structures of concept representations using functional magnetic resonance imaging (fMRI). STUDY DESIGN: We quantified various concept representations in patients' brains from fMRI activity evoked by movie scenes using encoding modeling. We then constructed semantic brain networks by evaluating the similarity of these semantic representations and conducted graph theory-based network analyses. STUDY RESULTS: Neurotypical networks had small-world properties similar to those of natural languages, suggesting small-worldness as a universal property in semantic knowledge networks. Conversely, small-worldness was significantly reduced in networks of schizophrenia patients and was correlated with psychological measures of delusions. Patients' semantic networks were partitioned into more distinct categories and had more random within-category structures than those of controls. CONCLUSIONS: The differences in conceptual representations manifest altered semantic clustering and associative intrusions that underlie thought disorders. This is the first study to provide pathophysiological evidence for the loosening of associations as reflected in randomization of semantic networks in schizophrenia. Our method provides a promising approach for understanding the neural basis of altered or creative inner experiences of individuals with mental illness or exceptional abilities, respectively.

A three-dimensional spatiotemporal receptive field model explains responses of area MT neurons to naturalistic movies.

Area MT has been an important target for studies of motion processing. However, previous neurophysiological studies of MT have used simple stimuli that do not contain many of the motion signals that occur during natural vision. In this study we sought to determine whether views of area MT neurons developed using simple stimuli can account for MT responses under more naturalistic conditions. We recorded responses from macaque area MT neurons during stimulation with naturalistic movies. We then used a quantitative modeling framework to discover which specific mechanisms best predict neuronal responses under these challenging conditions. We find that the simplest model that accurately predicts responses of MT neurons consists of a bank of V1-like filters, each followed by a compressive nonlinearity, a divisive nonlinearity, and linear pooling. Inspection of the fit models shows that the excitatory receptive fields of MT neurons tend to lie on a single plane within the three-dimensional spatiotemporal frequency domain, and suppressive receptive fields lie off this plane. However, most excitatory receptive fields form a partial ring in the plane and avoid low temporal frequencies. This receptive field organization ensures that most MT neurons are tuned for velocity but do not tend to respond to ambiguous static textures that are aligned with the direction of motion. In sum, MT responses to naturalistic movies are largely consistent with predictions based on simple stimuli. However, models fit using naturalistic stimuli reveal several novel properties of MT receptive fields that had not been shown in prior experiments.

Representations and decodability of diverse cognitive functions are preserved across the human cortex, cerebellum, and subcortex.

Which part of the brain contributes to our complex cognitive processes? Studies have revealed contributions of the cerebellum and subcortex to higher-order cognitive functions; however, it has been unclear whether such functional representations are preserved across the cortex, cerebellum, and subcortex. In this study, we use functional magnetic resonance imaging data with 103 cognitive tasks and construct three voxel-wise encoding and decoding models independently using cortical, cerebellar, and subcortical voxels. Representational similarity analysis reveals that the structure of task representations is preserved across the three brain parts. Principal component analysis visualizes distinct organizations of abstract cognitive functions in each part of the cerebellum and subcortex. More than 90% of the cognitive tasks are decodable from the cerebellum and subcortical activities, even for the novel tasks not included in model training. Furthermore, we show that the cerebellum and subcortex have sufficient information to reconstruct activity in the cerebral cortex.