Reconstructing visual illusory experiences from human brain activity.

Visual illusions provide valuable insights into the brain's interpretation of the world given sensory inputs. However, the precise manner in which brain activity translates into illusory experiences remains largely unknown. Here, we leverage a brain decoding technique combined with deep neural network (DNN) representations to reconstruct illusory percepts as images from brain activity. The reconstruction model was trained on natural images to establish a link between brain activity and perceptual features and then tested on two types of illusions: illusory lines and neon color spreading. Reconstructions revealed lines and colors consistent with illusory experiences, which varied across the source visual cortical areas. This framework offers a way to materialize subjective experiences, shedding light on the brain's internal representations of the world.

Diversity of aphantasia revealed by multiple assessments of visual imagery, multisensory imagery, and cognitive style.

Aphantasia-a condition wherein individuals have a reduced or absent construction of voluntary visual imagery-is diagnosed using either the Vividness of Visual Imagery Questionnaire (VVIQ) or self-identification. However, a significant discrepancy exists between the proportions of aphantasia in the populations assessed using these two criteria. It is unclear why the reported proportions differ excessively and what percentage of people cannot form visual imagery. We investigated the replicability of the proportion of people with aphantasia using both criteria in the same population of participants. Therefore, we explored the potential causes of the discrepancy and characteristics of putative aphantasia in terms of multisensory imagery, cognitive style, and face recognition ability. First, we conducted an online sampling study (Study 1: N = 2,871) using the VVIQ, self-identification of a reduction in visual imagery, Questionnaire upon Mental Imagery (QMI), and Verbalizer-Visualizer Questionnaire (VVQ). We found that 3.7 and 12.1% fulfilled the VVIQ and self-identification criteria, respectively, roughly replicating the proportions reported in previous studies. The self-identification criterion-but not the VVIQ criterion-contains items related to face recognition; hence, we suspected that face recognition ability was factor contributing to this discrepancy and conducted another online sampling study (Study 2: N = 774). We found a significant correlation between VVIQ and face recognition ability in the control group with self-identification, but not in the group defined by low VVIQ (VVIQ ≤32). As the participants in the control group with self-identification tended to exhibit moderately high VVIQ scores but low face recognition ability, we reason that the discrepancy can be partially explained by the contamination of individual differences in face recognition ability. Additional analyses of Study 1 revealed that the aphantasia group included participants who lacked all types of sensory imagery or only visual imagery in multisensory imagery and exhibited a non-specific cognitive style. This study indicates that the VVIQ alone may be insufficient to diagnose individuals who report an inability to form visual imagery. Furthermore, we highlight the importance of multiple assessments-along with the VVIQ-to better understand the diversity of imagery in aphantasia.

Inter-individual deep image reconstruction via hierarchical neural code conversion.

The sensory cortex is characterized by general organizational principles such as topography and hierarchy. However, measured brain activity given identical input exhibits substantially different patterns across individuals. Although anatomical and functional alignment methods have been proposed in functional magnetic resonance imaging (fMRI) studies, it remains unclear whether and how hierarchical and fine-grained representations can be converted between individuals while preserving the encoded perceptual content. In this study, we trained a method of functional alignment called neural code converter that predicts a target subject's brain activity pattern from a source subject given the same stimulus, and analyzed the converted patterns by decoding hierarchical visual features and reconstructing perceived images. The converters were trained on fMRI responses to identical sets of natural images presented to pairs of individuals, using the voxels on the visual cortex that covers from V1 through the ventral object areas without explicit labels of the visual areas. We decoded the converted brain activity patterns into the hierarchical visual features of a deep neural network using decoders pre-trained on the target subject and then reconstructed images via the decoded features. Without explicit information about the visual cortical hierarchy, the converters automatically learned the correspondence between visual areas of the same levels. Deep neural network feature decoding at each layer showed higher decoding accuracies from corresponding levels of visual areas, indicating that hierarchical representations were preserved after conversion. The visual images were reconstructed with recognizable silhouettes of objects even with relatively small numbers of data for converter training. The decoders trained on pooled data from multiple individuals through conversions led to a slight improvement over those trained on a single individual. These results demonstrate that the hierarchical and fine-grained representation can be converted by functional alignment, while preserving sufficient visual information to enable inter-individual visual image reconstruction.

Attention modulates neural representation to render reconstructions according to subjective appearance.

Stimulus images can be reconstructed from visual cortical activity. However, our perception of stimuli is shaped by both stimulus-induced and top-down processes, and it is unclear whether and how reconstructions reflect top-down aspects of perception. Here, we investigate the effect of attention on reconstructions using fMRI activity measured while subjects attend to one of two superimposed images. A state-of-the-art method is used for image reconstruction, in which brain activity is translated (decoded) to deep neural network (DNN) features of hierarchical layers then to an image. Reconstructions resemble the attended rather than unattended images. They can be modeled by superimposed images with biased contrasts, comparable to the appearance during attention. Attentional modulations are found in a broad range of hierarchical visual representations and mirror the brain-DNN correspondence. Our results demonstrate that top-down attention counters stimulus-induced responses, modulating neural representations to render reconstructions in accordance with subjective appearance.

fMRI lag structure during waking up from early sleep stages.

The brain mechanisms by which we transition from sleep to a conscious state remain largely unknown in humans, partly because of methodological challenges. Here we study a pre-existing dataset of waking up participants originally designed for a study of dreaming (Horikawa, Tamaki, Miyawaki, & Kamitani, 2013) and suggest that suddenly awakening from early sleep stages results from a two-stage process that involves a sequence of cortical and subcortical brain activity. First, subcortical and sensorimotor structures seem to be recruited before most cortical regions, followed by fast, ignition-like whole-brain activation-with frontal regions engaging a little after the rest of the brain. Second, a comparably slower and possibly mirror-reversed stage might take place, with cortical regions activating before subcortical structures and the cerebellum. This pattern of activation points to a key role of subcortical structures for the initiation and maintenance of conscious states.

The Neural Representation of Visually Evoked Emotion Is High-Dimensional, Categorical, and Distributed across Transmodal Brain Regions.

Central to our subjective lives is the experience of different emotions. Recent behavioral work mapping emotional responses to 2,185 videos found that people experience upward of 27 distinct emotions occupying a high-dimensional space, and that emotion categories, more so than affective dimensions (e.g., valence), organize self-reports of subjective experience. Here, we sought to identify the neural substrates of this high-dimensional space of emotional experience using fMRI responses to all 2,185 videos. Our analyses demonstrated that (1) dozens of video-evoked emotions were accurately predicted from fMRI patterns in multiple brain regions with different regional configurations for individual emotions; (2) emotion categories better predicted cortical and subcortical responses than affective dimensions, outperforming visual and semantic covariates in transmodal regions; and (3) emotion-related fMRI responses had a cluster-like organization efficiently characterized by distinct categories. These results support an emerging theory of the high-dimensional emotion space, illuminating its neural foundations distributed across transmodal regions.

Peripheral blood metabolome predicts mood change-related activity in mouse model of bipolar disorder.

Bipolar disorder is a major mental illness characterized by severe swings in mood and activity levels which occur with variable amplitude and frequency. Attempts have been made to identify mood states and biological features associated with mood changes to compensate for current clinical diagnosis, which is mainly based on patients' subjective reports. Here, we used infradian (a cycle > 24 h) cyclic locomotor activity in a mouse model useful for the study of bipolar disorder as a proxy for mood changes. We show that metabolome patterns in peripheral blood could retrospectively predict the locomotor activity levels. We longitudinally monitored locomotor activity in the home cage, and subsequently collected peripheral blood and performed metabolomic analyses. We then constructed cross-validated linear regression models based on blood metabolome patterns to predict locomotor activity levels of individual mice. Our analysis revealed a significant correlation between actual and predicted activity levels, indicative of successful predictions. Pathway analysis of metabolites used for successful predictions showed enrichment in mitochondria metabolism-related terms, such as "Warburg effect" and "citric acid cycle." In addition, we found that peripheral blood metabolome patterns predicted expression levels of genes implicated in bipolar disorder in the hippocampus, a brain region responsible for mood regulation, suggesting that the brain-periphery axis is related to mood-change-associated behaviors. Our results may serve as a basis for predicting individual mood states through blood metabolomics in bipolar disorder and other mood disorders and may provide potential insight into systemic metabolic activity in relation to mood changes.

End-to-End Deep Image Reconstruction From Human Brain Activity.

Deep neural networks (DNNs) have recently been applied successfully to brain decoding and image reconstruction from functional magnetic resonance imaging (fMRI) activity. However, direct training of a DNN with fMRI data is often avoided because the size of available data is thought to be insufficient for training a complex network with numerous parameters. Instead, a pre-trained DNN usually serves as a proxy for hierarchical visual representations, and fMRI data are used to decode individual DNN features of a stimulus image using a simple linear model, which are then passed to a reconstruction module. Here, we directly trained a DNN model with fMRI data and the corresponding stimulus images to build an end-to-end reconstruction model. We accomplished this by training a generative adversarial network with an additional loss term that was defined in high-level feature space (feature loss) using up to 6,000 training data samples (natural images and fMRI responses). The above model was tested on independent datasets and directly reconstructed image using an fMRI pattern as the input. Reconstructions obtained from our proposed method resembled the test stimuli (natural and artificial images) and reconstruction accuracy increased as a function of training-data size. Ablation analyses indicated that the feature loss that we employed played a critical role in achieving accurate reconstruction. Our results show that the end-to-end model can learn a direct mapping between brain activity and perception.

Characterization of deep neural network features by decodability from human brain activity.

Achievements of near human-level performance in object recognition by deep neural networks (DNNs) have triggered a flood of comparative studies between the brain and DNNs. Using a DNN as a proxy for hierarchical visual representations, our recent study found that human brain activity patterns measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into DNN feature values given the same inputs. However, not all DNN features are equally decoded, indicating a gap between the DNN and human vision. Here, we present a dataset derived from DNN feature decoding analyses, which includes fMRI signals of five human subjects during image viewing, decoded feature values of DNNs (AlexNet and VGG19), and decoding accuracies of individual DNN features with their rankings. The decoding accuracies of individual features were highly correlated between subjects, suggesting the systematic differences between the brain and DNNs. We hope the present dataset will contribute to revealing the gap between the brain and DNNs and provide an opportunity to make use of the decoded features for further applications.

Deep image reconstruction from human brain activity.

The mental contents of perception and imagery are thought to be encoded in hierarchical representations in the brain, but previous attempts to visualize perceptual contents have failed to capitalize on multiple levels of the hierarchy, leaving it challenging to reconstruct internal imagery. Recent work showed that visual cortical activity measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into the hierarchical features of a pre-trained deep neural network (DNN) for the same input image, providing a way to make use of the information from hierarchical visual features. Here, we present a novel image reconstruction method, in which the pixel values of an image are optimized to make its DNN features similar to those decoded from human brain activity at multiple layers. We found that our method was able to reliably produce reconstructions that resembled the viewed natural images. A natural image prior introduced by a deep generator neural network effectively rendered semantically meaningful details to the reconstructions. Human judgment of the reconstructions supported the effectiveness of combining multiple DNN layers to enhance the visual quality of generated images. While our model was solely trained with natural images, it successfully generalized to artificial shapes, indicating that our model was not simply matching to exemplars. The same analysis applied to mental imagery demonstrated rudimentary reconstructions of the subjective content. Our results suggest that our method can effectively combine hierarchical neural representations to reconstruct perceptual and subjective images, providing a new window into the internal contents of the brain.

Generic decoding of seen and imagined objects using hierarchical visual features.

Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. We present a decoding approach for arbitrary objects using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features, including those derived from a deep convolutional neural network, can be predicted from fMRI patterns, and that greater accuracy is achieved for low-/high-level features with lower-/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, decoding of imagined objects reveals progressive recruitment of higher-to-lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.

Position Information Encoded by Population Activity in Hierarchical Visual Areas.

Neurons in high-level visual areas respond to more complex visual features with broader receptive fields (RFs) compared to those in low-level visual areas. Thus, high-level visual areas are generally considered to carry less information regarding the position of seen objects in the visual field. However, larger RFs may not imply loss of position information at the population level. Here, we evaluated how accurately the position of a seen object could be predicted (decoded) from activity patterns in each of six representative visual areas with different RF sizes [V1-V4, lateral occipital complex (LOC), and fusiform face area (FFA)]. We collected functional magnetic resonance imaging (fMRI) responses while human subjects viewed a ball randomly moving in a two-dimensional field. To estimate population RF sizes of individual fMRI voxels, RF models were fitted for individual voxels in each brain area. The voxels in higher visual areas showed larger estimated RFs than those in lower visual areas. Then, the ball's position in a separate session was predicted by maximum likelihood estimation using the RF models of individual voxels. We also tested a model-free multivoxel regression (support vector regression, SVR) to predict the position. We found that regardless of the difference in RF size, all visual areas showed similar prediction accuracies, especially on the horizontal dimension. Higher areas showed slightly lower accuracies on the vertical dimension, which appears to be attributed to the narrower spatial distributions of the RF centers. The results suggest that much position information is preserved in population activity through the hierarchical visual pathway regardless of RF sizes and is potentially available in later processing for recognition and behavior.

Hierarchical Neural Representation of Dreamed Objects Revealed by Brain Decoding with Deep Neural Network Features.

Dreaming is generally thought to be generated by spontaneous brain activity during sleep with patterns common to waking experience. This view is supported by a recent study demonstrating that dreamed objects can be predicted from brain activity during sleep using statistical decoders trained with stimulus-induced brain activity. However, it remains unclear whether and how visual image features associated with dreamed objects are represented in the brain. In this study, we used a deep neural network (DNN) model for object recognition as a proxy for hierarchical visual feature representation, and DNN features for dreamed objects were analyzed with brain decoding of fMRI data collected during dreaming. The decoders were first trained with stimulus-induced brain activity labeled with the feature values of the stimulus image from multiple DNN layers. The decoders were then used to decode DNN features from the dream fMRI data, and the decoded features were compared with the averaged features of each object category calculated from a large-scale image database. We found that the feature values decoded from the dream fMRI data positively correlated with those associated with dreamed object categories at mid- to high-level DNN layers. Using the decoded features, the dreamed object category could be identified at above-chance levels by matching them to the averaged features for candidate categories. The results suggest that dreaming recruits hierarchical visual feature representations associated with objects, which may support phenomenal aspects of dream experience.

Circadian Gene Circuitry Predicts Hyperactive Behavior in a Mood Disorder Mouse Model.

Bipolar disorder, also known as manic-depressive illness, causes swings in mood and activity levels at irregular intervals. Such changes are difficult to predict, and their molecular basis remains unknown. Here, we use infradian (longer than a day) cyclic activity levels in αCaMKII (Camk2a) mutant mice as a proxy for such mood-associated changes. We report that gene-expression patterns in the hippocampal dentate gyrus could retrospectively predict whether the mice were in a state of high or low locomotor activity (LA). Expression of a subset of circadian genes, as well as levels of cAMP and pCREB, possible upstream regulators of circadian genes, were correlated with LA states, suggesting that the intrinsic molecular circuitry changes concomitant with infradian oscillatory LA. Taken together, these findings shed light onto the molecular basis of how irregular biological rhythms and behavior are controlled by the brain.

[Exploring dream contents by neuroimaging].

Dreaming is a subjective experience during sleep that is often accompanied by vivid perceptual and emotional contents. Because of its fundamentally subjective nature, the objective study of dream contents has been challenging. However, since the discovery of rapid eye movements during sleep, scientific knowledge on the relationship between dreaming and physiological measures including brain activity has accumulated. Recent advances in neuroimaging analysis methods have made it possible to uncover direct links between specific dream contents and brain activity patterns. In this review, we first give a historical overview on dream researches with a focus on the neurophysiological and behavioral signatures of dreaming. We then discuss our recent study in which visual dream contents were predicted, or decoded, from brain activity during sleep onset periods using machine learning-based pattern recognition of functional MRI data. We suggest that advanced analytical tools combined with neural and behavioral databases will reveal the relevance of spontaneous brain activity during sleep to waking experiences.