Reconstructing visual illusory experiences from human brain activity.

Visual illusions provide valuable insights into the brain's interpretation of the world given sensory inputs. However, the precise manner in which brain activity translates into illusory experiences remains largely unknown. Here, we leverage a brain decoding technique combined with deep neural network (DNN) representations to reconstruct illusory percepts as images from brain activity. The reconstruction model was trained on natural images to establish a link between brain activity and perceptual features and then tested on two types of illusions: illusory lines and neon color spreading. Reconstructions revealed lines and colors consistent with illusory experiences, which varied across the source visual cortical areas. This framework offers a way to materialize subjective experiences, shedding light on the brain's internal representations of the world.

Inter-individual deep image reconstruction via hierarchical neural code conversion.

The sensory cortex is characterized by general organizational principles such as topography and hierarchy. However, measured brain activity given identical input exhibits substantially different patterns across individuals. Although anatomical and functional alignment methods have been proposed in functional magnetic resonance imaging (fMRI) studies, it remains unclear whether and how hierarchical and fine-grained representations can be converted between individuals while preserving the encoded perceptual content. In this study, we trained a method of functional alignment called neural code converter that predicts a target subject's brain activity pattern from a source subject given the same stimulus, and analyzed the converted patterns by decoding hierarchical visual features and reconstructing perceived images. The converters were trained on fMRI responses to identical sets of natural images presented to pairs of individuals, using the voxels on the visual cortex that covers from V1 through the ventral object areas without explicit labels of the visual areas. We decoded the converted brain activity patterns into the hierarchical visual features of a deep neural network using decoders pre-trained on the target subject and then reconstructed images via the decoded features. Without explicit information about the visual cortical hierarchy, the converters automatically learned the correspondence between visual areas of the same levels. Deep neural network feature decoding at each layer showed higher decoding accuracies from corresponding levels of visual areas, indicating that hierarchical representations were preserved after conversion. The visual images were reconstructed with recognizable silhouettes of objects even with relatively small numbers of data for converter training. The decoders trained on pooled data from multiple individuals through conversions led to a slight improvement over those trained on a single individual. These results demonstrate that the hierarchical and fine-grained representation can be converted by functional alignment, while preserving sufficient visual information to enable inter-individual visual image reconstruction.

Multiple types of navigational information are independently encoded in the population activities of the dentate gyrus neurons.

The dentate gyrus (DG) plays critical roles in cognitive functions, such as learning, memory, and spatial coding, and its dysfunction is implicated in various neuropsychiatric disorders. However, it remains largely unknown how information is represented in this region. Here, we recorded neuronal activity in the DG using Ca2+ imaging in freely moving mice and analyzed this activity using machine learning. The activity patterns of populations of DG neurons enabled us to successfully decode position, speed, and motion direction in an open field, as well as current and future location in a T-maze, and each individual neuron was diversely and independently tuned to these multiple information types. Our data also showed that each type of information is unevenly distributed in groups of DG neurons, and different types of information are independently encoded in overlapping, but different, populations of neurons. In alpha-calcium/calmodulin-dependent kinase II (αCaMKII) heterozygous knockout mice, which present deficits in spatial remote and working memory, the decoding accuracy of position in the open field and future location in the T-maze were selectively reduced. These results suggest that multiple types of information are independently distributed in DG neurons.

Neurofeedback Training without Explicit Phantom Hand Movements and Hand-Like Visual Feedback to Modulate Pain: A Randomized Crossover Feasibility Trial.

Phantom limb pain is attributed to abnormal sensorimotor cortical representations, although the causal relationship between phantom limb pain and sensorimotor cortical representations suffers from the potentially confounding effects of phantom hand movements. We developed neurofeedback training to change sensorimotor cortical representations without explicit phantom hand movements or hand-like visual feedback. We tested the feasibility of neurofeedback training in fourteen patients with phantom limb pain. Neurofeedback training was performed in a single-blind, randomized, crossover trial using two decoders constructed using motor cortical currents measured during phantom hand movements; the motor cortical currents contralateral or ipsilateral to the phantom hand (contralateral and ipsilateral training) were estimated from magnetoencephalograms. Patients were instructed to control the size of a disk, which was proportional to the decoding results, but to not move their phantom hands or other body parts. The pain assessed by the visual analogue scale was significantly greater after contralateral training than after ipsilateral training. Classification accuracy of phantom hand movements significantly increased only after contralateral training. These results suggested that the proposed neurofeedback training changed phantom hand representation and modulated pain without explicit phantom hand movements or hand-like visual feedback, thus showing the relation between the phantom hand representations and pain. PERSPECTIVE: Our work demonstrates the feasibility of using neurofeedback training to change phantom hand representation and modulate pain perception without explicit phantom hand movements and hand-like visual feedback. The results enhance the mechanistic understanding of certain treatments, such as mirror therapy, that change the sensorimotor cortical representation.

Decoding distributed oscillatory signals driven by memory and perception in the prefrontal cortex.

Sensory perception and memory recall generate different conscious experiences. Although externally and internally driven neural activities signifying the same perceptual content overlap in the sensory cortex, their distribution in the prefrontal cortex (PFC), an area implicated in both perception and memory, remains elusive. Here, we test whether the local spatial configurations and frequencies of neural oscillations driven by perception and memory recall overlap in the macaque PFC using high-density electrocorticography and multivariate pattern analysis. We find that dynamically changing oscillatory signals distributed across the PFC in the delta-, theta-, alpha-, and beta-band ranges carry significant, but mutually different, information predicting the same feature of memory-recalled internal targets and passively perceived external objects. These findings suggest that the frequency-specific distribution of oscillatory neural signals in the PFC serves cortical signatures responsible for distinguishing between different types of cognition driven by external perception and internal memory.

Voluntary control of semantic neural representations by imagery with conflicting visual stimulation.

Neural representations of visual perception are affected by mental imagery and attention. Although attention is known to modulate neural representations, it is unknown how imagery changes neural representations when imagined and perceived images semantically conflict. We hypothesized that imagining an image would activate a neural representation during its perception even while watching a conflicting image. To test this hypothesis, we developed a closed-loop system to show images inferred from electrocorticograms using a visual semantic space. The successful control of the feedback images demonstrated that the semantic vector inferred from electrocorticograms became closer to the vector of the imagined category, even while watching images from different categories. Moreover, modulation of the inferred vectors by mental imagery depended asymmetrically on the perceived and imagined categories. Shared neural representation between mental imagery and perception was still activated by the imagery under semantically conflicting perceptions depending on the semantic category.

Attention modulates neural representation to render reconstructions according to subjective appearance.

Stimulus images can be reconstructed from visual cortical activity. However, our perception of stimuli is shaped by both stimulus-induced and top-down processes, and it is unclear whether and how reconstructions reflect top-down aspects of perception. Here, we investigate the effect of attention on reconstructions using fMRI activity measured while subjects attend to one of two superimposed images. A state-of-the-art method is used for image reconstruction, in which brain activity is translated (decoded) to deep neural network (DNN) features of hierarchical layers then to an image. Reconstructions resemble the attended rather than unattended images. They can be modeled by superimposed images with biased contrasts, comparable to the appearance during attention. Attentional modulations are found in a broad range of hierarchical visual representations and mirror the brain-DNN correspondence. Our results demonstrate that top-down attention counters stimulus-induced responses, modulating neural representations to render reconstructions in accordance with subjective appearance.

Natural and Artificial Intelligence: A brief introduction to the interplay between AI and neuroscience research.

Neuroscience and artificial intelligence (AI) share a long history of collaboration. Advances in neuroscience, alongside huge leaps in computer processing power over the last few decades, have given rise to a new generation of in silico neural networks inspired by the architecture of the brain. These AI systems are now capable of many of the advanced perceptual and cognitive abilities of biological systems, including object recognition and decision making. Moreover, AI is now increasingly being employed as a tool for neuroscience research and is transforming our understanding of brain functions. In particular, deep learning has been used to model how convolutional layers and recurrent connections in the brain's cerebral cortex control important functions, including visual processing, memory, and motor control. Excitingly, the use of neuroscience-inspired AI also holds great promise for understanding how changes in brain networks result in psychopathologies, and could even be utilized in treatment regimes. Here we discuss recent advancements in four areas in which the relationship between neuroscience and AI has led to major advancements in the field; (1) AI models of working memory, (2) AI visual processing, (3) AI analysis of big neuroscience datasets, and (4) computational psychiatry.

Brain hierarchy score: Which deep neural networks are hierarchically brain-like?

Achievement of human-level image recognition by deep neural networks (DNNs) has spurred interest in whether and how DNNs are brain-like. Both DNNs and the visual cortex perform hierarchical processing, and correspondence has been shown between hierarchical visual areas and DNN layers in representing visual features. Here, we propose the brain hierarchy (BH) score as a metric to quantify the degree of hierarchical correspondence based on neural decoding and encoding analyses where DNN unit activations and human brain activity are predicted from each other. We find that BH scores for 29 pre-trained DNNs with various architectures are negatively correlated with image recognition performance, thus indicating that recently developed high-performance DNNs are not necessarily brain-like. Experimental manipulations of DNN models suggest that single-path sequential feedforward architecture with broad spatial integration is critical to brain-like hierarchy. Our method may provide new ways to design DNNs in light of their representational homology to the brain.

fMRI lag structure during waking up from early sleep stages.

The brain mechanisms by which we transition from sleep to a conscious state remain largely unknown in humans, partly because of methodological challenges. Here we study a pre-existing dataset of waking up participants originally designed for a study of dreaming (Horikawa, Tamaki, Miyawaki, & Kamitani, 2013) and suggest that suddenly awakening from early sleep stages results from a two-stage process that involves a sequence of cortical and subcortical brain activity. First, subcortical and sensorimotor structures seem to be recruited before most cortical regions, followed by fast, ignition-like whole-brain activation-with frontal regions engaging a little after the rest of the brain. Second, a comparably slower and possibly mirror-reversed stage might take place, with cortical regions activating before subcortical structures and the cerebellum. This pattern of activation points to a key role of subcortical structures for the initiation and maintenance of conscious states.

BCI training to move a virtual hand reduces phantom limb pain: A randomized crossover trial.

OBJECTIVE: To determine whether training with a brain-computer interface (BCI) to control an image of a phantom hand, which moves based on cortical currents estimated from magnetoencephalographic signals, reduces phantom limb pain. METHODS: Twelve patients with chronic phantom limb pain of the upper limb due to amputation or brachial plexus root avulsion participated in a randomized single-blinded crossover trial. Patients were trained to move the virtual hand image controlled by the BCI with a real decoder, which was constructed to classify intact hand movements from motor cortical currents, by moving their phantom hands for 3 days ("real training"). Pain was evaluated using a visual analogue scale (VAS) before and after training, and at follow-up for an additional 16 days. As a control, patients engaged in the training with the same hand image controlled by randomly changing values ("random training"). The 2 trainings were randomly assigned to the patients. This trial is registered at UMIN-CTR (UMIN000013608). RESULTS: VAS at day 4 was significantly reduced from the baseline after real training (mean [SD], 45.3 [24.2]-30.9 [20.6], 1/100 mm; p = 0.009 < 0.025), but not after random training (p = 0.047 > 0.025). Compared to VAS at day 1, VAS at days 4 and 8 was significantly reduced by 32% and 36%, respectively, after real training and was significantly lower than VAS after random training (p < 0.01). CONCLUSION: Three-day training to move the hand images controlled by BCI significantly reduced pain for 1 week. CLASSIFICATION OF EVIDENCE: This study provides Class III evidence that BCI reduces phantom limb pain.

The Neural Representation of Visually Evoked Emotion Is High-Dimensional, Categorical, and Distributed across Transmodal Brain Regions.

Central to our subjective lives is the experience of different emotions. Recent behavioral work mapping emotional responses to 2,185 videos found that people experience upward of 27 distinct emotions occupying a high-dimensional space, and that emotion categories, more so than affective dimensions (e.g., valence), organize self-reports of subjective experience. Here, we sought to identify the neural substrates of this high-dimensional space of emotional experience using fMRI responses to all 2,185 videos. Our analyses demonstrated that (1) dozens of video-evoked emotions were accurately predicted from fMRI patterns in multiple brain regions with different regional configurations for individual emotions; (2) emotion categories better predicted cortical and subcortical responses than affective dimensions, outperforming visual and semantic covariates in transmodal regions; and (3) emotion-related fMRI responses had a cluster-like organization efficiently characterized by distinct categories. These results support an emerging theory of the high-dimensional emotion space, illuminating its neural foundations distributed across transmodal regions.

Peripheral blood metabolome predicts mood change-related activity in mouse model of bipolar disorder.

Bipolar disorder is a major mental illness characterized by severe swings in mood and activity levels which occur with variable amplitude and frequency. Attempts have been made to identify mood states and biological features associated with mood changes to compensate for current clinical diagnosis, which is mainly based on patients' subjective reports. Here, we used infradian (a cycle > 24 h) cyclic locomotor activity in a mouse model useful for the study of bipolar disorder as a proxy for mood changes. We show that metabolome patterns in peripheral blood could retrospectively predict the locomotor activity levels. We longitudinally monitored locomotor activity in the home cage, and subsequently collected peripheral blood and performed metabolomic analyses. We then constructed cross-validated linear regression models based on blood metabolome patterns to predict locomotor activity levels of individual mice. Our analysis revealed a significant correlation between actual and predicted activity levels, indicative of successful predictions. Pathway analysis of metabolites used for successful predictions showed enrichment in mitochondria metabolism-related terms, such as "Warburg effect" and "citric acid cycle." In addition, we found that peripheral blood metabolome patterns predicted expression levels of genes implicated in bipolar disorder in the hippocampus, a brain region responsible for mood regulation, suggesting that the brain-periphery axis is related to mood-change-associated behaviors. Our results may serve as a basis for predicting individual mood states through blood metabolomics in bipolar disorder and other mood disorders and may provide potential insight into systemic metabolic activity in relation to mood changes.

End-to-End Deep Image Reconstruction From Human Brain Activity.

Deep neural networks (DNNs) have recently been applied successfully to brain decoding and image reconstruction from functional magnetic resonance imaging (fMRI) activity. However, direct training of a DNN with fMRI data is often avoided because the size of available data is thought to be insufficient for training a complex network with numerous parameters. Instead, a pre-trained DNN usually serves as a proxy for hierarchical visual representations, and fMRI data are used to decode individual DNN features of a stimulus image using a simple linear model, which are then passed to a reconstruction module. Here, we directly trained a DNN model with fMRI data and the corresponding stimulus images to build an end-to-end reconstruction model. We accomplished this by training a generative adversarial network with an additional loss term that was defined in high-level feature space (feature loss) using up to 6,000 training data samples (natural images and fMRI responses). The above model was tested on independent datasets and directly reconstructed image using an fMRI pattern as the input. Reconstructions obtained from our proposed method resembled the test stimuli (natural and artificial images) and reconstruction accuracy increased as a function of training-data size. Ablation analyses indicated that the feature loss that we employed played a critical role in achieving accurate reconstruction. Our results show that the end-to-end model can learn a direct mapping between brain activity and perception.

Characterization of deep neural network features by decodability from human brain activity.

Achievements of near human-level performance in object recognition by deep neural networks (DNNs) have triggered a flood of comparative studies between the brain and DNNs. Using a DNN as a proxy for hierarchical visual representations, our recent study found that human brain activity patterns measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into DNN feature values given the same inputs. However, not all DNN features are equally decoded, indicating a gap between the DNN and human vision. Here, we present a dataset derived from DNN feature decoding analyses, which includes fMRI signals of five human subjects during image viewing, decoded feature values of DNNs (AlexNet and VGG19), and decoding accuracies of individual DNN features with their rankings. The decoding accuracies of individual features were highly correlated between subjects, suggesting the systematic differences between the brain and DNNs. We hope the present dataset will contribute to revealing the gap between the brain and DNNs and provide an opportunity to make use of the decoded features for further applications.

Deep image reconstruction from human brain activity.

The mental contents of perception and imagery are thought to be encoded in hierarchical representations in the brain, but previous attempts to visualize perceptual contents have failed to capitalize on multiple levels of the hierarchy, leaving it challenging to reconstruct internal imagery. Recent work showed that visual cortical activity measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into the hierarchical features of a pre-trained deep neural network (DNN) for the same input image, providing a way to make use of the information from hierarchical visual features. Here, we present a novel image reconstruction method, in which the pixel values of an image are optimized to make its DNN features similar to those decoded from human brain activity at multiple layers. We found that our method was able to reliably produce reconstructions that resembled the viewed natural images. A natural image prior introduced by a deep generator neural network effectively rendered semantically meaningful details to the reconstructions. Human judgment of the reconstructions supported the effectiveness of combining multiple DNN layers to enhance the visual quality of generated images. While our model was solely trained with natural images, it successfully generalized to artificial shapes, indicating that our model was not simply matching to exemplars. The same analysis applied to mental imagery demonstrated rudimentary reconstructions of the subjective content. Our results suggest that our method can effectively combine hierarchical neural representations to reconstruct perceptual and subjective images, providing a new window into the internal contents of the brain.

What you saw is what you will hear: Two new illusions with audiovisual postdictive effects.

Neuroscience investigations are most often focused on the prediction of future perception or decisions based on prior brain states or stimulus presentations. However, the brain can also process information retroactively, such that later stimuli impact conscious percepts of the stimuli that have already occurred (called "postdiction"). Postdictive effects have thus far been mostly unimodal (such as apparent motion), and the models for postdiction have accordingly been limited to early sensory regions of one modality. We have discovered two related multimodal illusions in which audition instigates postdictive changes in visual perception. In the first illusion (called the "Illusory Audiovisual Rabbit"), the location of an illusory flash is influenced by an auditory beep-flash pair that follows the perceived illusory flash. In the second illusion (called the "Invisible Audiovisual Rabbit"), a beep-flash pair following a real flash suppresses the perception of the earlier flash. Thus, we showed experimentally that these two effects are influenced significantly by postdiction. The audiovisual rabbit illusions indicate that postdiction can bridge the senses, uncovering a relatively-neglected yet critical type of neural processing underlying perceptual awareness. Furthermore, these two new illusions broaden the Double Flash Illusion, in which a single real flash is doubled by two sounds. Whereas the double flash indicated that audition can create an illusory flash, these rabbit illusions expand audition's influence on vision to the suppression of a real flash and the relocation of an illusory flash. These new additions to auditory-visual interactions indicate a spatio-temporally fine-tuned coupling of the senses to generate perception.

Sparse Ordinal Logistic Regression and Its Application to Brain Decoding.

Brain decoding with multivariate classification and regression has provided a powerful framework for characterizing information encoded in population neural activity. Classification and regression models are respectively used to predict discrete and continuous variables of interest. However, cognitive and behavioral parameters that we wish to decode are often ordinal variables whose values are discrete but ordered, such as subjective ratings. To date, there is no established method of predicting ordinal variables in brain decoding. In this study, we present a new algorithm, sparse ordinal logistic regression (SOLR), that combines ordinal logistic regression with Bayesian sparse weight estimation. We found that, in both simulation and analyses using real functional magnetic resonance imaging (fMRI) data, SOLR outperformed ordinal logistic regression with non-sparse regularization, indicating that sparseness leads to better decoding performance. SOLR also outperformed classification and linear regression models with the same type of sparseness, indicating the advantage of the modeling tailored to ordinal outputs. Our results suggest that SOLR provides a principled and effective method of decoding ordinal variables.

Training in Use of Brain-Machine Interface-Controlled Robotic Hand Improves Accuracy Decoding Two Types of Hand Movements.

Objective: Brain-machine interfaces (BMIs) are useful for inducing plastic changes in cortical representation. A BMI first decodes hand movements using cortical signals and then converts the decoded information into movements of a robotic hand. By using the BMI robotic hand, the cortical representation decoded by the BMI is modulated to improve decoding accuracy. We developed a BMI based on real-time magnetoencephalography (MEG) signals to control a robotic hand using decoded hand movements. Subjects were trained to use the BMI robotic hand freely for 10 min to evaluate plastic changes in the cortical representation due to the training. Method: We trained nine young healthy subjects with normal motor function. In open-loop conditions, they were instructed to grasp or open their right hands during MEG recording. Time-averaged MEG signals were then used to train a real decoder to control the robotic arm in real time. Then, subjects were instructed to control the BMI-controlled robotic hand by moving their right hands for 10 min while watching the robot's movement. During this closed-loop session, subjects tried to improve their ability to control the robot. Finally, subjects performed the same offline task to compare cortical activities related to the hand movements. As a control, we used a random decoder trained by the MEG signals with shuffled movement labels. We performed the same experiments with the random decoder as a crossover trial. To evaluate the cortical representation, cortical currents were estimated using a source localization technique. Hand movements were also decoded by a support vector machine using the MEG signals during the offline task. The classification accuracy of the movements was compared among offline tasks. Results: During the BMI training with the real decoder, the subjects succeeded in improving their accuracy in controlling the BMI robotic hand with correct rates of 0.28 ± 0.13 to 0.50 ± 0.11 (p = 0.017, n = 8, paired Student's t-test). Moreover, the classification accuracy of hand movements during the offline task was significantly increased after BMI training with the real decoder from 62.7 ± 6.5 to 70.0 ± 11.1% (p = 0.022, n = 8, t(7) = 2.93, paired Student's t-test), whereas accuracy did not significantly change after BMI training with the random decoder from 63.0 ± 8.8 to 66.4 ± 9.0% (p = 0.225, n = 8, t(7) = 1.33). Conclusion: BMI training is a useful tool to train the cortical activity necessary for BMI control and to induce some plastic changes in the activity.

MEG-BMI to Control Phantom Limb Pain.

A brachial plexus root avulsion (BPRA) causes intractable pain in the insensible affected hands. Such pain is partly due to phantom limb pain, which is neuropathic pain occurring after the amputation of a limb and partial or complete deafferentation. Previous studies suggested that the pain was attributable to maladaptive plasticity of the sensorimotor cortex. However, there is little evidence to demonstrate the causal links between the pain and the cortical representation, and how much cortical factors affect the pain. Here, we applied lesioning of the dorsal root entry zone (DREZotomy) and training with a brain-machine interface (BMI) based on real-time magnetoencephalography signals to reconstruct affected hand movements with a robotic hand. The DREZotomy successfully reduced the shooting pain after BPRA, but a part of the pain remained. The BMI training successfully induced some plastic changes in the sensorimotor representation of the phantom hand movements and helped control the remaining pain. When the patient tried to control the robotic hand by moving their phantom hand through association with the representation of the intact hand, this especially decreased the pain while decreasing the classification accuracy of the phantom hand movements. These results strongly suggested that pain after the BPRA was partly attributable to cortical representation of phantom hand movements and that the BMI training controlled the pain by inducing appropriate cortical reorganization. For the treatment of chronic pain, we need to know how to modulate the cortical representation by novel methods.