Multiple types of navigational information are independently encoded in the population activities of the dentate gyrus neurons.

The dentate gyrus (DG) plays critical roles in cognitive functions, such as learning, memory, and spatial coding, and its dysfunction is implicated in various neuropsychiatric disorders. However, it remains largely unknown how information is represented in this region. Here, we recorded neuronal activity in the DG using Ca2+ imaging in freely moving mice and analyzed this activity using machine learning. The activity patterns of populations of DG neurons enabled us to successfully decode position, speed, and motion direction in an open field, as well as current and future location in a T-maze, and each individual neuron was diversely and independently tuned to these multiple information types. Our data also showed that each type of information is unevenly distributed in groups of DG neurons, and different types of information are independently encoded in overlapping, but different, populations of neurons. In alpha-calcium/calmodulin-dependent kinase II (αCaMKII) heterozygous knockout mice, which present deficits in spatial remote and working memory, the decoding accuracy of position in the open field and future location in the T-maze were selectively reduced. These results suggest that multiple types of information are independently distributed in DG neurons.

Inter-individual deep image reconstruction via hierarchical neural code conversion.

The sensory cortex is characterized by general organizational principles such as topography and hierarchy. However, measured brain activity given identical input exhibits substantially different patterns across individuals. Although anatomical and functional alignment methods have been proposed in functional magnetic resonance imaging (fMRI) studies, it remains unclear whether and how hierarchical and fine-grained representations can be converted between individuals while preserving the encoded perceptual content. In this study, we trained a method of functional alignment called neural code converter that predicts a target subject's brain activity pattern from a source subject given the same stimulus, and analyzed the converted patterns by decoding hierarchical visual features and reconstructing perceived images. The converters were trained on fMRI responses to identical sets of natural images presented to pairs of individuals, using the voxels on the visual cortex that covers from V1 through the ventral object areas without explicit labels of the visual areas. We decoded the converted brain activity patterns into the hierarchical visual features of a deep neural network using decoders pre-trained on the target subject and then reconstructed images via the decoded features. Without explicit information about the visual cortical hierarchy, the converters automatically learned the correspondence between visual areas of the same levels. Deep neural network feature decoding at each layer showed higher decoding accuracies from corresponding levels of visual areas, indicating that hierarchical representations were preserved after conversion. The visual images were reconstructed with recognizable silhouettes of objects even with relatively small numbers of data for converter training. The decoders trained on pooled data from multiple individuals through conversions led to a slight improvement over those trained on a single individual. These results demonstrate that the hierarchical and fine-grained representation can be converted by functional alignment, while preserving sufficient visual information to enable inter-individual visual image reconstruction.

Deep image reconstruction from human brain activity.

The mental contents of perception and imagery are thought to be encoded in hierarchical representations in the brain, but previous attempts to visualize perceptual contents have failed to capitalize on multiple levels of the hierarchy, leaving it challenging to reconstruct internal imagery. Recent work showed that visual cortical activity measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into the hierarchical features of a pre-trained deep neural network (DNN) for the same input image, providing a way to make use of the information from hierarchical visual features. Here, we present a novel image reconstruction method, in which the pixel values of an image are optimized to make its DNN features similar to those decoded from human brain activity at multiple layers. We found that our method was able to reliably produce reconstructions that resembled the viewed natural images. A natural image prior introduced by a deep generator neural network effectively rendered semantically meaningful details to the reconstructions. Human judgment of the reconstructions supported the effectiveness of combining multiple DNN layers to enhance the visual quality of generated images. While our model was solely trained with natural images, it successfully generalized to artificial shapes, indicating that our model was not simply matching to exemplars. The same analysis applied to mental imagery demonstrated rudimentary reconstructions of the subjective content. Our results suggest that our method can effectively combine hierarchical neural representations to reconstruct perceptual and subjective images, providing a new window into the internal contents of the brain.

Heterogeneous Redistribution of Facial Subcategory Information Within and Outside the Face-Selective Domain in Primate Inferior Temporal Cortex.

The inferior temporal cortex (ITC) contains neurons selective to multiple levels of visual categories. However, the mechanisms by which these neurons collectively construct hierarchical category percepts remain unclear. By comparing decoding accuracy with simultaneously acquired electrocorticogram (ECoG), local field potentials (LFPs), and multi-unit activity in the macaque ITC, we show that low-frequency LFPs/ECoG in the early evoked visual response phase contain sufficient coarse category (e.g., face) information, which is homogeneous and enhanced by spatial summation of up to several millimeters. Late-induced high-frequency LFPs additionally carry spike-coupled finer category (e.g., species, view, and identity of the face) information, which is heterogeneous and reduced by spatial summation. Face-encoding neural activity forms a cluster in similar cortical locations regardless of whether it is defined by early evoked low-frequency signals or late-induced high-gamma signals. By contrast, facial subcategory-encoding activity is distributed, not confined to the face cluster, and dynamically increases its heterogeneity from the early evoked to late-induced phases. These findings support a view that, in contrast to the homogeneous and static coarse category-encoding neural cluster, finer category-encoding clusters are heterogeneously distributed even outside their parent category cluster and dynamically increase heterogeneity along with the local cortical processing in the ITC.

Alternating zones selective to faces and written words in the human ventral occipitotemporal cortex.

Recognition of faces and written words is associated with category-specific brain activation in the ventral occipitotemporal cortex (vOT). However, topological and functional relationships between face-selective and word-selective vOT regions remain unclear. In this study, we collected data from patients with intractable epilepsy who underwent high-density recording of surface field potentials in the vOT. "Faces" and "letterstrings" induced outstanding category-selective responses among the 24 visual categories tested, particularly in high-γ band powers. Strikingly, within-hemispheric analysis revealed alternation of face-selective and letterstring-selective zones within the vOT. Two distinct face-selective zones located anterior and posterior portions of the mid-fusiform sulcus whereas letterstring-selective zones alternated between and outside of these 2 face-selective zones. Further, a classification analysis indicated that activity patterns of these zones mostly represent dedicated categories. Functional connectivity analysis using Granger causality indicated asymmetrically directed causal influences from face-selective to letterstring-selective regions. These results challenge the prevailing view that different categories are represented in distinct contiguous regions in the vOT.

Inter-subject neural code converter for visual image representation.

Brain activity patterns differ from person to person, even for an identical stimulus. In functional brain mapping studies, it is important to align brain activity patterns between subjects for group statistical analyses. While anatomical templates are widely used for inter-subject alignment in functional magnetic resonance imaging (fMRI) studies, they are not sufficient to identify the mapping between voxel-level functional responses representing specific mental contents. Recent work has suggested that statistical learning methods could be used to transform individual brain activity patterns into a common space while preserving representational contents. Here, we propose a flexible method for functional alignment, "neural code converter," which converts one subject's brain activity pattern into another's representing the same content. The neural code converter was designed to learn statistical relationships between fMRI activity patterns of paired subjects obtained while they saw an identical series of stimuli. It predicts the signal intensity of individual voxels of one subject from a pattern of multiple voxels of the other subject. To test this method, we used fMRI activity patterns measured while subjects observed visual images consisting of random and structured patches. We show that fMRI activity patterns for visual images not used for training the converter could be predicted from those of another subject where brain activity was recorded for the same stimuli. This confirms that visual images can be accurately reconstructed from the predicted activity patterns alone. Furthermore, we show that a classifier trained only on predicted fMRI activity patterns could accurately classify measured fMRI activity patterns. These results demonstrate that the neural code converter can translate neural codes between subjects while preserving contents related to visual images. While this method is useful for functional alignment and decoding, it may also provide a basis for brain-to-brain communication using the converted pattern for designing brain stimulation.

Decoding visual object categories from temporal correlations of ECoG signals.

How visual object categories are represented in the brain is one of the key questions in neuroscience. Studies on low-level visual features have shown that relative timings or phases of neural activity between multiple brain locations encode information. However, whether such temporal patterns of neural activity are used in the representation of visual objects is unknown. Here, we examined whether and how visual object categories could be predicted (or decoded) from temporal patterns of electrocorticographic (ECoG) signals from the temporal cortex in five patients with epilepsy. We used temporal correlations between electrodes as input features, and compared the decoding performance with features defined by spectral power and phase from individual electrodes. While using power or phase alone, the decoding accuracy was significantly better than chance, correlations alone or those combined with power outperformed other features. Decoding performance with correlations was degraded by shuffling the order of trials of the same category in each electrode, indicating that the relative time series between electrodes in each trial is critical. Analysis using a sliding time window revealed that decoding performance with correlations began to rise earlier than that with power. This earlier increase in performance was replicated by a model using phase differences to encode categories. These results suggest that activity patterns arising from interactions between multiple neuronal units carry additional information on visual object categories.

How emotion is experienced and expressed in multiple cultures: a large-scale experiment across North America, Europe, and Japan.

Core to understanding emotion are subjective experiences and their expression in facial behavior. Past studies have largely focused on six emotions and prototypical facial poses, reflecting limitations in scale and narrow assumptions about the variety of emotions and their patterns of expression. We examine 45,231 facial reactions to 2,185 evocative videos, largely in North America, Europe, and Japan, collecting participants' self-reported experiences in English or Japanese and manual and automated annotations of facial movement. Guided by Semantic Space Theory, we uncover 21 dimensions of emotion in the self-reported experiences of participants in Japan, the United States, and Western Europe, and considerable cross-cultural similarities in experience. Facial expressions predict at least 12 dimensions of experience, despite massive individual differences in experience. We find considerable cross-cultural convergence in the facial actions involved in the expression of emotion, and culture-specific display tendencies-many facial movements differ in intensity in Japan compared to the U.S./Canada and Europe but represent similar experiences. These results quantitatively detail that people in dramatically different cultures experience and express emotion in a high-dimensional, categorical, and similar but complex fashion.

Regulation of motor representation by phase-amplitude coupling in the sensorimotor cortex.

High-γ amplitude (80-150 Hz) represents motor information, such as movement types, on the sensorimotor cortex. In several cortical areas, high-γ amplitudes are coupled with low-frequency phases, e.g., α and θ (phase-amplitude coupling, PAC). However, such coupling has not been studied in the sensorimotor cortex; thus, its potential functional role has yet to be explored. We investigated PAC of high-γ amplitude in the sensorimotor cortex during waiting for and the execution of movements using electrocorticographic (ECoG) recordings in humans. ECoG signals were recorded from the sensorimotor cortices of 4 epilepsy patients while they performed three different hand movements. A subset of electrodes showed high-γ activity selective to movement type around the timing of motor execution, while the same electrodes showed nonselective high-γ activity during the waiting period (>2 s before execution). Cross frequency coupling analysis revealed that the high-γ amplitude during waiting was strongly coupled with the α phase (10-14 Hz) at the electrodes with movement-selective high-γ amplitudes during execution. This coupling constituted the high-γ amplitude peaking around the trough of the α oscillation, and its strength and phase were not predictive of movement type. As the coupling attenuated toward the timing of motor execution, the high-γ amplitude appeared to be released from the α phase to build a motor representation with phase-independent activity. Our results suggest that PAC modulates motor representation in the sensorimotor cortex by holding and releasing high-γ activity in movement-selective cortical regions.

Electrocorticographic control of a prosthetic arm in paralyzed patients.

OBJECTIVE: Paralyzed patients may benefit from restoration of movement afforded by prosthetics controlled by electrocorticography (ECoG). Although ECoG shows promising results in human volunteers, it is unclear whether ECoG signals recorded from chronically paralyzed patients provide sufficient motor information, and if they do, whether they can be applied to control a prosthetic. METHODS: We recorded ECoG signals from sensorimotor cortices of 12 patients while they executed or attempted to execute 3 to 5 simple hand and elbow movements. Sensorimotor function was severely impaired in 3 patients due to peripheral nervous system lesion or amputation, moderately impaired due to central nervous system lesions sparing the cortex in 4 patients, and normal in 5 patients. Time frequency and decoding analyses were performed with the patients' ECoG signals. RESULTS: In all patients, the high gamma power (80-150 Hz) of the ECoG signals during movements was clearly responsive to movement types and provided the best information for classifying different movement types. The classification performance was significantly better than chance in all patients, although differences between ECoG power modulations during different movement types were significantly less in patients with severely impaired motor function. In the impaired patients, cortical representations tended to overlap each other. Finally, using the classification method in real time, a moderately impaired patient and 3 nonparalyzed patients successfully controlled a prosthetic arm. INTERPRETATION: ECoG signals appear useful for prosthetic arm control and may provide clinically feasible motor restoration for patients with paralysis but no injury of the sensorimotor cortex.

Modular encoding and decoding models derived from bayesian canonical correlation analysis.

Neural encoding and decoding provide perspectives for understanding neural representations of sensory inputs. Recent functional magnetic resonance imaging (fMRI) studies have succeeded in building prediction models for encoding and decoding numerous stimuli by representing a complex stimulus as a combination of simple elements. While arbitrary visual images were reconstructed using a modular model that combined the outputs of decoder modules for multiscale local image bases (elements), the shapes of the image bases were heuristically determined. In this work, we propose a method to establish mappings between the stimulus and the brain by automatically extracting modules from measured data. We develop a model based on Bayesian canonical correlation analysis, in which each module is modeled by a latent variable that relates a set of pixels in a visual image to a set of voxels in an fMRI activity pattern. The estimated mapping from a latent variable to pixels can be regarded as an image basis. We show that the model estimates a modular representation with spatially localized multiscale image bases. Further, using the estimated mappings, we derive encoding and decoding models that produce accurate predictions for brain activity and stimulus images. Our approach thus provides a novel means of revealing neural representations of stimuli by automatically extracting modules, which can be used to generate effective prediction models for encoding and decoding.

Reconstructing visual illusory experiences from human brain activity.

Visual illusions provide valuable insights into the brain's interpretation of the world given sensory inputs. However, the precise manner in which brain activity translates into illusory experiences remains largely unknown. Here, we leverage a brain decoding technique combined with deep neural network (DNN) representations to reconstruct illusory percepts as images from brain activity. The reconstruction model was trained on natural images to establish a link between brain activity and perceptual features and then tested on two types of illusions: illusory lines and neon color spreading. Reconstructions revealed lines and colors consistent with illusory experiences, which varied across the source visual cortical areas. This framework offers a way to materialize subjective experiences, shedding light on the brain's internal representations of the world.

Relationship between BOLD amplitude and pattern classification of orientation-selective activity in the human visual cortex.

Orientation-selective responses can be decoded from fMRI activity patterns in the human visual cortex, using multivariate pattern analysis (MVPA). To what extent do these feature-selective activity patterns depend on the strength and quality of the sensory input, and might the reliability of these activity patterns be predicted by the gross amplitude of the stimulus-driven BOLD response? Observers viewed oriented gratings that varied in luminance contrast (4, 20 or 100%) or spatial frequency (0.25, 1.0 or 4.0 cpd). As predicted, activity patterns in early visual areas led to better discrimination of orientations presented at high than low contrast, with greater effects of contrast found in area V1 than in V3. A second experiment revealed generally better decoding of orientations at low or moderate as compared to high spatial frequencies. Interestingly however, V1 exhibited a relative advantage at discriminating high spatial frequency orientations, consistent with the finer scale of representation in the primary visual cortex. In both experiments, the reliability of these orientation-selective activity patterns was well predicted by the average BOLD amplitude in each region of interest, as indicated by correlation analyses, as well as decoding applied to a simple model of voxel responses to simulated orientation columns. Moreover, individual differences in decoding accuracy could be predicted by the signal-to-noise ratio of an individual's BOLD response. Our results indicate that decoding accuracy can be well predicted by incorporating the amplitude of the BOLD response into simple simulation models of cortical selectivity; such models could prove useful in future applications of fMRI pattern classification.

Attention modulates neural representation to render reconstructions according to subjective appearance.

Stimulus images can be reconstructed from visual cortical activity. However, our perception of stimuli is shaped by both stimulus-induced and top-down processes, and it is unclear whether and how reconstructions reflect top-down aspects of perception. Here, we investigate the effect of attention on reconstructions using fMRI activity measured while subjects attend to one of two superimposed images. A state-of-the-art method is used for image reconstruction, in which brain activity is translated (decoded) to deep neural network (DNN) features of hierarchical layers then to an image. Reconstructions resemble the attended rather than unattended images. They can be modeled by superimposed images with biased contrasts, comparable to the appearance during attention. Attentional modulations are found in a broad range of hierarchical visual representations and mirror the brain-DNN correspondence. Our results demonstrate that top-down attention counters stimulus-induced responses, modulating neural representations to render reconstructions in accordance with subjective appearance.

Decoding distributed oscillatory signals driven by memory and perception in the prefrontal cortex.

Sensory perception and memory recall generate different conscious experiences. Although externally and internally driven neural activities signifying the same perceptual content overlap in the sensory cortex, their distribution in the prefrontal cortex (PFC), an area implicated in both perception and memory, remains elusive. Here, we test whether the local spatial configurations and frequencies of neural oscillations driven by perception and memory recall overlap in the macaque PFC using high-density electrocorticography and multivariate pattern analysis. We find that dynamically changing oscillatory signals distributed across the PFC in the delta-, theta-, alpha-, and beta-band ranges carry significant, but mutually different, information predicting the same feature of memory-recalled internal targets and passively perceived external objects. These findings suggest that the frequency-specific distribution of oscillatory neural signals in the PFC serves cortical signatures responsible for distinguishing between different types of cognition driven by external perception and internal memory.

Neurofeedback Training without Explicit Phantom Hand Movements and Hand-Like Visual Feedback to Modulate Pain: A Randomized Crossover Feasibility Trial.

Phantom limb pain is attributed to abnormal sensorimotor cortical representations, although the causal relationship between phantom limb pain and sensorimotor cortical representations suffers from the potentially confounding effects of phantom hand movements. We developed neurofeedback training to change sensorimotor cortical representations without explicit phantom hand movements or hand-like visual feedback. We tested the feasibility of neurofeedback training in fourteen patients with phantom limb pain. Neurofeedback training was performed in a single-blind, randomized, crossover trial using two decoders constructed using motor cortical currents measured during phantom hand movements; the motor cortical currents contralateral or ipsilateral to the phantom hand (contralateral and ipsilateral training) were estimated from magnetoencephalograms. Patients were instructed to control the size of a disk, which was proportional to the decoding results, but to not move their phantom hands or other body parts. The pain assessed by the visual analogue scale was significantly greater after contralateral training than after ipsilateral training. Classification accuracy of phantom hand movements significantly increased only after contralateral training. These results suggested that the proposed neurofeedback training changed phantom hand representation and modulated pain without explicit phantom hand movements or hand-like visual feedback, thus showing the relation between the phantom hand representations and pain. PERSPECTIVE: Our work demonstrates the feasibility of using neurofeedback training to change phantom hand representation and modulate pain perception without explicit phantom hand movements and hand-like visual feedback. The results enhance the mechanistic understanding of certain treatments, such as mirror therapy, that change the sensorimotor cortical representation.

Voluntary control of semantic neural representations by imagery with conflicting visual stimulation.

Neural representations of visual perception are affected by mental imagery and attention. Although attention is known to modulate neural representations, it is unknown how imagery changes neural representations when imagined and perceived images semantically conflict. We hypothesized that imagining an image would activate a neural representation during its perception even while watching a conflicting image. To test this hypothesis, we developed a closed-loop system to show images inferred from electrocorticograms using a visual semantic space. The successful control of the feedback images demonstrated that the semantic vector inferred from electrocorticograms became closer to the vector of the imagined category, even while watching images from different categories. Moreover, modulation of the inferred vectors by mental imagery depended asymmetrically on the perceived and imagined categories. Shared neural representation between mental imagery and perception was still activated by the imagery under semantically conflicting perceptions depending on the semantic category.

Simultaneous recording of ECoG and intracortical neuronal activity using a flexible multichannel electrode-mesh in visual cortex.

Electrocorticogram (ECoG) is a well-balanced methodology for stably mapping brain surface local field potentials (LFPs) over a wide cortical region with high signal fidelity and minimal invasiveness to the brain tissue. To directly compare surface ECoG signals with intracortical neuronal activity immediately underneath, we fabricated a flexible multichannel electrode array with mesh-form structure using micro-electro-mechanical systems. A Parylene-C-based "electrode-mesh" for rats contained a 6×6 gold electrode array with 1-mm interval. Specifically, the probe had 800×800 µm(2) fenestrae in interelectrode spaces, through which simultaneous penetration of microelectrode was capable. This electrode-mesh was placed acutely or chronically on the dural/pial surface of the visual cortex of Long-Evans rats for up to 2 weeks. We obtained reliable trial-wise profiles of visually evoked ECoG signals through individual eye stimulation. Visually evoked ECoG signals from the electrode-mesh exhibited as well or larger signal amplitudes as intracortical LFPs and less across-trial variability than conventional silver-ball ECoG. Ocular selectivity of ECoG responses was correlated with that of intracortical spike/LFP activities. Moreover, single-trial ECoG signals carried sufficient information for predicting the stimulated eye with a correct performance approaching 90%, and the decoding was significantly generalized across sessions over 6 hours. Electrode impedance or signal quality did not obviously deteriorate for 2 weeks following implantation. These findings open up a methodology to directly explore ECoG signals with reference to intracortical neuronal sources and would provide a key to developing minimally invasive next-generation brain-machine interfaces.

Brain hierarchy score: Which deep neural networks are hierarchically brain-like?

Achievement of human-level image recognition by deep neural networks (DNNs) has spurred interest in whether and how DNNs are brain-like. Both DNNs and the visual cortex perform hierarchical processing, and correspondence has been shown between hierarchical visual areas and DNN layers in representing visual features. Here, we propose the brain hierarchy (BH) score as a metric to quantify the degree of hierarchical correspondence based on neural decoding and encoding analyses where DNN unit activations and human brain activity are predicted from each other. We find that BH scores for 29 pre-trained DNNs with various architectures are negatively correlated with image recognition performance, thus indicating that recently developed high-performance DNNs are not necessarily brain-like. Experimental manipulations of DNN models suggest that single-path sequential feedforward architecture with broad spatial integration is critical to brain-like hierarchy. Our method may provide new ways to design DNNs in light of their representational homology to the brain.

Natural and Artificial Intelligence: A brief introduction to the interplay between AI and neuroscience research.

Neuroscience and artificial intelligence (AI) share a long history of collaboration. Advances in neuroscience, alongside huge leaps in computer processing power over the last few decades, have given rise to a new generation of in silico neural networks inspired by the architecture of the brain. These AI systems are now capable of many of the advanced perceptual and cognitive abilities of biological systems, including object recognition and decision making. Moreover, AI is now increasingly being employed as a tool for neuroscience research and is transforming our understanding of brain functions. In particular, deep learning has been used to model how convolutional layers and recurrent connections in the brain's cerebral cortex control important functions, including visual processing, memory, and motor control. Excitingly, the use of neuroscience-inspired AI also holds great promise for understanding how changes in brain networks result in psychopathologies, and could even be utilized in treatment regimes. Here we discuss recent advancements in four areas in which the relationship between neuroscience and AI has led to major advancements in the field; (1) AI models of working memory, (2) AI visual processing, (3) AI analysis of big neuroscience datasets, and (4) computational psychiatry.