Multimodal neural networks better explain multivoxel patterns in the hippocampus.

The human hippocampus possesses "concept cells", neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP (Radford et al., 2021). Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate that "multimodality" stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus.

Theta-phase dependent neuronal coding during sequence learning in human single neurons.

The ability to maintain a sequence of items in memory is a fundamental cognitive function. In the rodent hippocampus, the representation of sequentially organized spatial locations is reflected by the phase of action potentials relative to the theta oscillation (phase precession). We investigated whether the timing of neuronal activity relative to the theta brain oscillation also reflects sequence order in the medial temporal lobe of humans. We used a task in which human participants learned a fixed sequence of pictures and recorded single neuron and local field potential activity with implanted electrodes. We report that spikes for three consecutive items in the sequence (the preferred stimulus for each cell, as well as the stimuli immediately preceding and following it) were phase-locked at distinct phases of the theta oscillation. Consistent with phase precession, spikes were fired at progressively earlier phases as the sequence advanced. These findings generalize previous findings in the rodent hippocampus to the human temporal lobe and suggest that encoding stimulus information at distinct oscillatory phases may play a role in maintaining sequential order in memory.

Human Hippocampal Neurons Track Moments in a Sequence of Events.

An indispensable feature of episodic memory is our ability to temporally piece together different elements of an experience into a coherent memory. Hippocampal time cells-neurons that represent temporal information-may play a critical role in this process. Although these cells have been repeatedly found in rodents, it is still unclear to what extent similar temporal selectivity exists in the human hippocampus. Here, we show that temporal context modulates the firing activity of human hippocampal neurons during structured temporal experiences. We recorded neuronal activity in the human brain while patients of either sex learned predictable sequences of pictures. We report that human time cells fire at successive moments in this task. Furthermore, time cells also signaled inherently changing temporal contexts during empty 10 s gap periods between trials while participants waited for the task to resume. Finally, population activity allowed for decoding temporal epoch identity, both during sequence learning and during the gap periods. These findings suggest that human hippocampal neurons could play an essential role in temporally organizing distinct moments of an experience in episodic memory.SIGNIFICANCE STATEMENT Episodic memory refers to our ability to remember the what, where, and when of a past experience. Representing time is an important component of this form of memory. Here, we show that neurons in the human hippocampus represent temporal information. This temporal signature was observed both when participants were actively engaged in a memory task, as well as during 10-s-long gaps when they were asked to wait before performing the task. Furthermore, the activity of the population of hippocampal cells allowed for decoding one temporal epoch from another. These results suggest a robust representation of time in the human hippocampus.

Representational Content of Oscillatory Brain Activity during Object Recognition: Contrasting Cortical and Deep Neural Network Hierarchies.

Numerous theories propose a key role for brain oscillations in visual perception. Most of these theories postulate that sensory information is encoded in specific oscillatory components (e.g., power or phase) of specific frequency bands. These theories are often tested with whole-brain recording methods of low spatial resolution (EEG or MEG), or depth recordings that provide a local, incomplete view of the brain. Opportunities to bridge the gap between local neural populations and whole-brain signals are rare. Here, using representational similarity analysis (RSA) in human participants we explore which MEG oscillatory components (power and phase, across various frequency bands) correspond to low or high-level visual object representations, using brain representations from fMRI, or layer-wise representations in seven recent deep neural networks (DNNs), as a template for low/high-level object representations. The results showed that around stimulus onset and offset, most transient oscillatory signals correlated with low-level brain patterns (V1). During stimulus presentation, sustained ß (∼20 Hz) and γ (>60 Hz) power best correlated with V1, while oscillatory phase components correlated with IT representations. Surprisingly, this pattern of results did not always correspond to low-level or high-level DNN layer activity. In particular, sustained ß band oscillatory power reflected high-level DNN layers, suggestive of a feed-back component. These results begin to bridge the gap between whole-brain oscillatory signals and object representations supported by local neuronal activations.

Tentative fMRI signatures of perceptual echoes in early visual cortex.

The visual Impulse Response Function (IRF) can be estimated by cross-correlating random luminance sequences with concurrently recorded EEG. It typically contains a strong 10 Hz oscillatory component, suggesting that visual information reverberates in the human brain as a "perceptual echo". The neural origin of these echoes remains unknown. To address this question, we recorded EEG and fMRI in two separate sessions. In both sessions, a disk whose luminance followed a random (white noise) sequence was presented in the upper left quadrant. Individual IRFs were derived from the EEG session. These IRFs were then used as "response templates" to reconstruct an estimate of the EEG during the fMRI session, by convolution with the corresponding random luminance sequences. The 7-14 Hz (alpha, the main frequency component of the IRF) envelope of the reconstructed EEG was finally used as an fMRI regressor, to determine which brain voxels co-varied with the oscillations elicited by the luminance sequence, i.e., the "perceptual echoes". The reconstructed envelope of EEG alpha was significantly correlated with BOLD responses in V1 and V2. Surprisingly, this correlation was visible outside, but not within the directly (retinotopically) stimulated region. We tentatively interpret this lack of alpha modulation as a BOLD saturation effect, since the overall stimulus-induced BOLD response was inversely related, across voxels, to the signal variability over time. In conclusion, our results suggest that perceptual echoes originate in early visual cortex, driven by widespread activity in V1 and V2, not retinotopically restricted, but possibly reflecting the propagation of a travelling alpha wave.

The Architecture of Human Memory: Insights from Human Single-Neuron Recordings.

Deciphering the mechanisms of human memory is a central goal of neuroscience, both from the point of view of the fundamental biology of memory and for its translational relevance. Here, we review some contributions that recordings from neurons in humans implanted with electrodes for clinical purposes have made toward this goal. Recordings from the medial temporal lobe, including the hippocampus, reveal the existence of two classes of cells: those encoding highly selective and invariant representations of abstract concepts, and memory-selective cells whose activity is related to familiarity and episodic retrieval. Insights derived from observing these cells in behaving humans include that semantic representations are activated before episodic representations, that memory content and memory strength are segregated, and that the activity of both types of cells is related to subjective awareness as expected from a substrate for declarative memory. Visually selective cells can remain persistently active for several seconds, thereby revealing a cellular substrate for working memory in humans. An overarching insight is that the neural code of human memory is interpretable at the single-neuron level. Jointly, intracranial recording studies are starting to reveal aspects of the building blocks of human memory at the single-cell level. This work establishes a bridge to cellular-level work in animals on the one hand, and the extensive literature on noninvasive imaging in humans on the other hand. More broadly, this work is a step toward a detailed mechanistic understanding of human memory that is needed to develop therapies for human memory disorders.

Recording local field potential and neuronal activity with tetrodes in epileptic patients.

BACKGROUND: Recordings with tetrodes have proven to be more effective in isolating single neuron spiking activity than with single microwires. However, tetrodes have never been used in humans. We report on the characteristics, safety, compatibility with clinical intracranial recordings in epileptic patients, and performance, of a new type of hybrid electrode equipped with tetrodes. NEW METHOD: 240 standard clinical macroelectrodes and 102 hybrid electrodes were implanted in 28 patients. Hybrids (diameter 800 µm) are made of 6 or 9 macro-contacts and 2 or 3 tetrodes (diameter 70-80 µm). RESULTS: No clinical complication or adverse event was associated with the hybrids. Impedance and noise of recordings were stable over time. The design enabled multiscale spatial analyses that revealed physiopathological events which were sometimes specific to one tetrode, but could not be recorded on the macro-contacts. After spike sorting, the single-unit yield was similar to other hybrid electrodes and was sometimes as high as >10 neurons per tetrode. COMPARISON WITH EXISTING METHOD(S): This new hybrid electrode has a smaller diameter than other available hybrid electrodes. It provides novel spatial information due to the configuration of the tetrodes. The single-unit yield appears promising. CONCLUSIONS: This new hybrid electrode is safe, easy to use, and works satisfactorily for conducting multi-scale seizure and physiological analyses.

Reconstructing faces from fMRI patterns using deep generative neural networks.

Although distinct categories are reliably decoded from fMRI brain responses, it has proved more difficult to distinguish visually similar inputs, such as different faces. Here, we apply a recently developed deep learning system to reconstruct face images from human fMRI. We trained a variational auto-encoder (VAE) neural network using a GAN (Generative Adversarial Network) unsupervised procedure over a large data set of celebrity faces. The auto-encoder latent space provides a meaningful, topologically organized 1024-dimensional description of each image. We then presented several thousand faces to human subjects, and learned a simple linear mapping between the multi-voxel fMRI activation patterns and the 1024 latent dimensions. Finally, we applied this mapping to novel test images, translating fMRI patterns into VAE latent codes, and codes into face reconstructions. The system not only performed robust pairwise decoding (>95% correct), but also accurate gender classification, and even decoded which face was imagined, rather than seen.

Neuronal spiking activity highlights a gradient of epileptogenicity in human tuberous sclerosis lesions.

OBJECTIVE: The mechanisms underlying epileptogenicity in tuberous sclerosis complex (TSC) are poorly understood. METHODS: We analysed neuronal spiking activity (84 neurons), fast ripples (FRs), local field potentials and intracranial electroencephalogram during interictal epileptiform discharges (IEDs) in the tuber and perituber of a patient using novel hybrid electrodes equipped with tetrodes. RESULTS: IEDs were recorded in the tuber and perituber. FRs were recorded only in the tuber and only with the microelectrodes. A larger proportion of neurons in the tuber (57%) than in the perituber (17%) had firing-rates modulated around IEDs. CONCLUSIONS: A multi-scale analysis of neuronal activity, FRs and IEDs indicates a gradient of epileptogenicity running from the tuber to the perituber. SIGNIFICANCE: We demonstrate, for the first time in vivo, a gradient of epileptogenicity from the tuber to the perituber, which paves the way for future models of epilepsy in TSC. Our results also question the extent of the neurosurgical resection, including or not the perituber, that needs to be made in these patients.

Near-optimal integration of facial form and motion.

Human perception consists of the continuous integration of sensory cues pertaining to the same object. While it has been fairly well shown that humans use an optimal strategy when integrating low-level cues proportional to their relative reliability, the integration processes underlying high-level perception are much less understood. Here we investigate cue integration in a complex high-level perceptual system, the human face processing system. We tested cue integration of facial form and motion in an identity categorization task and found that an optimal model could successfully predict subjects' identity choices. Our results suggest that optimal cue integration may be implemented across different levels of the visual processing hierarchy.

The Effects of Context and Attention on Spiking Activity in Human Early Visual Cortex.

Here we report the first quantitative analysis of spiking activity in human early visual cortex. We recorded multi-unit activity from two electrodes in area V2/V3 of a human patient implanted with depth electrodes as part of her treatment for epilepsy. We observed well-localized multi-unit receptive fields with tunings for contrast, orientation, spatial frequency, and size, similar to those reported in the macaque. We also observed pronounced gamma oscillations in the local-field potential that could be used to estimate the underlying spiking response properties. Spiking responses were modulated by visual context and attention. We observed orientation-tuned surround suppression: responses were suppressed by image regions with a uniform orientation and enhanced by orientation contrast. Additionally, responses were enhanced on regions that perceptually segregated from the background, indicating that neurons in the human visual cortex are sensitive to figure-ground structure. Spiking responses were also modulated by object-based attention. When the patient mentally traced a curve through the neurons' receptive fields, the accompanying shift of attention enhanced neuronal activity. These results demonstrate that the tuning properties of cells in the human early visual cortex are similar to those in the macaque and that responses can be modulated by both contextual factors and behavioral relevance. Our results, therefore, imply that the macaque visual system is an excellent model for the human visual cortex.

Multivoxel Object Representations in Adult Human Visual Cortex Are Flexible: An Associative Learning Study.

Learning associations between co-occurring events enables us to extract structure from our environment. Medial-temporal lobe structures are critical for associative learning. However, the role of the ventral visual pathway (VVP) in associative learning is not clear. Do multivoxel object representations in the VVP reflect newly formed associations? We show that VVP multivoxel representations become more similar to each other after human participants learn arbitrary new associations between pairs of unrelated objects (faces, houses, cars, chairs). Participants were scanned before and after 15 days of associative learning. To evaluate how object representations changed, a classifier was trained on discriminating two nonassociated categories (e.g., faces/houses) and tested on discriminating their paired associates (e.g., cars/chairs). Because the associations were arbitrary and counterbalanced across participants, there was initially no particular reason for this cross-classification decision to tend toward either alternative. Nonetheless, after learning, cross-classification performance increased in the VVP (but not hippocampus), on average by 3.3%, with some voxels showing increases of up to 10%. For example, a chair multivoxel representation that initially resembled neither face nor house representations was, after learning, classified as more similar to that of faces for participants who associated chairs with faces and to that of houses for participants who associated chairs with houses. Additionally, learning produced long-lasting perceptual consequences. In a behavioral priming experiment performed several months later, the change in cross-classification performance was correlated with the degree of priming. Thus, VVP multivoxel representations are not static but become more similar to each other after associative learning.

Learning of anticipatory responses in single neurons of the human medial temporal lobe.

Neuronal processes underlying the formation of new associations in the human brain are not yet well understood. Here human participants, implanted with depth electrodes in the brain, learned arbitrary associations between images presented in an ordered, predictable sequence. During learning we recorded from medial temporal lobe (MTL) neurons that responded to at least one of the pictures in the sequence (the preferred stimulus). We report that as a result of learning, single MTL neurons show asymmetric shifts in activity and start firing earlier in the sequence in anticipation of their preferred stimulus. These effects appear relatively early in learning, after only 11 exposures to the stimulus sequence. The anticipatory neuronal responses emerge while the subjects became faster in reporting the next item in the sequence. These results demonstrate flexible representations that could support learning of new associations between stimuli in a sequence, in single neurons in the human MTL.

Concept cells through associative learning of high-level representations.

In this issue of Neuron, Quian Quiroga et al. (2014) show that neurons in the human medial temporal lobe (MTL) follow subjects' perceptual states rather than the features of the visual input. Patients with MTL damage however have intact perceptual abilities but suffer instead from extreme forgetfulness. Thus, the reported MTL neurons could create new memories of the current perceptual state.

A need for more information uptake but not focused attention to access basic-level representations.

Complex visual scenes can be categorized at the superordinate level (e.g., animal/non-animal or vehicle/non-vehicle) without focused attention. However, rapid visual categorization at the basic level (e.g., dog/non-dog or car/non-car) requires additional processing time. Such finer categorization might, thus, require attentional resources. This hypothesis was tested in the current study with a dual-task paradigm in which subjects performed a basic-level categorization task in peripheral vision either alone (single-task condition) or concurrently with an attentionally demanding letter discrimination task (dual-task condition). Our results indicate that basic-level categorization of either biological (dog/non-dog animal) or man-made (car/non-car vehicle) stimuli requires more information uptake but can, nevertheless, be performed when attention is not fully available, presumably because it is supported by hardwired, specialized neuronal networks.

Neural correlates of the continuous Wagon Wheel Illusion: a functional MRI study.

After prolonged viewing of a continuous periodic motion stimulus at frequencies around 10 Hz, observers experience a fleeting impression of reversed motion: the continuous Wagon Wheel Illusion (c-WWI). To account for this phenomenon it has been proposed that attentional mechanisms discretely sample motion information. Alternative accounts argue that the illusion relies on the spurious activation of motion detectors, which under the effect of adaptation could trigger a reversed percept. We investigated the neural correlates of the c-WWI using fMRI (3T). Subjects viewed a vertically bisected ring containing a radial grating unambiguously rotating at 10 Hz; they continuously reported the perceived motion direction within each half of the ring. The two halves always rotated in opposite directions, allowing us to separately explore illusory reversals occurring within each hemifield. Comparing BOLD activity during illusory (c-WWI) or real perceptual periods revealed systematic differences in right parietal regions, in addition to the right motion complex MT+. This activation pattern did not depend on the side on which the illusion occurred, and could not be accounted for by purely perceptual switch-related activity-known to encompass parietal regions during other bistable effects. This first characterization of the fMRI correlates of the c-WWI may have implications for the different theoretical explanations of the phenomenon.

Reading the mind's eye: decoding category information during mental imagery.

Category information for visually presented objects can be read out from multi-voxel patterns of fMRI activity in ventral-temporal cortex. What is the nature and reliability of these patterns in the absence of any bottom-up visual input, for example, during visual imagery? Here, we first ask how well category information can be decoded for imagined objects and then compare the representations evoked during imagery and actual viewing. In an fMRI study, four object categories (food, tools, faces, buildings) were either visually presented to subjects, or imagined by them. Using pattern classification techniques, we could reliably decode category information (including for non-special categories, i.e., food and tools) from ventral-temporal cortex in both conditions, but only during actual viewing from retinotopic areas. Interestingly, in temporal cortex when the classifier was trained on the viewed condition and tested on the imagery condition, or vice versa, classification performance was comparable to within the imagery condition. The above results held even when we did not use information in the specialized category-selective areas. Thus, the patterns of representation during imagery and actual viewing are in fact surprisingly similar to each other. Consistent with this observation, the maps of "diagnostic voxels" (i.e., the classifier weights) for the perception and imagery classifiers were more similar in ventral-temporal cortex than in retinotopic cortex. These results suggest that in the absence of any bottom-up input, cortical back projections can selectively re-activate specific patterns of neural activity.

Attention and biased competition in multi-voxel object representations.

The biased-competition theory accounts for attentional effects at the single-neuron level: It predicts that the neuronal response to simultaneously-presented stimuli is a weighted average of the response to isolated stimuli, and that attention biases the weights in favor of the attended stimulus. Perception, however, relies not on single neurons but on larger neuronal populations. The responses of such populations are in part reflected in large-scale multivoxel fMRI activation patterns. Because the pooling of neuronal responses into blood-oxygen-level-dependent signals is nonlinear, fMRI effects of attention need not mirror those observed at the neuronal level. Thus, to bridge the gap between neuronal responses and human perception, it is fundamental to understand attentional influences in large-scale multivariate representations of simultaneously-presented objects. Here, we ask how responses to simultaneous stimuli are combined in multivoxel fMRI patterns, and how attention affects the paired response. Objects from four categories were presented singly, or in pairs such that each category was attended, unattended, or attention was divided between the two. In a multidimensional voxel space, the response to simultaneously-presented categories was well described as a weighted average. The weights were biased toward the preferred category in category-selective regions. Consistent with single-unit reports, attention shifted the weights by approximately 30% in favor of the attended stimulus. These findings extend the biased-competition framework to the realm of large-scale multivoxel brain activations.

Category selectivity in the ventral visual pathway confers robustness to clutter and diverted attention.

Are objects coded by a small number of neurons or cortical regions that respond preferentially to the object in question, or by more distributed patterns of responses, including neurons or regions that respond only weakly? Distributed codes can represent a larger number of alternative items than sparse codes but produce ambiguities when multiple items are represented simultaneously (the "superposition" problem). Recent studies found category information in the distributed pattern of response across the ventral visual pathway, including in regions that do not "prefer" the object in question. However, these studies measured neural responses to isolated objects, a situation atypical of real-world vision, where multiple objects are usually present simultaneously ("clutter"). We report that information in the spatial pattern of fMRI response about standard object categories is severely disrupted by clutter and eliminated when attention is diverted. However, information about preferred categories in category-specific regions is undiminished by clutter and partly preserved under diverted attention. These findings indicate that in natural conditions, the pattern of fMRI response provides robust category information only for objects coded in selective cortical regions and highlight the vulnerability of distributed representations to clutter and the advantages of sparse cortical codes in mitigating clutter costs.

Top-down biases win against focal attention in the fusiform face area.

Many studies have reported that BOLD activity in visual cortex is enhanced in the presence of selective attention. These reports are seemingly at odds with psychophysical data showing that observers are able to efficiently categorize natural stimuli in the near-absence of focal attention. To reconcile these two lines of evidence, we study the effects of attentional modulation on face-selective responses in the fusiform face area (FFA) using fMRI. Different from previous fMRI studies in which an "attended" condition (where subjects make a behavioral report on faces) is compared to an "unattended" condition (where the faces are task irrelevant), we included a third condition in which focal attention was not fully available to the faces yet they remained task relevant. Thus we were able to distinguish between the effects of spatial attention and a task-based component of attention. Whether or not subjects had to spatially attend to the faces made no difference to the amplitude of BOLD activity in the FFA provided the faces had to be discriminated. As expected, we observed a decrease in BOLD activity in the FFA when faces were task irrelevant. This pattern of modulation of the BOLD response as a function of the subject's behavior was region specific as it did not extend to the parahippocampal place area. These results point to a coherent picture of how spatial attention and top-down task-based attention interact in visual cortex.