Semi-Supervised Multimodal Representation Learning Through a Global Workspace.

Recent deep learning models can efficiently combine inputs from different modalities (e.g., images and text) and learn to align their latent representations or to translate signals from one domain to another (as in image captioning or text-to-image generation). However, current approaches mainly rely on brute-force supervised training over large multimodal datasets. In contrast, humans (and other animals) can learn useful multimodal representations from only sparse experience with matched cross-modal data. Here, we evaluate the capabilities of a neural network architecture inspired by the cognitive notion of a "global workspace" (GW): a shared representation for two (or more) input modalities. Each modality is processed by a specialized system (pretrained on unimodal data and subsequently frozen). The corresponding latent representations are then encoded to and decoded from a single shared workspace. Importantly, this architecture is amenable to self-supervised training via cycle-consistency: encoding-decoding sequences should approximate the identity function. For various pairings of vision-language modalities and across two datasets of varying complexity, we show that such an architecture can be trained to align and translate between two modalities with very little need for matched data (from four to seven times less than a fully supervised approach). The GW representation can be used advantageously for downstream classification and cross-modal retrieval tasks and for robust transfer learning. Ablation studies reveal that both the shared workspace and the self-supervised cycle-consistency training are critical to the system's performance.

A Traveling Waves Perspective on Temporal Binding.

Brain oscillations are involved in many cognitive processes, and several studies have investigated their role in cognition. In particular, the phase of certain oscillations has been related to temporal binding and integration processes, with some authors arguing that perception could be an inherently rhythmic process. However, previous research on oscillations mostly overlooked their spatial component: how oscillations propagate through the brain as traveling waves, with systematic phase delays between brain regions. Here, we argue that interpreting oscillations as traveling waves is a useful paradigm shift to understand their role in temporal binding and address controversial results. After a brief definition of traveling waves, we propose an original view on temporal integration that considers this new perspective. We first focus on cortical dynamics, then speculate about the role of thalamic nuclei in modulating the waves, and on the possible consequences for rhythmic temporal binding. In conclusion, we highlight the importance of considering oscillations as traveling waves when investigating their role in cognitive functions.

Natural scene reconstruction from fMRI signals using generative latent diffusion.

In neural decoding research, one of the most intriguing topics is the reconstruction of perceived natural images based on fMRI signals. Previous studies have succeeded in re-creating different aspects of the visuals, such as low-level properties (shape, texture, layout) or high-level features (category of objects, descriptive semantics of scenes) but have typically failed to reconstruct these properties together for complex scene images. Generative AI has recently made a leap forward with latent diffusion models capable of generating high-complexity images. Here, we investigate how to take advantage of this innovative technology for brain decoding. We present a two-stage scene reconstruction framework called "Brain-Diffuser". In the first stage, starting from fMRI signals, we reconstruct images that capture low-level properties and overall layout using a VDVAE (Very Deep Variational Autoencoder) model. In the second stage, we use the image-to-image framework of a latent diffusion model (Versatile Diffusion) conditioned on predicted multimodal (text and visual) features, to generate final reconstructed images. On the publicly available Natural Scenes Dataset benchmark, our method outperforms previous models both qualitatively and quantitatively. When applied to synthetic fMRI patterns generated from individual ROI (region-of-interest) masks, our trained model creates compelling "ROI-optimal" scenes consistent with neuroscientific knowledge. Thus, the proposed methodology can have an impact on both applied (e.g. brain-computer interface) and fundamental neuroscience.

Mathematical Derivation of Wave Propagation Properties in Hierarchical Neural Networks with Predictive Coding Feedback Dynamics.

Sensory perception (e.g., vision) relies on a hierarchy of cortical areas, in which neural activity propagates in both directions, to convey information not only about sensory inputs but also about cognitive states, expectations and predictions. At the macroscopic scale, neurophysiological experiments have described the corresponding neural signals as both forward and backward-travelling waves, sometimes with characteristic oscillatory signatures. It remains unclear, however, how such activity patterns relate to specific functional properties of the perceptual apparatus. Here, we present a mathematical framework, inspired by neural network models of predictive coding, to systematically investigate neural dynamics in a hierarchical perceptual system. We show that stability of the system can be systematically derived from the values of hyper-parameters controlling the different signals (related to bottom-up inputs, top-down prediction and error correction). Similarly, it is possible to determine in which direction, and at what speed neural activity propagates in the system. Different neural assemblies (reflecting distinct eigenvectors of the connectivity matrices) can simultaneously and independently display different properties in terms of stability, propagation speed or direction. We also derive continuous-limit versions of the system, both in time and in neural space. Finally, we analyze the possible influence of transmission delays between layers, and reveal the emergence of oscillations.

Distinct roles of forward and backward alpha-band waves in spatial visual attention.

Previous research has associated alpha-band [8-12 Hz] oscillations with inhibitory functions: for instance, several studies showed that visual attention increases alpha-band power in the hemisphere ipsilateral to the attended location. However, other studies demonstrated that alpha oscillations positively correlate with visual perception, hinting at different processes underlying their dynamics. Here, using an approach based on traveling waves, we demonstrate that there are two functionally distinct alpha-band oscillations propagating in different directions. We analyzed EEG recordings from three datasets of human participants performing a covert visual attention task (one new dataset with N = 16, two previously published datasets with N = 16 and N = 31). Participants were instructed to detect a brief target by covertly attending to the screen's left or right side. Our analysis reveals two distinct processes: allocating attention to one hemifield increases top-down alpha-band waves propagating from frontal to occipital regions ipsilateral to the attended location, both with and without visual stimulation. These top-down oscillatory waves correlate positively with alpha-band power in frontal and occipital regions. Yet, different alpha-band waves propagate from occipital to frontal regions and contralateral to the attended location. Crucially, these forward waves were present only during visual stimulation, suggesting a separate mechanism related to visual processing. Together, these results reveal two distinct processes reflected by different propagation directions, demonstrating the importance of considering oscillations as traveling waves when characterizing their functional role.

Random Tactile Noise Stimulation Reveals Beta-Rhythmic Impulse Response Function of the Somatosensory System.

Both passive tactile stimulation and motor actions result in dynamic changes in beta band (15-30 Hz Hz) oscillations over somatosensory cortex. Similar to alpha band (8-12 Hz) power decrease in the visual system, beta band power also decreases following stimulation of the somatosensory system. This relative suppression of α and ß oscillations is generally interpreted as an increase in cortical excitability. Here, next to traditional single-pulse stimuli, we used a random intensity continuous right index finger tactile stimulation (white noise), which enabled us to uncover an impulse response function of the somatosensory system. Contrary to previous findings, we demonstrate a burst-like initial increase rather than decrease of beta activity following white noise stimulation (human participants, N = 18, 8 female). These ß bursts, on average, lasted for 3 cycles, and their frequency was correlated with resonant frequency of somatosensory cortex, as measured by a multifrequency steady-state somatosensory evoked potential paradigm. Furthermore, beta band bursts shared spectro-temporal characteristics with evoked and resting-state ß oscillations. Together, our findings not only reveal a novel oscillatory signature of somatosensory processing that mimics the previously reported visual impulse response functions, but also point to a common oscillatory generator underlying spontaneous ß bursts in the absence of tactile stimulation and phase-locked ß bursts following stimulation, the frequency of which is determined by the resonance properties of the somatosensory system.SIGNIFICANCE STATEMENT The investigation of the transient nature of oscillations has gained great popularity in recent years. The findings of bursting activity, rather than sustained oscillations in the beta band, have provided important insights into its role in movement planning, working memory, inhibition, and reactivation of neural ensembles. In this study, we show that also in response to tactile stimulation the somatosensory system responds with ∼3 cycle oscillatory beta band bursts, whose spectro-temporal characteristics are shared with evoked and resting-state beta band oscillatory signatures of the somatosensory system. As similar bursts have been observed in the visual domain, these oscillatory signatures might reflect an important supramodal mechanism in sensory processing.

On the role of feedback in image recognition under noise and adversarial attacks: A predictive coding perspective.

Brain-inspired machine learning is gaining increasing consideration, particularly in computer vision. Several studies investigated the inclusion of top-down feedback connections in convolutional networks; however, it remains unclear how and when these connections are functionally helpful. Here we address this question in the context of object recognition under noisy conditions. We consider deep convolutional networks (CNNs) as models of feed-forward visual processing and implement Predictive Coding (PC) dynamics through feedback connections (predictive feedback) trained for reconstruction or classification of clean images. First, we show that the accuracy of the network implementing PC dynamics is significantly larger compared to its equivalent forward network. Importantly, to directly assess the computational role of predictive feedback in various experimental situations, we optimize and interpret the hyper-parameters controlling the network's recurrent dynamics. That is, we let the optimization process determine whether top-down connections and predictive coding dynamics are functionally beneficial. Across different model depths and architectures (3-layer CNN, ResNet18, and EfficientNetB0) and against various types of noise (CIFAR100-C), we find that the network increasingly relies on top-down predictions as the noise level increases; in deeper networks, this effect is most prominent at lower layers. All in all, our results provide novel insights relevant to Neuroscience by confirming the computational role of feedback connections in sensory systems, and to Machine Learning by revealing how these can improve the robustness of current vision models.

Multimodal neural networks better explain multivoxel patterns in the hippocampus.

The human hippocampus possesses "concept cells", neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP (Radford et al., 2021). Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate that "multimodality" stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus.

Periodic attention operates faster during more complex visual search.

Attention has been found to sample visual information periodically, in a wide range of frequencies below 20 Hz. This periodicity may be supported by brain oscillations at corresponding frequencies. We propose that part of the discrepancy in periodic frequencies observed in the literature is due to differences in attentional demands, resulting from heterogeneity in tasks performed. To test this hypothesis, we used visual search and manipulated task complexity, i.e., target discriminability (high, medium, low) and number of distractors (set size), while electro-encephalography was simultaneously recorded. We replicated previous results showing that the phase of pre-stimulus low-frequency oscillations predicts search performance. Crucially, such effects were observed at increasing frequencies within the theta-alpha range (6-18 Hz) for decreasing target discriminability. In medium and low discriminability conditions, correct responses were further associated with higher post-stimulus phase-locking than incorrect ones, in increasing frequency and latency. Finally, the larger the set size, the later the post-stimulus effect peaked. Together, these results suggest that increased complexity (lower discriminability or larger set size) requires more attentional cycles to perform the task, partially explaining discrepancies between reports of attentional sampling. Low-frequency oscillations structure the temporal dynamics of neural activity and aid top-down, attentional control for efficient visual processing.

Understanding the Computational Demands Underlying Visual Reasoning.

Visual understanding requires comprehending complex visual relations between objects within a scene. Here, we seek to characterize the computational demands for abstract visual reasoning. We do this by systematically assessing the ability of modern deep convolutional neural networks (CNNs) to learn to solve the synthetic visual reasoning test (SVRT) challenge, a collection of 23 visual reasoning problems. Our analysis reveals a novel taxonomy of visual reasoning tasks, which can be primarily explained by both the type of relations (same-different versus spatial-relation judgments) and the number of relations used to compose the underlying rules. Prior cognitive neuroscience work suggests that attention plays a key role in humans' visual reasoning ability. To test this hypothesis, we extended the CNNs with spatial and feature-based attention mechanisms. In a second series of experiments, we evaluated the ability of these attention networks to learn to solve the SVRT challenge and found the resulting architectures to be much more efficient at solving the hardest of these visual reasoning tasks. Most important, the corresponding improvements on individual tasks partially explained our novel taxonomy. Overall, this work provides a granular computational account of visual reasoning and yields testable neuroscience predictions regarding the differential need for feature-based versus spatial attention depending on the type of visual reasoning problem.

α Phase-Amplitude Tradeoffs Predict Visual Perception.

Spontaneous α oscillations (∼10 Hz) have been associated with various cognitive functions, including perception. Their phase and amplitude independently predict cortical excitability and subsequent perceptual performance. However, the causal role of α phase-amplitude tradeoffs on visual perception remains ill-defined. We aimed to fill this gap and tested two clear predictions from the pulsed inhibition theory according to which α oscillations are associated with periodic functional inhibition. (1) High-α amplitude induces cortical inhibition at specific phases, associated with low perceptual performance, while at opposite phases, inhibition decreases (potentially increasing excitation) and perceptual performance increases. (2) Low-α amplitude is less susceptible to these phasic (periodic) pulses of inhibition, leading to overall higher perceptual performance. Here, cortical excitability was assessed in humans using phosphene (illusory) perception induced by single pulses of transcranial magnetic stimulation (TMS) applied over visual cortex at perceptual threshold, and its postpulse evoked activity recorded with simultaneous electroencephalography (EEG). We observed that prepulse α phase modulates the probability to perceive a phosphene, predominantly for high-α amplitude, with a nonoptimal phase for phosphene perception between -π/2 and -π/4. The prepulse nonoptimal phase further leads to an increase in postpulse-evoked activity [event-related potential (ERP)], in phosphene-perceived trials specifically. Together, these results show that α oscillations create periodic inhibitory moments when α amplitude is high, leading to periodic decrease of perceptual performance. This study provides strong causal evidence in favor of the pulsed inhibition theory.

Nudging the N170 forward with prior stimulation-Bridging the gap between N170 and recognition potential.

Evoked response potentials are often divided up into numerous components, each with their own body of literature. But is there less variety than we might suppose? In this study, we nudge one component into looking like another. Both the N170 and recognition potential (RP) are N1 components in response to familiar objects. However, the RP is often measured with a forward mask that ends at stimulus onset whereas the N170 is often measured with no masking at all. This study investigates how inter-stimulus interval (ISI) may delay and distort the N170 into an RP by manipulating the temporal gap (ISI) between forward mask and target. The results revealed reverse relationships between the ISI on the one hand, and the N170 latency, single-trial N1 jitter (an approximation of N1 width) and reaction time on the other hand. Importantly, we find that scalp topographies have a unique signature at the N1 peak across all conditions, from the longest gap (N170) to the shortest (RP). These findings prove that the mask-delayed N1 is still the same N170, even under conditions that are normally associated with a different component like the RP. In general, our results suggest greater synthesis in the study of event related potential components.

Predictive coding feedback results in perceived illusory contours in a recurrent neural network.

Modern feedforward convolutional neural networks (CNNs) can now solve some computer vision tasks at super-human levels. However, these networks only roughly mimic human visual perception. One difference from human vision is that they do not appear to perceive illusory contours (e.g. Kanizsa squares) in the same way humans do. Physiological evidence from visual cortex suggests that the perception of illusory contours could involve feedback connections. Would recurrent feedback neural networks perceive illusory contours like humans? In this work we equip a deep feedforward convolutional network with brain-inspired recurrent dynamics. The network was first pretrained with an unsupervised reconstruction objective on a natural image dataset, to expose it to natural object contour statistics. Then, a classification decision head was added and the model was finetuned on a form discrimination task: squares vs. randomly oriented inducer shapes (no illusory contour). Finally, the model was tested with the unfamiliar "illusory contour" configuration: inducer shapes oriented to form an illusory square. Compared with feedforward baselines, the iterative "predictive coding" feedback resulted in more illusory contours being classified as physical squares. The perception of the illusory contour was measurable in the luminance profile of the image reconstructions produced by the model, demonstrating that the model really "sees" the illusion. Ablation studies revealed that natural image pretraining and feedback error correction are both critical to the perception of the illusion. Finally we validated our conclusions in a deeper network (VGG): adding the same predictive coding feedback dynamics again leads to the perception of illusory contours.

Theta-phase dependent neuronal coding during sequence learning in human single neurons.

The ability to maintain a sequence of items in memory is a fundamental cognitive function. In the rodent hippocampus, the representation of sequentially organized spatial locations is reflected by the phase of action potentials relative to the theta oscillation (phase precession). We investigated whether the timing of neuronal activity relative to the theta brain oscillation also reflects sequence order in the medial temporal lobe of humans. We used a task in which human participants learned a fixed sequence of pictures and recorded single neuron and local field potential activity with implanted electrodes. We report that spikes for three consecutive items in the sequence (the preferred stimulus for each cell, as well as the stimuli immediately preceding and following it) were phase-locked at distinct phases of the theta oscillation. Consistent with phase precession, spikes were fired at progressively earlier phases as the sequence advanced. These findings generalize previous findings in the rodent hippocampus to the human temporal lobe and suggest that encoding stimulus information at distinct oscillatory phases may play a role in maintaining sequential order in memory.

Occipital alpha-TMS causally modulates temporal order judgements: Evidence for discrete temporal windows in vision.

Recent advances in neuroscience have challenged the view of conscious visual perception as a continuous process. Behavioral performance, reaction times and some visual illusions all undergo periodic fluctuations that can be traced back to oscillatory activity in the brain. These findings have given rise to the idea of a discrete sampling mechanism in the visual system. In this study we seek to investigate the causal relationship between occipital alpha oscillations and Temporal Order Judgements using neural entrainment via rhythmic TMS in 18 human subjects (9 females). We find that certain phases of the entrained oscillation facilitate temporal order perception of two visual stimuli, whereas others hinder it. Our findings support the idea that the visual system periodically compresses information into discrete packages within which temporal order information is lost. SIGNIFICANCE STATEMENT: Neural entrainment via TMS serves as a valuable tool to interfere with cortical rhythms and observe changes in perception. Here, using α-rhythmic TMS-pulses, we demonstrate the effect of the phase of entrained oscillations on performance in a temporal order judgment task. In extension of previous work, we 1. causally influenced brain rhythms far more directly using TMS, and 2. showed that previous results on discrete perception cannot simply be explained by rhythmic fluctuations in visibility. Our findings support the idea that the temporal organization of visual processing is discrete rather than continuous, and is causally modulated by cortical rhythms. To our knowledge, this is the first study providing causal evidence via TMS for an endogenous periodic modulation of time perception.

Deep learning and the Global Workspace Theory.

Recent advances in deep learning have allowed artificial intelligence (AI) to reach near human-level performance in many sensory, perceptual, linguistic, and cognitive tasks. There is a growing need, however, for novel, brain-inspired cognitive architectures. The Global Workspace Theory (GWT) refers to a large-scale system integrating and distributing information among networks of specialized modules to create higher-level forms of cognition and awareness. We argue that the time is ripe to consider explicit implementations of this theory using deep-learning techniques. We propose a roadmap based on unsupervised neural translation between multiple latent spaces (neural networks trained for distinct tasks, on distinct sensory inputs and/or modalities) to create a unique, amodal Global Latent Workspace (GLW). Potential functional advantages of GLW are reviewed, along with neuroscientific implications.

Conscious perception and perceptual echoes: a binocular rivalry study.

Alpha rhythms (∼10Hz) in the human brain are classically associated with idling activities, being predominantly observed during quiet restfulness with closed eyes. However, recent studies demonstrated that alpha (∼10Hz) rhythms can directly relate to visual stimulation, resulting in oscillations, which can last for as long as one second. This alpha reverberation, dubbed perceptual echoes (PE), suggests that the visual system actively samples and processes visual information within the alpha-band frequency. Although PE have been linked to various visual functions, their underlying mechanisms and functional role are not completely understood. In this study, we investigated the relationship between conscious perception and the generation and the amplitude of PE. Specifically, we displayed two coloured Gabor patches with different orientations on opposite sides of the screen, and using a set of dichoptic mirrors, we induced a binocular rivalry between the two stimuli. We asked participants to continuously report which one of two Gabor patches they consciously perceived, while recording their EEG signals. Importantly, the luminance of each patch fluctuated randomly over time, generating random sequences from which we estimated two impulse-response functions (IRFs) reflecting the PE generated by the perceived (dominant) and non-perceived (suppressed) stimulus, respectively. We found that the alpha power of the PE generated by the consciously perceived stimulus was comparable with that of the PE generated during monocular vision (control condition) and higher than the PE induced by the suppressed stimulus. Moreover, confirming previous findings, we found that all PEs propagated as a travelling wave from posterior to frontal brain regions, irrespective of conscious perception. All in all our results demonstrate a correlation between conscious perception and PE, suggesting that the synchronization of neural activity plays an important role in visual sampling and conscious perception.

Tentative fMRI signatures of perceptual echoes in early visual cortex.

The visual Impulse Response Function (IRF) can be estimated by cross-correlating random luminance sequences with concurrently recorded EEG. It typically contains a strong 10 Hz oscillatory component, suggesting that visual information reverberates in the human brain as a "perceptual echo". The neural origin of these echoes remains unknown. To address this question, we recorded EEG and fMRI in two separate sessions. In both sessions, a disk whose luminance followed a random (white noise) sequence was presented in the upper left quadrant. Individual IRFs were derived from the EEG session. These IRFs were then used as "response templates" to reconstruct an estimate of the EEG during the fMRI session, by convolution with the corresponding random luminance sequences. The 7-14 Hz (alpha, the main frequency component of the IRF) envelope of the reconstructed EEG was finally used as an fMRI regressor, to determine which brain voxels co-varied with the oscillations elicited by the luminance sequence, i.e., the "perceptual echoes". The reconstructed envelope of EEG alpha was significantly correlated with BOLD responses in V1 and V2. Surprisingly, this correlation was visible outside, but not within the directly (retinotopically) stimulated region. We tentatively interpret this lack of alpha modulation as a BOLD saturation effect, since the overall stimulus-induced BOLD response was inversely related, across voxels, to the signal variability over time. In conclusion, our results suggest that perceptual echoes originate in early visual cortex, driven by widespread activity in V1 and V2, not retinotopically restricted, but possibly reflecting the propagation of a travelling alpha wave.

Representational Content of Oscillatory Brain Activity during Object Recognition: Contrasting Cortical and Deep Neural Network Hierarchies.

Numerous theories propose a key role for brain oscillations in visual perception. Most of these theories postulate that sensory information is encoded in specific oscillatory components (e.g., power or phase) of specific frequency bands. These theories are often tested with whole-brain recording methods of low spatial resolution (EEG or MEG), or depth recordings that provide a local, incomplete view of the brain. Opportunities to bridge the gap between local neural populations and whole-brain signals are rare. Here, using representational similarity analysis (RSA) in human participants we explore which MEG oscillatory components (power and phase, across various frequency bands) correspond to low or high-level visual object representations, using brain representations from fMRI, or layer-wise representations in seven recent deep neural networks (DNNs), as a template for low/high-level object representations. The results showed that around stimulus onset and offset, most transient oscillatory signals correlated with low-level brain patterns (V1). During stimulus presentation, sustained ß (∼20 Hz) and γ (>60 Hz) power best correlated with V1, while oscillatory phase components correlated with IT representations. Surprisingly, this pattern of results did not always correspond to low-level or high-level DNN layer activity. In particular, sustained ß band oscillatory power reflected high-level DNN layers, suggestive of a feed-back component. These results begin to bridge the gap between whole-brain oscillatory signals and object representations supported by local neuronal activations.