End-to-end neural system identification with neural information flow.

Neural information flow (NIF) provides a novel approach for system identification in neuroscience. It models the neural computations in multiple brain regions and can be trained end-to-end via stochastic gradient descent from noninvasive data. NIF models represent neural information processing via a network of coupled tensors, each encoding the representation of the sensory input contained in a brain region. The elements of these tensors can be interpreted as cortical columns whose activity encodes the presence of a specific feature in a spatiotemporal location. Each tensor is coupled to the measured data specific to a brain region via low-rank observation models that can be decomposed into the spatial, temporal and feature receptive fields of a localized neuronal population. Both these observation models and the convolutional weights defining the information processing within regions are learned end-to-end by predicting the neural signal during sensory stimulation. We trained a NIF model on the activity of early visual areas using a large-scale fMRI dataset recorded in a single participant. We show that we can recover plausible visual representations and population receptive fields that are consistent with empirical findings.

Classifying sex and strain from mouse ultrasonic vocalizations using deep learning.

Vocalizations are widely used for communication between animals. Mice use a large repertoire of ultrasonic vocalizations (USVs) in different social contexts. During social interaction recognizing the partner's sex is important, however, previous research remained inconclusive whether individual USVs contain this information. Using deep neural networks (DNNs) to classify the sex of the emitting mouse from the spectrogram we obtain unprecedented performance (77%, vs. SVM: 56%, Regression: 51%). Performance was even higher (85%) if the DNN could also use each mouse's individual properties during training, which may, however, be of limited practical value. Splitting estimation into two DNNs and using 24 extracted features per USV, spectrogram-to-features and features-to-sex (60%) failed to reach single-step performance. Extending the features by each USVs spectral line, frequency and time marginal in a semi-convolutional DNN resulted in a performance mid-way (64%). Analyzing the network structure suggests an increase in sparsity of activation and correlation with sex, specifically in the fully-connected layers. A detailed analysis of the USV structure, reveals a subset of male vocalizations characterized by a few acoustic features, while the majority of sex differences appear to rely on a complex combination of many features. The same network architecture was also able to achieve above-chance classification for cortexless mice, which were considered indistinguishable before. In summary, spectrotemporal differences between male and female USVs allow at least their partial classification, which enables sexual recognition between mice and automated attribution of USVs during analysis of social interactions.

Generative adversarial networks for reconstructing natural images from brain activity.

We explore a method for reconstructing visual stimuli from brain activity. Using large databases of natural images we trained a deep convolutional generative adversarial network capable of generating gray scale photos, similar to stimuli presented during two functional magnetic resonance imaging experiments. Using a linear model we learned to predict the generative model's latent space from measured brain activity. The objective was to create an image similar to the presented stimulus image through the previously trained generator. Using this approach we were able to reconstruct structural and some semantic features of a proportion of the natural images sets. A behavioural test showed that subjects were capable of identifying a reconstruction of the original stimulus in 67.2% and 66.4% of the cases in a pairwise comparison for the two natural image datasets respectively. Our approach does not require end-to-end training of a large generative model on limited neuroimaging data. Rapid advances in generative modeling promise further improvements in reconstruction performance.

Convolutional neural network-based encoding and decoding of visual object recognition in space and time.

Representations learned by deep convolutional neural networks (CNNs) for object recognition are a widely investigated model of the processing hierarchy in the human visual system. Using functional magnetic resonance imaging, CNN representations of visual stimuli have previously been shown to correspond to processing stages in the ventral and dorsal streams of the visual system. Whether this correspondence between models and brain signals also holds for activity acquired at high temporal resolution has been explored less exhaustively. Here, we addressed this question by combining CNN-based encoding models with magnetoencephalography (MEG). Human participants passively viewed 1,000 images of objects while MEG signals were acquired. We modelled their high temporal resolution source-reconstructed cortical activity with CNNs, and observed a feed-forward sweep across the visual hierarchy between 75 and 200 ms after stimulus onset. This spatiotemporal cascade was captured by the network layer representations, where the increasingly abstract stimulus representation in the hierarchical network model was reflected in different parts of the visual cortex, following the visual ventral stream. We further validated the accuracy of our encoding model by decoding stimulus identity in a left-out validation set of viewed objects, achieving state-of-the-art decoding accuracy.

Decoding of task-relevant and task-irrelevant intracranial EEG representations.

Natural stimuli consist of multiple properties. However, not all of these properties are equally relevant in a given situation. In this study, we applied multivariate classification algorithms to intracranial electroencephalography data of human epilepsy patients performing an auditory Stroop task. This allowed us to identify neuronal representations of task-relevant and irrelevant pitch and semantic information of spoken words in a subset of patients. When properties were relevant, representations could be detected after about 350ms after stimulus onset. When irrelevant, the association with gamma power differed for these properties. Patients with more reliable representations of irrelevant pitch showed increased gamma band activity (35-64Hz), suggesting that attentional resources allow an increase in gamma power in some but not all patients. This effect was not observed for irrelevant semantics, possibly because the more automatic processing of this property allowed for less variation in free resources. Processing of different properties of the same stimulus seems therefore to be dependent on the characteristics of the property.

Probabilistic model-based functional parcellation reveals a robust, fine-grained subdivision of the striatum.

The striatum is involved in many different aspects of behaviour, reflected by the variety of cortical areas that provide input to this structure. This input is topographically organized and is likely to result in functionally specific signals. Such specificity can be examined using functional clustering approaches. Here, we propose a Bayesian model-based functional clustering approach applied solely to resting state striatal functional MRI timecourses to identify intrinsic striatal functional modules. Data from two sets of ten participants were used to obtain parcellations and examine their robustness. This stable clustering was used to initialize a more constrained model in order to obtain individualized parcellations in 57 additional participants. Resulting cluster time courses were used to examine functional connectivity between clusters and related to the rest of the brain in a GLM analysis. We find six distinct clusters in each hemisphere, with clear inter-hemispheric correspondence and functional relevance. These clusters exhibit functional connectivity profiles that further underscore their homologous nature and are consistent with existing notions on segregation and integration in parallel cortico-basal ganglia loops. Our findings suggest that multiple territories within both the affective and motor regions can be distinguished solely using resting state functional MRI from these regions.

MEG-based decoding of the spatiotemporal dynamics of visual category perception.

Visual processing is a complex task which is best investigated using sensitive multivariate analysis methods that can capture representation-specific brain activity over both time and space. In this study, we applied a multivariate decoding algorithm to MEG data of subjects engaged in passive viewing of images of faces, scenes, bodies and tools. We used reconstructed source-space time courses as input to the algorithm in order to localize brain regions involved in optimal image discrimination. Applying this method to the interval of 115 to 315 ms after stimulus onset, we show a focal localization of regression coefficients in the inferior occipital, middle occipital, and lingual gyrus that drive decoding of the different perceived image categories. Classifier accuracy was highest (over 90% correctly classified trials, compared to a chance level accuracy of 50%) when dissociating the perception of faces from perception of other object categories. Furthermore, we applied this method to each single time point to extract the temporal evolution of visual perception. This allowed for the detection of differences in visual category perception as early as 85 ms after stimulus onset. Furthermore, localizing the corresponding regression coefficients of each time point allowed us to capture the spatiotemporal dynamics of visual category perception. This revealed initial involvement of sources in the inferior occipital, inferior temporal and superior occipital gyrus. During sustained stimulation additional sources in the anterior inferior temporal gyrus and superior parietal gyrus became involved. We conclude that decoding of source-space MEG data provides a suitable method to investigate the spatiotemporal dynamics of ongoing cognitive processing.

Between-subject variability in the influence of mental imagery on conscious perception.

Mental imagery and visual perception rely on similar neural mechanisms, but the function of this overlap remains unclear. One idea is that imagery can influence perception. Previous research has shown that imagining a stimulus prior to binocular presentation of rivalling stimuli increases the chance of perceiving the imagined stimulus. In this study we investigated how this effect interacts with bottom-up sensory input by comparing psychometric response curves for congruent and incongruent imagery in humans. A Bayesian hierarchical model was used, allowing us to simultaneously study group-level effects as well as effects for individual participants. We found strong effects of both imagery as well as its interaction with sensory evidence within individual participants. However, the direction of these effects were highly variable between individuals, leading to weak effects at the group level. This highlights the heterogeneity of conscious perception and emphasizes the need for individualized investigation of such complex cognitive processes.

Parkinsonian rest tremor can be detected accurately based on neuronal oscillations recorded from the subthalamic nucleus.

OBJECTIVE: To investigate the possibility of tremor detection based on deep brain activity. METHODS: We re-analyzed recordings of local field potentials (LFPs) from the subthalamic nucleus in 10 PD patients (12 body sides) with spontaneously fluctuating rest tremor. Power in several frequency bands was estimated and used as input to Hidden Markov Models (HMMs) which classified short data segments as either tremor-free rest or rest tremor. HMMs were compared to direct threshold application to individual power features. RESULTS: Applying a threshold directly to band-limited power was insufficient for tremor detection (mean area under the curve [AUC] of receiver operating characteristic: 0.64, STD: 0.19). Multi-feature HMMs, in contrast, allowed for accurate detection (mean AUC: 0.82, STD: 0.15), using four power features obtained from a single contact pair. Within-patient training yielded better accuracy than across-patient training (0.84vs. 0.78, p=0.03), yet tremor could often be detected accurately with either approach. High frequency oscillations (>200Hz) were the best performing individual feature. CONCLUSIONS: LFP-based markers of tremor are robust enough to allow for accurate tremor detection in short data segments, provided that appropriate statistical models are used. SIGNIFICANCE: LFP-based markers of tremor could be useful control signals for closed-loop deep brain stimulation.

Distinct Top-down and Bottom-up Brain Connectivity During Visual Perception and Imagery.

Research suggests that perception and imagination engage neuronal representations in the same visual areas. However, the underlying mechanisms that differentiate sensory perception from imagination remain unclear. Here, we examine the directed coupling (effective connectivity) between fronto-parietal and visual areas during perception and imagery. We found an increase in bottom-up coupling during perception relative to baseline and an increase in top-down coupling during both perception and imagery, with a much stronger increase during imagery. Modulation of the coupling from frontal to early visual areas was common to both perception and imagery. Furthermore, we show that the experienced vividness during imagery was selectively associated with increases in top-down connectivity to early visual cortex. These results highlight the importance of top-down processing in internally as well as externally driven visual experience.

Predictability of depression severity based on posterior alpha oscillations.

OBJECTIVE: We aimed to integrate neural data and an advanced machine learning technique to predict individual major depressive disorder (MDD) patient severity. METHODS: MEG data was acquired from 22 MDD patients and 22 healthy controls (HC) resting awake with eyes closed. Individual power spectra were calculated by a Fourier transform. Sources were reconstructed via beamforming technique. Bayesian linear regression was applied to predict depression severity based on the spatial distribution of oscillatory power. RESULTS: In MDD patients, decreased theta (4-8 Hz) and alpha (8-14 Hz) power was observed in fronto-central and posterior areas respectively, whereas increased beta (14-30 Hz) power was observed in fronto-central regions. In particular, posterior alpha power was negatively related to depression severity. The Bayesian linear regression model showed significant depression severity prediction performance based on the spatial distribution of both alpha (r=0.68, p=0.0005) and beta power (r=0.56, p=0.007) respectively. CONCLUSIONS: Our findings point to a specific alteration of oscillatory brain activity in MDD patients during rest as characterized from MEG data in terms of spectral and spatial distribution. SIGNIFICANCE: The proposed model yielded a quantitative and objective estimation for the depression severity, which in turn has a potential for diagnosis and monitoring of the recovery process.

On the use of interaction error potentials for adaptive brain computer interfaces.

We propose an adaptive classification method for the Brain Computer Interfaces (BCI) which uses Interaction Error Potentials (IErrPs) as a reinforcement signal and adapts the classifier parameters when an error is detected. We analyze the quality of the proposed approach in relation to the misclassification of the IErrPs. In addition we compare static versus adaptive classification performance using artificial and MEG data. We show that the proposed adaptive framework significantly improves the static classification methods.