How well do models of visual cortex generalize to out of distribution samples?

Unit activity in particular deep neural networks (DNNs) are remarkably similar to the neuronal population responses to static images along the primate ventral visual cortex. Linear combinations of DNN unit activities are widely used to build predictive models of neuronal activity in the visual cortex. Nevertheless, prediction performance in these models is often investigated on stimulus sets consisting of everyday objects under naturalistic settings. Recent work has revealed a generalization gap in how predicting neuronal responses to synthetically generated out-of-distribution (OOD) stimuli. Here, we investigated how the recent progress in improving DNNs' object recognition generalization, as well as various DNN design choices such as architecture, learning algorithm, and datasets have impacted the generalization gap in neural predictivity. We came to a surprising conclusion that the performance on none of the common computer vision OOD object recognition benchmarks is predictive of OOD neural predictivity performance. Furthermore, we found that adversarially robust models often yield substantially higher generalization in neural predictivity, although the degree of robustness itself was not predictive of neural predictivity score. These results suggest that improving object recognition behavior on current benchmarks alone may not lead to more general models of neurons in the primate ventral visual cortex.

Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks.

Primates, including humans, can typically recognize objects in visual images at a glance despite naturally occurring identity-preserving image transformations (e.g., changes in viewpoint). A primary neuroscience goal is to uncover neuron-level mechanistic models that quantitatively explain this behavior by predicting primate performance for each and every image. Here, we applied this stringent behavioral prediction test to the leading mechanistic models of primate vision (specifically, deep, convolutional, artificial neural networks; ANNs) by directly comparing their behavioral signatures against those of humans and rhesus macaque monkeys. Using high-throughput data collection systems for human and monkey psychophysics, we collected more than one million behavioral trials from 1472 anonymous humans and five male macaque monkeys for 2400 images over 276 binary object discrimination tasks. Consistent with previous work, we observed that state-of-the-art deep, feedforward convolutional ANNs trained for visual categorization (termed DCNNIC models) accurately predicted primate patterns of object-level confusion. However, when we examined behavioral performance for individual images within each object discrimination task, we found that all tested DCNNIC models were significantly nonpredictive of primate performance and that this prediction failure was not accounted for by simple image attributes nor rescued by simple model modifications. These results show that current DCNNIC models cannot account for the image-level behavioral patterns of primates and that new ANN models are needed to more precisely capture the neural mechanisms underlying primate object vision. To this end, large-scale, high-resolution primate behavioral benchmarks such as those obtained here could serve as direct guides for discovering such models.SIGNIFICANCE STATEMENT Recently, specific feedforward deep convolutional artificial neural networks (ANNs) models have dramatically advanced our quantitative understanding of the neural mechanisms underlying primate core object recognition. In this work, we tested the limits of those ANNs by systematically comparing the behavioral responses of these models with the behavioral responses of humans and monkeys at the resolution of individual images. Using these high-resolution metrics, we found that all tested ANN models significantly diverged from primate behavior. Going forward, these high-resolution, large-scale primate behavioral benchmarks could serve as direct guides for discovering better ANN models of the primate visual system.

Spectrotemporal dynamics of the EEG during working memory encoding and maintenance predicts individual behavioral capacity.

We investigated the effect of memory load on encoding and maintenance of information in working memory. Electroencephalography (EEG) signals were recorded while participants performed a modified Sternberg visual memory task. Independent component analysis (ICA) was used to factorise the EEG signals into distinct temporal activations to perform spectrotemporal analysis and localisation of source activities. We found 'encoding' and 'maintenance' operations were correlated with negative and positive changes in α-band power, respectively. Transient activities were observed during encoding of information in the bilateral cuneus, precuneus, inferior parietal gyrus and fusiform gyrus, and a sustained activity in the inferior frontal gyrus. Strong correlations were also observed between changes in α-power and behavioral performance during both encoding and maintenance. Furthermore, it was also found that individuals with higher working memory capacity experienced stronger neural oscillatory responses during the encoding of visual objects into working memory. Our results suggest an interplay between two distinct neural pathways and different spatiotemporal operations during the encoding and maintenance of information which predict individual differences in working memory capacity observed at the behavioral level.

Using modular connectome-based predictive modeling to reveal brain-behavior relationships of individual differences in working memory.

Working memory plays a crucial role in our daily lives, and brain imaging has been used to predict working memory performance. Here, we present an improved connectome-based predictive modeling approach for building a predictive model of individual working memory performance from whole-brain functional connectivity. The model was built using n-back task-based fMRI and resting-state fMRI data from the Human Connectome Project. Compared to prior models, our model was more interpretable, demonstrated a closer connection to the known anatomical and functional network. The model also demonstrates strong generalization on nine other cognitive behaviors from the HCP database and can well predict the working memory performance of healthy individuals in external datasets. By comparing the differences in prediction effects of different brain networks and anatomical feature analysis on n-back tasks, we found the essential role of some networks in differentiating between high and low working memory loads conditions.

Auditory cortex supports verbal working memory capacity.

Working memory (WM) is a fundamental construct of human cognition. The neural basis of auditory WM is thought to reflect a distributed brain network consisting of canonical memory and central executive brain regions including frontal lobe and hippocampus. Yet, the role of auditory (sensory) cortex in supporting active memory representations remains controversial. Here, we recorded neuroelectric activity via electroencephalogram as listeners actively performed an auditory version of the Sternberg memory task. Memory load was taxed by parametrically manipulating the number of auditory tokens (letter sounds) held in memory. Source analysis of scalp potentials showed that sustained neural activity maintained in auditory cortex (AC) prior to memory retrieval closely scaled with behavioral performance. Brain-behavior correlations revealed that lateralized modulations in left (but not right) AC were predictive of individual differences in auditory WM capacity. Our findings confirm a prominent role of AC, traditionally viewed as a sensory-perceptual processor, in actively maintaining memory traces and dictating individual differences in behavioral WM limits.

Computational models of category-selective brain regions enable high-throughput tests of selectivity.

Cortical regions apparently selective to faces, places, and bodies have provided important evidence for domain-specific theories of human cognition, development, and evolution. But claims of category selectivity are not quantitatively precise and remain vulnerable to empirical refutation. Here we develop artificial neural network-based encoding models that accurately predict the response to novel images in the fusiform face area, parahippocampal place area, and extrastriate body area, outperforming descriptive models and experts. We use these models to subject claims of category selectivity to strong tests, by screening for and synthesizing images predicted to produce high responses. We find that these high-response-predicted images are all unambiguous members of the hypothesized preferred category for each region. These results provide accurate, image-computable encoding models of each category-selective region, strengthen evidence for domain specificity in the brain, and point the way for future research characterizing the functional organization of the brain with unprecedented computational precision.

Neural population control via deep image synthesis.

Particular deep artificial neural networks (ANNs) are today's most accurate models of the primate brain's ventral visual stream. Using an ANN-driven image synthesis method, we found that luminous power patterns (i.e., images) can be applied to primate retinae to predictably push the spiking activity of targeted V4 neural sites beyond naturally occurring levels. This method, although not yet perfect, achieves unprecedented independent control of the activity state of entire populations of V4 neural sites, even those with overlapping receptive fields. These results show how the knowledge embedded in today's ANN models might be used to noninvasively set desired internal brain states at neuron-level resolution, and suggest that more accurate ANN models would produce even more accurate control.

Learning stable and predictive network-based patterns of schizophrenia and its clinical symptoms.

Schizophrenia is often associated with disrupted brain connectivity. However, identifying specific neuroimaging-based patterns pathognomonic for schizophrenia and related symptom severity remains a challenging open problem requiring large-scale data-driven analyses emphasizing not only statistical significance but also stability across multiple datasets, contexts and cohorts. Accurate prediction on previously unseen subjects, or generalization, is also essential for any useful biomarker of schizophrenia. In order to build a predictive model based on functional network feature patterns, we studied whole-brain fMRI functional networks, both at the voxel level and lower-resolution supervoxel level. Targeting Auditory Oddball task data on the FBIRN fMRI dataset (n = 95), we considered node-degree and link-weight network features and evaluated stability and generalization accuracy of statistically significant feature sets in discriminating patients vs. CONTROLS: We also applied sparse multivariate regression (elastic net) to whole-brain functional connectivity features, for the first time, to derive stable predictive features for symptom severity. Whole-brain link-weight features achieved 74% accuracy in identifying patients and were more stable than voxel-wise node-degrees. Link-weight features predicted severity of several negative and positive symptom scales, including inattentiveness and bizarre behavior. The most-significant, stable and discriminative functional connectivity changes involved increased correlations between thalamus and primary motor/primary sensory cortex, and between precuneus (BA7) and thalamus, putamen, and Brodmann areas BA9 and BA44. Precuneus, along with BA6 and primary sensory cortex, was also involved in predicting severity of several symptoms. Overall, the proposed multi-step methodology may help identify more reliable multivariate patterns allowing for accurate prediction of schizophrenia and its symptoms severity.