Predictive Processing of Scene Layout Depends on Naturalistic Depth of Field.

Boundary extension is a classic memory illusion in which observers remember more of a scene than was presented. According to predictive-processing accounts, boundary extension reflects the integration of visual input and expectations of what is beyond a scene's boundaries. According to normalization accounts, boundary extension rather reflects one end of a normalization process toward a scene's typically experienced viewing distance, such that close-up views give boundary extension but distant views give boundary contraction. Here, across four experiments (N = 125 adults), we found that boundary extension strongly depends on depth of field, as determined by the aperture settings on a camera. Photographs with naturalistic depth of field led to larger boundary extension than photographs with unnaturalistic depth of field, even when distant views were shown. We propose that boundary extension reflects a predictive mechanism with adaptive value that is strongest for naturalistic views of scenes. The current findings indicate that depth of field is an important variable to consider in the study of scene perception and memory.

Spectral brain signatures of aesthetic natural perception in the α and ß frequency bands.

During our everyday lives, visual beauty is often conveyed by sustained and dynamic visual stimulation, such as when we walk through an enchanting forest or watch our pets playing. Here, I devised an MEG experiment that mimics such situations: participants viewed 8 s videos of everyday situations and rated their beauty. Using multivariate analysis, I linked aesthetic ratings to 1) sustained MEG broadband responses and 2) spectral MEG responses in the α and ß frequency bands. These effects were not accounted for by a set of high- and low-level visual descriptors of the videos, suggesting that they are genuinely related to aesthetic perception. My findings provide the first characterization of spectral brain signatures linked to aesthetic experiences in the real world.NEW & NOTEWORTHY In the real world, aesthetic experiences arise from complex and dynamic inputs. This study shows that such aesthetic experiences are represented in a spectral neural code: cortical α and ß activity track our judgments of the aesthetic appearance of natural videos, providing a new starting point for studying the neural correlates of beauty through rhythmic brain activity.

Transferring and generalizing deep-learning-based neural encoding models across subjects.

Recent studies have shown the value of using deep learning models for mapping and characterizing how the brain represents and organizes information for natural vision. However, modeling the relationship between deep learning models and the brain (or encoding models), requires measuring cortical responses to large and diverse sets of natural visual stimuli from single subjects. This requirement limits prior studies to few subjects, making it difficult to generalize findings across subjects or for a population. In this study, we developed new methods to transfer and generalize encoding models across subjects. To train encoding models specific to a target subject, the models trained for other subjects were used as the prior models and were refined efficiently using Bayesian inference with a limited amount of data from the target subject. To train encoding models for a population, the models were progressively trained and updated with incremental data from different subjects. For the proof of principle, we applied these methods to functional magnetic resonance imaging (fMRI) data from three subjects watching tens of hours of naturalistic videos, while a deep residual neural network driven by image recognition was used to model visual cortical processing. Results demonstrate that the methods developed herein provide an efficient and effective strategy to establish both subject-specific and population-wide predictive models of cortical representations of high-dimensional and hierarchical visual features.

Reliable individual differences in fine-grained cortical functional architecture.

Fine-grained functional organization of cortex is not well-conserved across individuals. As a result, individual differences in cortical functional architecture are confounded by topographic idiosyncrasies-i.e., differences in functional-anatomical correspondence. In this study, we used hyperalignment to align information encoded in topographically variable patterns to study individual differences in fine-grained cortical functional architecture in a common representational space. We characterized the structure of individual differences using three common functional indices, and assessed the reliability of this structure across independent samples of data in a natural vision paradigm. Hyperalignment markedly improved the reliability of individual differences across all three indices by resolving topographic idiosyncrasies and accommodating information encoded in spatially fine-grained response patterns. Our results demonstrate that substantial individual differences in cortical functional architecture exist at fine spatial scales, but are inaccessible with anatomical normalization alone.

Decoding naturalistic experiences from human brain activity via distributed representations of words.

Natural visual scenes induce rich perceptual experiences that are highly diverse from scene to scene and from person to person. Here, we propose a new framework for decoding such experiences using a distributed representation of words. We used functional magnetic resonance imaging (fMRI) to measure brain activity evoked by natural movie scenes. Then, we constructed a high-dimensional feature space of perceptual experiences using skip-gram, a state-of-the-art distributed word embedding model. We built a decoder that associates brain activity with perceptual experiences via the distributed word representation. The decoder successfully estimated perceptual contents consistent with the scene descriptions by multiple annotators. Our results illustrate three advantages of our decoding framework: (1) three types of perceptual contents could be decoded in the form of nouns (objects), verbs (actions), and adjectives (impressions) contained in 10,000 vocabulary words; (2) despite using such a large vocabulary, we could decode novel words that were absent in the datasets to train the decoder; and (3) the inter-individual variability of the decoded contents co-varied with that of the contents of scene descriptions. These findings suggest that our decoding framework can recover diverse aspects of perceptual experiences in naturalistic situations and could be useful in various scientific and practical applications.

What is changing when: Decoding visual information in movies from human intracranial recordings.

The majority of visual recognition studies have focused on the neural responses to repeated presentations of static stimuli with abrupt and well-defined onset and offset times. In contrast, natural vision involves unique renderings of visual inputs that are continuously changing without explicitly defined temporal transitions. Here we considered commercial movies as a coarse proxy to natural vision. We recorded intracranial field potential signals from 1,284 electrodes implanted in 15 patients with epilepsy while the subjects passively viewed commercial movies. We could rapidly detect large changes in the visual inputs within approximately 100 ms of their occurrence, using exclusively field potential signals from ventral visual cortical areas including the inferior temporal gyrus and inferior occipital gyrus. Furthermore, we could decode the content of those visual changes even in a single movie presentation, generalizing across the wide range of transformations present in a movie. These results present a methodological framework for studying cognition during dynamic and natural vision.

Task-evoked functional connectivity does not explain functional connectivity differences between rest and task conditions.

During complex tasks, patterns of functional connectivity differ from those in the resting state. However, what accounts for such differences remains unclear. Brain activity during a task reflects an unknown mixture of spontaneous and task-evoked activities. The difference in functional connectivity between a task state and the resting state may reflect not only task-evoked functional connectivity, but also changes in spontaneously emerging networks. Here, we characterized the differences in apparent functional connectivity between the resting state and when human subjects were watching a naturalistic movie. Such differences were marginally explained by the task-evoked functional connectivity involved in processing the movie content. Instead, they were mostly attributable to changes in spontaneous networks driven by ongoing activity during the task. The execution of the task reduced the correlations in ongoing activity among different cortical networks, especially between the visual and non-visual sensory or motor cortices. Our results suggest that task-evoked activity is not independent from spontaneous activity, and that engaging in a task may suppress spontaneous activity and its inter-regional correlation.

Deep recurrent neural network reveals a hierarchy of process memory during dynamic natural vision.

The human visual cortex extracts both spatial and temporal visual features to support perception and guide behavior. Deep convolutional neural networks (CNNs) provide a computational framework to model cortical representation and organization for spatial visual processing, but unable to explain how the brain processes temporal information. To overcome this limitation, we extended a CNN by adding recurrent connections to different layers of the CNN to allow spatial representations to be remembered and accumulated over time. The extended model, or the recurrent neural network (RNN), embodied a hierarchical and distributed model of process memory as an integral part of visual processing. Unlike the CNN, the RNN learned spatiotemporal features from videos to enable action recognition. The RNN better predicted cortical responses to natural movie stimuli than the CNN, at all visual areas, especially those along the dorsal stream. As a fully observable model of visual processing, the RNN also revealed a cortical hierarchy of temporal receptive window, dynamics of process memory, and spatiotemporal representations. These results support the hypothesis of process memory, and demonstrate the potential of using the RNN for in-depth computational understanding of dynamic natural vision.

Saccade-based termination responses in macaque V1 and visual perception.

Neurons in visual areas of the brain are generally characterized by the increase in firing rate that occurs when a stimulus is flashed on in the receptive field (RF). However, neurons also increase their firing rate when a stimulus is turned off. These "termination responses" or "after-discharges" that occur with flashed stimuli have been observed in area V1 and they may be important for vision as stimulus terminations have been shown to influence visual perception. The goal of the present study was to determine the strength of termination responses in the more natural situation in which eye movements move a stimulus out of an RF. We find that termination responses do occur in macaque V1 when termination results from a saccadic eye movement, but they are smaller in amplitude compared to flashed-off stimuli. Furthermore, there are termination responses even in the absence of visual stimulation. These findings demonstrate that termination responses are a component of naturalistic vision. They appear to be based on both visual and nonvisual signals in visual cortex. We speculate that the weakening of termination responses might be a neural correlate of saccadic suppression, the loss of perceptual sensitivity around the time of saccades.

Attention Selectively Reshapes the Geometry of Distributed Semantic Representation.

Humans prioritize different semantic qualities of a complex stimulus depending on their behavioral goals. These semantic features are encoded in distributed neural populations, yet it is unclear how attention might operate across these distributed representations. To address this, we presented participants with naturalistic video clips of animals behaving in their natural environments while the participants attended to either behavior or taxonomy. We used models of representational geometry to investigate how attentional allocation affects the distributed neural representation of animal behavior and taxonomy. Attending to animal behavior transiently increased the discriminability of distributed population codes for observed actions in anterior intraparietal, pericentral, and ventral temporal cortices. Attending to animal taxonomy while viewing the same stimuli increased the discriminability of distributed animal category representations in ventral temporal cortex. For both tasks, attention selectively enhanced the discriminability of response patterns along behaviorally relevant dimensions. These findings suggest that behavioral goals alter how the brain extracts semantic features from the visual world. Attention effectively disentangles population responses for downstream read-out by sculpting representational geometry in late-stage perceptual areas.

Single-unit activity during natural vision: diversity, consistency, and spatial sensitivity among AF face patch neurons.

Several visual areas within the STS of the macaque brain respond strongly to faces and other biological stimuli. Determining the principles that govern neural responses in this region has proven challenging, due in part to the inherently complex stimulus domain of dynamic biological stimuli that are not captured by an easily parameterized stimulus set. Here we investigated neural responses in one fMRI-defined face patch in the anterior fundus (AF) of the STS while macaques freely view complex videos rich with natural social content. Longitudinal single-unit recordings allowed for the accumulation of each neuron's responses to repeated video presentations across sessions. We found that individual neurons, while diverse in their response patterns, were consistently and deterministically driven by the video content. We used principal component analysis to compute a family of eigenneurons, which summarized 24% of the shared population activity in the first two components. We found that the most prominent component of AF activity reflected an interaction between visible body region and scene layout. Close-up shots of faces elicited the strongest neural responses, whereas far away shots of faces or close-up shots of hindquarters elicited weak or inhibitory responses. Sensitivity to the apparent proximity of faces was also observed in gamma band local field potential. This category-selective sensitivity to spatial scale, together with the known exchange of anatomical projections of this area with regions involved in visuospatial analysis, suggests that the AF face patch may be specialized in aspects of face perception that pertain to the layout of a social scene.

Functional MRI mapping of dynamic visual features during natural viewing in the macaque.

The ventral visual pathway of the primate brain is specialized to respond to stimuli in certain categories, such as the well-studied face selective patches in the macaque inferotemporal cortex. To what extent does response selectivity determined using brief presentations of isolated stimuli predict activity during the free viewing of a natural, dynamic scene, where features are superimposed in space and time? To approach this question, we obtained fMRI activity from the brains of three macaques viewing extended video clips containing a range of social and nonsocial content and compared the fMRI time courses to a family of feature models derived from the movie content. Starting with more than two dozen feature models extracted from each movie, we created functional maps based on features whose time courses were nearly orthogonal, focusing primarily on faces, motion content, and contrast level. Activity mapping using the face feature model readily yielded functional regions closely resembling face patches obtained using a block design in the same animals. Overall, the motion feature model dominated responses in nearly all visually driven areas, including the face patches as well as ventral visual areas V4, TEO, and TE. Control experiments presenting dynamic movies, whose content was free of animals, demonstrated that biological movement critically contributed to the predominance of motion in fMRI responses. These results highlight the value of natural viewing paradigms for studying the brain's functional organization and also underscore the paramount contribution of magnocellular input to the ventral visual pathway during natural vision.

Functional selectivity in the human occipitotemporal cortex during natural vision: evidence from combined intracranial EEG and eye-tracking.

Eye movements are a constant and essential component of natural vision, yet, most of our knowledge about the human visual system comes from experiments that restrict them. This experimental constraint is mostly in place to control visual stimuli presentation and to avoid artifacts in non-invasive measures of brain activity, however, this limitation can be overcome with intracranial EEG (iEEG) recorded from epilepsy patients. Moreover, the high-frequency components of the iEEG signal (between about 50 and 150Hz) can provide a proxy of population-level spiking activity in any cortical area during free-viewing. We combined iEEG with high precision eye-tracking to study fine temporal dynamics and functional specificity in the fusiform face (FFA) and visual word form area (VWFA) while patients inspected natural pictures containing faces and text. We defined the first local measure of visual (electrophysiological) responsiveness adapted to free-viewing in humans: amplitude modulations in the high-frequency activity range (50-150Hz) following fixations (fixation-related high-frequency response). We showed that despite the large size of receptive fields in the ventral occipito-temporal cortex, neural activity during natural vision of realistic cluttered scenes is mostly dependent upon the category of the foveated stimulus - suggesting that category-specificity is preserved during free-viewing and that attention mechanisms might filter out the influence of objects surrounding the fovea.

Structurally-constrained encoding framework using a multi-voxel reduced-rank latent model for human natural vision.

Objective. Voxel-wise visual encoding models based on convolutional neural networks (CNNs) have emerged as one of the prominent predictive tools of human brain activity via functional magnetic resonance imaging signals. While CNN-based models imitate the hierarchical structure of the human visual cortex to generate explainable features in response to natural visual stimuli, there is still a need for a brain-inspired model to predict brain responses accurately based on biomedical data.Approach. To bridge this gap, we propose a response prediction module called the Structurally Constrained Multi-Output (SCMO) module to include homologous correlations that arise between a group of voxels in a cortical region and predict more accurate responses.Main results. This module employs all the responses across a visual area to predict individual voxel-wise BOLD responses and therefore accounts for the population activity and collective behavior of voxels. Such a module can determine the relationships within each visual region by creating a structure matrix that represents the underlying voxel-to-voxel interactions. Moreover, since each response module in visual encoding tasks relies on the image features, we conducted experiments using two different feature extraction modules to assess the predictive performance of our proposed module. Specifically, we employed a recurrent CNN that integrates both feedforward and recurrent interactions, as well as the popular AlexNet model that utilizes feedforward connections.Significance.We demonstrate that the proposed framework provides a reliable predictive ability to generate brain responses across multiple areas, outperforming benchmark models in terms of stability and coherency of features.

Orthogonal neural representations support perceptual judgements of natural stimuli.

In natural behavior, observers must separate relevant information from a barrage of irrelevant information. Many studies have investigated the neural underpinnings of this ability using artificial stimuli presented on simple backgrounds. Natural viewing, however, carries a set of challenges that are inaccessible using artificial stimuli, including neural responses to background objects that are task-irrelevant. An emerging body of evidence suggests that the visual abilities of humans and animals can be modeled through the linear decoding of task-relevant information from visual cortex. This idea suggests the hypothesis that irrelevant features of a natural scene should impair performance on a visual task only if their neural representations intrude on the linear readout of the task relevant feature, as would occur if the representations of task-relevant and irrelevant features are not orthogonal in the underlying neural population. We tested this hypothesis using human psychophysics and monkey neurophysiology, in response to parametrically variable naturalistic stimuli. We demonstrate that 1) the neural representation of one feature (the position of a central object) in visual area V4 is orthogonal to those of several background features, 2) the ability of human observers to precisely judge object position was largely unaffected by task-irrelevant variation in those background features, and 3) many features of the object and the background are orthogonally represented by V4 neural responses. Our observations are consistent with the hypothesis that orthogonal neural representations can support stable perception of objects and features despite the tremendous richness of natural visual scenes.

Gamma Responses to Colored Natural Stimuli Can Be Predicted from Local Low-Level Stimulus Features.

The role of gamma rhythm (30-80 Hz) in visual processing is debated; stimuli like gratings and hue patches generate strong gamma, but many natural images do not. Could image gamma responses be predicted by approximating images as gratings or hue patches? Surprisingly, this question remains unanswered, since the joint dependence of gamma on multiple features is poorly understood. We recorded local field potentials and electrocorticogram from two female monkeys while presenting natural images and parametric stimuli varying along several feature dimensions. Gamma responses to different grating/hue features were separable, allowing for a multiplicative model based on individual features. By fitting a hue patch to the image around the receptive field, this simple model could predict gamma responses to chromatic images across scales with reasonably high accuracy. Our results provide a simple "baseline" model to predict gamma from local image properties, against which more complex models of natural vision can be tested.

Spontaneous Beta Band Rhythms in the Predictive Coding of Natural Stimuli.

The regularity of the physical world and the biomechanics of the human body movements generate distributions of highly probable states that are internalized by the brain in the course of a lifetime. In Bayesian terms, the brain exploits prior knowledge, especially under conditions when sensory input is unavailable or uncertain, to predictively anticipate the most likely outcome of upcoming stimuli and movements. These internal models, formed during development, yet still malleable in adults, continuously adapt through the learning of novel stimuli and movements.Traditionally, neural beta (ß) oscillations are considered essential for maintaining sensorimotor and cognitive representations, and for temporal coding of expectations. However, recent findings show that fluctuations of ß band power in the resting state strongly correlate between cortical association regions. Moreover, central (hub) regions form strong interactions over time with different brain regions/networks (dynamic core). ß band centrality fluctuations of regions of the dynamic core predict global efficiency peaks suggesting a mechanism for network integration. Furthermore, this temporal architecture is surprisingly stable, both in topology and dynamics, during the observation of ecological natural visual scenes, whereas synthetic temporally scrambled stimuli modify it. We propose that spontaneous ß rhythms may function as a long-term "prior" of frequent environmental stimuli and behaviors.

Interaction of contexts in context-dependent orientation estimation.

The processing of a visual stimulus is known to be influenced by the statistics in recent visual history and by the stimulus' visual surround. Such contextual influences lead to perceptually salient phenomena, such as the tilt aftereffect and the tilt illusion. Despite much research on the influence of an isolated context, it is not clear how multiple, possibly competing sources of contextual influence interact. Here, using psychophysical methods, we compared the combined influence of multiple contexts to the sum of the isolated context influences. The results showed large deviations from linear additivity for adjacent or overlapping contexts, and remarkably, clear additivity when the contexts were sufficiently separated. Specifically, for adjacent or overlapping contexts, the combined effect was often lower than the sum of the isolated component effects (sub-additivity), or was more influenced by one component than another (selection). For contexts that were separated in time (600 ms), the combined effect measured the exact sum of the isolated component effects (in degrees of bias). Overall, the results imply an initial compressive transformation during visual processing, followed by selection between the processed parts.

Dynamic Signal Compression for Robust Motion Vision in Flies.

Sensory systems need to reliably extract information from highly variable natural signals. Flies, for instance, use optic flow to guide their course and are remarkably adept at estimating image velocity regardless of image statistics. Current circuit models, however, cannot account for this robustness. Here, we demonstrate that the Drosophila visual system reduces input variability by rapidly adjusting its sensitivity to local contrast conditions. We exhaustively map functional properties of neurons in the motion detection circuit and find that local responses are compressed by surround contrast. The compressive signal is fast, integrates spatially, and derives from neural feedback. Training convolutional neural networks on estimating the velocity of natural stimuli shows that this dynamic signal compression can close the performance gap between model and organism. Overall, our work represents a comprehensive mechanistic account of how neural systems attain the robustness to carry out survival-critical tasks in challenging real-world environments.

Functionally Distinct Gamma Range Activity Revealed by Stimulus Tuning in Human Visual Cortex.

Neocortical gamma activity has long been hypothesized as a mechanism for synchronizing brain regions to support visual perception and cognition more broadly. Although early studies focused on narrowband gamma oscillations (∼20-60 Hz), recent work has emphasized a more broadband "high-gamma" response (∼70-150+ Hz). These responses are often conceptually or analytically treated as synonymous markers of gamma activity. Using high-density intracranial recordings from the human visual cortex, we challenge this view by showing distinct spectral, temporal, and functional properties of narrow and broadband gamma. Across four experiments, narrowband gamma was strongly selective for gratings and long-wavelength colors, displaying a delayed response onset, sustained temporal profile, and contrast-dependent peak frequency. In addition, induced narrowband gamma oscillations lacked phase consistency across stimulus repetitions and displayed highly focal inter-site synchronization. In contrast, broadband gamma was consistently observed for all presented stimuli, displaying a rapid response onset, transient temporal profile, and invariant spectral properties. We exploited stimulus tuning to highlight the functional dissociation of these distinct signals, reconciling prior inconsistencies across species and stimuli regarding the ubiquity of visual gamma oscillations during natural vision. The occurrence of visual narrowband gamma oscillations, unlike broadband high gamma, appears contingent on specific structural and chromatic stimulus attributes intersecting with the receptive field. Together, these findings have important implications for the study, analysis, and functional interpretation of neocortical gamma-range activity.