A framework for generalizable neural networks for robust estimation of eyelids and pupils.

Deep neural networks (DNNs) have enabled recent advances in the accuracy and robustness of video-oculography. However, to make robust predictions, most DNN models require extensive and diverse training data, which is costly to collect and label. In this work, we seek to improve the codevelop pylids, a pupil- and eyelid-estimation DNN model based on DeepLabCut. We show that performance of pylids-based pupil estimation can be related to the distance of test data from the distribution of training data. Based on this principle, we explore methods for efficient data selection for training our DNN. We show that guided sampling of new data points from the training data approaches state-of-the-art pupil and eyelid estimation with fewer training data points. We also demonstrate the benefit of using an efficient sampling method to select data augmentations for training DNNs. These sampling methods aim to minimize the time and effort required to label and train DNNs while promoting model generalization on new diverse datasets.

Object representation in inferior temporal cortex is organized hierarchically in a mosaic-like structure.

There are two dominant models for the functional organization of brain regions underlying object recognition. One model postulates category-specific modules while the other proposes a distributed representation of objects with generic visual features. Functional imaging techniques relying on metabolic signals, such as fMRI and optical intrinsic signal imaging (OISI), have been used to support both models, but due to the indirect nature of the measurements in these techniques, the existing data for one model cannot be used to support the other model. Here, we used large-scale multielectrode recordings over a large surface of anterior inferior temporal (IT) cortex, and densely mapped stimulus-evoked neuronal responses. We found that IT cortex is subdivided into distinct domains characterized by similar patterns of responses to the objects in our stimulus set. Each domain spanned several millimeters on the cortex. Some of these domains represented faces ("face" domains) or monkey bodies ("monkey-body" domains). We also identified domains with low responsiveness to faces ("anti-face" domains). Meanwhile, the recording sites within domains that displayed category selectivity showed heterogeneous tuning profiles to different exemplars within each category. This local heterogeneity was consistent with the stimulus-evoked feature columns revealed by OISI. Taken together, our study revealed that regions with common functional properties (domains) consist of a finer functional structure (columns) in anterior IT cortex. The "domains" and previously proposed "patches" are rather like "mosaics" where a whole mosaic is characterized by overall similarity in stimulus responses and pieces of the mosaic correspond to feature columns.

Cortical representation of medial axis structure.

The identity of an object is not only specified by its parts but also by the relations among the parts. Rearranging parts can produce a completely different object, in the same manner as rearranging the phonemes in "fur" can yield "rough." How does the visual system represent the relative positions of parts? Between-part relations can be characterized by specifying the relations between the medial axes (imaginary lines through the centers) of an object's parts. A functional magnetic resonance imaging multivoxel classification study tested whether the medial axis structure is represented in the human visual system independent of part identity and overall object orientation. Stimuli were line drawings of novel 3-part geometrical objects, which differed in the relations between their parts' medial axes (i.e., in their medial axis structures), the geons that composed each object, and the objects' orientations in plane and in depth. In regions of interest throughout visual cortex, a support vector machine classifier was trained to distinguish objects that shared either the same medial axis structures or the same orientations. By the level of V3, different medial axis structures were more accurately classified than different orientations, indicating a change in the representation of shape compared with earlier visual areas.

Occipital-temporal cortical tuning to semantic and affective features of natural images predicts associated behavioral responses.

In everyday life, people need to respond appropriately to many types of emotional stimuli. Here, we investigate whether human occipital-temporal cortex (OTC) shows co-representation of the semantic category and affective content of visual stimuli. We also explore whether OTC transformation of semantic and affective features extracts information of value for guiding behavior. Participants viewed 1620 emotional natural images while functional magnetic resonance imaging data were acquired. Using voxel-wise modeling we show widespread tuning to semantic and affective image features across OTC. The top three principal components underlying OTC voxel-wise responses to image features encoded stimulus animacy, stimulus arousal and interactions of animacy with stimulus valence and arousal. At low to moderate dimensionality, OTC tuning patterns predicted behavioral responses linked to each image better than regressors directly based on image features. This is consistent with OTC representing stimulus semantic category and affective content in a manner suited to guiding behavior.

Human Scene-Selective Areas Represent 3D Configurations of Surfaces.

It has been argued that scene-selective areas in the human brain represent both the 3D structure of the local visual environment and low-level 2D features (such as spatial frequency) that provide cues for 3D structure. To evaluate the degree to which each of these hypotheses explains variance in scene-selective areas, we develop an encoding model of 3D scene structure and test it against a model of low-level 2D features. We fit the models to fMRI data recorded while subjects viewed visual scenes. The fit models reveal that scene-selective areas represent the distance to and orientation of large surfaces, at least partly independent of low-level features. Principal component analysis of the model weights reveals that the most important dimensions of 3D structure are distance and openness. Finally, reconstructions of the stimuli based on the model weights demonstrate that our model captures unprecedented detail about the local visual environment from scene-selective areas.

No Evidence for Automatic Remapping of Stimulus Features or Location Found with fMRI.

The input to our visual system shifts every time we move our eyes. To maintain a stable percept of the world, visual representations must be updated with each saccade. Near the time of a saccade, neurons in several visual areas become sensitive to the regions of visual space that their receptive fields occupy after the saccade. This process, known as remapping, transfers information from one set of neurons to another, and may provide a mechanism for visual stability. However, it is not clear whether remapping transfers information about stimulus features in addition to information about stimulus location. To investigate this issue, we recorded blood-oxygen-level dependent (BOLD) functional magnetic resonance imaging (fMRI) responses while human subjects viewed images of faces and houses (two visual categories with many feature differences). Immediately after some image presentations, subjects made a saccade that moved the previously stimulated location to the opposite side of the visual field. We then used a combination of univariate analyses and multivariate pattern analyses to test whether information about stimulus location and stimulus features were remapped to the ipsilateral hemisphere after the saccades. We found no reliable indication of stimulus feature remapping in any region. However, we also found no reliable indication of stimulus location remapping, despite the fact that our paradigm was highly similar to previous fMRI studies of remapping. The absence of location remapping in our study precludes strong conclusions regarding feature remapping. However, these results also suggest that measurement of location remapping with fMRI depends strongly on the details of the experimental paradigm used. We highlight differences in our approach from the original fMRI studies of remapping, discuss potential reasons for the failure to generalize prior location remapping results, and suggest directions for future research.

Fourier power, subjective distance, and object categories all provide plausible models of BOLD responses in scene-selective visual areas.

Perception of natural visual scenes activates several functional areas in the human brain, including the Parahippocampal Place Area (PPA), Retrosplenial Complex (RSC), and the Occipital Place Area (OPA). It is currently unclear what specific scene-related features are represented in these areas. Previous studies have suggested that PPA, RSC, and/or OPA might represent at least three qualitatively different classes of features: (1) 2D features related to Fourier power; (2) 3D spatial features such as the distance to objects in a scene; or (3) abstract features such as the categories of objects in a scene. To determine which of these hypotheses best describes the visual representation in scene-selective areas, we applied voxel-wise modeling (VM) to BOLD fMRI responses elicited by a set of 1386 images of natural scenes. VM provides an efficient method for testing competing hypotheses by comparing predictions of brain activity based on encoding models that instantiate each hypothesis. Here we evaluated three different encoding models that instantiate each of the three hypotheses listed above. We used linear regression to fit each encoding model to the fMRI data recorded from each voxel, and we evaluated each fit model by estimating the amount of variance it predicted in a withheld portion of the data set. We found that voxel-wise models based on Fourier power or the subjective distance to objects in each scene predicted much of the variance predicted by a model based on object categories. Furthermore, the response variance explained by these three models is largely shared, and the individual models explain little unique variance in responses. Based on an evaluation of previous studies and the data we present here, we conclude that there is currently no good basis to favor any one of the three alternative hypotheses about visual representation in scene-selective areas. We offer suggestions for further studies that may help resolve this issue.

Pycortex: an interactive surface visualizer for fMRI.

Surface visualizations of fMRI provide a comprehensive view of cortical activity. However, surface visualizations are difficult to generate and most common visualization techniques rely on unnecessary interpolation which limits the fidelity of the resulting maps. Furthermore, it is difficult to understand the relationship between flattened cortical surfaces and the underlying 3D anatomy using tools available currently. To address these problems we have developed pycortex, a Python toolbox for interactive surface mapping and visualization. Pycortex exploits the power of modern graphics cards to sample volumetric data on a per-pixel basis, allowing dense and accurate mapping of the voxel grid across the surface. Anatomical and functional information can be projected onto the cortical surface. The surface can be inflated and flattened interactively, aiding interpretation of the correspondence between the anatomical surface and the flattened cortical sheet. The output of pycortex can be viewed using WebGL, a technology compatible with modern web browsers. This allows complex fMRI surface maps to be distributed broadly online without requiring installation of complex software.

Neural encoding of relative position.

Late ventral visual areas generally consist of cells having a significant degree of translation invariance. Such a "bag of features" representation is useful for the recognition of individual objects; however, it seems unable to explain our ability to parse a scene into multiple objects and to understand their spatial relationships. We review several schemes (e.g., global features and serial attention) for how to reconcile bag-of-features representation with our ability to understand relationships; we review structural description theories that, in contrast, suggest that a neural binding mechanism assigns the features of each object in a scene to a separate "slot" to which relational information for that object is explicitly bound. Four functional magnetic resonance imaging-adaptation experiments assessed how ventral stream regions respond to rearrangements of two objects in a minimal scene that depict scene translations and relational changes. Changes of relative position (e.g., elephant above bus changing to bus above elephant) produced larger releases of adaptation in the anterior lateral occipital complex (LOC) than physically equivalent translations, providing evidence that spatial relations are explicitly encoded in the anterior LOC in agreement with structural description theories.

Adaptation in the fusiform face area (FFA): image or person?

Viewing a sequence of faces of two different people results in a greater Blood Oxygenation Level Dependent (BOLD) response in FFA compared to a sequence of identical faces. Changes in identity, however, necessarily involve changes in the image. Is the release from adaptation a result of a change in face identity, per se, or could it be an effect that would arise from any change in the image of a face? Subjects viewed a sequence of two faces that could be of the same or different person, and in the same or different orientation in depth. Critically, the physical similarity of view changes of the same person was scaled, by Gabor-jet differences, to be equivalent to that produced by an identity change. Both person and orientation changes produced equivalent releases from adaptation in FFA (relative to identical faces) suggesting that FFA is sensitive to the physical similarity of faces rather than to the individuals depicted in the images.

Adaptation to objects in the lateral occipital complex (LOC): shape or semantics?

A change in the basic-level class when viewing a sequence of two objects produces a large release from adaptation in LOC compared to when the images are identical. Is this due to a change in semantics or shape? In an fMRI-adaptation experiment, subjects viewed a sequence of two objects and judged whether the stimuli were identical in shape. Different-shaped stimuli could be from the same or different basic-level classes, where the physical similarities of the pairs in the two conditions were equated by a model of simple cell similarity. BOLD responses in LOC for the two conditions were equivalent, and higher than that of the identical condition, indicating that LOC is sensitive to shape rather than to basic-level semantics.