The influence of stereopsis on visual saliency in a proto-object based model of selective attention.

Some animals including humans use stereoscopic vision which reconstructs spatial information about the environment from the disparity between images captured by eyes in two separate adjacent locations. Like other sensory information, such stereoscopic information is expected to influence attentional selection. We develop a biologically plausible model of binocular vision to study its effect on bottom-up visual attention, i.e., visual saliency. In our model, the scene is organized in terms of proto-objects on which attention acts, rather than on unbound sets of elementary features. We show that taking into account the stereoscopic information improves the performance of the model in the prediction of human eye movements with statistically significant differences.

The cellular architecture of memory modules in Drosophila supports stochastic input integration.

The ability to associate neutral stimuli with valence information and to store these associations as memories forms the basis for decision making. To determine the underlying computational principles, we build a realistic computational model of a central decision module within the Drosophila mushroom body (MB), the fly's center for learning and memory. Our model combines the electron microscopy-based architecture of one MB output neuron (MBON-α3), the synaptic connectivity of its 948 presynaptic Kenyon cells (KCs), and its membrane properties obtained from patch-clamp recordings. We show that this neuron is electrotonically compact and that synaptic input corresponding to simulated odor input robustly drives its spiking behavior. Therefore, sparse innervation by KCs can efficiently control and modulate MBON activity in response to learning with minimal requirements on the specificity of synaptic localization. This architecture allows efficient storage of large numbers of memories using the flexible stochastic connectivity of the circuit.

Quantifying Phase-Amplitude Modulation in Neural Data.

Phase-amplitude modulation (the modulation of the amplitude of higher frequency oscillations by the phase of lower frequency oscillations) is a specific type of cross-frequency coupling that has been observed in neural recordings from multiple species in a range of behavioral contexts. Given its potential importance, care must be taken with how it is measured and quantified. Previous studies have quantified phase-amplitude modulation by measuring the distance of the amplitude distribution from a uniform distribution. While this method is of general applicability, it is not targeted to the specific modulation pattern frequently observed with low-frequency oscillations. Here we develop a new method that has increased specificity to detect modulation in the sinusoidal shape commonly observed in neural data.

Decomposing Executive Function into Distinct Processes Underlying Human Decision Making.

Executive function (EF) consists of higher level cognitive processes including working memory, cognitive flexibility, and inhibition which together enable goal-directed behaviors. Many neurological disorders are associated with EF dysfunctions which can lead to suboptimal behavior. To assess the roles of these processes, we introduce a novel behavioral task and modeling approach. The gamble-like task, with sub-tasks targeting different EF capabilities, allows for quantitative assessment of the main components of EF. We demonstrate that human participants exhibit dissociable variability in the component processes of EF. These results will allow us to map behavioral outcomes to EEG recordings in future work in order to map brain networks associated with EF deficits. Clinical relevance- This work will allow us to quantify EF deficits and corresponding brain activity in patient populations in future work.

Event-driven proto-object based saliency in 3D space to attract a robot's attention.

To interact with its environment, a robot working in 3D space needs to organise its visual input in terms of objects or their perceptual precursors, proto-objects. Among other visual cues, depth is a submodality used to direct attention to visual features and objects. Current depth-based proto-object attention models have been implemented for standard RGB-D cameras that produce synchronous frames. In contrast, event cameras are neuromorphic sensors that loosely mimic the function of the human retina by asynchronously encoding per-pixel brightness changes at very high temporal resolution, thereby providing advantages like high dynamic range, efficiency (thanks to their high degree of signal compression), and low latency. We propose a bio-inspired bottom-up attention model that exploits event-driven sensing to generate depth-based saliency maps that allow a robot to interact with complex visual input. We use event-cameras mounted in the eyes of the iCub humanoid robot to directly extract edge, disparity and motion information. Real-world experiments demonstrate that our system robustly selects salient objects near the robot in the presence of clutter and dynamic scene changes, for the benefit of downstream applications like object segmentation, tracking and robot interaction with external objects.

Standing out in a small crowd: The role of display size in attracting attention.

Strong evidence supporting the top-down modulation of attention has come from studies in which participants learned to suppress a singleton in a heterogeneous four-item display. These studies have been criticized on the grounds that the displays are so sparse that the singleton is not actually salient. We argue that similar evidence of suppression has been found with substantially larger displays where salience is not in question. Additionally, we examine the results of applying salience models to four-item displays, and find prominent markers of salience at the location of the singleton. We conclude that small heterogeneous displays do not preclude strong salience signals. Beyond that, we reflect on how further basic research on salience may speed resolution of the attentional capture debate.

A Neuromorphic Proto-Object Based Dynamic Visual Saliency Model With a Hybrid FPGA Implementation.

Computing and attending to salient regions of a visual scene is an innate and necessary preprocessing step for both biological and engineered systems performing high-level visual tasks including object detection, tracking, and classification. Computational bandwidth and speed are improved by preferentially devoting computational resources to salient regions of the visual field. The human brain computes saliency effortlessly, but modeling this task in engineered systems is challenging. We first present a neuromorphic dynamic saliency model, which is bottom-up, feed-forward, and based on the notion of proto-objects with neurophysiological spatio-temporal features requiring no training. Our neuromorphic model outperforms state-of-the-art dynamic visual saliency models in predicting human eye fixations (i.e., ground truth saliency). Secondly, we present a hybrid FPGA implementation of the model for real-time applications, capable of processing 112×84 resolution frames at 18.71 Hz running at a 100 MHz clock rate - a 23.77× speedup from the software implementation. Additionally, our fixed-point model of the FPGA implementation yields comparable results to the software implementation.

Naturalistic Spike Trains Drive State-Dependent Homeostatic Plasticity in Superficial Layers of Visual Cortex.

The history of neural activity determines the synaptic plasticity mechanisms employed in the brain. Previous studies report a rapid reduction in the strength of excitatory synapses onto layer 2/3 (L2/3) pyramidal neurons of the primary visual cortex (V1) following two days of dark exposure and subsequent re-exposure to light. The abrupt increase in visually driven activity is predicted to drive homeostatic plasticity, however, the parameters of neural activity that trigger these changes are unknown. To determine this, we first recorded spike trains in vivo from V1 layer 4 (L4) of dark exposed (DE) mice of both sexes that were re-exposed to light through homogeneous or patterned visual stimulation. We found that delivering the spike patterns recorded in vivo to L4 of V1 slices was sufficient to reduce the amplitude of miniature excitatory postsynaptic currents (mEPSCs) of V1 L2/3 neurons in DE mice, but not in slices obtained from normal reared (NR) controls. Unexpectedly, the same stimulation pattern produced an up-regulation of mEPSC amplitudes in V1 L2/3 neurons from mice that received 2 h of light re-exposure (LE). A Poisson spike train exhibiting the same average frequency as the patterns recorded in vivo was equally effective at depressing mEPSC amplitudes in L2/3 neurons in V1 slices prepared from DE mice. Collectively, our results suggest that the history of visual experience modifies the responses of V1 neurons to stimulation and that rapid homeostatic depression of excitatory synapses can be driven by non-patterned input activity.

Analysis of spiking synchrony in visual cortex reveals distinct types of top-down modulation signals for spatial and object-based attention.

The activity of a border ownership selective (BOS) neuron indicates where a foreground object is located relative to its (classical) receptive field (RF). A population of BOS neurons thus provides an important component of perceptual grouping, the organization of the visual scene into objects. In previous theoretical work, it has been suggested that this grouping mechanism is implemented by a population of dedicated grouping ("G") cells that integrate the activity of the distributed feature cells representing an object and, by feedback, modulate the same cells, thus making them border ownership selective. The feedback modulation by G cells is thought to also provide the mechanism for object-based attention. A recent modeling study showed that modulatory common feedback, implemented by synapses with N-methyl-D-aspartate (NMDA)-type glutamate receptors, accounts for the experimentally observed synchrony in spike trains of BOS neurons and the shape of cross-correlations between them, including its dependence on the attentional state. However, that study was limited to pairs of BOS neurons with consistent border ownership preferences, defined as two neurons tuned to respond to the same visual object, in which attention decreases synchrony. But attention has also been shown to increase synchrony in neurons with inconsistent border ownership selectivity. Here we extend the computational model from the previous study to fully understand these effects of attention. We postulate the existence of a second type of G-cell that represents spatial attention by modulating the activity of all BOS cells in a spatially defined area. Simulations of this model show that a combination of spatial and object-based mechanisms fully accounts for the observed pattern of synchrony between BOS neurons. Our results suggest that modulatory feedback from G-cells may underlie both spatial and object-based attention.

Proto-Object Based Saliency Model With Texture Detection Channel.

The amount of visual information projected from the retina to the brain exceeds the information processing capacity of the latter. Attention, therefore, functions as a filter to highlight important information at multiple stages of the visual pathway that requires further and more detailed analysis. Among other functions, this determines where to fixate since only the fovea allows for high resolution imaging. Visual saliency modeling, i.e. understanding how the brain selects important information to analyze further and to determine where to fixate next, is an important research topic in computational neuroscience and computer vision. Most existing bottom-up saliency models use low-level features such as intensity and color, while some models employ high-level features, like faces. However, little consideration has been given to mid-level features, such as texture, for visual saliency models. In this paper, we extend a biologically plausible proto-object based saliency model by adding simple texture channels which employ nonlinear operations that mimic the processing performed by primate visual cortex. The extended model shows statistically significant improved performance in predicting human fixations compared to the previous model. Comparing the performance of our model with others on publicly available benchmarking datasets, we find that our biologically plausible model matches the performance of other models, even though those were designed entirely for maximal performance with little regard to biological realism.

The formation of preference in risky choice.

A key question in decision-making is how people integrate amounts and probabilities to form preferences between risky alternatives. Here we rely on the general principle of integration-to-boundary to develop several biologically plausible process models of risky-choice, which account for both choices and response-times. These models allowed us to contrast two influential competing theories: i) within-alternative evaluations, based on multiplicative interaction between amounts and probabilities, ii) within-attribute comparisons across alternatives. To constrain the preference formation process, we monitored eye-fixations during decisions between pairs of simple lotteries, designed to systematically span the decision-space. The behavioral results indicate that the participants' eye-scanning patterns were associated with risk-preferences and expected-value maximization. Crucially, model comparisons showed that within-alternative process models decisively outperformed within-attribute ones, in accounting for choices and response-times. These findings elucidate the psychological processes underlying preference formation when making risky-choices, and suggest that compensatory, within-alternative integration is an adaptive mechanism employed in human decision-making.

Figure-Ground Organization in Natural Scenes: Performance of a Recurrent Neural Model Compared with Neurons of Area V2.

A crucial step in understanding visual input is its organization into meaningful components, in particular object contours and partially occluded background structures. This requires that all contours are assigned to either the foreground or the background (border ownership assignment). While earlier studies showed that neurons in primate extrastriate cortex signal border ownership for simple geometric shapes, recent studies show consistent border ownership coding also for complex natural scenes. In order to understand how the brain performs this task, we developed a biologically plausible recurrent neural network that is fully image computable. Our model uses local edge detector ( B ) cells and grouping ( G ) cells whose activity represents proto-objects based on the integration of local feature information. G cells send modulatory feedback connections to those B cells that caused their activation, making the B cells border ownership selective. We found close agreement between our model and neurophysiological results in terms of the timing of border ownership signals (BOSs) as well as the consistency of BOSs across scenes. We also benchmarked our model on the Berkeley Segmentation Dataset and achieved performance comparable to recent state-of-the-art computer vision approaches. Our proposed model provides insight into the cortical mechanisms of figure-ground organization.

Unique objects attract attention even when faint.

Locally contrasting objects, e.g. a red apple surrounded by green apples, attract attention. Does this generalize to differences in feature space? That is, do unique objects-regardless of their location-stand out from a collection of objects that are similar to one another, even when the unique object has lower local contrast with the background than the other objects? Behavioral data show indeed a preference for unique items but previous experiments enabled viewers to anticipate what response they were "supposed" to give. We developed a new experimental paradigm that minimizes such top-down effects. Pitting local contrast against global uniqueness, we show that unique stimuli attract attention even in not-anticipated, never-seen images, and even when the unique stimuli are faint (low contrast). A computational model explains how competition between objects in feature space favors dissimilar objects over those with similar features. The model explains how humans select unique objects, without a loss of performance on natural scenes.

Risk-taking bias in human decision-making is encoded via a right-left brain push-pull system.

A person's decisions vary even when options stay the same, like when a gambler changes bets despite constant odds of winning. Internal bias (e.g., emotion) contributes to this variability and is shaped by past outcomes, yet its neurobiology during decision-making is not well understood. To map neural circuits encoding bias, we administered a gambling task to 10 participants implanted with intracerebral depth electrodes in cortical and subcortical structures. We predicted the variability in betting behavior within and across patients by individual bias, which is estimated through a dynamical model of choice. Our analysis further revealed that high-frequency activity increased in the right hemisphere when participants were biased toward risky bets, while it increased in the left hemisphere when participants were biased away from risky bets. Our findings provide electrophysiological evidence that risk-taking bias is a lateralized push-pull neural system governing counterintuitive and highly variable decision-making in humans.

Analytically determining frequency and amplitude of spontaneous alpha oscillation in Jansen's neural mass model using the describing function method.

Spontaneous alpha oscillations are a ubiquitous phenomenon in the brain and play a key role in neural information processing and various cognitive functions. Jansen's neural mass model (NMM) was initially proposed to study the origin of alpha oscillations. Most of previous studies of the spontaneous alpha oscillations in the NMM were conducted using numerical methods. In this study, we aim to propose an analytical approach using the describing function method to elucidate the spontaneous alpha oscillation mechanism in the NMM. First, the sigmoid nonlinear function in the NMM is approximated by its describing function, allowing us to reformulate the NMM and derive its standard form composed of one nonlinear part and one linear part. Second, by conducting a theoretical analysis, we can assess whether or not the spontaneous alpha oscillation would occur in the NMM and, furthermore, accurately determine its amplitude and frequency. The results reveal analytically that the interaction between linearity and nonlinearity of the NMM plays a key role in generating the spontaneous alpha oscillations. Furthermore, strong nonlinearity and large linear strength are required to generate the spontaneous alpha oscillations.

Short-term depression and transient memory in sensory cortex.

Persistent neuronal activity is usually studied in the context of short-term memory localized in central cortical areas. Recent studies show that early sensory areas also can have persistent representations of stimuli which emerge quickly (over tens of milliseconds) and decay slowly (over seconds). Traditional positive feedback models cannot explain sensory persistence for at least two reasons: (i) They show attractor dynamics, with transient perturbations resulting in a quasi-permanent change of system state, whereas sensory systems return to the original state after a transient. (ii) As we show, those positive feedback models which decay to baseline lose their persistence when their recurrent connections are subject to short-term depression, a common property of excitatory connections in early sensory areas. Dual time constant network behavior has also been implemented by nonlinear afferents producing a large transient input followed by much smaller steady state input. We show that such networks require unphysiologically large onset transients to produce the rise and decay observed in sensory areas. Our study explores how memory and persistence can be implemented in another model class, derivative feedback networks. We show that these networks can operate with two vastly different time courses, changing their state quickly when new information is coming in but retaining it for a long time, and that these capabilities are robust to short-term depression. Specifically, derivative feedback networks with short-term depression that acts differentially on positive and negative feedback projections are capable of dynamically changing their time constant, thus allowing fast onset and slow decay of responses without requiring unrealistically large input transients.

A recurrent neural model for proto-object based contour integration and figure-ground segregation.

Visual processing of objects makes use of both feedforward and feedback streams of information. However, the nature of feedback signals is largely unknown, as is the identity of the neuronal populations in lower visual areas that receive them. Here, we develop a recurrent neural model to address these questions in the context of contour integration and figure-ground segregation. A key feature of our model is the use of grouping neurons whose activity represents tentative objects ("proto-objects") based on the integration of local feature information. Grouping neurons receive input from an organized set of local feature neurons, and project modulatory feedback to those same neurons. Additionally, inhibition at both the local feature level and the object representation level biases the interpretation of the visual scene in agreement with principles from Gestalt psychology. Our model explains several sets of neurophysiological results (Zhou et al. Journal of Neuroscience, 20(17), 6594-6611 2000; Qiu et al. Nature Neuroscience, 10(11), 1492-1499 2007; Chen et al. Neuron, 82(3), 682-694 2014), and makes testable predictions about the influence of neuronal feedback and attentional selection on neural responses across different visual areas. Our model also provides a framework for understanding how object-based attention is able to select both objects and the features associated with them.

Attentive pointing in natural scenes correlates with other measures of attention.

Finger pointing is a natural human behavior frequently used to draw attention to specific parts of sensory input. Since this pointing behavior is likely preceded and/or accompanied by the deployment of attention by the pointing person, we hypothesize that pointing can be used as a natural means of providing self-reports of attention and, in the case of visual input, visual salience. We here introduce a new method for assessing attentional choice by asking subjects to point to and tap the first place they look at on an image appearing on an electronic tablet screen. Our findings show that the tap data are well-correlated with other measures of attention, including eye fixations and selections of interesting image points, as well as with predictions of a saliency map model. We also develop an analysis method for comparing attentional maps (including fixations, reported points of interest, finger pointing, and computed salience) that takes into account the error in estimating those maps from a finite number of data points. This analysis strengthens our original findings by showing that the measured correlation between attentional maps drawn from identical underlying processes is systematically underestimated. The underestimation is strongest when the number of samples is small but it is always present. Our analysis method is not limited to data from attentional paradigms but, instead, it is broadly applicable to measures of similarity made between counts of multinomial data or probability distributions.

The Edge of Stability: Response Times and Delta Oscillations in Balanced Networks.

The standard architecture of neocortex is a network with excitation and inhibition in closely maintained balance. These networks respond fast and with high precision to their inputs and they allow selective amplification of patterned signals. The stability of such networks is known to depend on balancing the strengths of positive and negative feedback. We here show that a second condition is required for stability which depends on the relative strengths and time courses of fast (AMPA) and slow (NMDA) currents in the excitatory projections. This condition also determines the response time of the network. We show that networks which respond quickly to an input are necessarily close to an oscillatory instability which resonates in the delta range. This instability explains the existence of neocortical delta oscillations and the emergence of absence epilepsy. Although cortical delta oscillations are a network-level phenomenon, we show that in non-pathological networks, individual neurons receive sufficient information to keep the network in the fast-response regime without sliding into the instability.

Spike synchrony generated by modulatory common input through NMDA-type synapses.

Common excitatory input to neurons increases their firing rates and the strength of the spike correlation (synchrony) between them. Little is known, however, about the synchronizing effects of modulatory common input. Here, we show that modulatory common input with the slow synaptic kinetics of N-methyl-d-aspartate (NMDA) receptors enhances firing rates and also produces synchrony. Tight synchrony (correlations on the order of milliseconds) always increases with modulatory strength. Unexpectedly, the relationship between strength of modulation and strength of loose synchrony (tens of milliseconds) is not monotonic: The strongest loose synchrony is obtained for intermediate modulatory amplitudes. This finding explains recent neurophysiological results showing that in cortical areas V1 and V2, presumed modulatory top-down input due to contour grouping increases (loose and tight) synchrony but that additional modulatory input due to top-down attention does not change tight synchrony and actually decreases loose synchrony. These neurophysiological findings are understood from our model of integrate-and-fire neurons under the assumption that contour grouping as well as attention lead to additive modulatory common input through NMDA-type synapses. In contrast, circuits with common projections through model α-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid receptors did not exhibit the paradoxical decrease of synchrony with increased input. Our results suggest that NMDA receptors play a critical role in top-down response modulation in the visual cortex.