Prospective Optimization with Limited Resources.

The future is uncertain because some forthcoming events are unpredictable and also because our ability to foresee the myriad consequences of our own actions is limited. Here we studied how humans select actions under such extrinsic and intrinsic uncertainty, in view of an exponentially expanding number of prospects on a branching multivalued visual stimulus. A triangular grid of disks of different sizes scrolled down a touchscreen at a variable speed. The larger disks represented larger rewards. The task was to maximize the cumulative reward by touching one disk at a time in a rapid sequence, forming an upward path across the grid, while every step along the path constrained the part of the grid accessible in the future. This task captured some of the complexity of natural behavior in the risky and dynamic world, where ongoing decisions alter the landscape of future rewards. By comparing human behavior with behavior of ideal actors, we identified the strategies used by humans in terms of how far into the future they looked (their "depth of computation") and how often they attempted to incorporate new information about the future rewards (their "recalculation period"). We found that, for a given task difficulty, humans traded off their depth of computation for the recalculation period. The form of this tradeoff was consistent with a complete, brute-force exploration of all possible paths up to a resource-limited finite depth. A step-by-step analysis of the human behavior revealed that participants took into account very fine distinctions between the future rewards and that they abstained from some simple heuristics in assessment of the alternative paths, such as seeking only the largest disks or avoiding the smaller disks. The participants preferred to reduce their depth of computation or increase the recalculation period rather than sacrifice the precision of computation.

Sensory adaptation as optimal resource allocation.

Visual adaptation is expected to improve visual performance in the new environment. This expectation has been contradicted by evidence that adaptation sometimes decreases sensitivity for the adapting stimuli, and sometimes it changes sensitivity for stimuli very different from the adapting ones. We hypothesize that this pattern of results can be explained by a process that optimizes sensitivity for many stimuli, rather than changing sensitivity only for those stimuli whose statistics have changed. To test this hypothesis, we measured visual sensitivity across a broad range of spatiotemporal modulations of luminance, while varying the distribution of stimulus speeds. The manipulation of stimulus statistics caused a large-scale reorganization of visual sensitivity, forming the orderly pattern of sensitivity gains and losses. This pattern is predicted by a theory of distribution of receptive field characteristics in the visual system.

Intermittent regime of brain activity at the early, bias-guided stage of perceptual learning.

Perceptual learning improves visual performance. Among the plausible mechanisms of learning, reduction of perceptual bias has been studied the least. Perceptual bias may compensate for lack of stimulus information, but excessive reliance on bias diminishes visual discriminability. We investigated the time course of bias in a perceptual grouping task and studied the associated cortical dynamics in spontaneous and evoked EEG. Participants reported the perceived orientation of dot groupings in ambiguous dot lattices. Performance improved over a 1-hr period as indicated by the proportion of trials in which participants preferred dot groupings favored by dot proximity. The proximity-based responses were compromised by perceptual bias: Vertical groupings were sometimes preferred to horizontal ones, independent of dot proximity. In the evoked EEG activity, greater amplitude of the N1 component for horizontal than vertical responses indicated that the bias was most prominent in conditions of reduced visual discriminability. The prominence of bias decreased in the course of the experiment. Although the bias was still prominent, prestimulus activity was characterized by an intermittent regime of alternating modes of low and high alpha power. Responses were more biased in the former mode, indicating that perceptual bias was deployed actively to compensate for stimulus uncertainty. Thus, early stages of perceptual learning were characterized by episodes of greater reliance on prior visual preferences, alternating with episodes of receptivity to stimulus information. In the course of learning, the former episodes disappeared, and biases reappeared only infrequently.

Dopamine function and the efficiency of human movement.

To sustain successful behavior in dynamic environments, active organisms must be able to learn from the consequences of their actions and predict action outcomes. One of the most important discoveries in systems neuroscience over the last 15 years has been about the key role of the neurotransmitter dopamine in mediating such active behavior. Dopamine cell firing was found to encode differences between the expected and obtained outcomes of actions. Although activity of dopamine cells does not specify movements themselves, a recent study in humans has suggested that tonic levels of dopamine in the dorsal striatum may in part enable normal movement by encoding sensitivity to the energy cost of a movement, providing an implicit "motor motivational" signal for movement. We investigated the motivational hypothesis of dopamine by studying motor performance of patients with Parkinson disease who have marked dopamine depletion in the dorsal striatum and compared their performance with that of elderly healthy adults. All participants performed rapid sequential movements to visual targets associated with different risk and different energy costs, countered or assisted by gravity. In conditions of low energy cost, patients performed surprisingly well, similar to prescriptions of an ideal planner and healthy participants. As energy costs increased, however, performance of patients with Parkinson disease dropped markedly below the prescriptions for action by an ideal planner and below performance of healthy elderly participants. The results indicate that the ability for efficient planning depends on the energy cost of action and that the effect of energy cost on action is mediated by dopamine.

Prospective Optimization.

Human performance approaches that of an ideal observer and optimal actor in some perceptual and motor tasks. These optimal abilities depend on the capacity of the cerebral cortex to store an immense amount of information and to flexibly make rapid decisions. However, behavior only approaches these limits after a long period of learning while the cerebral cortex interacts with the basal ganglia, an ancient part of the vertebrate brain that is responsible for learning sequences of actions directed toward achieving goals. Progress has been made in understanding the algorithms used by the brain during reinforcement learning, which is an online approximation of dynamic programming. Humans also make plans that depend on past experience by simulating different scenarios, which is called prospective optimization. The same brain structures in the cortex and basal ganglia that are active online during optimal behavior are also active offline during prospective optimization. The emergence of general principles and algorithms for goal-directed behavior has consequences for the development of autonomous devices in engineering applications.

Traveling waves and trial averaging: the nature of single-trial and averaged brain responses in large-scale cortical signals.

Analyzing single trial brain activity remains a challenging problem in the neurosciences. We gain purchase on this problem by focusing on globally synchronous fields in within-trial evoked brain activity, rather than on localized peaks in the trial-averaged evoked response (ER). We analyzed data from three measurement modalities, each with different spatial resolutions: magnetoencephalogram (MEG), electroencephalogram (EEG) and electrocorticogram (ECoG). We first characterized the ER in terms of summation of phase and amplitude components over trials. Both contributed to the ER, as expected, but the ER topography was dominated by the phase component. This means the observed topography of cross-trial phase will not necessarily reflect the phase topography within trials. To assess the organization of within-trial phase, traveling wave (TW) components were quantified by computing the phase gradient. TWs were intermittent but ubiquitous in the within-trial evoked brain activity. At most task-relevant times and frequencies, the within-trial phase topography was described better by a TW than by the trial-average of phase. The trial-average of the TW components also reproduced the topography of the ER; we suggest that the ER topography arises, in large part, as an average over TW behaviors. These findings were consistent across the three measurement modalities. We conclude that, while phase is critical to understanding the topography of event-related activity, the preliminary step of collating cortical signals across trials can obscure the TW components in brain activity and lead to an underestimation of the coherent motion of cortical fields.

Spatially distributed computation in cortical circuits.

The traditional view of neural computation in the cerebral cortex holds that sensory neurons are specialized, i.e., selective for certain dimensions of sensory stimuli. This view was challenged by evidence of contextual interactions between stimulus dimensions in which a neuron's response to one dimension strongly depends on other dimensions. Here, we use methods of mathematical modeling, psychophysics, and electrophysiology to address shortcomings of the traditional view. Using a model of a generic cortical circuit, we begin with the simple demonstration that cortical responses are always distributed among neurons, forming characteristic waveforms, which we call neural waves. When stimulated by patterned stimuli, circuit responses arise by interference of neural waves. Results of this process depend on interaction between stimulus dimensions. Comparison of modeled responses with responses of biological vision makes it clear that the framework of neural wave interference provides a useful alternative to the standard concept of neural computation.

Duration of coherence intervals in electrical brain activity in perceptual organization.

We investigated the relationship between visual experience and temporal intervals of synchronized brain activity. Using high-density scalp electroencephalography, we examined how synchronized activity depends on visual stimulus information and on individual observer sensitivity. In a perceptual grouping task, we varied the ambiguity of visual stimuli and estimated observer sensitivity to this variation. We found that durations of synchronized activity in the beta frequency band were associated with both stimulus ambiguity and sensitivity: the lower the stimulus ambiguity and the higher individual observer sensitivity the longer were the episodes of synchronized activity. Durations of synchronized activity intervals followed an extreme value distribution, indicating that they were limited by the slowest mechanism among the multiple neural mechanisms engaged in the perceptual task. Because the degree of stimulus ambiguity is (inversely) related to the amount of stimulus information, the durations of synchronous episodes reflect the amount of stimulus information processed in the task. We therefore interpreted our results as evidence that the alternating episodes of desynchronized and synchronized electrical brain activity reflect, respectively, the processing of information within local regions and the transfer of information across regions.

A perceptual scaling approach to eyewitness identification.

Eyewitness misidentification accounts for 70% of verified erroneous convictions. To address this alarming phenomenon, research has focused on factors that influence likelihood of correct identification, such as the manner in which a lineup is conducted. Traditional lineups rely on overt eyewitness responses that confound two covert factors: strength of recognition memory and the criterion for deciding what memory strength is sufficient for identification. Here we describe a lineup that permits estimation of memory strength independent of decision criterion. Our procedure employs powerful techniques developed in studies of perception and memory: perceptual scaling and signal detection analysis. Using these tools, we scale memory strengths elicited by lineup faces, and quantify performance of a binary classifier tasked with distinguishing perpetrator from innocent suspect. This approach reveals structure of memory inaccessible using traditional lineups and renders accurate identifications uninfluenced by decision bias. The approach furthermore yields a quantitative index of individual eyewitness performance.

Mechanisms of Spatiotemporal Selectivity in Cortical Area MT.

Cortical sensory neurons are characterized by selectivity to stimulation. This selectivity was originally viewed as a part of the fundamental "receptive field" characteristic of neurons. This view was later challenged by evidence that receptive fields are modulated by stimuli outside of the classical receptive field. Here, we show that even this modified view of selectivity needs revision. We measured spatial frequency selectivity of neurons in cortical area MT of alert monkeys and found that their selectivity strongly depends on luminance contrast, shifting to higher spatial frequencies as contrast increases. The changes of preferred spatial frequency are large at low temporal frequency, and they decrease monotonically as temporal frequency increases. That is, even interactions among basic stimulus dimensions of luminance contrast, spatial frequency, and temporal frequency strongly influence neuronal selectivity. This dynamic nature of neuronal selectivity is inconsistent with the notion of stimulus preference as a stable characteristic of cortical neurons.

Dissociation of early evoked cortical activity in perceptual grouping.

Perceptual grouping is a multi-stage process, irreducible to a single mechanism localized anatomically or chronometrically. To understand how various grouping mechanisms interact, we combined a phenomenological report paradigm with high-density event-related potential (ERP) measurements, using a 256-channel electrode array. We varied the relative salience of competing perceptual organizations in multi-stable dot lattices and asked observers to report perceived groupings. The ability to discriminate groupings (the grouping sensitivity) was positively correlated with the amplitude of the earliest ERP peak C1 (about 60 ms after stimulus onset) over the middle occipital area. This early activity is believed to reflect spontaneous feed-forward processes preceding perceptual awareness. Grouping sensitivity was negatively correlated with the amplitude of the next peak P1 (about 110 ms), which is believed to reflect lateral and feedback interactions associated with perceptual awareness and attention. This dissociation between C1 and P1 activity implies that the recruitment of fast, spontaneous mechanisms for grouping leads to high grouping sensitivity. Observers who fail to recruit these mechanisms are trying to compensate by using later mechanisms, which depend less on stimulus properties such as proximity.

Viewing geometry determines how vision and haptics combine in size perception.

Vision and haptics have different limitations and advantages because they obtain information by different methods. If the brain combined information from the two senses optimally, it would rely more on the one providing more precise information for the current task. In this study, human observers judged the distance between two parallel surfaces in two within-modality experiments (vision-alone and haptics-alone) and in an intermodality experiment (vision and haptics together). In the within-modality experiments, the precision of visual estimates varied with surface orientation, as expected from geometric considerations; the precision of haptic estimates did not. An ideal observer that combines visual and haptic information weights them differently as a function of orientation. In the intermodality experiment, humans adjusted visual and haptic weights in a fashion quite similar to that of the ideal observer. As a result, combined size estimates are finer than is possible with either vision or haptics alone; indeed, they approach statistical optimality.

The lawful perception of apparent motion.

Visual apparent motion is the experience of motion from the successive stimulation of separate spatial locations. How spatial and temporal distances interact to determine the strength of apparent motion has been controversial. Some studies report space-time coupling: If we increase spatial or temporal distance between successive stimuli, we must also increase the other distance between them to maintain a constant strength of apparent motion (Korte's third law of motion). Other studies report space-time tradeoff: If we increase one of these distances, we must decrease the other to maintain a constant strength of apparent motion. In this article, we resolve the controversy. Starting from a normative theory of motion measurement and data on human spatiotemporal sensitivity, we conjecture that both coupling and tradeoff should occur, but at different speeds. We confirm the prediction in two experiments, using suprathreshold multistable apparent-motion displays called motion lattices. Our results show a smooth transition between the tradeoff and coupling as a function of speed: Tradeoff occurs at low speeds and coupling occurs at high speeds. From our data, we reconstruct the suprathreshold equivalence contours that are analogous to isosensitivity contours obtained at the threshold of visibility.

The economics of motion perception and invariants of visual sensitivity.

Neural systems face the challenge of optimizing their performance with limited resources, just as economic systems do. Here, we use tools of neoclassical economic theory to explore how a frugal visual system should use a limited number of neurons to optimize perception of motion. The theory prescribes that vision should allocate its resources to different conditions of stimulation according to the degree of balance between measurement uncertainties and stimulus uncertainties. We find that human vision approximately follows the optimal prescription. The equilibrium theory explains why human visual sensitivity is distributed the way it is and why qualitatively different regimes of apparent motion are observed at different speeds. The theory offers a new normative framework for understanding the mechanisms of visual sensitivity at the threshold of visibility and above the threshold and predicts large-scale changes in visual sensitivity in response to changes in the statistics of stimulation and system goals.

Optimality of human movement under natural variations of visual-motor uncertainty.

Biological movements are prone to error. Different movements lead to different errors, and the distributions of errors depend on movement amplitude and direction. Movement planning would benefit from taking this variability into account, by applying appropriate corrections for movements associated with the different shapes and sizes of error distributions. Here we asked whether the human nervous system can do so. In a game-like task, participants performed rapid sequences of goal-directed pointing movements in different directions, toward stimulus configurations presented at different eccentricities on a slanted touch screen. The task was to accumulate rewards by hitting target regions and to minimize losses by avoiding penalty regions. The distributions of endpoint errors varied in size and degree of anisotropy across stimulus locations. Our participants adjusted their movements toward the different locations accordingly. We compared human behavior with the optimal behavior predicted by ideal movement planner maximizing expected gain. In most cases, human behavior was indistinguishable from optimal. This is evidence that human movement planning approaches statistical optimality by representing the task-relevant movement variability.

Paradoxical perception of object identity in visual motion.

In the course of perceptual organization, incomplete optical stimulation can evoke the experience of complete objects with distinct perceptual identities. According to a well-known principle of perceptual organization, stimulus parts separated by shorter spatial distances are more likely to appear as parts of the same perceptual identity. Whereas this principle of proximity has been confirmed in many studies of perceptual grouping in static displays, we show that it does not generalize to perception of object identity in dynamic displays, where the parts are separated by spatial and temporal distances. We use ambiguous displays which contain multiple moving parts and which can be perceived two ways: as two large objects that gradually change their size or as multiple smaller objects that rotate independent of one another. Grouping over long and short distances corresponds to the perception of the respectively large and small objects. We find that grouping over long distances is often preferred to grouping over short distances, against predictions of the proximity principle. Even though these effects are observed at high luminance contrast, we show that they are consistent with results obtained at the threshold of luminance contrast, in agreement with predictions of a theory of efficient motion measurement. This is evidence that the perception of object identity can be explained by a computational principle of neural economy rather than by the empirical principle of proximity.

Adaptive Optimization of Visual Sensitivity.

Sensory systems adapt to environmental change. It has been argued that adaptation should have the effect of optimizing sensitivity to the new environment. Here we consider a framework in which this premise is made concrete using an economic normative theory of visual motion perception. In this framework, visual systems adapt to the environment by reallocating their limited neural resources. The allocation is optimal when uncertainties about different aspects of stimulation are balanced. This theory makes predictions about visual sensitivity as a function of environmental statistics. Adaptive optimization of the visual system should be manifested as a change in sensitivity for an observer and for the underlying motion-sensitive neurons. We review evidence supporting these predictions and examine effects of adaptation on the neuronal representation of visual motion.

Optimal compensation for changes in task-relevant movement variability.

Effective movement planning should take into account the consequences of possible errors in executing a planned movement. These errors can result from either sensory uncertainty or variability in movement planning and production. We examined the ability of humans to compensate for variability in sensory estimation and movement production under conditions in which variability is increased artificially by the experimenter. Subjects rapidly pointed at a target region that had an adjacent penalty region. Target and penalty hits yielded monetary rewards and losses. We manipulated the task-relevant variability by perturbing visual feedback of finger position during the movement. The feedback was shifted in a random direction with a random amplitude in each trial, causing an increase in the task-relevant variability. Subjects were unable to counteract this form of perturbation. Rewards and penalties were based on the perturbed, visually specified finger position. Subjects rapidly acquired an estimate of their new variability in <120 trials and adjusted their aim points accordingly. We compared subjects' performance to the performance of an optimal movement planner maximizing expected gain. Their performance was consistent with that expected from an optimal movement planner that perfectly compensated for externally imposed changes in task-relevant variability. When exposed to novel stimulus configurations, aim points shifted in the first trial without showing any detectable trend across trials. These results indicate that subjects are capable of changing their pointing strategy in the presence of externally imposed noise. Furthermore, they manage to update their estimate of task-relevant variability and to transfer this estimate to novel stimulus configurations.

Why is spatial stereoresolution so low?

Spatial stereoresolution (the finest detectable modulation of binocular disparity) is much poorer than luminance resolution (finest detectable luminance variation). In a series of psychophysical experiments, we examined four factors that could cause low stereoresolution: (1) the sampling properties of the stimulus, (2) the disparity gradient limit, (3) low-pass spatial filtering by mechanisms early in the visual process, and (4) the method by which binocular matches are computed. Our experimental results reveal the contributions of the first three factors. A theoretical analysis of binocular matching by interocular correlation reveals the contribution of the fourth: the highest attainable stereoresolution may be limited by (1) the smallest useful correlation window in the visual system, and (2) a matching process that estimates the disparity of image patches and assumes that disparity is constant across the patch. Both properties are observed in disparity-selective neurons in area V1 of the primate (Nienborg et al., 2004).