Neural correlates of decision making on whole body yaw rotation: An fNIRS study.

Prominent accounts of decision making state that decisions are made on the basis of an accumulation of sensory evidence, orchestrated by networks of prefrontal and parietal neural populations. Here we assess whether these findings generalize to decisions on self-motion. Participants were presented with whole body yaw rotations of different durations in a 2-Interval-Forced-Choice paradigm, and tasked to discriminate motions on the basis of their amplitude. The cortical hemodynamic response was recorded using functional near-infrared spectroscopy (fNIRS) while participants were performing the task. The imaging data was used to predict the specific response on individual experimental trials, and to predict whether the comparison stimulus would be judged larger than the reference. Classifier performance on the former variable was negligible. However, considerable performance was achieved for the latter variable, specifically using parietal imaging data. The findings provide support for the notion that activity in the parietal cortex reflects modality independent decision variables that represent the strength of the neural evidence in favor of a decision. The results are encouraging for the use of fNIRS as a method to perform neuroimaging in moving individuals.

Integration of visual and inertial cues in the perception of angular self-motion.

The brain is able to determine angular self-motion from visual, vestibular, and kinesthetic information. There is compelling evidence that both humans and non-human primates integrate visual and inertial (i.e., vestibular and kinesthetic) information in a statistically optimal fashion when discriminating heading direction. In the present study, we investigated whether the brain also integrates information about angular self-motion in a similar manner. Eight participants performed a 2IFC task in which they discriminated yaw-rotations (2-s sinusoidal acceleration) on peak velocity. Just-noticeable differences (JNDs) were determined as a measure of precision in unimodal inertial-only and visual-only trials, as well as in bimodal visual-inertial trials. The visual stimulus was a moving stripe pattern, synchronized with the inertial motion. Peak velocity of comparison stimuli was varied relative to the standard stimulus. Individual analyses showed that data of three participants showed an increase in bimodal precision, consistent with the optimal integration model; while data from the other participants did not conform to maximum-likelihood integration schemes. We suggest that either the sensory cues were not perceived as congruent, that integration might be achieved with fixed weights, or that estimates of visual precision obtained from non-moving observers do not accurately reflect visual precision during self-motion.

Neural correlates of oddball detection in self-motion heading: a high-density event-related potential study of vestibular integration.

The perception of self-motion is a product of the integration of information from both visual and non-visual cues, to which the vestibular system is a central contributor. It is well documented that vestibular dysfunction leads to impaired movement and balance, dizziness and falls, and yet our knowledge of the neuronal processing of vestibular signals remains relatively sparse. In this study, high-density electroencephalographic recordings were deployed to investigate the neural processes associated with vestibular detection of changes in heading. To this end, a self-motion oddball paradigm was designed. Participants were translated linearly 7.8 cm on a motion platform using a one second motion profile, at a 45° angle leftward or rightward of straight ahead. These headings were presented with a stimulus probability of 80-20 %. Participants responded when they detected the infrequent direction change via button-press. Event-related potentials (ERPs) were calculated in response to the standard (80 %) and target (20 %) movement directions. Statistical parametric mapping showed that ERPs to standard and target movements differed significantly from 490 to 950 ms post-stimulus. Topographic analysis showed that this difference had a typical P3 topography. Individual participant bootstrap analysis revealed that 93.3 % of participants exhibited a clear P3 component. These results indicate that a perceived change in vestibular heading can readily elicit a P3 response, wholly similar to that evoked by oddball stimuli presented in other sensory modalities. This vestibular-evoked P3 response may provide a readily and robustly detectable objective measure for the evaluation of vestibular integrity in various disease models.

Cross-Modal Transfer in Visual and Haptic Face Recognition.

We report four psychophysical experiments investigating cross-modal transfer in visual and haptic face recognition. We found surprisingly good haptic performance and cross-modal transfer for both modalities. Interestingly, transfer was asymmetric depending on which modality was learned first. These findings are discussed in relation to haptic object processing and face processing.

Orientation congruency effects for familiar objects: coordinate transformations in object recognition.

How do observers recognize objects after spatial transformations? Recent neurocomputational models have proposed that object recognition is based on coordinate transformations that align memory and stimulus representations. If the recognition of a misoriented object is achieved by adjusting a coordinate system (or reference frame), then recognition should be facilitated when the object is preceded by a different object in the same orientation. In the two experiments reported here, two objects were presented in brief masked displays that were in close temporal contiguity; the objects were in either congruent or incongruent picture-plane orientations. Results showed that naming accuracy was higher for congruent than for incongruent orientations. The congruency effect was independent of superordinate category membership (Experiment 1) and was found for objects with different main axes of elongation (Experiment 2). The results indicate congruency effects for common familiar objects even when they have dissimilar shapes. These findings are compatible with models in which object recognition is achieved by an adjustment of a perceptual coordinate system.

Grasp effects of the Ebbinghaus illusion: obstacle avoidance is not the explanation.

The perception-versus-action hypothesis states that visual information is processed in two different streams, one for visual awareness (or perception) and one for motor performance. Previous reports that the Ebbinghaus illusion deceives perception but not grasping seemed to indicate that this dichotomy between perception and action was fundamental enough to be reflected in the overt behavior of non-neurological, healthy humans. Contrary to this view we show that the Ebbinghaus illusion affects grasping to the same extent as perception. We also show that the grasp effects cannot be accounted for by non-perceptual obstacle avoidance mechanisms as has recently been suggested. Instead, even subtle variations of the Ebbinghaus illusion affect grasping in the same way as they affect perception. Our results suggest that the same signals are responsible for the perceptual effects and for the motor effects of the Ebbinghaus illusion. This casts doubt on one line of evidence, which used to strongly favor the perception-versus-action hypothesis.

Effects of visual illusions on grasping.

In 2 experiments, the Muller-Lyer illusion (F. C. Muller-Lyer, 1889; N = 16) and the parallel-lines illusion (W. Wundt, 1898; N = 26) clearly affected maximum preshape aperture in grasping (both ps < .001). The grasping effects were similar but not perfectly equal to the perceptual effects. Control experiments show that these differences can be attributed to problems in matching the perceptual task and the grasping task. A model is described stating the assumptions that are needed to compare the grasping effects and the perceptual effects of visual illusions. Further studies on the relationship between perception and grasping are reviewed. These studies provide no clear evidence for a dissociation between perception and grasping and therefore do not support the action versus perception hypothesis (A. D. Milner & M. A. Goodale, 1995).

Detection of animals in natural images using far peripheral vision.

It is generally believed that the acuity of the peripheral visual field is too poor to allow accurate object recognition and, that to be identified, most objects need to be brought into foveal vision by using saccadic eye movements. However, most measures of form vision in the periphery have been done at eccentricities below 10 degrees and have used relatively artificial stimuli such as letters, digits and compound Gabor patterns. Little is known about how such data would apply in the case of more naturalistic stimuli. Here humans were required to categorize briefly flashed (28 ms) unmasked photographs of natural scenes (39 degrees high, and 26 degrees across) on the basis of whether or not they contained an animal. The photographs appeared randomly in nine locations across virtually the entire extent of the horizontal visual field. Accuracy was 93.3% for central vision and decreased almost linearly with increasing eccentricity (89.8% at 13 degrees, 76.1% at 44.5 degrees and 71.2% at 57.5 degrees ). Even at the most extreme eccentricity, where the images were centred at 70.5 degrees, subjects scored 60.5% correct. No evidence was found for hemispheric specialization. This level of performance was achieved despite the fact that the position of the image was unpredictable, ruling out the use of precued attention to target locations. The results demonstrate that even high-level visual tasks involving object vision can be performed using the relatively coarse information provided by the peripheral retina.

A prior for global convexity in local shape-from-shading.

To solve the ill-posed problem of shape-from-shading, the visual system often relies on prior assumptions such as illumination from above or viewpoint from above. Here we demonstrate that a third prior assumption is used--namely that the surface is globally convex. We use complex surface shapes that are realistically rendered with computer graphics, and we find that performance in a local-shape-discrimination task is significantly higher when the shapes are globally convex than when they are globally concave. The results are surprising because the qualitative global shapes of the surfaces are perceptually unambiguous. The results generalise findings such as the hollow-potato illusion (Hill and Bruce 1994 Perception 23 1335-1337) which consider global shape perception only.

Viewpoint dependence in visual and haptic object recognition.

On the whole, people recognize objects best when they see the objects from a familiar view and worse when they see the objects from views that were previously occluded from sight. Unexpectedly, we found haptic object recognition to be viewpoint-specific as well, even though hand movements were unrestricted. This viewpoint dependence was due to the hands preferring the back "view" of the objects. Furthermore, when the sensory modalities (visual vs. haptic) differed between learning an object and recognizing it, recognition performance was best when the objects were rotated back-to-front between learning and recognition. Our data indicate that the visual system recognizes the front view of objects best, whereas the hand recognizes objects best from the back.

Effects of temporal association on recognition memory.

The influence of temporal association on the representation and recognition of objects was investigated. Observers were shown sequences of novel faces in which the identity of the face changed as the head rotated. As a result, observers showed a tendency to treat the views as if they were of the same person. Additional experiments revealed that this was only true if the training sequences depicted head rotations rather than jumbled views; in other words, the sequence had to be spatially as well as temporally smooth. Results suggest that we are continuously associating views of objects to support later recognition, and that we do so not only on the basis of the physical similarity, but also the correlated appearance in time of the objects.

Driving in the future: temporal visuomotor adaptation and generalization.

Rapid and accurate visuomotor coordination requires tight spatial and temporal sensorimotor synchronization. The introduction of a sensorimotor or intersensory misalignment (either spatial or temporal) impairs performance on most tasks. For more than a century, it has been known that a few minutes of exposure to a spatial misalignment can induce a recalibration of sensorimotor spatial relationships, a phenomenon that may be referred to as spatial visuomotor adaptation. Here, we use a high-fidelity driving simulator to demonstrate that the sensorimotor system can adapt to temporal misalignments on very complex tasks, a phenomenon that we refer to as temporal visuomotor adaptation. We demonstrate that adapting on a single street produces an adaptive state that generalizes to other streets. This shows that temporal visuomotor adaptation is not specific to a single visuomotor transformation, but generalizes across a class of transformations. Temporal visuomotor adaptation is strikingly parallel to spatial visuomotor adaptation, and has strong implications for the understanding of visuomotor coordination and intersensory integration.

Depth discrimination from shading under diffuse lighting.

The human visual system has a remarkable ability to interpret smooth patterns of light on a surface in terms of 3-D surface geometry. Classical studies of shape-from-shading perception have assumed that surface irradiance varies with the angle between the local surface normal and a collimated light source. This model holds, for example, on a sunny day. One common situation in which this model fails to hold, however, is under diffuse lighting such as on a cloudy day. Here we report on the first psychophysical experiments that address shape-from-shading under a uniform diffuse-lighting condition. Our hypothesis was that shape perception can be explained with a perceptual model that "dark means deep". We tested this hypothesis by comparing performance in a depth-discrimination task to performance in a brightness-discrimination task, using identical stimuli. We found a significant correlation between responses in the two tasks, supporting a dark-means-deep model. However, overall performance in the depth-discrimination task was superior to that predicted by a dark-means-deep model. This implies that humans use a more accurate model than dark-means-deep to perceive shape-from-shading under diffuse lighting.

Neuronal representation of object orientation.

The dissociation between object identity and object orientation observed in six patients with brain damage, has been taken as evidence for a view-invariant model of object recognition. However, there was also some indication that these patients were not generally agnosic for object orientation but were able to gain access to at least some information about objects' canonical upright. We studied a new case (KB) with spared knowledge of object identity and impaired perception of object orientation using a forced choice paradigm to contrast directly the patient's ability to perceive objects' canonical upright vs non-upright orientations. We presented 2D-pictures of objects with unambiguous canonical upright orientations in four different orientations (0 degrees, -90 degrees, +90 degrees, 180 degrees ). KB showed no impairment in identifying letters, objects, animals, or faces irrespective of their given orientation. Also, her knowledge of upright orientation of stimuli was perfectly preserved. In sharp contrast, KB was not able to judge the orientation when the stimuli were presented in a non-upright orientation. The findings give further support for a distributed view-based representation of objects in which neurons become tuned to the features present in certain views of an object. Since we see more upright than inverted animals and familiar objects, the statistics of these images leads to a larger number of neurons tuned for objects in an upright orientation. We suppose that probably for this reason KB's knowledge of upright orientation was found to be more robust against neuronal damage than knowledge of other orientations.

Perception, representation and recognition: a holistic view of recognition.

It is clear that humans have mental representations of their spatial environments and that these representations are useful, if not essential, in a wide variety of cognitive tasks such as identification of landmarks and objects, guiding actions and navigation and in directing spatial awareness and attention. Determining the properties of mental representation has long been a contentious issue (see Pinker, 1984). One method of probing the nature of human representation is by studying the extent to which representation can surpass or go beyond the visual (or sensory) experience from which it derives. From a strictly empiricist standpoint what is not sensed cannot be represented; except as a combination of things that have been experienced. But perceptual experience is always limited by our view of the world and the properties of our visual system. It is therefore not surprising when human representation is found to be highly dependent on the initial viewpoint of the observer and on any shortcomings thereof. However, representation is not a static entity; it evolves with experience. The debate as to whether human representation of objects is view-dependent or view-invariant that has dominated research journals recently may simply be a discussion concerning how much information is available in the retinal image during experimental tests and whether this information is sufficient for the task at hand. Here we review an approach to the study of the development of human spatial representation under realistic problem solving scenarios. This is facilitated by the use of realistic virtual environments, exploratory learning and redundancy in visual detail.

Grasping visual illusions: no evidence for a dissociation between perception and action.

Neuropsychological studies prompted the theory that the primate visual system might be organized into two parallel pathways, one for conscious perception and one for guiding action. Supporting evidence in healthy subjects seemed to come from a dissociation in visual illusions: In previous studies, the Ebbinghaus (or Titchener) illusion deceived perceptual judgments of size, but only marginally influenced the size estimates used in grasping. Contrary to those results, the findings from the present study show that there is no difference in the sizes of the perceptual and grasp illusions if the perceptual and grasping tasks are appropriately matched. We show that the differences found previously can be accounted for by a hitherto unknown, nonadditive effect in the illusion. We conclude that the illusion does not provide evidence for the existence of two distinct pathways for perception and action in the visual system.

Touch can change visual slant perception.

The visual system uses several signals to deduce the three-dimensional structure of the environment, including binocular disparity, texture gradients, shading and motion parallax. Although each of these sources of information is independently insufficient to yield reliable three-dimensional structure from everyday scenes, the visual system combines them by weighting the available information; altering the weights would therefore change the perceived structure. We report that haptic feedback (active touch) increases the weight of a consistent surface-slant signal relative to inconsistent signals. Thus, appearance of a subsequently viewed surface is changed: the surface appears slanted in the direction specified by the haptically reinforced signal.

View dependence in scene recognition after active learning.

Human spatial encoding of three-dimensional navigable space was studied, using a virtual environment simulation. This allowed subjects to become familiar with a realistic scene by making simulated rotational and translational movements during training. Subsequent tests determined whether subjects could generalize their recognition ability by identifying novel-perspective views and topographic floor plans of the scene. Results from picture recognition tests showed that familiar direction views were most easily recognized, although significant generalization to novel views was observed. Topographic floor plans were also easily identified. In further experiments, novel-view performance diminished when active training was replaced by passive viewing of static images of the scene. However, the ability to make self-initiated movements, as opposed to watching dynamic movie sequences, had no effect on performance. These results suggest that representation of navigable space is view dependent and highlight the importance of spatial-temporal continuity during learning.

Effects of parametric manipulation of inter-stimulus similarity on 3D object categorization.

To explore the nature of the representation space of 3D objects, we studied human performance in forced-choice categorization of objects composed of four geon-like parts emanating from a common center. Two categories were defined by prototypical objects, distinguished by qualitative properties of their parts (bulging vs waist-like limbs). Subjects were trained to discriminate between the two prototypes (shown briefly, from a number of viewpoints, in stereo) in a 1-interval forced-choice task, until they reached a 90% correct-response performance level. After training, in the first experiment, 11 subjects were tested on shapes obtained by varying the prototypical parameters both orthogonally (ORTHO) and in parallel (PARA) to the line connecting the prototypes in the parameter space. For the eight subjects who performed above chance, the error rate increased with the ORTHO parameter-space displacement between the stimulus and the corresponding prototype; the effect of the PARA displacement was weaker. Thus, the parameter-space location of the stimuli mattered more than the qualitative contrasts, which were always present. To find out whether both prototypes or just the nearest one to the test shape influenced the decision, in the second experiment we varied the similarity between the categories. Specifically, in the test stage trials the distance between the two prototypes could assume one of three values (FAR, INTERMEDIATE, and NEAR). For the 13 subjects who performed above chance, the error rate (on physically identical stimuli) in the NEAR condition was higher than in the other two conditions. The results of the two experiments contradict the prediction of theories that postulate exclusive reliance on qualitative contrasts, and support the notion of a representation space in which distances to more than one reference point or prototype are encoded (Edelman, 1998).

What object attributes determine canonical views?

We investigated preferred or canonical views for familiar and three-dimensional nonsense objects using computer-graphics psychophysics. We assessed the canonical views for objects by allowing participants to actively rotate realistically shaded three-dimensional models in real-time. Objects were viewed on a Silicon Graphics workstation and manipulated in virtual space with a three-degree-of-freedom input device. In the first experiment, participants adjusted each object to the viewpoint from which they would take a photograph if they planned to use the object to illustrate a brochure. In the second experiment, participants mentally imaged each object on the basis of the name and then adjusted the object to the viewpoint from which they imagined it. In both experiments, there was a large degree of consistency across participants in terms of the preferred view for a given object. Our results provide new insights on the geometrical, experiential, and functional attributes that determine canonical views under ecological conditions.