Gaze following requires early visual experience.

Gaze understanding­a suggested precursor for understanding others' intentions­requires recovery of gaze direction from the observed person's head and eye position. This challenging computation is naturally acquired at infancy without explicit external guidance, but can it be learned later if vision is extremely poor throughout early childhood? We addressed this question by studying gaze following in Ethiopian patients with early bilateral congenital cataracts diagnosed and treated by us only at late childhood. This sight restoration provided a unique opportunity to directly address basic issues on the roles of "nature" and "nurture" in development, as it caused a selective perturbation to the natural process, eliminating some gaze-direction cues while leaving others still available. Following surgery, the patients' visual acuity typically improved substantially, allowing discrimination of pupil position in the eye. Yet, the patients failed to show eye gaze-following effects and fixated less than controls on the eyes­two spontaneous behaviors typically seen in controls. Our model for unsupervised learning of gaze direction explains how head-based gaze following can develop under severe image blur, resembling preoperative conditions. It also suggests why, despite acquiring sufficient resolution to extract eye position, automatic eye gaze following is not established after surgery due to lack of detailed early visual experience. We suggest that visual skills acquired in infancy in an unsupervised manner will be difficult or impossible to acquire when internal guidance is no longer available, even when sufficient image resolution for the task is restored. This creates fundamental barriers to spontaneous vision recovery following prolonged deprivation in early age.

Object Representations in Human Visual Cortex Formed Through Temporal Integration of Dynamic Partial Shape Views.

We typically recognize visual objects using the spatial layout of their parts, which are present simultaneously on the retina. Therefore, shape extraction is based on integration of the relevant retinal information over space. The lateral occipital complex (LOC) can represent shape faithfully in such conditions. However, integration over time is sometimes required to determine object shape. To study shape extraction through temporal integration of successive partial shape views, we presented human participants (both men and women) with artificial shapes that moved behind a narrow vertical or horizontal slit. Only a tiny fraction of the shape was visible at any instant at the same retinal location. However, observers perceived a coherent whole shape instead of a jumbled pattern. Using fMRI and multivoxel pattern analysis, we searched for brain regions that encode temporally integrated shape identity. We further required that the representation of shape should be invariant to changes in the slit orientation. We show that slit-invariant shape information is most accurate in the LOC. Importantly, the slit-invariant shape representations matched the conventional whole-shape representations assessed during full-image runs. Moreover, when the same slit-dependent shape slivers were shuffled, thereby preventing their spatiotemporal integration, slit-invariant shape information was reduced dramatically. The slit-invariant representation of the various shapes also mirrored the structure of shape perceptual space as assessed by perceptual similarity judgment tests. Therefore, the LOC is likely to mediate temporal integration of slit-dependent shape views, generating a slit-invariant whole-shape percept. These findings provide strong evidence for a global encoding of shape in the LOC regardless of integration processes required to generate the shape percept.SIGNIFICANCE STATEMENT Visual objects are recognized through spatial integration of features available simultaneously on the retina. The lateral occipital complex (LOC) represents shape faithfully in such conditions even if the object is partially occluded. However, shape must sometimes be reconstructed over both space and time. Such is the case in anorthoscopic perception, when an object is moving behind a narrow slit. In this scenario, spatial information is limited at any moment so the whole-shape percept can only be inferred by integration of successive shape views over time. We find that LOC carries shape-specific information recovered using such temporal integration processes. The shape representation is invariant to slit orientation and is similar to that evoked by a fully viewed image. Existing models of object recognition lack such capabilities.

Practice improves peri-saccadic shape judgment but does not diminish target mislocalization.

Visual sensitivity is markedly reduced during an eye movement. Peri-saccadic vision is also characterized by a mislocalization of the briefly presented stimulus closer to the saccadic target. These features are commonly viewed as obligatory elements of peri-saccadic vision. However, practice improves performance in many perceptual tasks performed at threshold conditions. We wondered if this could also be the case with peri-saccadic perception. To test this, we used a paradigm in which subjects reported the orientation (or location) of an ellipse briefly presented during a saccade. Practice on peri-saccadic orientation discrimination led to long-lasting gains in that task but did not alter the classical mislocalization of the visual stimulus. Shape discrimination gains were largely generalized to other untrained conditions when the same stimuli were used (discrimination during a saccade in the opposite direction or at a different stimulus location than previously trained). However, performance dropped to baseline level when participants shifted to a novel Vernier discrimination task under identical saccade conditions. Furthermore, practice on the location task did not induce better stimulus localization or discrimination. These results suggest that the limited visual information available during a saccade may be better used with practice, possibly by focusing attention on the specific target features or a better readout of the available information. Saccadic mislocalization, by contrast, is robust and resistant to top-down modulations, suggesting that it involves an automatic process triggered by the upcoming execution of a saccade (e.g., an efference copy signal).

Lack of Automatic Imitation in Newly Sighted Individuals.

Viewing a hand action performed by another person facilitates a response-compatible action and slows a response-incompatible one, even when the viewed action is irrelevant to the task. This automatic imitation effect is taken as the clearest evidence for a direct mapping between action viewing and motor performance. But there is an ongoing debate whether this effect is innate or experience dependent. We tackled this issue by studying a unique group of newly sighted children who suffered from dense bilateral cataracts from early infancy and were surgically treated only years later. The newly sighted children were less affected by viewing task-irrelevant actions than were control children, even 2 years after the cataract-removal surgery. This strongly suggests that visually guided motor experience is necessary for the development of automatic imitation. At the very least, our results indicate that if imitation is based on innate mechanisms, these are clearly susceptible to long periods of visual deprivation.

Position and Identity Information Available in fMRI Patterns of Activity in Human Visual Cortex.

Parietal cortex is often implicated in visual processing of actions. Action understanding is essentially abstract, specific to the type or goal of action, but greatly independent of variations in the perceived position of the action. If certain parietal regions are involved in action understanding, then we expect them to show these generalization and selectivity properties. However, additional functions of parietal cortex, such as self-action control, may impose other demands by requiring an accurate representation of the location of graspable objects. Therefore, the dimensions along which responses are modulated may indicate the functional role of specific parietal regions. Here, we studied the degree of position invariance and hand/object specificity during viewing of tool-grasping actions. To that end, we characterize the information available about location, hand, and tool identity in the patterns of fMRI activation in various cortical areas: early visual cortex, posterior intraparietal sulcus, anterior superior parietal lobule, and the ventral object-specific lateral occipital complex. Our results suggest a gradient within the human dorsal stream: along the posterior-anterior axis, position information is gradually lost, whereas hand and tool identity information is enhanced. This may reflect a gradual transformation of visual input from an initial retinotopic representation in early visual areas to an abstract, position-invariant representation of viewed action in anterior parietal cortex. SIGNIFICANCE STATEMENT: Since the seminal study of Goodale and Milner (1992), there is general agreement that visual processing is largely divided between a ventral and dorsal stream specializing in object recognition and vision for action, respectively. Here, we address the specific representation of viewed actions. Specifically, we study the degree of position invariance and hand/object manipulation specificity in the human visual pathways, characterizing the information available in patterns of fMRI activation during viewing of object-grasping videos, which appeared in different retinal locations. We find converging evidence for a gradient within the dorsal stream: along the posterior-anterior axis, position information is gradually lost, whereas hand and action identity information is enhanced, leading to an abstract, position-invariant representation of viewed action in the anterior parietal cortex.

Fingerprints of Learned Object Recognition Seen in the fMRI Activation Patterns of Lateral Occipital Complex.

One feature of visual processing in the ventral stream is that cortical responses gradually depart from the physical aspects of the visual stimulus and become correlated with perceptual experience. Thus, unlike early retinotopic areas, the responses in the object-related lateral occipital complex (LOC) are typically immune to parameter changes (e.g., contrast, location, etc.) when these do not affect recognition. Here, we use a complementary approach to highlight changes in brain activity following a shift in the perceptual state (in the absence of any alteration in the physical image). Specifically, we focus on LOC and early visual cortex (EVC) and compare their functional magnetic resonance imaging (fMRI) responses to degraded object images, before and after fast perceptual learning that renders initially unrecognized objects identifiable. Using 3 complementary analyses, we find that, in LOC, unlike EVC, learned recognition is associated with a change in the multivoxel response pattern to degraded object images, such that the response becomes significantly more correlated with that evoked by the intact version of the same image. This provides further evidence that the coding in LOC reflects the recognition of visual objects.

Hands in motion: an upper-limb-selective area in the occipitotemporal cortex shows sensitivity to viewed hand kinematics.

Regions in the occipitotemporal cortex (OTC) show clear selectivity to static images of human body parts, and upper limbs in particular, with respect to other object categories. Such selectivity was previously attributed to shape aspects, which presumably vary across categories. Alternatively, it has been proposed that functional selectivity for upper limbs is driven by processing of their distinctive motion features. In the present study we show that selectivity to static upper-limb images and motion processing go hand in hand. Using resting-state and task-based functional MRI, we demonstrate that OTC voxels showing greater preference to static images of arms and hands also show stronger functional connectivity with motion coding regions within the human middle temporal complex (hMT+), but not with shape-selective midtier areas, such as hV4 or LO-1, suggesting a tight link between upper-limb selectivity and motion processing. To test this directly, we created a set of natural arm-movement videos where kinematic patterns were parametrically manipulated, while keeping shape information constant. Using multivariate pattern analysis, we show that the degree of (dis)similarity in arm-velocity profiles across the video set predicts, to a significant extent, the degree of (dis)similarity in multivoxel activation patterns in both upper-limb-selective OTC regions and the hMT+. Together, these results suggest that the functional specificity of upper-limb-selective regions may be partially determined by their involvement in the processing of upper-limb dynamics. We propose that the selectivity to static upper-limb images in the OTC may be a result of experience-dependent association between shape elements, which characterize upper limbs, and upper-limb-specific motion patterns.

Visual memory in unilateral spatial neglect: immediate recall versus delayed recognition.

Patients with unilateral spatial neglect (USN) often show impaired performance in spatial working memory tasks, apart from the difficulty retrieving "left-sided" spatial data from long-term memory, shown in the "piazza effect" by Bisiach and colleagues. This study's aim was to compare the effect of the spatial position of a visual object on immediate and delayed memory performance in USN patients. Specifically, immediate verbal recall performance, tested using a simultaneous presentation of four visual objects in four quadrants, was compared with memory in a later-provided recognition task, in which objects were individually shown at the screen center. Unlike healthy controls, USN patients showed a left-side disadvantage and a vertical bias in the immediate free recall task (69% vs. 42% recall for right- and left-sided objects, respectively). In the recognition task, the patients correctly recognized half of "old" items, and their correct rejection rate was 95.5%. Importantly, when the analysis focused on previously recalled items (in the immediate task), no statistically significant difference was found in the delayed recognition of objects according to their original quadrant of presentation. Furthermore, USN patients were able to recollect the correct original location of the recognized objects in 60% of the cases, well beyond chance level. This suggests that the memory trace formed in these cases was not only semantic but also contained a visuospatial tag. Finally, successful recognition of objects missed in recall trials points to formation of memory traces for neglected contralesional objects, which may become accessible to retrieval processes in explicit memory.

Multiple reference frames for saccadic planning in the human parietal cortex.

We apply functional magnetic resonance imaging and multivariate analysis methods to study the coordinate frame in which saccades are represented in the human cortex. Subjects performed a memory-guided saccade task in which equal-amplitude eye movements were executed from several starting points to various directions. Response patterns during the memory period for same-vector saccades were correlated in the frontal eye fields and the intraparietal sulcus (IPS), indicating a retinotopic representation. Interestingly, response patterns in the middle aspect of the IPS were also correlated for saccades made to the same destination point, even when their movement vector was different. Thus, this region also contains information about saccade destination in (at least) a head-centered coordinate frame. This finding may explain behavioral and neuropsychological studies demonstrating that eye movements are also anchored to an egocentric or an allocentric representation of space rather than strictly to the retinal visual input and that parietal cortex is involved in maintaining these representations of space.

The representation of visual and motor aspects of reaching movements in the human motor cortex.

The human primary motor cortex (M1) is robustly activated during visually guided hand movements. M1 multivoxel patterns of functional MRI activation are more correlated during repeated hand movements to the same targets than to greatly differing ones, and therefore potentially contain information about movement direction. It is unclear, however, whether direction specificity is due to the motor command, as implicitly assumed, or to the visual aspects of the task, such as the target location and the direction of the cursor's trajectory. To disambiguate the visual and motor components, different visual-to-motor transformations were applied during an fMRI scan, in which participants made visually guided hand movements in various directions. The first run was the "baseline" (i.e., visual and motor mappings were matched); in the second run ("rotation"), the cursor movement was rotated by 45° with respect to the joystick movement. As expected, positive correlations were seen between the M1 multivoxel patterns evoked by the baseline run and by the rotation run, when the two movements were matched in their movement direction but the visual aspects differed. Importantly, similar correlations were observed when the visual elements were matched but the direction of hand movement differed. This indicates that M1 is sensitive to both motor and visual components of the task. However, repeated observation of the cursor movement without concurrent joystick control did not elicit significant activation in M1 or any correlated patterns of activation. Thus, visual aspects of movement are encoded in M1 only when they are coupled with motor consequences.

Motion adaptation reveals that the motion vector is represented in multiple coordinate frames.

Accurately perceiving the velocity of an object during smooth pursuit is a complex challenge: although the object is moving in the world, it is almost still on the retina. Yet we can perceive the veridical motion of a visual stimulus in such conditions, suggesting a nonretinal representation of the motion vector. To explore this issue, we studied the frames of representation of the motion vector by evoking the well known motion aftereffect during smooth-pursuit eye movements (SPEM). In the retinotopic configuration, due to an accompanying smooth pursuit, a stationary adapting random-dot stimulus was actually moving on the retina. Motion adaptation could therefore only result from motion in retinal coordinates. In contrast, in the spatiotopic configuration, the adapting stimulus moved on the screen but was practically stationary on the retina due to a matched SPEM. Hence, adaptation here would suggest a representation of the motion vector in spatiotopic coordinates. We found that exposure to spatiotopic motion led to significant adaptation. Moreover, the degree of adaptation in that condition was greater than the adaptation induced by viewing a random-dot stimulus that moved only on the retina. Finally, pursuit of the same target, without a random-dot array background, yielded no adaptation. Thus, in our experimental conditions, adaptation is not induced by the SPEM per se. Our results suggest that motion computation is likely to occur in parallel in two distinct representations: a low-level, retinal-motion dependent mechanism and a high-level representation, in which the veridical motion is computed through integration of information from other sources.

Rapid formation of spatiotopic representations as revealed by inhibition of return.

Inhibition of return (IOR), a performance decrement for stimuli appearing at recently cued locations, occurs when the target and cue share the same screen position. This is in contrast to cue-based attention facilitation effects that were recently suggested to be mapped in a retinotopic reference frame, the prevailing representation throughout early visual processing stages. Here, we investigate the dynamics of IOR in both reference frames, using a modified cued-location saccadic reaction time task with an intervening saccade between cue and target presentation. Thus, on different trials, the target was present either at the same retinotopic location as the cue, or at the same screen position (e.g., spatiotopic location). IOR was primarily found for targets appearing at the same spatiotopic position as the initial cue, when the cue and target were presented at the same hemifield. This suggests that there is restricted information transfer of cue position across the two hemispheres. Moreover, the effect was maximal when the target was presented 10 ms after the intervening saccade ended and was attenuated in longer delays. In our case, therefore, the representation of previously attended locations (as revealed by IOR) is not remapped slowly after the execution of a saccade. Rather, either a retinotopic representation is remapped rapidly, adjacent to the end of the saccade (using a prospective motor command), or the positions of the cue and target are encoded in a spatiotopic reference frame, regardless of eye position. Spatial attention can therefore be allocated to target positions defined in extraretinal coordinates.

Functional organization of human motor cortex: directional selectivity for movement.

In monkeys, neurons in the hand representation of the primary motor cortex (M1) are often tuned to the direction of hand movement, and there is evidence that these neurons are clustered according to their "preferred" direction of movement. However, this organizational principle has yet to be demonstrated in M1 of humans. We conducted a functional magnetic resonance imaging (fMRI) study in which participants used a joystick to move a cursor from a central origin to one of five equidistant targets. The fMRI signal of individual voxels was sensitive to the directional aspects of the reaching task and manifested direction-specific adaptation. Furthermore, the correlation between multivoxel patterns of responses for different movement directions depended on the angular distance between them. We conclude that M1 neurons are likely to be organized in clusters according to their preferred direction, since only such a coarse-grained representation can lead to directional selectivity of voxels, encompassing millions of neurons. A simple model that estimates cluster size suggests that the diameter of these clusters is on the order of a few hundred micrometers.

Viewed actions are mapped in retinotopic coordinates in the human visual pathways.

Viewed object-oriented actions elicit widespread fMRI activation in the dorsal and ventral visual pathways. This activation is typically stronger in the hemisphere contralateral to the visual field in which action is seen. However, since in previous studies participants kept fixation at the same screen position throughout the scan, it was impossible to infer if the viewed actions are represented in retina-based coordinates or in a more elaborated coordinate system. Here, participants changed their gaze between experimental conditions, such that some conditions shared the same retinotopic coordinates (but differed in their screen position), while other pairs of conditions shared the opposite trait. The degree of similarity between the patterns of activation elicited by the various conditions was assessed using multivoxel pattern analysis methods. Regions of interest, showing robust overall activation, included the intraparietal sulcus (IPS) and the occipitotemporal cortex. In these areas, the correlation between activation patterns for conditions sharing the same retinotopic coordinates was significantly higher than that of those having different retinotopic coordinates. In contrast, the correlations between activation patterns for conditions with the same spatiotopic coordinates were not significantly greater than for non-spatiotopic conditions. These results suggest that viewed object-oriented actions are likely to be maintained in retinotopic-framed coordinates.

The presence of semantic content in a visual recognition memory task reduces the severity of neglect.

Patients with right hemisphere damage often show a lateral bias when asked to report the left side of mental images held in visual working memory (i.e. representational neglect). The neural basis of representational neglect is not well understood. One hypothesis suggests that it reflects a deficit in attentional-exploratory mechanisms, i.e. an inability to direct attention to the left side of the image. Another proposition states that intact visual working memory (VWM) is necessary for correctly creating a mental image. Here we examined two components of VWM in patients with unilateral spatial neglect (USN): memory for identity, and memory for spatial position. We manipulated the strength of memory representations by presenting two distinct categories of objects, in separate blocks. These were familiar namable objects (fruits, etc.), and unfamiliar abstract objects. The former category elicits stronger working-memory traces, thanks to preexisting visual and semantic representations in long-term memory. We hypothesized that if USN patients show a lateralized deficit in VWM, it should be more pronounced for abstract objects, due to their weaker working-memory traces. Importantly, to isolate a spatially lateralized deficit in memory from a failure to fully perceive the object-arrays, we ensured that all included patients perceived every item during the encoding phase. We used a working-memory task: participants viewed object arrays and had to memorize items' identities and spatial positions. Then, single objects were presented requiring 'old/new' recognition, and retrieval of 'old' items' original positions. Our results show a lateral bias in patients' recognition-memory performance. Remarkably, it was threefold milder for namable objects compared to abstract objects. We conclude that VWM lateralized deficit is substantial in USN patients and could play a role in representational neglect.

Learning to perceive shape from temporal integration following late emergence from blindness.

Visual perception requires massive use of inference because the 3D structure of the world is not directly provided by the sensory input.1 Particularly challenging is anorthoscopic vision-when an object moves behind a narrow slit such that only a tiny fraction of it is visible at any instant. Impressively, human observers correctly recognize objects in slit-viewing conditions by early childhood,2,3 via temporal integration of the contours available in each sliver.4,5 But can this capability be acquired if one has been effectively blind throughout childhood? We studied 23 Ethiopian children which had bilateral early-onset cataracts-resulting in extremely poor vision in infancy-and surgically treated only years later. We tested their anorthoscopic vision, precisely because it requires a cascade of demanding visual inference processes to perceive veridical shape. Failure to perform the task may allow mapping specific bottlenecks for late visual recovery. The patients' visual acuity typically improved substantially within 6 months post-surgery. Still, at this stage many were unable to recover shape under slit-viewing conditions, although they could infer the direction of global motion. However, when retested later, almost all patients could judge shape in slit-conditions necessitating temporal integration. This acquired capability often transferred to novel stimuli, in similar slit-viewing conditions. Thus, learning was not limited to the specific visual features of the original shapes. These results indicate that plasticity of sophisticated visual inference routines is preserved well into adolescence, and vision restoration after prolonged early-onset blindness is feasible to a greater extent than previously thought.

Development of multisensory integration following prolonged early-onset visual deprivation.

Adult humans make effortless use of multisensory signals and typically integrate them in an optimal fashion.1 This remarkable ability takes many years for normally sighted children to develop.2,3 Would individuals born blind or with extremely low vision still be able to develop multisensory integration later in life when surgically treated for sight restoration? Late acquisition of such capability would be a vivid example of the brain's ability to retain high levels of plasticity. We studied the development of multisensory integration in individuals suffering from congenital dense bilateral cataract, surgically treated years after birth. We assessed cataract-treated individuals' reliance on their restored visual abilities when estimating the size of an object simultaneously explored by touch. Within weeks to months after surgery, when combining information from vision and touch, they developed a multisensory weighting behavior similar to matched typically sighted controls. Next, we tested whether cataract-treated individuals benefited from integrating vision with touch by increasing the precision of size estimates, as it occurs when integrating signals in a statistically optimal fashion.1 For participants retested multiple times, such a benefit developed within months after surgery to levels of precision indistinguishable from optimal behavior. To summarize, the development of multisensory integration does not merely depend on age, but requires extensive multisensory experience with the world, rendered possible by the improved post-surgical visual acuity. We conclude that early exposure to multisensory signals is not essential for the development of multisensory integration, which can still be acquired even after many years of visual deprivation.

Two phases of V1 activity for visual recognition of natural images.

Present theories of visual recognition emphasize the role of interactive processing across populations of neurons within a given network, but the nature of these interactions remains unresolved. In particular, data describing the sufficiency of feedforward algorithms for conscious vision and studies revealing the functional relevance of feedback connections to the striate cortex seem to offer contradictory accounts of visual information processing. TMS is a good method to experimentally address this issue, given its excellent temporal resolution and its capacity to establish causal relations between brain function and behavior. We studied 20 healthy volunteers in a visual recognition task. Subjects were briefly presented with images of animals (birds or mammals) in natural scenes and were asked to indicate the animal category. MRI-guided stereotaxic single TMS pulses were used to transiently disrupt striate cortex function at different times after image onset (SOA). Visual recognition was significantly impaired when TMS was applied over the occipital pole at SOAs of 100 and 220 msec. The first interval has consistently been described in previous TMS studies and is explained as the interruption of the feedforward volley of activity. Given the late latency and discrete nature of the second peak, we hypothesize that it represents the disruption of a feedback projection to V1, probably from other areas in the visual network. These results provide causal evidence for the necessity of recurrent interactive processing, through feedforward and feedback connections, in visual recognition of natural complex images.

Superior serial memory in the blind: a case of cognitive compensatory adjustment.

In the absence of vision, perception of space is likely to be highly dependent on memory. As previously stated, the blind tend to code spatial information in the form of "route-like" sequential representations [1-3]. Thus, serial memory, indicating the order in which items are encountered, may be especially important for the blind to generate a mental picture of the world. In accordance, we find that the congenitally blind are remarkably superior to sighted peers in serial memory tasks. Specifically, subjects heard a list of 20 words and were instructed to recall the words according to their original order in the list. The blind recalled more words than the sighted (indicating better item memory), but their greatest advantage was in recalling longer word sequences (according to their original order). We further show that the serial memory superiority of the blind is not merely a result of their advantage in item recall per se (as we additionally confirm via a separate recognition memory task). These results suggest the refinement of a specific cognitive ability to compensate for blindness in humans.

Dissociation between ventral and dorsal fMRI activation during object and action recognition.

Neuropsychological case studies suggest the existence of two functionally separate visual streams: the ventral pathway, central for object recognition; and the dorsal pathway, engaged in visually guided actions. However, a clear dissociation between the functions of the two streams has not been decisively shown in intact humans. In this study, we demonstrate dissociation between dorsal and ventral fMRI activation patterns during observation of object manipulation video clips. Parietal areas, such as anterior intraparietal sulcus (aIPS) display grasp viewing-dependent adaptation (i.e., fMR adaptation during repeated viewing of the same object-grasping movement) as well as a contralateral preference for the viewed manipulating hand. Ventral regions, such as the fusiform gyrus, show similar characteristics (i.e., adaptation, contralateral preference), but these depend on object identity. Our results support the hypothesized functional specialization in the visual system and suggest that parietal areas (such as aIPS) are engaged in action recognition, as well as in action planning.