Learning to perceive shape from temporal integration following late emergence from blindness.

Visual perception requires massive use of inference because the 3D structure of the world is not directly provided by the sensory input.1 Particularly challenging is anorthoscopic vision-when an object moves behind a narrow slit such that only a tiny fraction of it is visible at any instant. Impressively, human observers correctly recognize objects in slit-viewing conditions by early childhood,2,3 via temporal integration of the contours available in each sliver.4,5 But can this capability be acquired if one has been effectively blind throughout childhood? We studied 23 Ethiopian children which had bilateral early-onset cataracts-resulting in extremely poor vision in infancy-and surgically treated only years later. We tested their anorthoscopic vision, precisely because it requires a cascade of demanding visual inference processes to perceive veridical shape. Failure to perform the task may allow mapping specific bottlenecks for late visual recovery. The patients' visual acuity typically improved substantially within 6 months post-surgery. Still, at this stage many were unable to recover shape under slit-viewing conditions, although they could infer the direction of global motion. However, when retested later, almost all patients could judge shape in slit-conditions necessitating temporal integration. This acquired capability often transferred to novel stimuli, in similar slit-viewing conditions. Thus, learning was not limited to the specific visual features of the original shapes. These results indicate that plasticity of sophisticated visual inference routines is preserved well into adolescence, and vision restoration after prolonged early-onset blindness is feasible to a greater extent than previously thought.

The presence of semantic content in a visual recognition memory task reduces the severity of neglect.

Patients with right hemisphere damage often show a lateral bias when asked to report the left side of mental images held in visual working memory (i.e. representational neglect). The neural basis of representational neglect is not well understood. One hypothesis suggests that it reflects a deficit in attentional-exploratory mechanisms, i.e. an inability to direct attention to the left side of the image. Another proposition states that intact visual working memory (VWM) is necessary for correctly creating a mental image. Here we examined two components of VWM in patients with unilateral spatial neglect (USN): memory for identity, and memory for spatial position. We manipulated the strength of memory representations by presenting two distinct categories of objects, in separate blocks. These were familiar namable objects (fruits, etc.), and unfamiliar abstract objects. The former category elicits stronger working-memory traces, thanks to preexisting visual and semantic representations in long-term memory. We hypothesized that if USN patients show a lateralized deficit in VWM, it should be more pronounced for abstract objects, due to their weaker working-memory traces. Importantly, to isolate a spatially lateralized deficit in memory from a failure to fully perceive the object-arrays, we ensured that all included patients perceived every item during the encoding phase. We used a working-memory task: participants viewed object arrays and had to memorize items' identities and spatial positions. Then, single objects were presented requiring 'old/new' recognition, and retrieval of 'old' items' original positions. Our results show a lateral bias in patients' recognition-memory performance. Remarkably, it was threefold milder for namable objects compared to abstract objects. We conclude that VWM lateralized deficit is substantial in USN patients and could play a role in representational neglect.

Artificial limb representation in amputees.

The human brain contains multiple hand-selective areas, in both the sensorimotor and visual systems. Could our brain repurpose neural resources, originally developed for supporting hand function, to represent and control artificial limbs? We studied individuals with congenital or acquired hand-loss (hereafter one-handers) using functional MRI. We show that the more one-handers use an artificial limb (prosthesis) in their everyday life, the stronger visual hand-selective areas in the lateral occipitotemporal cortex respond to prosthesis images. This was found even when one-handers were presented with images of active prostheses that share the functionality of the hand but not necessarily its visual features (e.g. a 'hook' prosthesis). Further, we show that daily prosthesis usage determines large-scale inter-network communication across hand-selective areas. This was demonstrated by increased resting state functional connectivity between visual and sensorimotor hand-selective areas, proportional to the intensiveness of everyday prosthesis usage. Further analysis revealed a 3-fold coupling between prosthesis activity, visuomotor connectivity and usage, suggesting a possible role for the motor system in shaping use-dependent representation in visual hand-selective areas, and/or vice versa. Moreover, able-bodied control participants who routinely observe prosthesis usage (albeit less intensively than the prosthesis users) showed significantly weaker associations between degree of prosthesis observation and visual cortex activity or connectivity. Together, our findings suggest that altered daily motor behaviour facilitates prosthesis-related visual processing and shapes communication across hand-selective areas. This neurophysiological substrate for prosthesis embodiment may inspire rehabilitation approaches to improve usage of existing substitutionary devices and aid implementation of future assistive and augmentative technologies.

Object Representations in Human Visual Cortex Formed Through Temporal Integration of Dynamic Partial Shape Views.

We typically recognize visual objects using the spatial layout of their parts, which are present simultaneously on the retina. Therefore, shape extraction is based on integration of the relevant retinal information over space. The lateral occipital complex (LOC) can represent shape faithfully in such conditions. However, integration over time is sometimes required to determine object shape. To study shape extraction through temporal integration of successive partial shape views, we presented human participants (both men and women) with artificial shapes that moved behind a narrow vertical or horizontal slit. Only a tiny fraction of the shape was visible at any instant at the same retinal location. However, observers perceived a coherent whole shape instead of a jumbled pattern. Using fMRI and multivoxel pattern analysis, we searched for brain regions that encode temporally integrated shape identity. We further required that the representation of shape should be invariant to changes in the slit orientation. We show that slit-invariant shape information is most accurate in the LOC. Importantly, the slit-invariant shape representations matched the conventional whole-shape representations assessed during full-image runs. Moreover, when the same slit-dependent shape slivers were shuffled, thereby preventing their spatiotemporal integration, slit-invariant shape information was reduced dramatically. The slit-invariant representation of the various shapes also mirrored the structure of shape perceptual space as assessed by perceptual similarity judgment tests. Therefore, the LOC is likely to mediate temporal integration of slit-dependent shape views, generating a slit-invariant whole-shape percept. These findings provide strong evidence for a global encoding of shape in the LOC regardless of integration processes required to generate the shape percept.SIGNIFICANCE STATEMENT Visual objects are recognized through spatial integration of features available simultaneously on the retina. The lateral occipital complex (LOC) represents shape faithfully in such conditions even if the object is partially occluded. However, shape must sometimes be reconstructed over both space and time. Such is the case in anorthoscopic perception, when an object is moving behind a narrow slit. In this scenario, spatial information is limited at any moment so the whole-shape percept can only be inferred by integration of successive shape views over time. We find that LOC carries shape-specific information recovered using such temporal integration processes. The shape representation is invariant to slit orientation and is similar to that evoked by a fully viewed image. Existing models of object recognition lack such capabilities.

Hands in motion: an upper-limb-selective area in the occipitotemporal cortex shows sensitivity to viewed hand kinematics.

Regions in the occipitotemporal cortex (OTC) show clear selectivity to static images of human body parts, and upper limbs in particular, with respect to other object categories. Such selectivity was previously attributed to shape aspects, which presumably vary across categories. Alternatively, it has been proposed that functional selectivity for upper limbs is driven by processing of their distinctive motion features. In the present study we show that selectivity to static upper-limb images and motion processing go hand in hand. Using resting-state and task-based functional MRI, we demonstrate that OTC voxels showing greater preference to static images of arms and hands also show stronger functional connectivity with motion coding regions within the human middle temporal complex (hMT+), but not with shape-selective midtier areas, such as hV4 or LO-1, suggesting a tight link between upper-limb selectivity and motion processing. To test this directly, we created a set of natural arm-movement videos where kinematic patterns were parametrically manipulated, while keeping shape information constant. Using multivariate pattern analysis, we show that the degree of (dis)similarity in arm-velocity profiles across the video set predicts, to a significant extent, the degree of (dis)similarity in multivoxel activation patterns in both upper-limb-selective OTC regions and the hMT+. Together, these results suggest that the functional specificity of upper-limb-selective regions may be partially determined by their involvement in the processing of upper-limb dynamics. We propose that the selectivity to static upper-limb images in the OTC may be a result of experience-dependent association between shape elements, which characterize upper limbs, and upper-limb-specific motion patterns.

Topographic representation of the human body in the occipitotemporal cortex.

Large-scale topographic representations of the body have long been established in the somatosensory and motor cortices. Using functional imaging, we identified a topographically organized body part map within the occipitotemporal cortex (OTC), with distinct clusters of voxels showing clear preference for different visually presented body parts. This representation was consistent both across hemispheres and participants. Using converging methods, the preference for specific body parts was demonstrated to be robust and did not merely reflect shape differences between the categories. Finally, execution of (unseen) movements with different body parts resulted in a limited topographic representation of the limbs and trunk, which partially overlapped with the visual body part map. This motor-driven activation in the OTC could not be explained solely by visual or motor imagery of the body parts. This suggests that visual and motor-related information converge within the OTC in a body part specific manner.

Superior serial memory in the blind: a case of cognitive compensatory adjustment.

In the absence of vision, perception of space is likely to be highly dependent on memory. As previously stated, the blind tend to code spatial information in the form of "route-like" sequential representations [1-3]. Thus, serial memory, indicating the order in which items are encountered, may be especially important for the blind to generate a mental picture of the world. In accordance, we find that the congenitally blind are remarkably superior to sighted peers in serial memory tasks. Specifically, subjects heard a list of 20 words and were instructed to recall the words according to their original order in the list. The blind recalled more words than the sighted (indicating better item memory), but their greatest advantage was in recalling longer word sequences (according to their original order). We further show that the serial memory superiority of the blind is not merely a result of their advantage in item recall per se (as we additionally confirm via a separate recognition memory task). These results suggest the refinement of a specific cognitive ability to compensate for blindness in humans.

Normal and abnormal fMRI activation patterns in the visual cortex after recovery from optic neuritis.

Recovery to normal or near normal visual acuity after an optic neuritis episode is common, despite frequent persistence of conduction abnormalities, evident in prolonged visual evoked potential (VEP) latencies. Improvement of visual function is commonly attributed to peripheral nerve recovery. However, central reorganization processes may also be involved. To assess this, we compared the patterns of fMRI activation, elicited by stimulation of the affected and the normal eye, along the visual cortical hierarchy. Activation was assessed in 8 subjects, which recovered clinically from an episode of optic neuritis but still had prolonged VEP latencies. In all patients, reduced fMRI activation was seen in V1 during stimulation of the affected eye, compared to the normal eye. The fMRI signal difference decreased in magnitude with progression along the visual hierarchy, and in some regions within the lateral occipital complex even showed the opposite preference (for the affected eye). These results may indicate a built-in robustness of the object-related areas to disruption of the visual input. Alternatively, it could reflect an adaptive functional reorganization of the cortical response to an abnormal input.

Memory of ordinal number categories in macaque monkeys.

What mechanism underlies serial order memory? Studying preverbal serial memory shows that macaque monkeys reproducing a sequence of items can acquire knowledge of item ordinal position. In our previous experiment, macaques were repeatedly presented with image lists (first shown sequentially and then simultaneously on a touch screen together with a distractor chosen randomly from other lists). The task was to touch list images in the correct order. The monkeys' natural tendency was to categorize images by their ordinal position or number because their most common error was touching the distractor when it had the same ordinal number (in its own list) as the correct image. Item-to-item associations were used to complete the categorization strategy. Proposing a dynamic image-salience hypothesis for serial recall (based on category-to-image influence and a salience computation for identifying touch targets), we now study the category label characteristics in the context of this hypothesis. We found that these category labels are absolute, ordinal-number-based categories (first, second, etc.), not relative memorized as relative distance from the beginning and the end of the list, and not based on fixed ranking of reward contingency/image familiarity. Even isolated from item-item associations, the categories demonstrate category tuning (as well as the corresponding overlap of adjacent ordinal number codes). Moreover, monkeys choose images by proximity of their category to the current touch number, irrespective of the accuracy of the preceding choice. Category tuning itself is symmetric relative to correct ordinal position, but is skewed by other factors (reward, etc.). Tuning width increases with list length, with a concurrent increased use of item-to-item associations for determining touch order.

Multi-item working memory--a behavioral study.

Macaque monkeys were trained to recognize the repetition of one of the images already seen in a sequence of random length. On average, performance decreased with sequence length. However, this was due to a complex combination of factors, as follows: performance was found to decrease with the separation in the sequence of the test (repetition image) from the cue (its first appearance in the sequence), for trials with sequences of fixed length. In contrast, performance improved as a function of sequence length, for equal cue-test separations. Reaction times followed a complementary trend: they increased with cue-test separation and decreased with sequence length. The frequency of false positives (FPs) indicates that images are not always removed from working memory between successive trials, and that the monkeys rarely confuse different images. The probability of miss errors depends on number of intervening stimulus presentations, while FPs depend on elapsed time. A simple two-state stochastic model of multi-item working memory is proposed that guides the account for the main effects of performance and false positives, as well as their interaction. In the model, images enter WM when they are presented, or by spontaneous jump-in. Misses are due to spontaneous jump-out of images previously seen.

Serial memory strategies in macaque monkeys: behavioral and theoretical aspects.

Serial memory is the ability to encode and retrieve a list of items in their correct temporal order. To study nonverbal strategies involved in serial memory, we trained four macaque monkeys on a novel delayed sequence-recall task and analysed the mechanisms underlying their performance in terms of a neural network model. Thirty fractal images, divided into 10 triplets, were presented repeatedly in fixed temporal order. On each trial the monkeys viewed three sequentially presented sample images, followed by a test stimulus consisting of the same triplet of images and a distractor image (chosen randomly from the remaining 27). The task was to touch the three images in their original order, avoiding the distractor. The monkeys' most common error was touching the distractor when it had the same ordinal position (in its own triplet) as the correct image. This finding suggests that monkeys naturally categorize images by their ordinal number. Additional, secondary strategies were eventually used to avoid distractor images. These include memory of the sample images (working memory) and associations between triplet members. Further direct evidence for ordinal number categorization was provided by a transfer of learning to untrained images of the same ordinal category, following reassignment of image categories within each triplet. We propose a generic three-tier neuronal framework that can explain the components and complex set of characteristics of the observed behavior. This framework, with its intermediate level representing ordinal categories, can also explain the transfer of learning following category reassignment.