A self-training program for sensory substitution devices.

Sensory Substitution Devices (SSDs) convey visual information through audition or touch, targeting blind and visually impaired individuals. One bottleneck towards adopting SSDs in everyday life by blind users, is the constant dependency on sighted instructors throughout the learning process. Here, we present a proof-of-concept for the efficacy of an online self-training program developed for learning the basics of the EyeMusic visual-to-auditory SSD tested on sighted blindfolded participants. Additionally, aiming to identify the best training strategy to be later re-adapted for the blind, we compared multisensory vs. unisensory as well as perceptual vs. descriptive feedback approaches. To these aims, sighted participants performed identical SSD-stimuli identification tests before and after ~75 minutes of self-training on the EyeMusic algorithm. Participants were divided into five groups, differing by the feedback delivered during training: auditory-descriptive, audio-visual textual description, audio-visual perceptual simultaneous and interleaved, and a control group which had no training. At baseline, before any EyeMusic training, participants SSD objects' identification was significantly above chance, highlighting the algorithm's intuitiveness. Furthermore, self-training led to a significant improvement in accuracy between pre- and post-training tests in each of the four feedback groups versus control, though no significant difference emerged among those groups. Nonetheless, significant correlations between individual post-training success rates and various learning measures acquired during training, suggest a trend for an advantage of multisensory vs. unisensory feedback strategies, while no trend emerged for perceptual vs. descriptive strategies. The success at baseline strengthens the conclusion that cross-modal correspondences facilitate learning, given SSD algorithms are based on such correspondences. Additionally, and crucially, the results highlight the feasibility of self-training for the first stages of SSD learning, and suggest that for these initial stages, unisensory training, easily implemented also for blind and visually impaired individuals, may suffice. Together, these findings will potentially boost the use of SSDs for rehabilitation.

Decoding Natural Sounds in Early "Visual" Cortex of Congenitally Blind Individuals.

Complex natural sounds, such as bird singing, people talking, or traffic noise, induce decodable fMRI activation patterns in early visual cortex of sighted blindfolded participants [1]. That is, early visual cortex receives non-visual and potentially predictive information from audition. However, it is unclear whether the transfer of auditory information to early visual areas is an epiphenomenon of visual imagery or, alternatively, whether it is driven by mechanisms independent from visual experience. Here, we show that we can decode natural sounds from activity patterns in early "visual" areas of congenitally blind individuals who lack visual imagery. Thus, visual imagery is not a prerequisite of auditory feedback to early visual cortex. Furthermore, the spatial pattern of sound decoding accuracy in early visual cortex was remarkably similar in blind and sighted individuals, with an increasing decoding accuracy gradient from foveal to peripheral regions. This suggests that the typical organization by eccentricity of early visual cortex develops for auditory feedback, even in the lifelong absence of vision. The same feedback to early visual cortex might support visual perception in the sighted [1] and drive the recruitment of this area for non-visual functions in blind individuals [2, 3].

Reading in the dark: neural correlates and cross-modal plasticity for learning to read entire words without visual experience.

Cognitive neuroscience has long attempted to determine the ways in which cortical selectivity develops, and the impact of nature vs. nurture on it. Congenital blindness (CB) offers a unique opportunity to test this question as the brains of blind individuals develop without visual experience. Here we approach this question through the reading network. Several areas in the visual cortex have been implicated as part of the reading network, and one of the main ones among them is the VWFA, which is selective to the form of letters and words. But what happens in the CB brain? On the one hand, it has been shown that cross-modal plasticity leads to the recruitment of occipital areas, including the VWFA, for linguistic tasks. On the other hand, we have recently demonstrated VWFA activity for letters in contrast to other visual categories when the information is provided via other senses such as touch or audition. Which of these tasks is more dominant? By which mechanism does the CB brain process reading? Using fMRI and visual-to-auditory sensory substitution which transfers the topographical features of the letters we compare reading with semantic and scrambled conditions in a group of CB. We found activation in early auditory and visual cortices during the early processing phase (letter), while the later phase (word) showed VWFA and bilateral dorsal-intraparietal activations for words. This further supports the notion that many visual regions in general, even early visual areas, also maintain a predilection for task processing even when the modality is variable and in spite of putative lifelong linguistic cross-modal plasticity. Furthermore, we find that the VWFA is recruited preferentially for letter and word form, while it was not recruited, and even exhibited deactivation, for an immediately subsequent semantic task suggesting that despite only short sensory substitution experience orthographic task processing can dominate semantic processing in the VWFA. On a wider scope, this implies that at least in some cases cross-modal plasticity which enables the recruitment of areas for new tasks may be dominated by sensory independent task specific activation.

Flexibility and Stability in Sensory Processing Revealed Using Visual-to-Auditory Sensory Substitution.

The classical view of sensory processing involves independent processing in sensory cortices and multisensory integration in associative areas. This hierarchical structure has been challenged by evidence of multisensory responses in sensory areas, and dynamic weighting of sensory inputs in associative areas, thus far reported independently. Here, we used a visual-to-auditory sensory substitution algorithm (SSA) to manipulate the information conveyed by sensory inputs while keeping the stimuli intact. During scan sessions before and after SSA learning, subjects were presented with visual images and auditory soundscapes. The findings reveal 2 dynamic processes. First, crossmodal attenuation of sensory cortices changed direction after SSA learning from visual attenuations of the auditory cortex to auditory attenuations of the visual cortex. Secondly, associative areas changed their sensory response profile from strongest response for visual to that for auditory. The interaction between these phenomena may play an important role in multisensory processing. Consistent features were also found in the sensory dominance in sensory areas and audiovisual convergence in associative area Middle Temporal Gyrus. These 2 factors allow for both stability and a fast, dynamic tuning of the system when required.

Reading with sounds: sensory substitution selectively activates the visual word form area in the blind.

Using a visual-to-auditory sensory-substitution algorithm, congenitally fully blind adults were taught to read and recognize complex images using "soundscapes"--sounds topographically representing images. fMRI was used to examine key questions regarding the visual word form area (VWFA): its selectivity for letters over other visual categories without visual experience, its feature tolerance for reading in a novel sensory modality, and its plasticity for scripts learned in adulthood. The blind activated the VWFA specifically and selectively during the processing of letter soundscapes relative to both textures and visually complex object categories and relative to mental imagery and semantic-content controls. Further, VWFA recruitment for reading soundscapes emerged after 2 hr of training in a blind adult on a novel script. Therefore, the VWFA shows category selectivity regardless of input sensory modality, visual experience, and long-term familiarity or expertise with the script. The VWFA may perform a flexible task-specific rather than sensory-specific computation, possibly linking letter shapes to phonology.

'Visual' acuity of the congenitally blind using visual-to-auditory sensory substitution.

Sensory Substitution Devices (SSDs) convey visual information through sounds or touch, thus theoretically enabling a form of visual rehabilitation in the blind. However, for clinical use, these devices must provide fine-detailed visual information which was not yet shown for this or other means of visual restoration. To test the possible functional acuity conveyed by such devices, we used the Snellen acuity test conveyed through a high-resolution visual-to-auditory SSD (The vOICe). We show that congenitally fully blind adults can exceed the World Health Organization (WHO) blindness acuity threshold using SSDs, reaching the highest acuity reported yet with any visual rehabilitation approach. This demonstrates the potential capacity of SSDs as inexpensive, non-invasive visual rehabilitation aids, alone or when supplementing visual prostheses.

Cortical activity during tactile exploration of objects in blind and sighted humans.

PURPOSE: Recent studies show evidence of multisensory representation in the functionally normal visual cortex, but this idea remains controversial. Occipital cortex activation is often claimed to be a reflection of mental visual imagery processes triggered by other modalities. However, if the occipital cortex is genuinely active during touch, this might be the basis for the massive cross-modal plasticity observed in the congenitally blind. METHODS: To address these issues, we used fMRI to compare patterns of activation evoked by a tactile object recognition (TOR) task (right or left hand) in 8 sighted and 8 congenitally blind subjects, with several other control tasks. RESULTS: TOR robustly activated object selective regions in the lateral occipital complex (LOC/LOtv) in the blind (similar to the patterns of activation found in the sighted), indicating that object identification per se (i.e. in the absence of visual imagery) is sufficient to evoke responses in the LOC/LOtv. Importantly, there was negligible occipital activation for hand movements (imitating object palpations) in the occipital cortex, in both groups. Moreover, in both groups, TOR activation in the LOC/LOtv was bilateral, regardless of the palpating hand (similar to the lack of strong visual field preference in the LOC/LOtv for viewed objects). Finally, the most prominent enhancement in TOR activation in the congenitally blind (compared to their sighted peers) was found in the posterior occipital cortex. CONCLUSIONS: These findings suggest that visual imagery is not an obligatory condition for object activation in visual cortex. It also demonstrates the massive plasticity in visual cortex of the blind for tactile object recognition that involves both the ventral and dorsal occipital areas, probably to support the high demand for this function in the blind.

Disentangling unisensory and multisensory components in audiovisual integration using a novel multifrequency fMRI spectral analysis.

Events in the world are mediated through multiple sensory inputs which are processed separately in specific unisensory areas according to the division-of-labor principle, and then need to be integrated to create a unified percept. How this is done is still not clear. For instance, recent evidence showed crossmodal activation in primary areas. We developed a novel approach to study multisensory integration using multifrequency spectral analysis to investigate the processing of audio and visual streams in a multisensory context. Auditory and visual stimuli were delivered in the same experimental condition, each in different presentation frequencies, and thus could be detected by applying Fourier spectral analysis in their different presentation frequencies. The cochleotopic and retinotopic organization of primary auditory and visual areas were found to remain intact in spite of the multisensory context. Auditory responses were also found in the Precuneus, suggesting that it might be a new auditory area responsive to pure tone stimuli, and serving as one end of a novel sensory preference gradient stretching across POG to the calcarine sulcus. Additional audiovisual areal convergence was detected both in areas in the middle of sensory preference gradients, and in primary auditory areas. Interestingly, the in/out synchronization rate of the auditory and visual streams yielded a third interaction frequency, which could be analyzed independently to reveal higher-order audiovisual interaction responses. These results were detected in one compact and natural multisensory experimental condition, which has several advantages over previous approaches. The method can be further implemented to study any type of interaction, within and across sensory modalities.

Negative BOLD in sensory cortices during verbal memory: a component in generating internal representations?

People tend to close their eyes when trying to retrieve an event or a visual image from memory. However the brain mechanisms behind this phenomenon remain poorly understood. Recently, we showed that during visual mental imagery, auditory areas show a much more robust deactivation than during visual perception. Here we ask whether this is a special case of a more general phenomenon involving retrieval of intrinsic, internally stored information, which would result in crossmodal deactivations in other sensory cortices which are irrelevant to the task at hand. To test this hypothesis, a group of 9 sighted individuals were scanned while performing a memory retrieval task for highly abstract words (i.e., with low imaginability scores). We also scanned a group of 10 congenitally blind, which by definition do not have any visual imagery per se. In sighted subjects, both auditory and visual areas were robustly deactivated during memory retrieval, whereas in the blind the auditory cortex was deactivated while visual areas, shown previously to be relevant for this task, presented a positive BOLD signal. These results suggest that deactivation may be most prominent in task-irrelevant sensory cortices whenever there is a need for retrieval or manipulation of internally stored representations. Thus, there is a task-dependent balance of activation and deactivation that might allow maximization of resources and filtering out of non relevant information to enable allocation of attention to the required task. Furthermore, these results suggest that the balance between positive and negative BOLD might be crucial to our understanding of a large variety of intrinsic and extrinsic tasks including high-level cognitive functions, sensory processing and multisensory integration.

Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex.

The lateral-occipital tactile-visual area (LOtv) is activated when objects are recognized by vision or touch. We report here that the LOtv is also activated in sighted and blind humans who recognize objects by extracting shape information from visual-to-auditory sensory substitution soundscapes. Recognizing objects by their typical sounds or learning to associate specific soundscapes with specific objects do not activate this region. This suggests that LOtv is driven by the presence of shape information.

Early 'visual' cortex activation correlates with superior verbal memory performance in the blind.

The visual cortex may be more modifiable than previously considered. Using functional magnetic resonance imaging (fMRI) in ten congenitally blind human participants, we found robust occipital activation during a verbal-memory task (in the absence of any sensory input), as well as during verb generation and Braille reading. We also found evidence for reorganization and specialization of the occipital cortex, along the anterior-posterior axis. Whereas anterior regions showed preference for Braille, posterior regions (including V1) showed preference for verbal-memory and verb generation (which both require memory of verbal material). No such occipital activation was found in sighted subjects. This difference between the groups was mirrored by superior performance of the blind in various verbal-memory tasks. Moreover, the magnitude of V1 activation during the verbal-memory condition was highly correlated with the blind individual's abilities in a variety of verbal-memory tests, suggesting that the additional occipital activation may have a functional role.