Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device.

Current advancements in both technology and science allow us to manipulate our sensory modalities in new and unexpected ways. In the present study, we explore the potential of expanding what we perceive through our natural senses by utilizing a visual-to-auditory sensory substitution device (SSD), the EyeMusic, an algorithm that converts images to sound. The EyeMusic was initially developed to allow blind individuals to create a spatial representation of information arriving from a video feed at a slow sampling rate. In this study, we aimed to use the EyeMusic for the blind areas of sighted individuals. We use it in this initial proof-of-concept study to test the ability of sighted subjects to combine visual information with surrounding auditory sonification representing visual information. Participants in this study were tasked with recognizing and adequately placing the stimuli, using sound to represent the areas outside the standard human visual field. As such, the participants were asked to report shapes' identities as well as their spatial orientation (front/right/back/left), requiring combined visual (90° frontal) and auditory input (the remaining 270°) for the successful performance of the task (content in both vision and audition was presented in a sweeping clockwise motion around the participant). We found that participants were successful at a highly above chance level after a brief 1-h-long session of online training and one on-site training session of an average of 20 min. They could even draw a 2D representation of this image in some cases. Participants could also generalize, recognizing new shapes they were not explicitly trained on. Our findings provide an initial proof of concept indicating that sensory augmentation devices and techniques can potentially be used in combination with natural sensory information in order to expand the natural fields of sensory perception.

Loss of action-related function and connectivity in the blind extrastriate body area.

The Extrastriate Body Area (EBA) participates in the visual perception and motor actions of body parts. We recently showed that EBA's perceptual function develops independently of visual experience, responding to stimuli with body-part information in a supramodal fashion. However, it is still unclear if the EBA similarly maintains its action-related function. Here, we used fMRI to study motor-evoked responses and connectivity patterns in the congenitally blind brain. We found that, unlike the case of perception, EBA does not develop an action-related response without visual experience. In addition, we show that congenital blindness alters EBA's connectivity profile in a counterintuitive way-functional connectivity with sensorimotor cortices dramatically decreases, whereas connectivity with perception-related visual occipital cortices remains high. To the best of our knowledge, we show for the first time that action-related functions and connectivity in the visual cortex could be contingent on visuomotor experience. We further discuss the role of the EBA within the context of visuomotor control and predictive coding theory.

Gyrification in relation to cortical thickness in the congenitally blind.

Greater cortical gyrification (GY) is linked with enhanced cognitive abilities and is also negatively related to cortical thickness (CT). Individuals who are congenitally blind (CB) exhibits remarkable functional brain plasticity which enables them to perform certain non-visual and cognitive tasks with supranormal abilities. For instance, extensive training using touch and audition enables CB people to develop impressive skills and there is evidence linking these skills to cross-modal activations of primary visual areas. There is a cascade of anatomical, morphometric and functional-connectivity changes in non-visual structures, volumetric reductions in several components of the visual system, and CT is also increased in CB. No study to date has explored GY changes in this population, and no study has explored how variations in CT are related to GY changes in CB. T1-weighted 3D structural magnetic resonance imaging scans were acquired to examine the effects of congenital visual deprivation in cortical structures in a healthy sample of 11 CB individuals (6 male) and 16 age-matched sighted controls (SC) (10 male). In this report, we show for the first time an increase in GY in several brain areas of CB individuals compared to SC, and a negative relationship between GY and CT in the CB brain in several different cortical areas. We discuss the implications of our findings and the contributions of developmental factors and synaptogenesis to the relationship between CT and GY in CB individuals compared to SC. F.

A case study in phenomenology of visual experience with retinal prosthesis versus visual-to-auditory sensory substitution.

The phenomenology of the blind has provided an age-old, unparalleled means of exploring the enigmatic link between the brain and mind. This paper delves into the unique phenomenological experience of a man who became blind in adulthood. He subsequently underwent both an Argus II retinal prosthesis implant and training, and extensive training on the EyeMusic visual to auditory sensory substitution device (SSD), thereby becoming the first reported case to date of dual proficiency with both devices. He offers a firsthand account into what he considers the great potential of combining sensory substitution devices with visual prostheses as part of a complete visual restoration protocol. While the Argus II retinal prosthesis alone provided him with immediate visual percepts by way of electrically stimulated phosphenes elicited by the device, the EyeMusic SSD requires extensive training from the onset. Yet following the extensive training program with the EyeMusic sensory substitution device, our subject reports that the sensory substitution device allowed him to experience a richer, more complex perceptual experience, that felt more "second nature" to him, while the Argus II prosthesis (which also requires training) did not allow him to achieve the same levels of automaticity and transparency. Following long-term use of the EyeMusic SSD, our subject reported that visual percepts representing mainly, but not limited to, colors portrayed by the EyeMusic SSD are elicited in association with auditory stimuli, indicating the acquisition of a high level of automaticity. Finally, the case study indicates an additive benefit to the combination of both devices on the user's subjective phenomenological visual experience.

Body Ownership of Anatomically Implausible Hands in Virtual Reality.

Manipulating sensory and motor cues can cause an illusionary perception of ownership of a fake body part. Presumably, the illusion can work as long as the false body part's position and appearance are anatomically plausible. Here, we introduce an illusion that challenges past assumptions on body ownership. We used virtual reality to switch and mirror participants' views of their hands. When a participant moves their physical hand, they see the incongruent virtual hand moving. The result is an anatomically implausible configuration of the fake hand. Despite the hand switch, participants reported significant body ownership sensations over the virtual hands. In the first between-group experiment, we found that the strength of body ownership over the incongruent hands was similar to that of congruent hands. Whereas, in the second within-group experiment, anatomical incongruency significantly decreased body ownership. Still, participants reported significant body ownership sensations of the switched hands. Curiously, we found that perceived levels of agency mediate the effect of anatomical congruency on body ownership. These findings offer a fresh perspective on the relationship between anatomical plausibility and assumed body ownership. We propose that goal-directed and purposeful actions can override anatomical plausibility constraints and discuss this in the context of the immersive properties of virtual reality.

Applying a novel visual-to-touch sensory substitution for studying tactile reference frames.

Perceiving the spatial location and physical dimensions of touched objects is crucial for goal-directed actions. To achieve this, our brain transforms skin-based coordinates into a reference frame by integrating visual and posture information. In the current study, we examine the role of posture in mapping tactile sensations to a visual image. We developed a new visual-to-touch sensory substitution device that transforms images into a sequence of vibrations on the arm. 52 blindfolded participants performed spatial recognition tasks in three different arm postures and had to switch postures between trial blocks. As participants were not told which side of the device is down and which is up, they could choose how to map its vertical axis in their responses. Contrary to previous findings, we show that new proprioceptive inputs can be overridden in mapping tactile sensations. We discuss the results within the context of the spatial task and the various sensory contributions to the process.