Enhancing learning outcomes through multisensory integration: A fMRI study of audio-visual training in virtual reality.

The integration of information from different sensory modalities is a fundamental process that enhances perception and performance in real and virtual environments (VR). Understanding these mechanisms, especially during learning tasks that exploit novel multisensory cue combinations provides opportunities for the development of new rehabilitative interventions. This study aimed to investigate how functional brain changes support behavioural performance improvements during an audio-visual (AV) learning task. Twenty healthy participants underwent a 30 min daily VR training for four weeks. The task was an AV adaptation of a 'scanning training' paradigm that is commonly used in hemianopia rehabilitation. Functional magnetic resonance imaging (fMRI) and performance data were collected at baseline, after two and four weeks of training, and four weeks post-training. We show that behavioural performance, operationalised as mean reaction time reduction in VR, significantly improves. In separate tests in a controlled laboratory environment, we showed that the behavioural performance gains in the VR training environment transferred to a significant mean RT reduction for the trained AV voluntary task on a computer screen. Enhancements were observed in both the visual-only and AV conditions, with the latter demonstrating a faster response time supported by the presence of audio cues. The behavioural learning effect also transfers to two additional tasks that were tested: a visual search task and an involuntary visual task. Our fMRI results reveal an increase in functional activation (BOLD signal) in multisensory brain regions involved in early-stage AV processing: the thalamus, the caudal inferior parietal lobe and cerebellum. These functional changes were only observed for the trained, multisensory, task and not for unimodal visual stimulation. Functional activation changes in the thalamus were significantly correlated to behavioural performance improvements. This study demonstrates that incorporating spatial auditory cues to voluntary visual training in VR leads to augmented brain activation changes in multisensory integration, resulting in measurable performance gains across tasks. The findings highlight the potential of VR-based multisensory training as an effective method for enhancing cognitive function and as a potentially valuable tool in rehabilitative programmes.

Audio-visual stimulation for visual compensatory functions in stroke survivors with visual field defect: a systematic review.

BACKGROUND: Hemianopia is a complete or partial blindness in the visual fields of both eyes, commonly caused by cerebral infarction. It has been hypothesized that systematic audio-visual (AV) stimulation of the blind hemifield can improve accuracy and search times, probably due to the stimulation of bimodal representations in the superior colliculus (SC), an important multisensory structure involved in both the initiation and execution of saccades. METHODS: A narrative synthesis of the findings is presented to highlight how AV rehabilitation impacts on patients with hemianopia including visual oculomotor function, functional ability in activities of daily living, hemianopic dyslexia, visual scanning and searching tasks, maintaining of functional ability post training and the effect on brain multisensory integration by using neuroimaging. RESULTS: Sixteen studies were included (fourteen articles (188 participants) and two literature reviews). Results were grouped into AV training of hemianopia in adults and in children and then further grouped according to the AV task type: tasks measuring the training effects by comparing visual stimulation training to audio-visual training, localization abilities in homonymous hemianopia (HH) and AV integration in patients with HH. CONCLUSION: Systematic AV training may improve the processing of visual information by recruiting subcortical pathways, and because most of the patients with visual cortex damage have an intact SC, it might be useful to use the bimodal AV training to activate retinotectal functions. Nevertheless, the underlying mechanisms supporting the reported positive effects are not currently understood. Systematic functional and/or structural imaging studies may help in understanding the underlying mechanism and inform the design of optimal training paradigms.