Somatosensory omissions reveal action-related predictive processing.

The intricate relation between action and somatosensory perception has been studied extensively in the past decades. Generally, a forward model is thought to predict the somatosensory consequences of an action. These models propose that when an action is reliably coupled to a tactile stimulus, unexpected absence of the stimulus should elicit prediction error. Although such omission responses have been demonstrated in the auditory modality, it remains unknown whether this mechanism generalizes across modalities. This study therefore aimed to record action-induced somatosensory omission responses using EEG in humans. Self-paced button presses were coupled to somatosensory stimuli in 88% of trials, allowing a prediction, or in 50% of trials, not allowing a prediction. In the 88% condition, stimulus omission resulted in a neural response consisting of multiple components, as revealed by temporal principal component analysis. The oN1 response suggests similar sensory sources as stimulus-evoked activity, but an origin outside primary cortex. Subsequent oN2 and oP3 responses, as previously observed in the auditory domain, likely reflect modality-unspecific higher order processes. Together, findings straightforwardly demonstrate somatosensory predictions during action and provide evidence for a partially amodal mechanism of prediction error generation.

Temporal Audiovisual Motion Prediction in 2D- vs. 3D-Environments.

Predicting motion is essential for many everyday life activities, e.g., in road traffic. Previous studies on motion prediction failed to find consistent results, which might be due to the use of very different stimulus material and behavioural tasks. Here, we directly tested the influence of task (detection, extrapolation) and stimulus features (visual vs. audiovisual and three-dimensional vs. non-three-dimensional) on temporal motion prediction in two psychophysical experiments. In both experiments a ball followed a trajectory toward the observer and temporarily disappeared behind an occluder. In audiovisual conditions a moving white noise (congruent or non-congruent to visual motion direction) was presented concurrently. In experiment 1 the ball reappeared on a predictable or a non-predictable trajectory and participants detected when the ball reappeared. In experiment 2 the ball did not reappear after occlusion and participants judged when the ball would reach a specified position at two possible distances from the occluder (extrapolation task). Both experiments were conducted in three-dimensional space (using stereoscopic screen and polarised glasses) and also without stereoscopic presentation. Participants benefitted from visually predictable trajectories and concurrent sounds during detection. Additionally, visual facilitation was more pronounced for non-3D stimulation during detection task. In contrast, for a more complex extrapolation task group mean results indicated that auditory information impaired motion prediction. However, a post hoc cross-validation procedure (split-half) revealed that participants varied in their ability to use sounds during motion extrapolation. Most participants selectively profited from either near or far extrapolation distances but were impaired for the other one. We propose that interindividual differences in extrapolation efficiency might be the mechanism governing this effect. Together, our results indicate that both a realistic experimental environment and subject-specific differences modulate the ability of audiovisual motion prediction and need to be considered in future research.

Possible anatomical pathways for short-latency multisensory integration processes in primary sensory cortices.

Multisensory integration does not only recruit higher-level association cortex, but also low-level and even primary sensory cortices. Here, we will describe and quantify two types of anatomical pathways, a thalamocortical and a corticocortical that possibly underlie short-latency multisensory integration processes in the primary auditory (A1), somatosensory (S1), and visual cortex (V1). Results were obtained from Mongolian gerbils, a common model-species in neuroscience, using simultaneous injections of different retrograde tracers into A1, S1, and V1. Several auditory, visual, and somatosensory thalamic nuclei project not only to the primary sensory area of their own (matched) but also to areas of other (non-matched) modalities. The crossmodal output ratios of these nuclei, belonging to both core and non-core sensory pathways, vary between 0.4 and 63.5 % of the labeled neurons. Approximately 0.3 % of the sensory thalamic input to A1, 5.0 % to S1, and 2.1 % to V1 arise from non-matched nuclei. V1 has most crossmodal corticocortical connections, projecting strongest to S1 and receiving a similar amount of moderate inputs from A1 and S1. S1 is mainly interconnected with V1. A1 has slightly more projections to V1 than S1, but gets just faint inputs from there. Concerning the layer-specific distribution of the retrogradely labeled somata in cortex, V1 provides the most pronounced feedforward-type outputs and receives (together with S1) most pronounced feedback-type inputs. In contrast, A1 has most pronounced feedback-type outputs and feedforward-type inputs in this network. Functionally, the different sets of thalamocortical and corticocortical connections could underlie distinctive types of integration mechanisms for different modality pairings.

Task-demands and audio-visual stimulus configurations modulate neural activity in the human thalamus.

Recent electrophysiological studies have reported short latency modulations in cortical regions for multisensory stimuli, thereby suggesting a subcortical, possibly thalamic origin of these modulations. Concurrently, there is an ongoing debate, whether multisensory interplay reflects automatic, bottom-up driven processes or relies on top-down influences. Here, we dissociated the effects of task set and stimulus configurations on BOLD-signals in the human thalamus with event-related functional magnetic resonance imaging (fMRI). We orthogonally manipulated temporal and spatial congruency of audio-visual stimulus configurations, while subjects judged either their temporal or spatial congruency. Voxel-based fMRI results revealed increased fMRI-signals for the temporal versus spatial task in posterior and central thalamus, respectively. A more sensitive region of interest (ROI)-analysis confirmed that the posterior thalamic nuclei showed a preference for the temporal task and central thalamic nuclei for the spatial task. Moreover, the ROI-analysis also revealed enhanced fMRI-signals for spatially incongruent stimuli in the central thalamus. Together, our results demonstrate that both audio-visual stimulus configurations and task-related processing of spatial or temporal stimulus features selectively modulate thalamic processing and thus are in a position to influence cortical processing at an early stage.

Neural basis of multisensory looming signals.

Approaching or looming signals are often related to extremely relevant environmental events (e.g. threats or collisions) making these signals critical for survival. However, the neural network underlying multisensory looming processing is not yet fully understood. Using functional magnetic resonance imaging (fMRI) we identified the neural correlates of audiovisual looming processing in humans: audiovisual looming (vs. receding) signals enhance fMRI-responses in low-level visual and auditory areas plus multisensory cortex (superior temporal sulcus; plus parietal and frontal structures). When characterizing the fMRI-response profiles for multisensory looming stimuli, we found significant enhancements relative to the mean and maximum of unisensory responses in looming-sensitive visual and auditory cortex plus STS. Superadditive enhancements were observed in visual cortex. Subject-specific region-of-interest analyses further revealed superadditive response profiles within all sensory-specific looming-sensitive structures plus bilateral STS for audiovisual looming vs. summed unisensory looming conditions. Finally, we observed enhanced connectivity of bilateral STS with low-level visual areas in the context of looming processing. This enhanced coupling of STS with unisensory regions might potentially serve to enhance the salience of unisensory stimulus features and is accompanied by superadditive fMRI-responses. We suggest that this preference in neural signaling for looming stimuli effectively informs animals to avoid potential threats or collisions.

Coding of multisensory temporal patterns in human superior temporal sulcus.

Philosophers, psychologists, and neuroscientists have long been interested in how the temporal aspects of perception are represented in the brain. In the present study, we investigated the neural basis of the temporal perception of synchrony/asynchrony for audiovisual speech stimuli using functional magnetic resonance imaging (fMRI). Subjects judged the temporal relation of (a)synchronous audiovisual speech streams, and indicated any changes in their perception of the stimuli over time. Differential hemodynamic responses for synchronous versus asynchronous stimuli were observed in the multisensory superior temporal sulcus complex (mSTS-c) and prefrontal cortex. Within mSTS-c we found adjacent regions expressing an enhanced BOLD-response to the different physical (a)synchrony conditions. These regions were further modulated by the subjects' perceptual state. By calculating the distances between the modulated regions within mSTS-c in single-subjects we demonstrate that the "auditory leading (A(L))" and "visual leading (V(L)) areas" lie closer to "synchrony areas" than to each other. Moreover, analysis of interregional connectivity indicates a stronger functional connection between multisensory prefrontal cortex and mSTS-c during the perception of asynchrony. Taken together, these results therefore suggest the presence of distinct sub-regions within the human STS-c for the maintenance of temporal relations for audiovisual speech stimuli plus differential functional connectivity with prefrontal regions. The respective local activity in mSTS-c is dependent both upon the physical properties of the stimuli presented and upon the subjects' perception of (a)synchrony.

Reaction times reflect subjective auditory perception of tone sequences in macaque monkeys.

Perceptually ambiguous stimuli are useful for testing psychological and neuronal models of perceptual organization, e.g. for studying brain processes that underlie sequential segregation and integration. This is because the same stimuli may give rise to different subjective experiences. For humans, a tone sequence that alternates between a low-frequency and a high-frequency tone is perceptually bistable, and can be perceived as one or two streams. In the current study we present a new method based on response times (RTs) which allows identification ambiguous and unambiguous stimuli for subjects who cannot verbally report their subjective experience. We required two macaque monkeys (macaca fascicularis) to detect the termination of a sequence of light flashes which were either presented alone, or synchronized in different ways with a sequence of alternating low and high tones. We found that the monkeys responded faster to the termination of the flash sequence when the tone sequence terminated shortly before the flash sequence and thus predicted the termination of the flash sequence. This RT gain depended on the frequency separation of the tones. RT gains were largest when the frequency separation was small and the tones were presumably heard mainly as one stream. RT gains were smallest when the frequency separation was large and the tones were presumably mainly heard as two streams. RT gain was of intermediate size for intermediate frequency separations. Similar results were obtained from human subjects. We conclude that the observed RT gains reflect the perceptual organization of the tone sequence, and that tone sequences with an intermediate frequency separation, as for humans, are perceptually ambiguous for monkeys.

Mandatory processing of irrelevant fearful face features in visual search.

Faces expressing fear may attract attention in an automatic bottom-up fashion. Here we address this issue with magneto-encephalographic (MEG) recordings in subjects performing a demanding visual search combined with the presentation of irrelevant neutral or fearful faces. The impact of the faces' emotional expression on attentional selection was assessed by analyzing the N2pc component--a modulation of the event-related magnetic field response known to reflect attentional focusing in visual search. We observed that lateralized fearful faces elicited an N2pc approximately between 240 and 400 msec in ventral extrastriate cortex that was independent of the N2pc reflecting target selection in visual search. Despite their clear influence on neural processing, fearful faces did not significantly influence behavioral performance. To clarify this discrepancy, we further performed an MEG experiment in which the demands of the search task were reduced. Under those conditions, lateralized fearful faces elicited an N2pc response that was again independent of the N2pc response to the search target. Behavioral performance was, however, influenced in a significant manner, suggesting that for behavioral effects to appear, sufficient attentional resources need to be left unoccupied by the search task--a notion put forward by the perceptual load theory. Our observations are taken to indicate that irrelevant fearful faces influence attentional processing in extrastriate visual cortex in an automatic fashion and independent of other task-relevant attentional operations. However, this may not necessarily be echoed at the behavioral level as long as task-relevant selection operations exhaust attentional resources.

Endoscopic eye tracking system for fMRI.

Here we introduce a new video-based real-time eye tracking system suitable for functional magnetic resonance imaging (fMRI) application. The described system monitors the subject's eye, which is illuminated with infrared light, directly at the headcoil using an endoscopic fibre optical system. This endoscopic technique assures reliable, easy-to-use and fast adjustment. It requires only a minimal amount of equipment at the headcoil and inside the examination room. Moreover, the short distance between the image acquisition optics and the eye provides high spatial tracking resolution. Interference from physiological head movement is effectively reduced by simultaneous tracking of both eye and head movements.

Top-down and bottom-up modulation of language related areas--an fMRI study.

BACKGROUND: One major problem for cognitive neuroscience is to describe the interaction between stimulus and task driven neural modulation. We used fMRI to investigate this interaction in the human brain. Ten male subjects performed a passive listening and a semantic categorization task in a factorial design. In both tasks, words were presented auditorily at three different rates. RESULTS: We found: (i) as word presentation rate increased hemodynamic responses increased bilaterally in the superior temporal gyrus including Heschl's gyrus (HG), the planum temporale (PT), and the planum polare (PP); (ii) compared to passive listening, semantic categorization produced increased bilateral activations in the ventral inferior frontal gyrus (IFG) and middle frontal gyrus (MFG); (iii) hemodynamic responses in the left dorsal IFG increased linearly with increasing word presentation rate only during the semantic categorization task; (iv) in the semantic task hemodynamic responses decreased bilaterally in the insula with increasing word presentation rates; and (v) in parts of the HG the hemodynamic response increased with increasing word presentation rates during passive listening more strongly. CONCLUSION: The observed "rate effect" in primary and secondary auditory cortex is in accord with previous findings and suggests that these areas are driven by low-level stimulus attributes. The bilateral effect of semantic categorization is also in accord with previous studies and emphasizes the role of these areas in semantic operations. The interaction between semantic categorization and word presentation in the left IFG indicates that this area has linguistic functions not present in the right IFG. Finally, we speculate that the interaction between semantic categorization and word presentation rates in HG and the insula might reflect an inhibition of the transfer of unnecessary information from the temporal to frontal regions of the brain.