Seeing and extrapolating motion trajectories share common informative activation patterns in primary visual cortex.

The natural environment is dynamic and moving objects become constantly occluded, engaging the brain in a challenging completion process to estimate where and when the object might reappear. Although motion extrapolation is critical in daily life-imagine crossing the street while an approaching car is occluded by a larger standing vehicle-its neural underpinnings are still not well understood. While the engagement of low-level visual cortex during dynamic occlusion has been postulated, most of the previous group-level fMRI-studies failed to find evidence for an involvement of low-level visual areas during occlusion. In this fMRI-study, we therefore used individually defined retinotopic maps and multivariate pattern analysis to characterize the neural basis of visible and occluded changes in motion direction in humans. To this end, participants learned velocity-direction change pairings (slow motion-upwards; fast motion-downwards or vice versa) during a training phase without occlusion and judged the change in stimulus direction, based on its velocity, during a following test phase with occlusion. We find that occluded motion direction can be predicted from the activity patterns during visible motion within low-level visual areas, supporting the notion of a mental representation of motion trajectory in these regions during occlusion.

Macroscopic information-based taste representations in insular cortex are shaped by stimulus concentration.

Taste processing is an essential ability in all animals signaling potential harm or benefit of ingestive behavior. However, current evidence for cortical taste representations remains contradictory. To address this issue, high-resolution functional MRI (fMRI) and multivariate pattern analysis were used to characterize taste-related informational content in human insular cortex, which contains primary gustatory cortex. Human participants judged pleasantness and intensity of low- and high-concentration tastes (salty, sweet, sour, and bitter) in two fMRI experiments on two different days to test for task- and concentration-invariant taste representations. We observed patterns of fMRI activity within insular cortex narrowly tuned to specific tastants consistently across tasks in all participants. Fewer patterns responded to more than one taste category. Importantly, changes in taste concentration altered the spatial layout of putative taste-specific patterns with distinct, almost nonoverlapping patterns for each taste category at different concentration levels. Together, our results point at macroscopic representations in human insular cortex as a complex function of taste category and concentration rather than representations based solely on taste identity.

Context dependency of time-based event-related expectations for different modalities.

Expectations about the temporal occurrence of events (when) are often tied with the expectations about certain event-related properties (what and where) happening at these time points. For instance, slowly waking up in the morning we expect our alarm clock to go off; however, the longer we do not hear it the more likely we already missed it. However, most current evidence for complex time-based event-related expectations (TBEEs) is based on the visual modality. Here we tested whether implicit TBEEs can act cross-modally. To this end, visual and auditory stimulus streams were presented which contained early and late targets embedded among distractors (to maximise temporal target uncertainty). Foreperiod-modality-contingencies were manipulated run-wise: visual targets either occurred early in 80% of trials and auditory targets occurred late in 80% of trials or vice versa. Participants showed increased sensitivity for expected auditory early/visual late targets which increased over time while the opposite pattern was observed for visual early/auditory late targets. A benefit in reaction times was only found for auditory early trials. Together, this pattern of results suggests that implicit context-dependent TBEEs for auditory targets after short foreperiods (be they correct or not) dominated and determined which modality became more expected at the late position irrespective of the veridical statistical regularity. Hence, TBEEs in cross-modal and uncertain environments are context-dependent, shaped by the dominant modality in temporal tasks (i.e., auditory) and only boost performance cross-modally when expectations about the event after the short foreperiod match with the run-wise context (i.e., auditory early/visual late).

The spatio-temporal profile of multisensory integration.

Task-irrelevant visual stimuli can enhance auditory perception. However, while there is some neurophysiological evidence for mechanisms that underlie the phenomenon, the neural basis of visually induced effects on auditory perception remains unknown. Combining fMRI and EEG with psychophysical measurements in two independent studies, we identified the neural underpinnings and temporal dynamics of visually induced auditory enhancement. Lower- and higher-intensity sounds were paired with a non-informative visual stimulus, while participants performed an auditory detection task. Behaviourally, visual co-stimulation enhanced auditory sensitivity. Using fMRI, enhanced BOLD signals were observed in primary auditory cortex for low-intensity audiovisual stimuli which scaled with subject-specific enhancement in perceptual sensitivity. Concordantly, a modulation of event-related potentials could already be observed over frontal electrodes at an early latency (30-80 ms), which again scaled with subject-specific behavioural benefits. Later modulations starting around 280 ms, that is in the time range of the P3, did not fit this pattern of brain-behaviour correspondence. Hence, the latency of the corresponding fMRI-EEG brain-behaviour modulation points at an early interplay of visual and auditory signals in low-level auditory cortex, potentially mediated by crosstalk at the level of the thalamus. However, fMRI signals in primary auditory cortex, auditory thalamus and the P50 for higher-intensity auditory stimuli were also elevated by visual co-stimulation (in the absence of any behavioural effect) suggesting a general, intensity-independent integration mechanism. We propose that this automatic interaction occurs at the level of the thalamus and might signify a first step of audiovisual interplay necessary for visually induced perceptual enhancement of auditory perception.

Strength of Temporal White Matter Pathways Predicts Semantic Learning.

Learning the associations between words and meanings is a fundamental human ability. Although the language network is cortically well defined, the role of the white matter pathways supporting novel word-to-meaning mappings remains unclear. Here, by using contextual and cross-situational word learning, we tested whether learning the meaning of a new word is related to the integrity of the language-related white matter pathways in 40 adults (18 women). The arcuate, uncinate, inferior-fronto-occipital and inferior-longitudinal fasciculi were virtually dissected using manual and automatic deterministic fiber tracking. Critically, the automatic method allowed assessing the white matter microstructure along the tract. Results demonstrate that the microstructural properties of the left inferior-longitudinal fasciculus predict contextual learning, whereas the left uncinate was associated with cross-situational learning. In addition, we identified regions of special importance within these pathways: the posterior middle temporal gyrus, thought to serve as a lexical interface and specifically related to contextual learning; the anterior temporal lobe, known to be an amodal hub for semantic processing and related to cross-situational learning; and the white matter near the hippocampus, a structure fundamental for the initial stages of new-word learning and, remarkably, related to both types of word learning. No significant associations were found for the inferior-fronto-occipital fasciculus or the arcuate. While previous results suggest that learning new phonological word forms is mediated by the arcuate fasciculus, these findings show that the temporal pathways are the crucial neural substrate supporting one of the most striking human abilities: our capacity to identify correct associations between words and meanings under referential indeterminacy.SIGNIFICANCE STATEMENT The language-processing network is cortically (i.e., gray matter) well defined. However, the role of the white matter pathways that support novel word learning within this network remains unclear. In this work, we dissected language-related (arcuate, uncinate, inferior-fronto-occipital, and inferior-longitudinal) fasciculi using manual and automatic tracking. We found the left inferior-longitudinal fasciculus to be predictive of word-learning success in two word-to-meaning tasks: contextual and cross-situational learning paradigms. The left uncinate was predictive of cross-situational word learning. No significant correlations were found for the arcuate or the inferior-fronto-occipital fasciculus. While previous results showed that learning new phonological word forms is supported by the arcuate fasciculus, these findings demonstrate that learning new word-to-meaning associations is mainly dependent on temporal white matter pathways.

The role of auditory cortex in the spatial ventriloquism aftereffect.

Cross-modal recalibration allows the brain to maintain coherent sensory representations of the world. Using functional magnetic resonance imaging (fMRI), the present study aimed at identifying the neural mechanisms underlying recalibration in an audiovisual ventriloquism aftereffect paradigm. Participants performed a unimodal sound localization task, before and after they were exposed to adaptation blocks, in which sounds were paired with spatially disparate visual stimuli offset by 14° to the right. Behavioral results showed a significant rightward shift in sound localization following adaptation, indicating a ventriloquism aftereffect. Regarding fMRI results, left and right planum temporale (lPT/rPT) were found to respond more to contralateral sounds than to central sounds at pretest. Contrasting posttest with pretest blocks revealed significantly enhanced fMRI-signals in space-sensitive lPT after adaptation, matching the behavioral rightward shift in sound localization. Moreover, a region-of-interest analysis in lPT/rPT revealed that the lPT activity correlated positively with the localization shift for right-side sounds, whereas rPT activity correlated negatively with the localization shift for left-side and central sounds. Finally, using functional connectivity analysis, we observed enhanced coupling of the lPT with left and right inferior parietal areas as well as left motor regions following adaptation and a decoupling of lPT/rPT with contralateral auditory cortex, which scaled with participants' degree of adaptation. Together, the fMRI results suggest that cross-modal spatial recalibration is accomplished by an adjustment of unisensory representations in low-level auditory cortex. Such persistent adjustments of low-level sensory representations seem to be mediated by the interplay with higher-level spatial representations in parietal cortex.

How task demands shape brain responses to visual food cues.

Several previous imaging studies have aimed at identifying the neural basis of visual food cue processing in humans. However, there is little consistency of the functional magnetic resonance imaging (fMRI) results across studies. Here, we tested the hypothesis that this variability across studies might - at least in part - be caused by the different tasks employed. In particular, we assessed directly the influence of task set on brain responses to food stimuli with fMRI using two tasks (colour vs. edibility judgement, between-subjects design). When participants judged colour, the left insula, the left inferior parietal lobule, occipital areas, the left orbitofrontal cortex and other frontal areas expressed enhanced fMRI responses to food relative to non-food pictures. However, when judging edibility, enhanced fMRI responses to food pictures were observed in the superior and middle frontal gyrus and in medial frontal areas including the pregenual anterior cingulate cortex and ventromedial prefrontal cortex. This pattern of results indicates that task sets can significantly alter the neural underpinnings of food cue processing. We propose that judging low-level visual stimulus characteristics - such as colour - triggers stimulus-related representations in the visual and even in gustatory cortex (insula), whereas discriminating abstract stimulus categories activates higher order representations in both the anterior cingulate and prefrontal cortex. Hum Brain Mapp 38:2897-2912, 2017. © 2017 Wiley Periodicals, Inc.

Audio-visual synchrony modulates the ventriloquist illusion and its neural/spatial representation in the auditory cortex.

An essential task of our perceptual systems is to bind together the distinctive features of single objects and events into unitary percepts, even when those features are registered in different sensory modalities. In cases where auditory and visual inputs are spatially incongruent, they may still be perceived as belonging to a single event at the location of the visual stimulus - a phenomenon known as the 'ventriloquist illusion'. The present study examined how audio-visual temporal congruence influences the ventriloquist illusion and characterized its neural underpinnings with functional magnetic resonance imaging (fMRI). Behaviorally, the ventriloquist illusion was reduced for asynchronous versus synchronous audio-visual stimuli, in accordance with previous reports. Neural activity patterns associated with the ventriloquist effect were consistently observed in the planum temporale (PT), with a reduction in illusion-related fMRI-signals ipsilateral to visual stimulation for central sounds perceived peripherally and a contralateral increase in illusion-related fMRI-signals for peripheral sounds perceived centrally. Moreover, it was found that separate but adjacent regions within the PT were preferentially activated for ventriloquist illusions produced by synchronous and asynchronous audio-visual stimulation. We conclude that the left-right balance of neural activity in the PT represents the neural code that underlies the ventriloquist illusion, with greater activity in the cerebral hemisphere contralateral to the direction of the perceived shift of sound location.

Entering into a self-regulated learning mode prevents detrimental effects of feedback removal on memory.

Incentives can decrease performance by undermining intrinsic motivation. How such an interplay of external reinforcers and internal self-regulation influences memory processes, however, is less known. Here, we investigated their interaction on memory performance while learning the meaning of new-words from their context. Specifically, participants inferred congruent meanings of new-words from semantic context (congruent trials) or lack of congruence (incongruent trials), while receiving external feedback in the first or second half of trials only. Removing feedback during learning of congruent word meanings lowered subsequent recognition rates a day later, whereas recognition remained high in the group, which received feedback only in the second half. In contrast, feedback did not substantially alter recognition rates for learning that new-words had no congruent meanings. Our findings suggest that external reinforcers can selectively impair memories if internal self-regulated processes are not already established, but whether they do so depends on what is being learned (specific word-meanings vs. unspecific incongruence). This highlights the relevance of self-regulated learning in education to support stable memory formation.

The Potential Role of Gustatory Function as an Early Diagnostic Marker for the Risk of Alzheimer's Disease in Subjective Cognitive Decline.

Background: Patients with subjective cognitive decline (SCD) report memory deterioration and are at an increased risk of converting to Alzheimer's disease (AD) although psychophysical testing does not reveal any cognitive deficit. Objective: Here, gustatory function is investigated as a potential predictor for an increased risk of progressive cognitive decline indicating higher AD risk in SCD. Methods: Measures of smell and taste perception as well as neuropsychological data were assessed in patients with subjective cognitive decline (SCD): Subgroups with an increased likelihood of the progression to preclinical AD (SCD+) and those with a lower likelihood (SCD-) were compared to healthy controls (HC), patients with mild cognitive impairment and AD patients. The Sniffin' Sticks test contained 12 items with different qualities and taste was measured with 32 taste stripes (sweet, salty, bitter, sour) of different concentration. Results: Only taste was able to distinguish between HC/SCD- and SCD+ patients. Conclusion: This study provides a first hint of taste as a more sensitive marker than smell for detecting preclinical AD in SCD. Longitudinal observation of cognition and pathology are necessary to further evaluate taste perception as a predictor of pathological objective decline in cognition.

Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments.

Although much traditional sensory research has studied each sensory modality in isolation, there has been a recent explosion of interest in causal interplay between different senses. Various techniques have now identified numerous multisensory convergence zones in the brain. Some convergence may arise surprisingly close to low-level sensory-specific cortex, and some direct connections may exist even between primary sensory cortices. A variety of multisensory phenomena have now been reported in which sensory-specific brain responses and perceptual judgments concerning one sense can be affected by relations with other senses. We survey recent progress in this multisensory field, foregrounding human studies against the background of invasive animal work and highlighting possible underlying mechanisms. These include rapid feedforward integration, possible thalamic influences, and/or feedback from multisensory regions to sensory-specific brain areas. Multisensory interplay is more prevalent than classic modular approaches assumed, and new methods are now available to determine the underlying circuits.

Task-load-dependent activation of dopaminergic midbrain areas in the absence of reward.

Dopamine release in cortical and subcortical structures plays a central role in reward-related neural processes. Within this context, dopaminergic inputs are commonly assumed to play an activating role, facilitating behavioral and cognitive operations necessary to obtain a prospective reward. Here, we provide evidence from human fMRI that this activating role can also be mediated by task-demand-related processes and thus extends beyond situations that only entail extrinsic motivating factors. Using a visual discrimination task in which varying levels of task demands were precued, we found enhanced hemodynamic activity in the substantia nigra (SN) for high task demands in the absence of reward or similar extrinsic motivating factors. This observation thus indicates that the SN can also be activated in an endogenous fashion. In parallel to its role in reward-related processes, reward-independent activation likely serves to recruit the processing resources needed to meet enhanced task demands. Simultaneously, activity in a wide network of cortical and subcortical control regions was enhanced in response to high task demands, whereas areas of the default-mode network were deactivated more strongly. The present observations suggest that the SN represents a core node within a broader neural network that adjusts the amount of available neural and behavioral resources to changing situational opportunities and task requirements, which is often driven by extrinsic factors but can also be controlled endogenously.

Cross-modal perceptual enhancement of unisensory targets is uni-directional and does not affect temporal expectations.

Temporal structures in the environment can shape temporal expectations (TE); and previous studies demonstrated that TEs interact with multisensory interplay (MSI) when multisensory stimuli are presented synchronously. Here, we tested whether other types of MSI - evoked by asynchronous yet temporally flanking irrelevant stimuli - result in similar performance patterns. To this end, we presented sequences of 12 stimuli (10 Hz) which consisted of auditory (A), visual (V) or alternating auditory-visual stimuli (e.g. A-V-A-V-…) with either auditory or visual targets (Exp. 1). Participants discriminated target frequencies (auditory pitch or visual spatial frequency) embedded in these sequences. To test effects of TE, the proportion of early and late temporal target positions was manipulated run-wise. Performance for unisensory targets was affected by temporally flanking distractors, with auditory temporal flankers selectively improving visual target perception (Exp. 1). However, no effect of temporal expectation was observed. Control experiments (Exp. 2-3) tested whether this lack of TE effect was due to the higher presentation frequency in Exp. 1 relative to previous experiments. Importantly, even at higher stimulation frequencies redundant multisensory targets (Exp. 2-3) reliably modulated TEs. Together, our results indicate that visual target detection was enhanced by MSI. However, this cross-modal enhancement - in contrast to the redundant target effect - was still insufficient to generate TEs. We posit that unisensory target representations were either instable or insufficient for the generation of TEs while less demanding MSI still occurred; highlighting the need for robust stimulus representations when generating temporal expectations.

Sound-induced enhancement of low-intensity vision: multisensory influences on human sensory-specific cortices and thalamic bodies relate to perceptual enhancement of visual detection sensitivity.

Combining information across modalities can affect sensory performance. We studied how co-occurring sounds modulate behavioral visual detection sensitivity (d'), and neural responses, for visual stimuli of higher or lower intensity. Co-occurrence of a sound enhanced human detection sensitivity for lower- but not higher-intensity visual targets. Functional magnetic resonance imaging (fMRI) linked this to boosts in activity-levels for sensory-specific visual and auditory cortex, plus multisensory superior temporal sulcus (STS), specifically for a lower-intensity visual event when paired with a sound. Thalamic structures in visual and auditory pathways, the lateral and medial geniculate bodies, respectively (LGB, MGB), showed a similar pattern. Subject-by-subject psychophysical benefits correlated with corresponding fMRI signals in visual, auditory, and multisensory regions. We also analyzed differential "coupling" patterns of LGB and MGB with other regions in the different experimental conditions. Effective-connectivity analyses showed enhanced coupling of sensory-specific thalamic bodies with the affected cortical sites during enhanced detection of lower-intensity visual events paired with sounds. Coupling strength between visual and auditory thalamus with cortical regions, including STS, covaried parametrically with the psychophysical benefit for this specific multisensory context. Our results indicate that multisensory enhancement of detection sensitivity for low-contrast visual stimuli by co-occurring sounds reflects a brain network involving not only established multisensory STS and sensory-specific cortex but also visual and auditory thalamus.

Substantia nigra activity level predicts trial-to-trial adjustments in cognitive control.

Effective adaptation to the demands of a changing environment requires flexible cognitive control. The medial and the lateral frontal cortices are involved in such control processes, putatively in close interplay with the BG. In particular, dopaminergic projections from the midbrain (i.e., from the substantia nigra [SN] and the ventral tegmental area) have been proposed to play a pivotal role in modulating the activity in these areas for cognitive control purposes. In that dopaminergic involvement has been strongly implicated in reinforcement learning, these ideas suggest functional links between reinforcement learning, where the outcome of actions shapes behavior over time, and cognitive control in a more general context, where no direct reward is involved. Here, we provide evidence from functional MRI in humans that activity in the SN predicts systematic subsequent trial-to-trial RT prolongations that are thought to reflect cognitive control in a stop-signal paradigm. In particular, variations in the activity level of the SN in one trial predicted the degree of RT prolongation on the subsequent trial, consistent with a modulating output signal from the SN being involved in enhancing cognitive control. This link between SN activity and subsequent behavioral adjustments lends support to theoretical accounts that propose dopaminergic control signals that shape behavior both in the presence and in the absence of direct reward. This SN-based modulatory mechanism is presumably mediated via a wider network that determines response speed in this task, including frontal and parietal control regions, along with the BG and the associated subthalamic nucleus.

Feature-based attention modulates direction-selective hemodynamic activity within human MT.

Attending to the spatial location or to nonspatial features of a stimulus modulates neural activity in cortical areas that process its perceptual attributes. The feature-based attentional selection of the direction of a moving stimulus is associated with increased firing of individual neurons tuned to the direction of the movement in area V5/MT, while responses of neurons tuned to opposite directions are suppressed. However, it is not known how these multiplicatively scaled responses of individual neurons tuned to different motion-directions are integrated at the population level, in order to facilitate the processing of stimuli that match the perceptual goals. Using functional magnetic resonance imaging (fMRI) the present study revealed that attending to the movement direction of a dot field enhances the response in a number of areas including the human MT region (hMT) as a function of the coherence of the stimulus. Attending the opposite direction, however, lead to a suppressed response in hMT that was inversely correlated with stimulus-coherence. These findings demonstrate that the multiplicative scaling of single-neuron responses by feature-based attention results in an enhanced direction-selective population response within those cortical modules that processes the physical attributes of the attended stimuli. Our results provide strong support for the validity of the "feature similarity gain model" on the integrated population response as quantified by parametric fMRI in humans.

Minimal interplay between explicit knowledge, dynamics of learning and temporal expectations in different, complex uni- and multisensory contexts.

While temporal expectations (TE) generally improve reactions to temporally predictable events, it remains unknown how the learning of temporal regularities (one time point more likely than another time point) and explicit knowledge about temporal regularities contribute to performance improvements; and whether any contributions generalise across modalities. Here, participants discriminated the frequency of diverging auditory, visual or audio-visual targets embedded in auditory, visual or audio-visual distractor sequences. Temporal regularities were manipulated run-wise (early vs. late target within sequence). Behavioural performance (accuracy, RT) plus measures from a computational learning model all suggest that learning of temporal regularities occurred but did not generalise across modalities, and that dynamics of learning (size of TE effect across runs) and explicit knowledge have little to no effect on the strength of TE. Remarkably, explicit knowledge affects performance-if at all-in a context-dependent manner: Only under complex task regimes (here, unknown target modality) might it partially help to resolve response conflict while it is lowering performance in less complex environments.

Neural basis of the ventriloquist illusion.

The ventriloquist creates the illusion that his or her voice emerges from the visibly moving mouth of the puppet [1]. This well-known illusion exemplifies a basic principle of how auditory and visual information is integrated in the brain to form a unified multimodal percept. When auditory and visual stimuli occur simultaneously at different locations, the more spatially precise visual information dominates the perceived location of the multimodal event. Previous studies have examined neural interactions between spatially disparate auditory and visual stimuli [2-5], but none has found evidence for a visual influence on the auditory cortex that could be directly linked to the illusion of a shifted auditory percept. Here we utilized event-related brain potentials combined with event-related functional magnetic resonance imaging to demonstrate on a trial-by-trial basis that a precisely timed biasing of the left-right balance of auditory cortex activity by the discrepant visual input underlies the ventriloquist illusion. This cortical biasing may reflect a fundamental mechanism for integrating the auditory and visual components of environmental events, which ensures that the sounds are adaptively localized to the more reliable position provided by the visual input.

Explicitly versus implicitly driven temporal expectations: No evidence for altered perceptual processing due to top-down modulations.

Learning the statistical regularities of environmental events is a powerful tool for enhancing performance. However, it remains unclear whether this often implicit type of behavioral facilitation can be proactively modulated by explicit knowledge about temporal regularities. Only recently, Menceloglu and colleagues (Attention, Perception & Psychophysics, 79(1), 169-179, 2017) tested for differences between implicit versus explicit statistical learning of temporal regularities by using a within-paradigm manipulation of metacognitive temporal knowledge. The authors reported that temporal expectations were enhanced if participants had explicit knowledge about temporal regularities. Here, we attempted to replicate and extend their results, and to provide a mechanistic framework for any effects by means of computational modelling. Participants performed a letter-discrimination task, with target letters embedded in congruent or incongruent flankers. Temporal predictability was manipulated block-wise, with targets occurring more often after either a short or a long delay period. During the delay a sound was presented in half of the trials. Explicit knowledge about temporal regularities was manipulated by changing instructions: Participants received no information (implicit), information about the most likely cue-target delay (explicit), or received 100% valid cues on each trial (highly explicit). We replicated previous effects of target-flanker congruence and sound presence. However, no evidence was found for an effect of explicit knowledge on temporal expectations using Bayesian statistics. Concordantly, computational modelling suggested that explicit knowledge may only influence non-perceptual processing such as response criteria. Together, our results indicate that explicit metacognitive knowledge does not necessarily alter sensory representations or temporal expectations but rather affects response strategies.

Delayed striate cortical activation during spatial attention.

Recordings of event-related potentials (ERPs) and event-related magnetic fields (ERMFs) were combined with functional magnetic resonance imaging (fMRI) to study visual cortical activity in humans during spatial attention. While subjects attended selectively to stimulus arrays in one visual field, fMRI revealed stimulus-related activations in the contralateral primary visual cortex and in multiple extrastriate areas. ERP and ERMF recordings showed that attention did not affect the initial evoked response at 60-90 ms poststimulus that was localized to primary cortex, but a similarly localized late response at 140-250 ms was enhanced to attended stimuli. These findings provide evidence that the primary visual cortex participates in the selective processing of attended stimuli by means of delayed feedback from higher visual-cortical areas.