Dissonant music engages early visual processing.

The neuroscientific examination of music processing in audio-visual contexts offers a valuable framework to assess how auditory information influences the emotional encoding of visual information. Using fMRI during naturalistic film viewing, we investigated the neural mechanisms underlying the effect of music on valence inferences during mental state attribution. Thirty-eight participants watched the same short-film accompanied by systematically controlled consonant or dissonant music. Subjects were instructed to think about the main character's intentions. The results revealed that increasing levels of dissonance led to more negatively valenced inferences, displaying the profound emotional impact of musical dissonance. Crucially, at the neuroscientific level and despite music being the sole manipulation, dissonance evoked the response of the primary visual cortex (V1). Functional/effective connectivity analysis showed a stronger coupling between the auditory ventral stream (AVS) and V1 in response to tonal dissonance and demonstrated the modulation of early visual processing via top-down feedback inputs from the AVS to V1. These V1 signal changes indicate the influence of high-level contextual representations associated with tonal dissonance on early visual cortices, serving to facilitate the emotional interpretation of visual information. Our results highlight the significance of employing systematically controlled music, which can isolate emotional valence from the arousal dimension, to elucidate the brain's sound-to-meaning interface and its distributive crossmodal effects on early visual encoding during naturalistic film viewing.

Speaking in gestures: Left dorsal and ventral frontotemporal brain systems underlie communication in conducting.

Conducting constitutes a well-structured system of signs anticipating information concerning the rhythm and dynamic of a musical piece. Conductors communicate the musical tempo to the orchestra, unifying the individual instrumental voices to form an expressive musical Gestalt. In a functional magnetic resonance imaging (fMRI) experiment, 12 professional conductors and 16 instrumentalists conducted real-time novel pieces with diverse complexity in orchestration and rhythm. For control, participants either listened to the stimuli or performed beat patterns, setting the time of a metronome or complex rhythms played by a drum. Activation of the left superior temporal gyrus (STG), supplementary and premotor cortex and Broca's pars opercularis (F3op) was shared in both musician groups and separated conducting from the other conditions. Compared to instrumentalists, conductors activated Broca's pars triangularis (F3tri) and the STG, which differentiated conducting from time beating and reflected the increase in complexity during conducting. In comparison to conductors, instrumentalists activated F3op and F3tri when distinguishing complex rhythm processing from simple rhythm processing. Fibre selection from a normative human connectome database, constructed using a global tractography approach, showed that the F3op and STG are connected via the arcuate fasciculus, whereas the F3tri and STG are connected via the extreme capsule. Like language, the anatomical framework characterising conducting gestures is located in the left dorsal system centred on F3op. This system reflected the sensorimotor mapping for structuring gestures to musical tempo. The ventral system centred on F3Tri may reflect the art of conductors to set this musical tempo to the individual orchestra's voices in a global, holistic way.

Using HVS Dual-Pathway and Contrast Sensitivity to Blindly Assess Image Quality.

Blind image quality assessment (BIQA) aims to evaluate image quality in a way that closely matches human perception. To achieve this goal, the strengths of deep learning and the characteristics of the human visual system (HVS) can be combined. In this paper, inspired by the ventral pathway and the dorsal pathway of the HVS, a dual-pathway convolutional neural network is proposed for BIQA tasks. The proposed method consists of two pathways: the "what" pathway, which mimics the ventral pathway of the HVS to extract the content features of distorted images, and the "where" pathway, which mimics the dorsal pathway of the HVS to extract the global shape features of distorted images. Then, the features from the two pathways are fused and mapped to an image quality score. Additionally, gradient images weighted by contrast sensitivity are used as the input to the "where" pathway, allowing it to extract global shape features that are more sensitive to human perception. Moreover, a dual-pathway multi-scale feature fusion module is designed to fuse the multi-scale features of the two pathways, enabling the model to capture both global features and local details, thus improving the overall performance of the model. Experiments conducted on six databases show that the proposed method achieves state-of-the-art performance.

The ventral pathway of the human brain: A continuous association tract system.

The brain hemispheres can be divided into an upper dorsal and a lower ventral system. Each system consists of distinct cortical regions connected via long association tracts. The tracts cross the central sulcus or the limen insulae to connect the frontal lobe with the posterior brain. The dorsal stream is associated with sensorimotor mapping. The ventral stream serves structural analysis and semantics in different domains, as visual, acoustic or space processing. How does the prefrontal cortex, regarded as the platform for the highest level of integration, incorporate information from these different domains? In the current view, the ventral pathway consists of several separate tracts, related to different modalities. Originally the assumption was that the ventral path is a continuum, covering all modalities. The latter would imply a very different anatomical basis for cognitive and clinical models of processing. To further define the ventral connections, we used cutting-edge in vivo global tractography on high-resolution diffusion tensor imaging (DTI) data from 100 normal subjects from the human connectome project and ex vivo preparation of fiber bundles in the extreme capsule of 8 humans using the Klingler technique. Our data showed that ventral stream tracts, traversing through the extreme capsule, form a continuous band of fibers that fan out anteriorly to the prefrontal cortex, and posteriorly to temporal, occipital and parietal cortical regions. Introduction of additional volumes of interest in temporal and occipital lobes differentiated between the inferior fronto-occipital fascicle (IFOF) and uncinate fascicle (UF). Unequivocally, in both experiments, in all subjects a connection between the inferior frontal and middle-to-posterior temporal cortical region, otherwise known as the temporo-frontal extreme capsule fascicle (ECF) from nonhuman primate brain-tracing experiments was identified. In the human brain, this tract connects the language domains of "Broca's area" and "Wernicke's area". The differentiation in the three tracts, IFOF, UF and ECF seems arbitrary, all three pass through the extreme capsule. Our data show that the ventral pathway represents a continuum. The three tracts merge seamlessly and streamlines showed considerable overlap in their anterior and posterior course. Terminal maps identified prefrontal cortex in the frontal lobe and association cortex in temporal, occipital and parietal lobes as streamline endings. This anatomical substrate potentially facilitates the prefrontal cortex to integrate information across different domains and modalities.

Self-limited focal epilepsy decreased regional brain activity in sensorimotor areas.

OBJECTIVE: The fractional amplitude of low-frequency fluctuation (fALFF) method was used to identify the regional brain activity deficits of self-limited focal epilepsy with centrotemporal spikes (SLFECS) relative to normal controls (NCs). METHODS: A total of 21 SLFECS (10 females, 11 males; mean age, 8.57 ± 1.5 years) and 21 status-matched (age, sex, and education) NCs (10 females, 11 males; mean age, 8.76 ± 2.19 years) were recruited. The fALFF method was applied to identify SLFECS-related regional brain alterations. Receiver operating characteristic (ROC) curve was applied to identify the ability of these regional brain areas in distinguishing the SLFECS group from the NCs group. The relationships between the regional brain activity deficits and clinical features were evaluated by Pearson's correlation analysis. RESULTS: Self-limited focal epilepsy with centrotemporal spikes was associated with widespread regional brain activity alterations, including left cuneus with higher fALFF values, and bilateral striatum, bilateral precentral gyrus, ventral and dorsal pathway of sensory area, left dorsolateral prefrontal cortex, and left Rolandic area with lower fALFF values. ROC curve revealed excellent AUC value (0.964) of these areas in distinguishing the SLFECS group from the NCs group with high degree of sensitivity (90.5%) and specificity (95.2%). Intelligence quotient score positively correlated with the fALFF value in the left striatum (r = 0.453, p = 0.039). CONCLUSIONS: The fALFF parameter could be served as a potential biomarker to identify the SLFECS-related regional brain deficits in the sensorimotor cortex and their pathways, which may be the etiology of paresthesia in SLFECS.

Timing of response onset and offset in macaque V4: stimulus and task dependence.

In the primate visual cortex, both the magnitude of the neuronal response and its timing can carry important information about the visual world, but studies typically focus only on response magnitude. Here, we examine the onset and offset latency of the responses of neurons in area V4 of awake, behaving macaques across several experiments in the context of a variety of stimuli and task paradigms. Our results highlight distinct contributions of stimuli and tasks to V4 response latency. We found that response onset latencies are shorter than typically cited (median = 75.5 ms), supporting a role for V4 neurons in rapid object and scene recognition functions. Moreover, onset latencies are longer for smaller stimuli and stimulus outlines, consistent with the hypothesis that longer latencies are associated with higher spatial frequency content. Strikingly, we found that onset latencies showed no significant dependence on stimulus occlusion, unlike in inferotemporal cortex, nor on task demands. Across the V4 population, onset latencies had a broad distribution, reflecting the diversity of feedforward, recurrent, and feedback connections that inform the responses of individual neurons. Response offset latencies, on the other hand, displayed the opposite tendency in their relationship to stimulus and task attributes: they are less influenced by stimulus appearance but are shorter in guided saccade tasks compared with fixation tasks. The observation that response latency is influenced by stimulus- and task-associated factors emphasizes a need to examine response timing alongside firing rate in determining the functional role of area V4.NEW & NOTEWORTHY Onset and offset timing of neuronal responses can provide information about visual environment and neuron's role in visual processing and its anatomical connectivity. In the first comprehensive examination of onset and offset latencies in the intermediate visual cortical area V4, we find neurons respond faster than previously reported, making them ideally suited to contribute to rapid object and scene recognition. While response onset reflects stimulus characteristics, timing of response offset is influenced more by behavioral task.

Attentional modulation differentially affects ventral and dorsal face areas in both normal participants and developmental prosopagnosics.

Face-selective cortical areas that can be divided into a ventral stream and a dorsal stream. Previous findings indicate selective attention to particular aspects of faces have different effects on the two streams. To better understand the organization of the face network and whether deficits in attentional modulation contribute to developmental prosopagnosia (DP), we assessed the effect of selective attention to different face aspects across eight face-selective areas. Our results from normal participants found that ROIs in the ventral pathway (OFA, FFA) responded strongly when attention was directed to identity and expression, and ROIs in the dorsal pathway (pSTS-FA, IFG-FA) responded the most when attention was directed to facial expression. Response profiles generated by attention to different face aspects were comparable in DPs and normals. Our results demonstrate attentional modulation affects the ventral and dorsal steam face areas differently and indicate deficits in attentional modulation do not contribute to DP.

Modeling diverse responses to filled and outline shapes in macaque V4.

Visual area V4 is an important midlevel cortical processing stage that subserves object recognition in primates. Studies investigating shape coding in V4 have largely probed neuronal responses with filled shapes, i.e., shapes defined by both a boundary and an interior fill. As a result, we do not know whether form-selective V4 responses are dictated by boundary features alone or if interior fill is also important. We studied 43 V4 neurons in two male macaque monkeys ( Macaca mulatta) with a set of 362 filled shapes and their corresponding outlines to determine how interior fill modulates neuronal responses in shape-selective neurons. Only a minority of neurons exhibited similar response strength and shape preferences for filled and outline stimuli. A majority responded preferentially to one stimulus category (either filled or outline shapes) and poorly to the other. Our findings are inconsistent with predictions of the hierarchical-max (HMax) V4 model that builds form selectivity from oriented boundary features and takes little account of attributes related to object surface, such as the phase of the boundary edge. We modified the V4 HMax model to include sensitivity to interior fill by either removing phase-pooling or introducing unoriented units at the V1 level; both modifications better explained our data without increasing the number of free parameters. Overall, our results suggest that boundary orientation and interior surface information are both maintained until at least the midlevel visual representation, consistent with the idea that object fill is important for recognition and perception in natural vision. NEW & NOTEWORTHY The shape of an object's boundary is critical for identification; consistent with this idea, models of object recognition predict that filled and outline versions of a shape are encoded similarly. We report that many neurons in a midlevel visual cortical area respond differently to filled and outline shapes and modify a biologically plausible model to account for our data. Our results suggest that representations of boundary shape and surface fill are interrelated in visual cortex.

What is changing when: Decoding visual information in movies from human intracranial recordings.

The majority of visual recognition studies have focused on the neural responses to repeated presentations of static stimuli with abrupt and well-defined onset and offset times. In contrast, natural vision involves unique renderings of visual inputs that are continuously changing without explicitly defined temporal transitions. Here we considered commercial movies as a coarse proxy to natural vision. We recorded intracranial field potential signals from 1,284 electrodes implanted in 15 patients with epilepsy while the subjects passively viewed commercial movies. We could rapidly detect large changes in the visual inputs within approximately 100 ms of their occurrence, using exclusively field potential signals from ventral visual cortical areas including the inferior temporal gyrus and inferior occipital gyrus. Furthermore, we could decode the content of those visual changes even in a single movie presentation, generalizing across the wide range of transformations present in a movie. These results present a methodological framework for studying cognition during dynamic and natural vision.

Functional Dynamics of Dorsal and Ventral Reading Networks in Bilinguals.

In today's world, bilingualism is increasingly common. However, it is still unclear how left-lateralized dorsal and ventral reading networks are tuned to reading in proficient second-language learners. Here, we investigated differences in functional regional activation and connectivity as a function of L1 and L2 reading, L2 orthographic depth, and task demands. Thirty-seven late bilinguals with the same L1 and either an opaque or transparent L2 performed perceptual and semantic reading tasks in L1 and L2 during functional magnetic resonance imaging (fMRI) scanning. Results revealed stronger regional recruitment for L2 versus L1 reading and stronger connectivity within the dorsal stream during L1 versus L2 reading. Differences in orthographic depth were associated with a segregated profile of left ventral occipitotemporal (vOT) coactivation with dorsal regions for the transparent L2 group and with ventral regions for the opaque L2 group. Finally, semantic versus perceptual demands modulated left vOT engagement, supporting the interactive account of the contribution of vOT to reading, and were associated with stronger coactivation within the ventral network. Our findings support a division of labor between ventral and dorsal reading networks, elucidating the critical role of the language used to read, L2 orthographic depth, and task demands on the functional dynamics of bilingual reading.

Planning Ahead: Object-Directed Sequential Actions Decoded from Human Frontoparietal and Occipitotemporal Networks.

Object-manipulation tasks (e.g., drinking from a cup) typically involve sequencing together a series of distinct motor acts (e.g., reaching toward, grasping, lifting, and transporting the cup) in order to accomplish some overarching goal (e.g., quenching thirst). Although several studies in humans have investigated the neural mechanisms supporting the planning of visually guided movements directed toward objects (such as reaching or pointing), only a handful have examined how manipulatory sequences of actions-those that occur after an object has been grasped-are planned and represented in the brain. Here, using event-related functional MRI and pattern decoding methods, we investigated the neural basis of real-object manipulation using a delayed-movement task in which participants first prepared and then executed different object-directed action sequences that varied either in their complexity or final spatial goals. Consistent with previous reports of preparatory brain activity in non-human primates, we found that activity patterns in several frontoparietal areas reliably predicted entire action sequences in advance of movement. Notably, we found that similar sequence-related information could also be decoded from pre-movement signals in object- and body-selective occipitotemporal cortex (OTC). These findings suggest that both frontoparietal and occipitotemporal circuits are engaged in transforming object-related information into complex, goal-directed movements.

Signal Propagation in the Human Visual Pathways: An Effective Connectivity Analysis.

Although the visual system has been extensively investigated, an integrated account of the spatiotemporal dynamics of long-range signal propagation along the human visual pathways is not completely known or validated. In this work, we used dynamic causal modeling approach to provide insights into the underlying neural circuit dynamics of pattern reversal visual-evoked potentials extracted from concurrent EEG-fMRI data. A recurrent forward-backward connectivity model, consisting of multiple interacting brain regions identified by EEG source localization aided by fMRI spatial priors, best accounted for the data dynamics. Sources were first identified in the thalamic area, primary visual cortex, as well as higher cortical areas along the ventral and dorsal visual processing streams. Consistent with hierarchical early visual processing, the model disclosed and quantified the neural temporal dynamics across the identified activity sources. This signal propagation is dominated by a feedforward process, but we also found weaker effective feedback connectivity. Using effective connectivity analysis, the optimal dynamic causal modeling revealed enhanced connectivity along the dorsal pathway but slightly suppressed connectivity along the ventral pathway. A bias was also found in favor of the right hemisphere consistent with functional attentional asymmetry. This study validates, for the first time, the long-range signal propagation timing in the human visual pathways. A similar modeling approach can potentially be used to understand other cognitive processes and dysfunctions in signal propagation in neurological and neuropsychiatric disorders. Significance statement: An integrated account of long-range visual signal propagation in the human brain is currently incomplete. Using computational neural modeling on our acquired concurrent EEG-fMRI data under a visual evoked task, we found not only a substantial forward propagation toward "higher-order" brain regions but also a weaker backward propagation. Asymmetry in our model's long-range connectivity accounted for the various observed activity biases. Importantly, the model disclosed the timing of signal propagation across these connectivity pathways and validates, for the first time, long-range signal propagation in the human visual system. A similar modeling approach could be used to identify neural pathways for other cognitive processes and their dysfunctions in brain disorders.

Three-Dimensional Face Recognition in Mild Cognitive Impairment: A Psychophysical and Structural MR Study.

OBJECTIVES: Mild cognitive impairment (MCI) has been associated with a high risk of conversion to Alzheimer's dementia. In addition to memory complaints, impairments in the visuospatial domain have been reported in this condition. We have previously shown that deficits in perceiving structure-from-motion (SFM) objects are reflected in functional reorganization of brain activity within the visual ventral stream. Here we aimed to identify structural correlates of psychophysical complex face and object recognition performance in amnestic MCI patients (n=30 vs. n=25 controls). This study was, therefore, motivated by evidence from recent studies showing that a combination of visual information across dorsal and ventral visual streams may be needed for the perception of three-dimensional (3D) SFM objects. METHODS: In our experimental paradigm, participants had to discriminate 3D SFM shapes (faces and objects) from 3D SFM meaningless (scrambled) shapes. RESULTS: Morphometric analysis established neuroanatomical evidence for impairment in MCI as demonstrated by smaller hippocampal volumes. We found association between cortical thickness and face recognition performance, comprising the occipital lobe and visual ventral stream fusiform regions (overlapping the known location of face fusiform area) in the right hemisphere, in MCI. CONCLUSIONS: We conclude that impairment of 3D visual integration exists at the MCI stage involving also the visual ventral stream and contributing to face recognition deficits. The specificity of such observed structure-function correlation for faces suggests a special role of this processing pathway in health and disease. (JINS, 2016, 22, 744-754).

Effective connectivity of brain regions related to visual word recognition: An fMRI study of Chinese reading.

Past neuroimaging studies have focused on identifying specialized functional brain systems for processing different components of reading, such as orthography, phonology, and semantics. More recently, a few experiments have been performed to look into the integration and interaction of distributed neural systems for visual word recognition, suggesting that lexical processing in alphabetic languages involves both ventral and dorsal neural pathways originating from the visual cortex. In the present functional magnetic resonance imaging study, we tested the multiple pathways model with Chinese character stimuli and examined how the neural systems interacted in reading Chinese. Using dynamic causal modeling, we demonstrated that visual word recognition in Chinese engages the ventral pathway from the visual cortex to the left ventral occipitotemporal cortex, but not the dorsal pathway from the visual cortex to the left parietal region. The ventral pathway, however, is linked to the superior parietal lobule and the left middle frontal gyrus (MFG) so that a dynamic neural network is formed, with information flowing from the visual cortex to the left ventral occipitotemporal cortex to the parietal lobule and then to the left MFG. The findings suggest that cortical dynamics is constrained by the differences in how visual orthographic symbols in writing systems are linked to spoken language.

Two 'what' pathways for action and object recognition.

The ventral visual stream is conceived as a pathway for object recognition. However, we also recognize the actions an object can be involved in. Here, we show that action recognition critically depends on a pathway in lateral occipitotemporal cortex, partially overlapping and topographically aligned with object representations that are precursors for action recognition. By contrast, object features that are more relevant for object recognition, such as color and texture, are typically found in ventral occipitotemporal cortex. We argue that occipitotemporal cortex contains similarly organized lateral and ventral 'what' pathways for action and object recognition, respectively. This account explains a number of observed phenomena, such as the duplication of object domains and the specific representational profiles in lateral and ventral cortex.

Early Emergence of Solid Shape Coding in Natural and Deep Network Vision.

Area V4 is the first object-specific processing stage in the ventral visual pathway, just as area MT is the first motion-specific processing stage in the dorsal pathway. For almost 50 years, coding of object shape in V4 has been studied and conceived in terms of flat pattern processing, given its early position in the transformation of 2D visual images. Here, however, in awake monkey recording experiments, we found that roughly half of V4 neurons are more tuned and responsive to solid, 3D shape-in-depth, as conveyed by shading, specularity, reflection, refraction, or disparity cues in images. Using 2-photon functional microscopy, we found that flat- and solid-preferring neurons were segregated into separate modules across the surface of area V4. These findings should impact early shape-processing theories and models, which have focused on 2D pattern processing. In fact, our analyses of early object processing in AlexNet, a standard visual deep network, revealed a similar distribution of sensitivities to flat and solid shape in layer 3. Early processing of solid shape, in parallel with flat shape, could represent a computational advantage discovered by both primate brain evolution and deep-network training.

Reunification of Object and View-Center Background Information in the Primate Medial Temporal Lobe.

Recent work has shown that the medial temporal lobe (MTL), including the hippocampus (HPC) and its surrounding limbic cortices, plays a role in scene perception in addition to episodic memory. The two basic factors of scene perception are the object ("what") and location ("where"). In this review, we first summarize the anatomical knowledge related to visual inputs to the MTL and physiological studies examining object-related information processed along the ventral pathway briefly. Thereafter, we discuss the space-related information, the processing of which was unclear, presumably because of its multiple aspects and a lack of appropriate task paradigm in contrast to object-related information. Based on recent electrophysiological studies using non-human primates and the existing literature, we proposed the "reunification theory," which explains brain mechanisms which construct object-location signals at each gaze. In this reunification theory, the ventral pathway signals a large-scale background image of the retina at each gaze position. This view-center background signal reflects the first person's perspective and specifies the allocentric location in the environment by similarity matching between images. The spatially invariant object signal and view-center background signal, both of which are derived from the same retinal image, are integrated again (i.e., reunification) along the ventral pathway-MTL stream, particularly in the perirhinal cortex. The conjunctive signal, which represents a particular object at a particular location, may play a role in scene perception in the HPC as a key constituent element of an entire scene.

Extreme capsule is a bottleneck for ventral pathway.

As neuroscience literature suggests, extreme capsule is considered a whiter matter tract. Nevertheless, it is not clear whether extreme capsule itself is an association fiber pathway or only a bottleneck for other association fibers to pass. Via our review, investigating anatomical position, connectivity and cognitive role of the bundles in extreme capsule, and by analyzing data from the dissection, it can be argued that extreme capsule is probably a bottleneck for the passage of uncinated fasciculus (UF) and inferior fronto-occipital fasciculus (IFOF), and these fasciculi are responsible for the respective roles in language processing.

Altered brain structures in the dorsal and ventral language pathways in individuals with and without developmental language disorder (DLD).

Developmental Language Disorder (DLD) is a neurodevelopmental disorder characterized by difficulty learning and using language, and this difficulty cannot be attributed to other developmental conditions. The aim of the current study was to examine structural differences in dorsal and ventral language pathways between adolescents and young adults with and without DLD (age range: 14-27 years) using anatomical magnetic resonance imaging (MRI) and diffusion tensor imaging (DTI). Results showed age-related structural brain differences in both dorsal and ventral pathways in individuals with DLD. These findings provide evidence for neuroanatomical correlates of persistent language deficits in adolescents/young adults with DLD, and further suggest that this brain-language relationship in DLD is better characterized by taking account the dynamic course of the disorder along development.

Age effects on the neural processing of object-context associations in briefly flashed natural scenes.

In daily life, fast visual recognition of surrounding objects is facilitated through context-based expectations. However the ability to rapidly and accurately recognize unexpected stimuli in a given environment is also crucial and this ability is impaired with age. The present fMRI study aimed at comparing in young and older adults the neural correlates of fast object processing. Patterns of cerebral activity were investigated in response to briefly-presented (100 ms) congruent and incongruent natural scenes. Participants were slower and less accurate when categorizing objects in incongruent relative to congruent contexts. This behavioral cost was notably more pronounced in the older group. Height and multivariate patterns of fMRI activity in context-selective regions were equivalent in both age groups, suggesting preserved processing of coarse scene features in older participants. Incongruent scenes elicited additional activity in the parahippocampal gyrus that possibly reflected simultaneous activation of rarely co-occurring neural representations. Contextual effects were observed in object-selective cortex for the young group only, and may be driven by detection of mismatch between actually perceived and previously-experienced associations. In the older group exclusively, increased bilateral prefrontal and left fusiform activity in response to incongruent scenes was observed. However this supplemental activity was not found to efficiently contribute to improve task performance in difficult visual conditions. Altogether these results suggest age-related changes in the interaction between object- and context-processing pathways, that may subserve impairment in fast identification of unexpected objects in natural scenes.