Metamodal Coupling of Vibrotactile and Auditory Speech Processing Systems through Matched Stimulus Representations.

It has been postulated that the brain is organized by "metamodal," sensory-independent cortical modules capable of performing tasks (e.g., word recognition) in both "standard" and novel sensory modalities. Still, this theory has primarily been tested in sensory-deprived individuals, with mixed evidence in neurotypical subjects, thereby limiting its support as a general principle of brain organization. Critically, current theories of metamodal processing do not specify requirements for successful metamodal processing at the level of neural representations. Specification at this level may be particularly important in neurotypical individuals, where novel sensory modalities must interface with existing representations for the standard sense. Here we hypothesized that effective metamodal engagement of a cortical area requires congruence between stimulus representations in the standard and novel sensory modalities in that region. To test this, we first used fMRI to identify bilateral auditory speech representations. We then trained 20 human participants (12 female) to recognize vibrotactile versions of auditory words using one of two auditory-to-vibrotactile algorithms. The vocoded algorithm attempted to match the encoding scheme of auditory speech while the token-based algorithm did not. Crucially, using fMRI, we found that only in the vocoded group did trained-vibrotactile stimuli recruit speech representations in the superior temporal gyrus and lead to increased coupling between them and somatosensory areas. Our results advance our understanding of brain organization by providing new insight into unlocking the metamodal potential of the brain, thereby benefitting the design of novel sensory substitution devices that aim to tap into existing processing streams in the brain.SIGNIFICANCE STATEMENT It has been proposed that the brain is organized by "metamodal," sensory-independent modules specialized for performing certain tasks. This idea has inspired therapeutic applications, such as sensory substitution devices, for example, enabling blind individuals "to see" by transforming visual input into soundscapes. Yet, other studies have failed to demonstrate metamodal engagement. Here, we tested the hypothesis that metamodal engagement in neurotypical individuals requires matching the encoding schemes between stimuli from the novel and standard sensory modalities. We trained two groups of subjects to recognize words generated by one of two auditory-to-vibrotactile transformations. Critically, only vibrotactile stimuli that were matched to the neural encoding of auditory speech engaged auditory speech areas after training. This suggests that matching encoding schemes is critical to unlocking the brain's metamodal potential.

From shape to meaning: Evidence for multiple fast feedforward hierarchies of concept processing in the human brain.

A number of fMRI studies have provided support for the existence of multiple concept representations in areas of the brain such as the anterior temporal lobe (ATL) and inferior parietal lobule (IPL). However, the interaction among different conceptual representations remains unclear. To better understand the dynamics of how the brain extracts meaning from sensory stimuli, we conducted a human high-density electroencephalography (EEG) study in which we first trained participants to associate pseudowords with various animal and tool concepts. After training, multivariate pattern classification of EEG signals in sensor and source space revealed the representation of both animal and tool concepts in the left ATL and tool concepts within the left IPL within 250 â€‹ms. Finally, we used Granger Causality analyses to show that orthography-selective sensors directly modulated activity in the parietal-tool selective cluster. Together, our results provide evidence for distinct but parallel "perceptual-to-conceptual" feedforward hierarchies in the brain.

Neural mechanisms of vibrotactile categorization.

The grouping of sensory stimuli into categories is fundamental to cognition. Previous research in the visual and auditory systems supports a two-stage processing hierarchy that underlies perceptual categorization: (a) a "bottom-up" perceptual stage in sensory cortices where neurons show selectivity for stimulus features and (b) a "top-down" second stage in higher level cortical areas that categorizes the stimulus-selective input from the first stage. In order to test the hypothesis that the two-stage model applies to the somatosensory system, 14 human participants were trained to categorize vibrotactile stimuli presented to their right forearm. Then, during an fMRI scan, participants actively categorized the stimuli. Representational similarity analysis revealed stimulus selectivity in areas including the left precentral and postcentral gyri, the supramarginal gyrus, and the posterior middle temporal gyrus. Crucially, we identified a single category-selective region in the left ventral precentral gyrus. Furthermore, an estimation of directed functional connectivity delivered evidence for robust top-down connectivity from the second to first stage. These results support the validity of the two-stage model of perceptual categorization for the somatosensory system, suggesting common computational principles and a unified theory of perceptual categorization across the visual, auditory, and somatosensory systems.

A crash in visual processing: Interference between feedforward and feedback of successive targets limits detection and categorization.

The human visual system can detect objects in streams of rapidly presented images at presentation rates of 70 Hz and beyond. Yet, target detection is often impaired when multiple targets are presented in quick temporal succession. Here, we provide evidence for the hypothesis that such impairments can arise from interference between "top-down" feedback signals and the initial "bottom-up" feedforward processing of the second target. Although it is has been recently shown that feedback signals are important for visual detection, this "crash" in neural processing affected both the detection and categorization of both targets. Moreover, experimentally reducing such interference between the feedforward and feedback portions of the two targets substantially improved participants' performance. The results indicate a key role of top-down re-entrant feedback signals and show how their interference with a successive target's feedforward process determine human behavior. These results are not just relevant for our understanding of how, when, and where capacity limits in the brain's processing abilities can arise, but also have ramifications spanning topics from consciousness to learning and attention.

Multivariate Pattern Analysis Reveals Category-Related Organization of Semantic Representations in Anterior Temporal Cortex.

UNLABELLED: The neural substrates of semantic representation have been the subject of much controversy. The study of semantic representations is complicated by difficulty in disentangling perceptual and semantic influences on neural activity, as well as in identifying stimulus-driven, "bottom-up" semantic selectivity unconfounded by top-down task-related modulations. To address these challenges, we trained human subjects to associate pseudowords (TPWs) with various animal and tool categories. To decode semantic representations of these TPWs, we used multivariate pattern classification of fMRI data acquired while subjects performed a semantic oddball detection task. Crucially, the classifier was trained and tested on disjoint sets of TPWs, so that the classifier had to use the semantic information from the training set to correctly classify the test set. Animal and tool TPWs were successfully decoded based on fMRI activity in spatially distinct subregions of the left medial anterior temporal lobe (LATL). In addition, tools (but not animals) were successfully decoded from activity in the left inferior parietal lobule. The tool-selective LATL subregion showed greater functional connectivity with left inferior parietal lobule and ventral premotor cortex, indicating that each LATL subregion exhibits distinct patterns of connectivity. Our findings demonstrate category-selective organization of semantic representations in LATL into spatially distinct subregions, continuing the lateral-medial segregation of activation in posterior temporal cortex previously observed in response to images of animals and tools, respectively. Together, our results provide evidence for segregation of processing hierarchies for different classes of objects and the existence of multiple, category-specific semantic networks in the brain. SIGNIFICANCE STATEMENT: The location and specificity of semantic representations in the brain are still widely debated. We trained human participants to associate specific pseudowords with various animal and tool categories, and used multivariate pattern classification of fMRI data to decode the semantic representations of the trained pseudowords. We found that: (1) animal and tool information was organized in category-selective subregions of medial left anterior temporal lobe (LATL); (2) tools, but not animals, were encoded in left inferior parietal lobe; and (3) LATL subregions exhibited distinct patterns of functional connectivity with category-related regions across cortex. Our findings suggest that semantic knowledge in LATL is organized in category-related subregions, providing evidence for the existence of multiple, category-specific semantic representations in the brain.

There Is a "U" in Clutter: Evidence for Robust Sparse Codes Underlying Clutter Tolerance in Human Vision.

The ability to recognize objects in clutter is crucial for human vision, yet the underlying neural computations remain poorly understood. Previous single-unit electrophysiology recordings in inferotemporal cortex in monkeys and fMRI studies of object-selective cortex in humans have shown that the responses to pairs of objects can sometimes be well described as a weighted average of the responses to the constituent objects. Yet, from a computational standpoint, it is not clear how the challenge of object recognition in clutter can be solved if downstream areas must disentangle the identity of an unknown number of individual objects from the confounded average neuronal responses. An alternative idea is that recognition is based on a subpopulation of neurons that are robust to clutter, i.e., that do not show response averaging, but rather robust object-selective responses in the presence of clutter. Here we show that simulations using the HMAX model of object recognition in cortex can fit the aforementioned single-unit and fMRI data, showing that the averaging-like responses can be understood as the result of responses of object-selective neurons to suboptimal stimuli. Moreover, the model shows how object recognition can be achieved by a sparse readout of neurons whose selectivity is robust to clutter. Finally, the model provides a novel prediction about human object recognition performance, namely, that target recognition ability should show a U-shaped dependency on the similarity of simultaneously presented clutter objects. This prediction is confirmed experimentally, supporting a simple, unifying model of how the brain performs object recognition in clutter. SIGNIFICANCE STATEMENT: The neural mechanisms underlying object recognition in cluttered scenes (i.e., containing more than one object) remain poorly understood. Studies have suggested that neural responses to multiple objects correspond to an average of the responses to the constituent objects. Yet, it is unclear how the identities of an unknown number of objects could be disentangled from a confounded average response. Here, we use a popular computational biological vision model to show that averaging-like responses can result from responses of clutter-tolerant neurons to suboptimal stimuli. The model also provides a novel prediction, that human detection ability should show a U-shaped dependency on target-clutter similarity, which is confirmed experimentally, supporting a simple, unifying account of how the brain performs object recognition in clutter.

Adding words to the brain's visual dictionary: novel word learning selectively sharpens orthographic representations in the VWFA.

The nature of orthographic representations in the human brain is still subject of much debate. Recent reports have claimed that the visual word form area (VWFA) in left occipitotemporal cortex contains an orthographic lexicon based on neuronal representations highly selective for individual written real words (RWs). This theory predicts that learning novel words should selectively increase neural specificity for these words in the VWFA. We trained subjects to recognize novel pseudowords (PWs) and used fMRI rapid adaptation to compare neural selectivity with RWs, untrained PWs (UTPWs), and trained PWs (TPWs). Before training, PWs elicited broadly tuned responses, whereas responses to RWs indicated tight tuning. After training, TPW responses resembled those of RWs, whereas UTPWs continued to show broad tuning. This change in selectivity was specific to the VWFA. Therefore, word learning appears to selectively increase neuronal specificity for the new words in the VWFA, thereby adding these words to the brain's visual dictionary.

Uncovering phonological and orthographic selectivity across the reading network using fMRI-RA.

Reading has been shown to rely on a dorsal brain circuit involving the temporoparietal cortex (TPC) for grapheme-to-phoneme conversion of novel words (Pugh et al., 2001), and a ventral stream involving left occipitotemporal cortex (OTC) (in particular in the so-called "visual word form area", VWFA) for visual identification of familiar words. In addition, portions of the inferior frontal cortex (IFC) have been posited to be an output of the dorsal reading pathway involved in phonology. While this dorsal versus ventral dichotomy for phonological and orthographic processing of words is widely accepted, it is not known if these brain areas are actually strictly sensitive to orthographic or phonological information. Using an fMRI rapid adaptation technique we probed the selectivity of the TPC, OTC, and IFC to orthographic and phonological features during single word reading. We found in two independent experiments using different task conditions in adult normal readers, that the TPC is exclusively sensitive to phonology and the VWFA in the OTC is exclusively sensitive to orthography. The dorsal IFC (BA 44), however, showed orthographic but not phonological selectivity. These results support the theory that reading involves a specific phonological-based temporoparietal region and a specific orthographic-based ventral occipitotemporal region. The dorsal IFC, however, was not sensitive to phonological processing, suggesting a more complex role for this region.

Behavioral and neuroimaging evidence for impaired executive function in "cognitively normal" older HIV-infected adults.

The increased prevalence of HIV among adults >50 years underscores the importance of improving our understanding of mechanisms causing HIV-associated neurocognitive disorders (HAND). Identifying novel and noninvasive diagnostic predictors of HAND prior to clinical manifestation is critical to ultimately identifying means of preventing progression to symptomatic HAND. Here, using a task-switching paradigm, in which subjects were cued (unpredictably) to perform a face-gender or a word-semantic task on superimposed face and word images, we examined the behavioral and neural profile of impaired cognitive control in older HIV + adults (N = 14, 9 HIV+). Functional magnetic resonance imaging (fMRI) and behavioral data were acquired while subjects were performing the face-gender or word-semantic task. We found that, despite comparable performance in standard neuropsychology tests that are designed to probe executive deficits, HIV-infected participants were significantly slower than uninfected controls in adapting to change in task demand, and the behavioral impairments can be quantitatively related to difference in fMRI signal at the dorsal anterior cingulate cortex (ACC). Due to the limited sample size of this hypothesis-generating study, we should take caution with these findings and future studies with a large and better matched sample size are needed. However, these rather novel findings in this study have a few important implications: first, the prevalence of cognitive impairments in HIV+ older adults might be even higher than previously proposed; second, ACC (in particularly its dorsal region) might be one of the key regions underlying cognitive impairments (in particularly executive functions) in HIV; and third, it might be beneficial to adopt paradigms developed and validated in cognitive neuroscience to study HAND, as these techniques might be more sensitive to some aspects of HIV-associated neurocognitive impairments than standard neuropsychology tests.

Task dependence of visual and category representations in prefrontal and inferior temporal cortices.

Visual categorization is an essential perceptual and cognitive process for assigning behavioral significance to incoming stimuli. Categorization depends on sensory processing of stimulus features as well as flexible cognitive processing for classifying stimuli according to the current behavioral context. Neurophysiological studies suggest that the prefrontal cortex (PFC) and the inferior temporal cortex (ITC) are involved in visual shape categorization. However, their precise roles in the perceptual and cognitive aspects of the categorization process are unclear, as the two areas have not been directly compared during changing task contexts. To address this, we examined the impact of task relevance on categorization-related activity in PFC and ITC by recording from both areas as monkeys alternated between a shape categorization and passive viewing tasks. As monkeys viewed the same stimuli in both tasks, the impact of task relevance on encoding in each area could be compared. While both areas showed task-dependent modulations of neuronal activity, the patterns of results differed markedly. PFC, but not ITC, neurons showed a modest increase in firing rates when stimuli were task relevant. PFC also showed significantly stronger category selectivity during the task compared with passive viewing, while task-dependent modulations of category selectivity in ITC were weak and occurred with a long latency. Finally, both areas showed an enhancement of stimulus selectivity during the task compared with passive viewing. Together, this suggests that the ITC and PFC show differing degrees of task-dependent flexibility and are preferentially involved in the perceptual and cognitive aspects of the categorization process, respectively.

Individual variability in location impacts orthographic selectivity in the "visual word form area".

Strong evidence exists for a key role of the human ventral occipitotemporal cortex (vOT) in reading, yet there have been conflicting reports about the specificity of this area in orthographic versus nonorthographic processing. We suggest that the inconsistencies in the literature can be explained by the method used to identify regions that respond to words. Here we provide evidence that the "visual word form area" (VWFA) shows word selectivity when identified at the individual subject level, but that intersubject variability in the location and size of the VWFA causes this selectivity to be washed out if defining the VWFA at the group level or based on coordinates from the literature. Our findings confirm the existence of a word-selective region in vOT while providing an explanation for why other studies have found a lack of word specificity in vOT.

Automatic phoneme category selectivity in the dorsal auditory stream.

Debates about motor theories of speech perception have recently been reignited by a burst of reports implicating premotor cortex (PMC) in speech perception. Often, however, these debates conflate perceptual and decision processes. Evidence that PMC activity correlates with task difficulty and subject performance suggests that PMC might be recruited, in certain cases, to facilitate category judgments about speech sounds (rather than speech perception, which involves decoding of sounds). However, it remains unclear whether PMC does, indeed, exhibit neural selectivity that is relevant for speech decisions. Further, it is unknown whether PMC activity in such cases reflects input via the dorsal or ventral auditory pathway, and whether PMC processing of speech is automatic or task-dependent. In a novel modified categorization paradigm, we presented human subjects with paired speech sounds from a phonetic continuum but diverted their attention from phoneme category using a challenging dichotic listening task. Using fMRI rapid adaptation to probe neural selectivity, we observed acoustic-phonetic selectivity in left anterior and left posterior auditory cortical regions. Conversely, we observed phoneme-category selectivity in left PMC that correlated with explicit phoneme-categorization performance measured after scanning, suggesting that PMC recruitment can account for performance on phoneme-categorization tasks. Structural equation modeling revealed connectivity from posterior, but not anterior, auditory cortex to PMC, suggesting a dorsal route for auditory input to PMC. Our results provide evidence for an account of speech processing in which the dorsal stream mediates automatic sensorimotor integration of speech and may be recruited to support speech decision tasks.

Time course of shape and category selectivity revealed by EEG rapid adaptation.

A hallmark of human cognition is the ability to rapidly assign meaning to sensory stimuli. It has been suggested that this fast visual object categorization ability is accomplished by a feedforward processing hierarchy consisting of shape-selective neurons in occipito-temporal cortex that feed into task circuits in frontal cortex computing conceptual category membership. We performed an EEG rapid adaptation study to test this hypothesis. Participants were trained to categorize novel stimuli generated with a morphing system that precisely controlled both stimulus shape and category membership. We subsequently performed EEG recordings while participants performed a category matching task on pairs of successively presented stimuli. We used space-time cluster analysis to identify channels and latencies exhibiting selective neural responses. Neural signals before 200 msec on posterior channels demonstrated a release from adaptation for shape changes, irrespective of category membership, compatible with a shape- but not explicitly category-selective neural representation. A subsequent cluster with anterior topography appeared after 200 msec and exhibited release from adaptation consistent with explicit categorization. These signals were subsequently modulated by perceptual uncertainty starting around 300 msec. The degree of category selectivity of the anterior signals was strongly predictive of behavioral performance. We also observed a posterior category-selective signal after 300 msec exhibiting significant functional connectivity with the initial anterior category-selective signal. In summary, our study supports the proposition that perceptual categorization is accomplished by the brain within a quarter second through a largely feedforward process culminating in frontal areas, followed by later category-selective signals in posterior regions.

Evidence for a Spoken Word Lexicon in the Auditory Ventral Stream.

The existence of a neural representation for whole words (i.e., a lexicon) is a common feature of many models of speech processing. Prior studies have provided evidence for a visual lexicon containing representations of whole written words in an area of the ventral visual stream known as the visual word form area. Similar experimental support for an auditory lexicon containing representations of spoken words has yet to be shown. Using functional magnetic resonance imaging rapid adaptation techniques, we provide evidence for an auditory lexicon in the auditory word form area in the human left anterior superior temporal gyrus that contains representations highly selective for individual spoken words. Furthermore, we show that familiarization with novel auditory words sharpens the selectivity of their representations in the auditory word form area. These findings reveal strong parallels in how the brain represents written and spoken words, showing convergent processing strategies across modalities in the visual and auditory ventral streams.

Functional correlates of the anterolateral processing hierarchy in human auditory cortex.

Converging evidence supports the hypothesis that an anterolateral processing pathway mediates sound identification in auditory cortex, analogous to the role of the ventral cortical pathway in visual object recognition. Studies in nonhuman primates have characterized the anterolateral auditory pathway as a processing hierarchy, composed of three anatomically and physiologically distinct initial stages: core, belt, and parabelt. In humans, potential homologs of these regions have been identified anatomically, but reliable and complete functional distinctions between them have yet to be established. Because the anatomical locations of these fields vary across subjects, investigations of potential homologs between monkeys and humans require these fields to be defined in single subjects. Using functional MRI, we presented three classes of sounds (tones, band-passed noise bursts, and conspecific vocalizations), equivalent to those used in previous monkey studies. In each individual subject, three regions showing functional similarities to macaque core, belt, and parabelt were readily identified. Furthermore, the relative sizes and locations of these regions were consistent with those reported in human anatomical studies. Our results demonstrate that the functional organization of the anterolateral processing pathway in humans is largely consistent with that of nonhuman primates. Because our scanning sessions last only 15 min/subject, they can be run in conjunction with other scans. This will enable future studies to characterize functional modules in human auditory cortex at a level of detail previously possible only in visual cortex. Furthermore, the approach of using identical schemes in both humans and monkeys will aid with establishing potential homologies between them.

Getting a handle on how the brain generates complexity.

Sensory processing in cortex across modalities appears to rely on a "simple-to-complex" hierarchical computational strategy in which neurons at later levels in the hierarchy combine inputs from earlier levels to create more complex neuronal selectivities. The specifics of this process are still poorly understood, however. In this issue of Network, Plebe shows how computational modeling of experimental data on neuronal tuning in secondary visual cortex can help us understand how the brain increases neuronal tuning complexity across the visual cortical hierarchy.

Appearance isn't everything: news on object representation in cortex.

How objects are represented in the visual system is one of the big questions in cognitive neuroscience. In this issue of Neuron, Mahon and colleagues present an intriguing study that suggests that properties of objects other than shape can influence the arrangement of object selectivities in visual areas. In the process, the study also points to important caveats regarding the ability of standard fMRI studies to make inferences about neuronal selectivity.

Categorization training results in shape- and category-selective human neural plasticity.

Object category learning is a fundamental ability, requiring the combination of "bottom-up" stimulus-driven with "top-down" task-specific information. It therefore may be a fruitful domain for study of the general neural mechanisms underlying cortical plasticity. A simple model predicts that category learning involves the formation of a task-independent shape-selective representation that provides input to circuits learning the categorization task, with the computationally appealing prediction of facilitated learning of additional, novel tasks over the same stimuli. Using fMRI rapid-adaptation techniques, we find that categorization training (on morphed "cars") induced a significant release from adaptation for small shape changes in lateral occipital cortex irrespective of category membership, compatible with the sharpening of a representation coding for physical appearance. In contrast, an area in lateral prefrontal cortex, selectively activated during categorization, showed sensitivity posttraining to explicit changes in category membership. Further supporting the model, categorization training also improved discrimination performance on the trained stimuli.

Prefrontal cortex activity during flexible categorization.

Items are categorized differently depending on the behavioral context. For instance, a lion can be categorized as an African animal or a type of cat. We recorded lateral prefrontal cortex (PFC) neural activity while monkeys switched between categorizing the same image set along two different category schemes with orthogonal boundaries. We found that each category scheme was largely represented by independent PFC neuronal populations and that activity reflecting a category distinction was weaker, but not absent, when that category was irrelevant. We suggest that the PFC represents competing category representations independently to reduce interference between them.

Evaluation of a shape-based model of human face discrimination using FMRI and behavioral techniques.

Understanding the neural mechanisms underlying object recognition is one of the fundamental challenges of visual neuroscience. While neurophysiology experiments have provided evidence for a "simple-to-complex" processing model based on a hierarchy of increasingly complex image features, behavioral and fMRI studies of face processing have been interpreted as incompatible with this account. We present a neurophysiologically plausible, feature-based model that quantitatively accounts for face discrimination characteristics, including face inversion and "configural" effects. The model predicts that face discrimination is based on a sparse representation of units selective for face shapes, without the need to postulate additional, "face-specific" mechanisms. We derive and test predictions that quantitatively link model FFA face neuron tuning, neural adaptation measured in an fMRI rapid adaptation paradigm, and face discrimination performance. The experimental data are in excellent agreement with the model prediction that discrimination performance should asymptote as faces become dissimilar enough to activate different neuronal populations.