Neural responses in human superior temporal cortex support coding of voice representations.

The ability to recognize abstract features of voice during auditory perception is an intricate feat of human audition. For the listener, this occurs in near-automatic fashion to seamlessly extract complex cues from a highly variable auditory signal. Voice perception depends on specialized regions of auditory cortex, including superior temporal gyrus (STG) and superior temporal sulcus (STS). However, the nature of voice encoding at the cortical level remains poorly understood. We leverage intracerebral recordings across human auditory cortex during presentation of voice and nonvoice acoustic stimuli to examine voice encoding at the cortical level in 8 patient-participants undergoing epilepsy surgery evaluation. We show that voice selectivity increases along the auditory hierarchy from supratemporal plane (STP) to the STG and STS. Results show accurate decoding of vocalizations from human auditory cortical activity even in the complete absence of linguistic content. These findings show an early, less-selective temporal window of neural activity in the STG and STS followed by a sustained, strongly voice-selective window. Encoding models demonstrate divergence in the encoding of acoustic features along the auditory hierarchy, wherein STG/STS responses are best explained by voice category and acoustics, as opposed to acoustic features of voice stimuli alone. This is in contrast to neural activity recorded from STP, in which responses were accounted for by acoustic features. These findings support a model of voice perception that engages categorical encoding mechanisms within STG and STS to facilitate feature extraction.

Learning nonnative speech sounds changes local encoding in the adult human cortex.

Adults can learn to identify nonnative speech sounds with training, albeit with substantial variability in learning behavior. Increases in behavioral accuracy are associated with increased separability for sound representations in cortical speech areas. However, it remains unclear whether individual auditory neural populations all show the same types of changes with learning, or whether there are heterogeneous encoding patterns. Here, we used high-resolution direct neural recordings to examine local population response patterns, while native English listeners learned to recognize unfamiliar vocal pitch patterns in Mandarin Chinese tones. We found a distributed set of neural populations in bilateral superior temporal gyrus and ventrolateral frontal cortex, where the encoding of Mandarin tones changed throughout training as a function of trial-by-trial accuracy ("learning effect"), including both increases and decreases in the separability of tones. These populations were distinct from populations that showed changes as a function of exposure to the stimuli regardless of trial-by-trial accuracy. These learning effects were driven in part by more variable neural responses to repeated presentations of acoustically identical stimuli. Finally, learning effects could be predicted from speech-evoked activity even before training, suggesting that intrinsic properties of these populations make them amenable to behavior-related changes. Together, these results demonstrate that nonnative speech sound learning involves a wide array of changes in neural representations across a distributed set of brain regions.

The nature of non-native speech sound representations.

Most current theories and models of second language speech perception are grounded in the notion that learners acquire speech sound categories in their target language. In this paper, this classic idea in speech perception is revisited, given that clear evidence for formation of such categories is lacking in previous research. To understand the debate on the nature of speech sound representations in a second language, an operational definition of "category" is presented, and the issues of categorical perception and current theories of second language learning are reviewed. Following this, behavioral and neuroimaging evidence for and against acquisition of categorical representations is described. Finally, recommendations for future work are discussed. The paper concludes with a recommendation for integration of behavioral and neuroimaging work and theory in this area.

A distributed dynamic brain network mediates linguistic tone representation and categorization.

Successful categorization requires listeners to represent the incoming sensory information, resolve the "blooming, buzzing confusion" inherent to noisy sensory signals, and leverage the accumulated evidence towards making a decision. Despite decades of intense debate, the neural systems underlying speech categorization remain unresolved. Here we assessed the neural representation and categorization of lexical tones by native Mandarin speakers (N = 31) across a range of acoustic and contextual variabilities (talkers, perceptual saliences, and stimulus-contexts) using functional magnetic imaging (fMRI) and an evidence accumulation model of decision-making. Univariate activation and multivariate pattern analyses reveal that the acoustic-variability-tolerant representations of tone category are observed within the middle portion of the left superior temporal gyrus (STG). Activation patterns in the frontal and parietal regions also contained category-relevant information that was differentially sensitive to various forms of variability. The robustness of neural representations of tone category in a distributed fronto-temporoparietal network is associated with trial-by-trial decision-making parameters. These findings support a hybrid model involving a representational core within the STG that operates dynamically within an extensive frontoparietal network to support the representation and categorization of linguistic pitch patterns.

Neural dynamics underlying the acquisition of distinct auditory category structures.

Despite the multidimensional and temporally fleeting nature of auditory signals we quickly learn to assign novel sounds to behaviorally relevant categories. The neural systems underlying the learning and representation of novel auditory categories are far from understood. Current models argue for a rigid specialization of hierarchically organized core regions that are fine-tuned to extracting and mapping relevant auditory dimensions to meaningful categories. Scaffolded within a dual-learning systems approach, we test a competing hypothesis: the spatial and temporal dynamics of emerging auditory-category representations are not driven by the underlying dimensions but are constrained by category structure and learning strategies. To test these competing models, we used functional Magnetic Resonance Imaging (fMRI) to assess representational dynamics during the feedback-based acquisition of novel non-speech auditory categories with identical dimensions but differing category structures: rule-based (RB) categories, hypothesized to involve an explicit sound-to-rule mapping network, and information integration (II) based categories, involving pre-decisional integration of dimensions via a procedural-based sound-to-reward mapping network. Adults were assigned to either the RB (n = 30, 19 females) or II (n = 30, 22 females) learning tasks. Despite similar behavioral learning accuracies, learning strategies derived from computational modeling and involvements of corticostriatal systems during feedback processing differed across tasks. Spatiotemporal multivariate representational similarity analysis revealed an emerging representation within an auditory sensory-motor pathway exclusively for the II learning task, prominently involving the superior temporal gyrus (STG), inferior frontal gyrus (IFG), and posterior precentral gyrus. In contrast, the RB learning task yielded distributed neural representations within regions involved in cognitive-control and attentional processes that emerged at different time points of learning. Our results unequivocally demonstrate that auditory learners' neural systems are highly flexible and show distinct spatial and temporal patterns that are not dimension-specific but reflect underlying category structures and learning strategies.

Cortical Tracking of Speech in Delta Band Relates to Individual Differences in Speech in Noise Comprehension in Older Adults.

OBJECTIVES: Understanding speech in adverse listening environments is challenging for older adults. Individual differences in pure tone averages and working memory are known to be critical indicators of speech in noise comprehension. Recent studies have suggested that tracking of the speech envelope in cortical oscillations <8 Hz may be an important mechanism related to speech comprehension by segmenting speech into words and phrases (delta, 1 to 4 Hz) or phonemes and syllables (theta, 4 to 8 Hz). The purpose of this study was to investigate the extent to which individual differences in pure tone averages, working memory, and cortical tracking of the speech envelope relate to speech in noise comprehension in older adults. DESIGN: Cortical tracking of continuous speech was assessed using electroencephalography in older adults (60 to 80 years). Participants listened to speech in quiet and in the presence of noise (time-reversed speech) and answered comprehension questions. Participants completed Forward Digit Span and Backward Digit Span as measures of working memory, and pure tone averages were collected. An index of reduction in noise (RIN) was calculated by normalizing the difference between raw cortical tracking in quiet and in noise. RESULTS: Comprehension question performance was greater for speech in quiet than for speech in noise. The relationship between RIN and speech in noise comprehension was assessed while controlling for the effects of individual differences in pure tone averages and working memory. Delta band RIN correlated with speech in noise comprehension, while theta band RIN did not. CONCLUSIONS: Cortical tracking by delta oscillations is robust to the effects of noise. These findings demonstrate that the magnitude of delta band RIN relates to individual differences in speech in noise comprehension in older adults. Delta band RIN may serve as a neural metric of speech in noise comprehension beyond the effects of pure tone averages and working memory.

The Role of the Human Auditory Corticostriatal Network in Speech Learning.

We establish a mechanistic account of how the mature human brain functionally reorganizes to acquire and represent new speech sounds. Native speakers of English learned to categorize Mandarin lexical tone categories produced by multiple talkers using trial-by-trial feedback. We hypothesized that the corticostriatal system is a key intermediary in mediating temporal lobe plasticity and the acquisition of new speech categories in adulthood. We conducted a functional magnetic resonance imaging experiment in which participants underwent a sound-to-category mapping task. Diffusion tensor imaging data were collected, and probabilistic fiber tracking analysis was employed to assay the auditory corticostriatal pathways. Multivariate pattern analysis showed that talker-invariant novel tone category representations emerged in the left superior temporal gyrus (LSTG) within a few hundred training trials. Univariate analysis showed that the putamen, a subregion of the striatum, was sensitive to positive feedback in correctly categorized trials. With learning, functional coupling between the putamen and LSTG increased during error processing. Furthermore, fiber tractography demonstrated robust structural connectivity between the feedback-sensitive striatal regions and the LSTG regions that represent the newly learned tone categories. Our convergent findings highlight a critical role for the auditory corticostriatal circuitry in mediating the acquisition of new speech categories.

Task-General and Acoustic-Invariant Neural Representation of Speech Categories in the Human Brain.

A significant neural challenge in speech perception includes extracting discrete phonetic categories from continuous and multidimensional signals despite varying task demands and surface-acoustic variability. While neural representations of speech categories have been previously identified in frontal and posterior temporal-parietal regions, the task dependency and dimensional specificity of these neural representations are still unclear. Here, we asked native Mandarin participants to listen to speech syllables carrying 4 distinct lexical tone categories across passive listening, repetition, and categorization tasks while they underwent functional magnetic resonance imaging (fMRI). We used searchlight classification and representational similarity analysis (RSA) to identify the dimensional structure underlying neural representation across tasks and surface-acoustic properties. Searchlight classification analyses revealed significant "cross-task" lexical tone decoding within the bilateral superior temporal gyrus (STG) and left inferior parietal lobule (LIPL). RSA revealed that the LIPL and LSTG, in contrast to the RSTG, relate to 2 critical dimensions (pitch height, pitch direction) underlying tone perception. Outside this core representational network, we found greater activation in the inferior frontal and parietal regions for stimuli that are more perceptually similar during tone categorization. Our findings reveal the specific characteristics of fronto-tempo-parietal regions that support speech representation and categorization processing.

Error patterns of native and non-native listeners' perception of speech in noise.

Speech perception in noise requires both bottom-up sampling of the stimulus and top-down reconstruction of the masked signal from a language model. Previous studies have provided mixed evidence about the exact role that linguistic knowledge plays in native and non-native listeners' perception of masked speech. This paper describes an analysis of whole utterance, content word, and morphosyntactic error patterns to test the prediction that non-native listeners are uniquely affected by energetic and informational masks because of limited information at multiple linguistic levels. The results reveal a consistent disadvantage for non-native listeners at all three levels in challenging listening environments.

Training-induced brain activation and functional connectivity differentiate multi-talker and single-talker speech training.

In second language acquisition studies, the high talker variability training approach has been frequently used to train participants to learn new speech patterns. However, the neuroplasticity induced by training is poorly understood. In the present study, native English speakers were trained on non-native pitch patterns (linguistic tones from Mandarin Chinese) in multi-talker (N = 16) or single-talker (N = 16) training conditions. We focused on two aspects of multi-talker training, voice processing and lexical phonology accessing, and used functional magnetic resonance imaging (fMRI) to measure the brain activation and functional connectivity (FC) of two regions of interest in a tone identification task conducted before and after training, namely the anterior part of the right superior temporal gyrus (aRSTG) and the posterior left superior temporal gyrus (pLSTG). The results showed distinct patterns of associations between neural signals and learning success for multi-talker training. Specifically, post-training brain activation in the aRSTG and FC strength between the aRSTG and pLSTG were correlated with learning success in the multi-talker training group but not in the single-talker group. These results suggest that talker variability in the training procedure may enhance neural efficiency in these brain areas and strengthen the cooperation between them. Our findings highlight the brain processing of newly learned speech patterns is influenced by the given training approach.

Stability and plasticity in neural encoding of linguistically relevant pitch patterns.

While lifelong language experience modulates subcortical encoding of pitch patterns, there is emerging evidence that short-term training introduced in adulthood also shapes subcortical pitch encoding. Here we use a cross-language design to examine the stability of language experience-dependent subcortical plasticity over multiple days. We then examine the extent to which behavioral relevance induced by sound-to-category training leads to plastic changes in subcortical pitch encoding in adulthood relative to adolescence, a period of ongoing maturation of subcortical and cortical auditory processing. Frequency-following responses (FFRs), which reflect phase-locked activity from subcortical neural ensembles, were elicited while participants passively listened to pitch patterns reflective of Mandarin tones. In experiment 1, FFRs were recorded across three consecutive days from native Chinese-speaking (n = 10) and English-speaking (n = 10) adults. In experiment 2, FFRs were recorded from native English-speaking adolescents (n = 20) and adults (n = 15) before, during, and immediately after a session of sound-to-category training, as well as a day after training ceased. Experiment 1 demonstrated the stability of language experience-dependent subcortical plasticity in pitch encoding across multiple days of passive exposure to linguistic pitch patterns. In contrast, experiment 2 revealed an enhancement in subcortical pitch encoding that emerged a day after the sound-to-category training, with some developmental differences observed. Taken together, these findings suggest that behavioral relevance is a critical component for the observation of plasticity in the subcortical encoding of pitch.NEW & NOTEWORTHY We examine the timescale of experience-dependent auditory plasticity to linguistically relevant pitch patterns. We find extreme stability in lifelong experience-dependent plasticity. We further demonstrate that subcortical function in adolescents and adults is modulated by a single session of sound-to-category training. Our results suggest that behavioral relevance is a necessary ingredient for neural changes in pitch encoding to be observed throughout human development. These findings contribute to the neurophysiological understanding of long- and short-term experience-dependent modulation of pitch.

Context-dependent plasticity in the subcortical encoding of linguistic pitch patterns.

We examined the mechanics of online experience-dependent auditory plasticity by assessing the influence of prior context on the frequency-following responses (FFRs), which reflect phase-locked responses from neural ensembles within the subcortical auditory system. FFRs were elicited to a Cantonese falling lexical pitch pattern from 24 native speakers of Cantonese in a variable context, wherein the falling pitch pattern randomly occurred in the context of two other linguistic pitch patterns; in a patterned context, wherein, the falling pitch pattern was presented in a predictable sequence along with two other pitch patterns, and in a repetitive context, wherein the falling pitch pattern was presented with 100% probability. We found that neural tracking of the stimulus pitch contour was most faithful and accurate when listening context was patterned and least faithful when the listening context was variable. The patterned context elicited more robust pitch tracking relative to the repetitive context, suggesting that context-dependent plasticity is most robust when the context is predictable but not repetitive. Our study demonstrates a robust influence of prior listening context that works to enhance online neural encoding of linguistic pitch patterns. We interpret these results as indicative of an interplay between contextual processes that are responsive to predictability as well as novelty in the presentation context. NEW & NOTEWORTHY: Human auditory perception in dynamic listening environments requires fine-tuning of sensory signal based on behaviorally relevant regularities in listening context, i.e., online experience-dependent plasticity. Our finding suggests what partly underlie online experience-dependent plasticity are interplaying contextual processes in the subcortical auditory system that are responsive to predictability as well as novelty in listening context. These findings add to the literature that looks to establish the neurophysiological bases of auditory system plasticity, a central issue in auditory neuroscience.

The Role of Corticostriatal Systems in Speech Category Learning.

One of the most difficult category learning problems for humans is learning nonnative speech categories. While feedback-based category training can enhance speech learning, the mechanisms underlying these benefits are unclear. In this functional magnetic resonance imaging study, we investigated neural and computational mechanisms underlying feedback-dependent speech category learning in adults. Positive feedback activated a large corticostriatal network including the dorsolateral prefrontal cortex, inferior parietal lobule, middle temporal gyrus, caudate, putamen, and the ventral striatum. Successful learning was contingent upon the activity of domain-general category learning systems: the fast-learning reflective system, involving the dorsolateral prefrontal cortex that develops and tests explicit rules based on the feedback content, and the slow-learning reflexive system, involving the putamen in which the stimuli are implicitly associated with category responses based on the reward value in feedback. Computational modeling of response strategies revealed significant use of reflective strategies early in training and greater use of reflexive strategies later in training. Reflexive strategy use was associated with increased activation in the putamen. Our results demonstrate a critical role for the reflexive corticostriatal learning system as a function of response strategy and proficiency during speech category learning.

Enhanced procedural learning of speech sound categories in a genetic variant of FOXP2.

A mutation of the forkhead box protein P2 (FOXP2) gene is associated with severe deficits in human speech and language acquisition. In rodents, the humanized form of FOXP2 promotes faster switching from declarative to procedural learning strategies when the two learning systems compete. Here, we examined a polymorphism of FOXP2 (rs6980093) in humans (214 adults; 111 females) for associations with non-native speech category learning success. Neurocomputational modeling results showed that individuals with the GG genotype shifted faster to procedural learning strategies, which are optimal for the task. These findings support an adaptive role for the FOXP2 gene in modulating the function of neural learning systems that have a direct bearing on human speech category learning.

The role of age and executive function in auditory category learning.

Auditory categorization is a natural and adaptive process that allows for the organization of high-dimensional, continuous acoustic information into discrete representations. Studies in the visual domain have identified a rule-based learning system that learns and reasons via a hypothesis-testing process that requires working memory and executive attention. The rule-based learning system in vision shows a protracted development, reflecting the influence of maturing prefrontal function on visual categorization. The aim of the current study was twofold: (a) to examine the developmental trajectory of rule-based auditory category learning from childhood through adolescence and into early adulthood and (b) to examine the extent to which individual differences in rule-based category learning relate to individual differences in executive function. A sample of 60 participants with normal hearing-20 children (age range=7-12years), 21 adolescents (age range=13-19years), and 19 young adults (age range=20-23years)-learned to categorize novel dynamic "ripple" sounds using trial-by-trial feedback. The spectrotemporally modulated ripple sounds are considered the auditory equivalent of the well-studied "Gabor" patches in the visual domain. Results reveal that auditory categorization accuracy improved with age, with young adults outperforming children and adolescents. Computational modeling analyses indicated that the use of the task-optimal strategy (i.e., a conjunctive rule-based learning strategy) improved with age. Notably, individual differences in executive flexibility significantly predicted auditory category learning success. The current findings demonstrate a protracted development of rule-based auditory categorization. The results further suggest that executive flexibility coupled with perceptual processes play important roles in successful rule-based auditory category learning.

Auditory categories with separable decision boundaries are learned faster with full feedback than with minimal feedback.

During visual category learning, full feedback (e.g., "Wrong, that was a category 4."), relative to minimal feedback (e.g., "Wrong."), enhances performance when the relevant dimensions are separable. This pattern is reversed with inseparable dimensions. Here, the interaction between trial-by-trial feedback and separability of dimensions in the auditory domain is examined. Participants were trained to categorize auditory stimuli along separable or inseparable dimensions. One group received full feedback, while the other group received minimal feedback. In the separable-dimensions condition, the full-feedback group achieved higher accuracy than did the minimal-feedback group. In the inseparable-dimensions condition, performance was equivalent across the feedback groups. These results altogether suggest that trial-by-trial feedback affects auditory category learning performance differentially for separable and inseparable categories.

The C957T polymorphism in the dopamine receptor D2 gene modulates domain-general category learning.

Adaptive learning from reward and punishment is vital for human survival. Striatal and frontal dopaminergic activities are associated with adaptive learning. For example, the C957T single nucleotide polymorphism of the dopamine receptor D2 (DRD2) gene alters striatal D2 receptor availability and affects individuals' adaptive learning ability. Specifically, individuals with the T/T genotype, which is associated with higher striatal D2 availability, show enhanced learning from negative outcomes. Prior work examining DRD2 genetic variability has focused primarily on frontally mediated reflective learning that is under effortful, conscious control. However, less is known about a more automatic, striatally mediated reflexive learning. Here we examined the extent to which this polymorphism differentially influences reflective and reflexive learning across visual and auditory modalities. We employed rule-based (RB) and information-integration (II) category learning paradigms that target reflective and reflexive learning, respectively. Results revealed an advantage in II category learning but poorer RB category learning in T/T homozygotes. The pattern of results was consistent across sensory modalities. These findings suggest that this DRD2 polymorphism exerts opposite influences on domain-general frontally mediated reflective learning and striatally mediated reflexive learning.

Music training for the development of auditory skills.

The effects of music training in relation to brain plasticity have caused excitement, evident from the popularity of books on this topic among scientists and the general public. Neuroscience research has shown that music training leads to changes throughout the auditory system that prime musicians for listening challenges beyond music processing. This effect of music training suggests that, akin to physical exercise and its impact on body fitness, music is a resource that tones the brain for auditory fitness. Therefore, the role of music in shaping individual development deserves consideration.

Influence of depressive symptoms on speech perception in adverse listening conditions.

It is widely acknowledged that individuals with elevated depressive symptoms exhibit deficits in inter-personal communication. Research has primarily focused on speech production in individuals with elevated depressive symptoms. Little is known about speech perception in individuals with elevated depressive symptoms, especially in challenging listening conditions. Here, we examined speech perception in young adults with low- or high-depressive (HD) symptoms in the presence of a range of maskers. Maskers were selected to reflect various levels of informational masking (IM), which refers to cognitive interference due to signal and masker similarity, and energetic masking (EM), which refers to peripheral interference due to signal degradation by the masker. Speech intelligibility data revealed that individuals with HD symptoms did not differ from those with low-depressive symptoms during EM, but they exhibited a selective deficit during IM. Since IM is a common occurrence in real-world social settings, this listening deficit may exacerbate communicative difficulties.

Human brainstem plasticity: the interaction of stimulus probability and auditory learning.

Two forms of brainstem plasticity are known to occur: an immediate stimulus probability-based and learning-dependent plasticity. Whether these kinds of plasticity interact is unknown. We examined this question in a training experiment involving three phases: (1) an initial baseline measurement, (2) a 9-session training paradigm, and (3) a retest measurement. At the outset of the experiment, auditory brainstem responses (ABR) were recorded to two unfamiliar pitch patterns presented in an oddball paradigm. Then half the participants underwent sound-to-meaning training where they learned to match these pitch patterns to novel words, with the remaining participants serving as controls who received no auditory training. Nine days after the baseline measurement, the pitch patterns were re-presented to all participants using the same oddball paradigm. Analysis of the baseline recordings revealed an effect of probability: when a sound was presented infrequently, the pitch contour was represented less accurately in the ABR than when it was presented frequently. After training, pitch tracking was more accurate for infrequent sounds, particularly for the pitch pattern that was encoded more poorly pre-training. However, the control group was stable over the same interval. Our results provide evidence that probability-based and learning-dependent plasticity interact in the brainstem.