Energetic and informational masking place dissociable demands on listening effort: Evidence from simultaneous electroencephalography and pupillometrya).

The task of processing speech masked by concurrent speech/noise can pose a substantial challenge to listeners. However, performance on such tasks may not directly reflect the amount of listening effort they elicit. Changes in pupil size and neural oscillatory power in the alpha range (8-12 Hz) are prominent neurophysiological signals known to reflect listening effort; however, measurements obtained through these two approaches are rarely correlated, suggesting that they may respond differently depending on the specific cognitive demands (and, by extension, the specific type of effort) elicited by specific tasks. This study aimed to compare changes in pupil size and alpha power elicited by different types of auditory maskers (highly confusable intelligible speech maskers, speech-envelope-modulated speech-shaped noise, and unmodulated speech-shaped noise maskers) in young, normal-hearing listeners. Within each condition, the target-to-masker ratio was set at the participant's individually estimated 75% correct point on the psychometric function. The speech masking condition elicited a significantly greater increase in pupil size than either of the noise masking conditions, whereas the unmodulated noise masking condition elicited a significantly greater increase in alpha oscillatory power than the speech masking condition, suggesting that the effort needed to solve these respective tasks may have different neural origins.

Distributed networks for auditory memory differentially contribute to recall precision.

Re-directing attention to objects in working memory can enhance their representational fidelity. However, how this attentional enhancement of memory representations is implemented across distinct, sensory and cognitive-control brain network is unspecified. The present fMRI experiment leverages psychophysical modelling and multivariate auditory-pattern decoding as behavioral and neural proxies of mnemonic fidelity. Listeners performed an auditory syllable pitch-discrimination task and received retro-active cues to selectively attend to a to-be-probed syllable in memory. Accompanied by increased neural activation in fronto-parietal and cingulo-opercular networks, valid retro-cues yielded faster and more perceptually sensitive responses in recalling acoustic detail of memorized syllables. Information about the cued auditory object was decodable from hemodynamic response patterns in superior temporal sulcus (STS), fronto-parietal, and sensorimotor regions. However, among these regions retaining auditory memory objects, neural fidelity in the left STS and its enhancement through attention-to-memory best predicted individuals' gain in auditory memory recall precision. Our results demonstrate how functionally discrete brain regions differentially contribute to the attentional enhancement of memory representations.

Role of the striatum in incidental learning of sound categories.

Humans are born as "universal listeners" without a bias toward any particular language. However, over the first year of life, infants' perception is shaped by learning native speech categories. Acoustically different sounds-such as the same word produced by different speakers-come to be treated as functionally equivalent. In natural environments, these categories often emerge incidentally without overt categorization or explicit feedback. However, the neural substrates of category learning have been investigated almost exclusively using overt categorization tasks with explicit feedback about categorization decisions. Here, we examined whether the striatum, previously implicated in category learning, contributes to incidental acquisition of sound categories. In the fMRI scanner, participants played a videogame in which sound category exemplars aligned with game actions and events, allowing sound categories to incidentally support successful game play. An experimental group heard nonspeech sound exemplars drawn from coherent category spaces, whereas a control group heard acoustically similar sounds drawn from a less structured space. Although the groups exhibited similar in-game performance, generalization of sound category learning and activation of the posterior striatum were significantly greater in the experimental than control group. Moreover, the experimental group showed brain-behavior relationships related to the generalization of all categories, while in the control group these relationships were restricted to the categories with structured sound distributions. Together, these results demonstrate that the striatum, through its interactions with the left superior temporal sulcus, contributes to incidental acquisition of sound category representations emerging from naturalistic learning environments.

Dopaminergic modulation of hemodynamic signal variability and the functional connectome during cognitive performance.

Dopamine underlies important aspects of cognition, and has been suggested to boost cognitive performance. However, how dopamine modulates the large-scale cortical dynamics during cognitive performance has remained elusive. Using functional MRI during a working memory task in healthy young human listeners, we investigated the effect of levodopa (l-dopa) on two aspects of cortical dynamics, blood oxygen-level-dependent (BOLD) signal variability and the functional connectome of large-scale cortical networks. We here show that enhanced dopaminergic signaling modulates the two potentially interrelated aspects of large-scale cortical dynamics during cognitive performance, and the degree of these modulations is able to explain inter-individual differences in l-dopa-induced behavioral benefits. Relative to placebo, l-dopa increased BOLD signal variability in task-relevant temporal, inferior frontal, parietal and cingulate regions. On the connectome level, however, l-dopa diminished functional integration across temporal and cingulo-opercular regions. This hypo-integration was expressed as a reduction in network efficiency and modularity in more than two thirds of the participants and to different degrees. Hypo-integration co-occurred with relative hyper-connectivity in paracentral lobule and precuneus, as well as posterior putamen. Both, l-dopa-induced BOLD signal variability modulation and functional connectome modulations proved predictive of an individual's l-dopa-induced benefits in behavioral performance, namely response speed and perceptual sensitivity. Lastly, l-dopa-induced modulations of BOLD signal variability were correlated with l-dopa-induced modulation of nodal connectivity and network efficiency. Our findings underline the role of dopamine in maintaining the dynamic range of, and communication between, cortical systems, and their explanatory power for inter-individual differences in benefits from dopamine during cognitive performance.

The Human Neural Alpha Response to Speech is a Proxy of Attentional Control.

Human alpha (~10 Hz) oscillatory power is a prominent neural marker of cognitive effort. When listeners attempt to process and retain acoustically degraded speech, alpha power enhances. It is unclear whether these alpha modulations reflect the degree of acoustic degradation per se or the degradation-driven demand to a listener's attentional control. Using an irrelevant-speech paradigm and measuring the electroencephalogram (EEG), the current experiment demonstrates that the neural alpha response to speech is a surprisingly clear proxy of top-down control, entirely driven by the listening goals of attending versus ignoring degraded speech. While (n = 23) listeners retained the serial order of 9 to-be-recalled digits, one to-be-ignored sentence was presented. Distractibility of the to-be-ignored sentence parametrically varied in acoustic detail (noise-vocoding), with more acoustic detail of distracting speech increasingly disrupting listeners' serial memory recall. Where previous studies had observed decreases in parietal and auditory alpha power with more acoustic detail (of target speech), alpha power here showed the opposite pattern and increased with more acoustic detail in the speech distractor. In sum, the neural alpha response reflects almost exclusively a listener's goal, which is decisive for whether more acoustic detail facilitates comprehension (of attended speech) or enhances distraction (of ignored speech).

Selective Attention to Auditory Memory Neurally Enhances Perceptual Precision.

Selective attention to a task-relevant stimulus facilitates encoding of that stimulus into a working memory representation. It is less clear whether selective attention also improves the precision of a stimulus already represented in memory. Here, we investigate the behavioral and neural dynamics of selective attention to representations in auditory working memory (i.e., auditory objects) using psychophysical modeling and model-based analysis of electroencephalographic signals. Human listeners performed a syllable pitch discrimination task where two syllables served as to-be-encoded auditory objects. Valid (vs neutral) retroactive cues were presented during retention to allow listeners to selectively attend to the to-be-probed auditory object in memory. Behaviorally, listeners represented auditory objects in memory more precisely (expressed by steeper slopes of a psychometric curve) and made faster perceptual decisions when valid compared to neutral retrocues were presented. Neurally, valid compared to neutral retrocues elicited a larger frontocentral sustained negativity in the evoked potential as well as enhanced parietal alpha/low-beta oscillatory power (9-18 Hz) during memory retention. Critically, individual magnitudes of alpha oscillatory power (7-11 Hz) modulation predicted the degree to which valid retrocues benefitted individuals' behavior. Our results indicate that selective attention to a specific object in auditory memory does benefit human performance not by simply reducing memory load, but by actively engaging complementary neural resources to sharpen the precision of the task-relevant object in memory. SIGNIFICANCE STATEMENT: Can selective attention improve the representational precision with which objects are held in memory? And if so, what are the neural mechanisms that support such improvement? These issues have been rarely examined within the auditory modality, in which acoustic signals change and vanish on a milliseconds time scale. Introducing a new auditory memory paradigm and using model-based electroencephalography analyses in humans, we thus bridge this gap and reveal behavioral and neural signatures of increased, attention-mediated working memory precision. We further show that the extent of alpha power modulation predicts the degree to which individuals' memory performance benefits from selective attention.

Evidence for Cerebellar Contributions to Adaptive Plasticity in Speech Perception.

Human speech perception rapidly adapts to maintain comprehension under adverse listening conditions. For example, with exposure listeners can adapt to heavily accented speech produced by a non-native speaker. Outside the domain of speech perception, adaptive changes in sensory and motor processing have been attributed to cerebellar functions. The present functional magnetic resonance imaging study investigates whether adaptation in speech perception also involves the cerebellum. Acoustic stimuli were distorted using a vocoding plus spectral-shift manipulation and presented in a word recognition task. Regions in the cerebellum that showed differences before versus after adaptation were identified, and the relationship between activity during adaptation and subsequent behavioral improvements was examined. These analyses implicated the right Crus I region of the cerebellum in adaptive changes in speech perception. A functional correlation analysis with the right Crus I as a seed region probed for cerebral cortical regions with covarying hemodynamic responses during the adaptation period. The results provided evidence of a functional network between the cerebellum and language-related regions in the temporal and parietal lobes of the cerebral cortex. Consistent with known cerebellar contributions to sensorimotor adaptation, cerebro-cerebellar interactions may support supervised learning mechanisms that rely on sensory prediction error signals in speech perception.

Electrophysiological correlates of perceptual prediction error are attenuated in dyslexia.

A perceptual adaptation deficit often accompanies reading difficulty in dyslexia, manifesting in poor perceptual learning of consistent stimuli and reduced neurophysiological adaptation to stimulus repetition. However, it is not known how adaptation deficits relate to differences in feedforward or feedback processes in the brain. Here we used electroencephalography (EEG) to interrogate the feedforward and feedback contributions to neural adaptation as adults with and without dyslexia viewed pairs of faces and words in a paradigm that manipulated whether there was a high probability of stimulus repetition versus a high probability of stimulus change. We measured three neural dependent variables: expectation (the difference between prestimulus EEG power with and without the expectation of stimulus repetition), feedforward repetition (the difference between event-related potentials (ERPs) evoked by an expected change and an unexpected repetition), and feedback-mediated prediction error (the difference between ERPs evoked by an unexpected change and an expected repetition). Expectation significantly modulated prestimulus theta- and alpha-band EEG in both groups. Unexpected repetitions of words, but not faces, also led to significant feedforward repetition effects in the ERPs of both groups. However, neural prediction error when an unexpected change occurred instead of an expected repetition was significantly weaker in dyslexia than the control group for both faces and words. These results suggest that the neural and perceptual adaptation deficits observed in dyslexia reflect the failure to effectively integrate perceptual predictions with feedforward sensory processing. In addition to reducing perceptual efficiency, the attenuation of neural prediction error signals would also be deleterious to the wide range of perceptual and procedural learning abilities that are critical for developing accurate and fluent reading skills.

Talker discontinuity disrupts attention to speech: Evidence from EEG and pupillometry.

Speech is processed less efficiently from discontinuous, mixed talkers than one consistent talker, but little is known about the neural mechanisms for processing talker variability. Here, we measured psychophysiological responses to talker variability using electroencephalography (EEG) and pupillometry while listeners performed a delayed recall of digit span task. Listeners heard and recalled seven-digit sequences with both talker (single- vs. mixed-talker digits) and temporal (0- vs. 500-ms inter-digit intervals) discontinuities. Talker discontinuity reduced serial recall accuracy. Both talker and temporal discontinuities elicited P3a-like neural evoked response, while rapid processing of mixed-talkers' speech led to increased phasic pupil dilation. Furthermore, mixed-talkers' speech produced less alpha oscillatory power during working memory maintenance, but not during speech encoding. Overall, these results are consistent with an auditory attention and streaming framework in which talker discontinuity leads to involuntary, stimulus-driven attentional reorientation to novel speech sources, resulting in the processing interference classically associated with talker variability.

Effects of talker continuity and speech rate on auditory working memory.

Speech processing is slower and less accurate when listeners encounter speech from multiple talkers compared to one continuous talker. However, interference from multiple talkers has been investigated only using immediate speech recognition or long-term memory recognition tasks. These tasks reveal opposite effects of speech processing time on speech recognition - while fast processing of multi-talker speech impedes immediate recognition, it also results in more abstract and less talker-specific long-term memories for speech. Here, we investigated whether and how processing multi-talker speech disrupts working memory maintenance, an intermediate stage between perceptual recognition and long-term memory. In a digit sequence recall task, listeners encoded seven-digit sequences and recalled them after a 5-s delay. Sequences were spoken by either a single talker or multiple talkers at one of three presentation rates (0-, 200-, and 500-ms inter-digit intervals). Listeners' recall was slower and less accurate for sequences spoken by multiple talkers than a single talker. Especially for the fastest presentation rate, listeners were less efficient when recalling sequences spoken by multiple talkers. Our results reveal that talker-specificity effects for speech working memory are most prominent when listeners must rapidly encode speech. These results suggest that, like immediate speech recognition, working memory for speech is susceptible to interference from variability across talkers. While many studies ascribe effects of talker variability to the need to calibrate perception to talker-specific acoustics, these results are also consistent with the idea that a sudden change of talkers disrupts attentional focus, interfering with efficient working-memory processing.

The Benefit of Attention-to-Memory Depends on the Interplay of Memory Capacity and Memory Load.

Humans can be cued to attend to an item in memory, which facilitates and enhances the perceptual precision in recalling this item. Here, we demonstrate that this facilitating effect of attention-to-memory hinges on the overall degree of memory load. The benefit an individual draws from attention-to-memory depends on her overall working memory performance, measured as sensitivity (d') in a retroactive cue (retro-cue) pitch discrimination task. While listeners maintained 2, 4, or 6 auditory syllables in memory, we provided valid or neutral retro-cues to direct listeners' attention to one, to-be-probed syllable in memory. Participants' overall memory performance (i.e., perceptual sensitivity d') was relatively unaffected by the presence of valid retro-cues across memory loads. However, a more fine-grained analysis using psychophysical modeling shows that valid retro-cues elicited faster pitch-change judgments and improved perceptual precision. Importantly, as memory load increased, listeners' overall working memory performance correlated with inter-individual differences in the degree to which precision improved (r = 0.39, p = 0.029). Under high load, individuals with low working memory profited least from attention-to-memory. Our results demonstrate that retrospective attention enhances perceptual precision of attended items in memory but listeners' optimal use of informative cues depends on their overall memory abilities.

Discovering functional units in continuous speech.

Language learning requires that listeners discover acoustically variable functional units like phonetic categories and words from an unfamiliar, continuous acoustic stream. Although many category learning studies have examined how listeners learn to generalize across the acoustic variability inherent in the signals that convey the functional units of language, these studies have tended to focus upon category learning across isolated sound exemplars. However, continuous input presents many additional learning challenges that may impact category learning. Listeners may not know the timescale of the functional unit, its relative position in the continuous input, or its relationship to other evolving input regularities. Moving laboratory-based studies of isolated category exemplars toward more natural input is important to modeling language learning, but very little is known about how listeners discover categories embedded in continuous sound. In 3 experiments, adult participants heard acoustically variable sound category instances embedded in acoustically variable and unfamiliar sound streams within a video game task. This task was inherently rich in multisensory regularities with the to-be-learned categories and likely to engage procedural learning without requiring explicit categorization, segmentation, or even attention to the sounds. After 100 min of game play, participants categorized familiar sound streams in which target words were embedded and generalized this learning to novel streams as well as isolated instances of the target words. The findings demonstrate that even without a priori knowledge, listeners can discover input regularities that have the best predictive control over the environment for both non-native speech and nonspeech signals, emphasizing the generality of the learning.

How may the basal ganglia contribute to auditory categorization and speech perception?

Listeners must accomplish two complementary perceptual feats in extracting a message from speech. They must discriminate linguistically-relevant acoustic variability and generalize across irrelevant variability. Said another way, they must categorize speech. Since the mapping of acoustic variability is language-specific, these categories must be learned from experience. Thus, understanding how, in general, the auditory system acquires and represents categories can inform us about the toolbox of mechanisms available to speech perception. This perspective invites consideration of findings from cognitive neuroscience literatures outside of the speech domain as a means of constraining models of speech perception. Although neurobiological models of speech perception have mainly focused on cerebral cortex, research outside the speech domain is consistent with the possibility of significant subcortical contributions in category learning. Here, we review the functional role of one such structure, the basal ganglia. We examine research from animal electrophysiology, human neuroimaging, and behavior to consider characteristics of basal ganglia processing that may be advantageous for speech category learning. We also present emerging evidence for a direct role for basal ganglia in learning auditory categories in a complex, naturalistic task intended to model the incidental manner in which speech categories are acquired. To conclude, we highlight new research questions that arise in incorporating the broader neuroscience research literature in modeling speech perception, and suggest how understanding contributions of the basal ganglia can inform attempts to optimize training protocols for learning non-native speech categories in adulthood.

Learning foreign sounds in an alien world: videogame training improves non-native speech categorization.

Although speech categories are defined by multiple acoustic dimensions, some are perceptually weighted more than others and there are residual effects of native-language weightings in non-native speech perception. Recent research on nonlinguistic sound category learning suggests that the distribution characteristics of experienced sounds influence perceptual cue weights: Increasing variability across a dimension leads listeners to rely upon it less in subsequent category learning (Holt & Lotto, 2006). The present experiment investigated the implications of this among native Japanese learning English /r/-/l/ categories. Training was accomplished using a videogame paradigm that emphasizes associations among sound categories, visual information, and players' responses to videogame characters rather than overt categorization or explicit feedback. Subjects who played the game for 2.5h across 5 days exhibited improvements in /r/-/l/ perception on par with 2-4 weeks of explicit categorization training in previous research and exhibited a shift toward more native-like perceptual cue weights.