Audiovisual Speech Perception Benefits are Stable from Preschool through Adolescence.

The ability to leverage visual cues in speech perception - especially in noisy backgrounds - is well established from infancy to adulthood. Yet, the developmental trajectory of audiovisual benefits stays a topic of debate. The inconsistency in findings can be attributed to relatively small sample sizes or tasks that are not appropriate for given age groups. We designed an audiovisual speech perception task that was cognitively and linguistically age-appropriate from preschool to adolescence and recruited a large sample ( N = 161) of children (age 4-15). We found that even the youngest children show reliable speech perception benefits when provided with visual cues and that these benefits are consistent throughout development when auditory and visual signals match. Individual variability is explained by how the child experiences their speech-in-noise performance rather than the quality of the signal itself. This underscores the importance of visual speech for young children who are regularly in noisy environments like classrooms and playgrounds.

Children with developmental dyslexia have equivalent audiovisual speech perception performance but their perceptual weights differ.

As reading is inherently a multisensory, audiovisual (AV) process where visual symbols (i.e., letters) are connected to speech sounds, the question has been raised whether individuals with reading difficulties, like children with developmental dyslexia (DD), have broader impairments in multisensory processing. This question has been posed before, yet it remains unanswered due to (a) the complexity and contentious etiology of DD along with (b) lack of consensus on developmentally appropriate AV processing tasks. We created an ecologically valid task for measuring multisensory AV processing by leveraging the natural phenomenon that speech perception improves when listeners are provided visual information from mouth movements (particularly when the auditory signal is degraded). We designed this AV processing task with low cognitive and linguistic demands such that children with and without DD would have equal unimodal (auditory and visual) performance. We then collected data in a group of 135 children (age 6.5-15) with an AV speech perception task to answer the following questions: (1) How do AV speech perception benefits manifest in children, with and without DD? (2) Do children all use the same perceptual weights to create AV speech perception benefits, and (3) what is the role of phonological processing in AV speech perception? We show that children with and without DD have equal AV speech perception benefits on this task, but that children with DD rely less on auditory processing in more difficult listening situations to create these benefits and weigh both incoming information streams differently. Lastly, any reported differences in speech perception in children with DD might be better explained by differences in phonological processing than differences in reading skills. RESEARCH HIGHLIGHTS: Children with versus without developmental dyslexia have equal audiovisual speech perception benefits, regardless of their phonological awareness or reading skills. Children with developmental dyslexia rely less on auditory performance to create audiovisual speech perception benefits. Individual differences in speech perception in children might be better explained by differences in phonological processing than differences in reading skills.

Sound Level Changes the Auditory Cortical Activation Detected with Functional Near-Infrared Spectroscopy.

BACKGROUND: Functional near-infrared spectroscopy (fNIRS) is a viable non-invasive technique for functional neuroimaging in the cochlear implant (CI) population; however, the effects of acoustic stimulus features on the fNIRS signal have not been thoroughly examined. This study examined the effect of stimulus level on fNIRS responses in adults with normal hearing or bilateral CIs. We hypothesized that fNIRS responses would correlate with both stimulus level and subjective loudness ratings, but that the correlation would be weaker with CIs due to the compression of acoustic input to electric output. METHODS: Thirteen adults with bilateral CIs and 16 with normal hearing (NH) completed the study. Signal-correlated noise, a speech-shaped noise modulated by the temporal envelope of speech stimuli, was used to determine the effect of stimulus level in an unintelligible speech-like stimulus between the range of soft to loud speech. Cortical activity in the left hemisphere was recorded. RESULTS: Results indicated a positive correlation of cortical activation in the left superior temporal gyrus with stimulus level in both NH and CI listeners with an additional correlation between cortical activity and perceived loudness for the CI group. The results are consistent with the literature and our hypothesis. CONCLUSIONS: These results support the potential of fNIRS to examine auditory stimulus level effects at a group level and the importance of controlling for stimulus level and loudness in speech recognition studies. Further research is needed to better understand cortical activation patterns for speech recognition as a function of both stimulus presentation level and perceived loudness.

The prevalence and developmental course of auditory processing differences in autistic children.

Auditory processing differences, including hyper- or hyposensitivity to sound, aversions to sound, and difficulty listening under noisy, real-world conditions, are commonly reported in autistic individuals. However, the developmental course and functional impact of these auditory processing differences are unclear. In this study, we investigate the prevalence, developmental trajectory, and functional impact of auditory processing differences in autistic children throughout childhood using a longitudinal study design. Auditory processing differences were measured using the Short Sensory Profile, a caregiver questionnaire, in addition to adaptive behaviors and disruptive/concerning behaviors at 3, 6, and 9 years of age. Our results showed that auditory processing differences were reported in greater than 70% of the autistic children in our sample at all three timepoints, maintained a high prevalence through 9 years of age, and were associated with increased disruptive/concerning behaviors and difficulty with adaptive behaviors. Furthermore, in our sample of children, auditory processing differences at age 3 years predicted disruptive/concerning behaviors and difficulty with adaptive behaviors at age 9 years. These findings warrant further investigations of the potential benefit of incorporating measures of auditory processing during routine clinical evaluations as well as interventions targeting auditory processing differences in autistic children.

Reduced Glx and GABA Inductions in the Anterior Cingulate Cortex and Caudate Nucleus Are Related to Impaired Control of Attention in Attention-Deficit/Hyperactivity Disorder.

Attention-deficit/hyperactivity disorder (ADHD) is a neurodevelopmental disorder that impairs the control of attention and behavioral inhibition in affected individuals. Recent genome-wide association findings have revealed an association between glutamate and GABA gene sets and ADHD symptoms. Consistently, people with ADHD show altered glutamate and GABA content in the brain circuitry that is important for attention control function. Yet, it remains unknown how glutamate and GABA content in the attention control circuitry change when people are controlling their attention, and whether these changes can predict impaired attention control in people with ADHD. To study these questions, we recruited 18 adults with ADHD (31-51 years) and 16 adults without ADHD (28-54 years). We studied glutamate + glutamine (Glx) and GABA content in the fronto-striatal circuitry while participants performed attention control tasks. We found that Glx and GABA concentrations at rest did not differ between participants with ADHD or without ADHD. However, while participants were performing the attention control tasks, participants with ADHD showed smaller Glx and GABA increases than participants without ADHD. Notably, smaller GABA increases in participants with ADHD significantly predicted their poor task performance. Together, these findings provide the first demonstration showing that attention control deficits in people with ADHD may be related to insufficient responses of the GABAergic system in the fronto-striatal circuitry.

Audiovisual Speech Processing in Relationship to Phonological and Vocabulary Skills in First Graders.

PURPOSE: It is generally accepted that adults use visual cues to improve speech intelligibility in noisy environments, but findings regarding visual speech benefit in children are mixed. We explored factors that contribute to audiovisual (AV) gain in young children's speech understanding. We examined whether there is an AV benefit to speech-in-noise recognition in children in first grade and if visual salience of phonemes influences their AV benefit. We explored if individual differences in AV speech enhancement could be explained by vocabulary knowledge, phonological awareness, or general psychophysical testing performance. METHOD: Thirty-seven first graders completed online psychophysical experiments. We used an online single-interval, four-alternative forced-choice picture-pointing task with age-appropriate consonant-vowel-consonant words to measure auditory-only, visual-only, and AV word recognition in noise at -2 and -8 dB SNR. We obtained standard measures of vocabulary and phonological awareness and included a general psychophysical test to examine correlations with AV benefits. RESULTS: We observed a significant overall AV gain among children in first grade. This effect was mainly attributed to the benefit at -8 dB SNR, for visually distinct targets. Individual differences were not explained by any of the child variables. Boys showed lower auditory-only performances, leading to significantly larger AV gains. CONCLUSIONS: This study shows AV benefit, of distinctive visual cues, to word recognition in challenging noisy conditions in first graders. The cognitive and linguistic constraints of the task may have minimized the impact of individual differences of vocabulary and phonological awareness on AV benefit. The gender difference should be studied on a larger sample and age range.

Analysis methods for measuring passive auditory fNIRS responses generated by a block-design paradigm.

Significance: Functional near-infrared spectroscopy (fNIRS) is an increasingly popular tool in auditory research, but the range of analysis procedures employed across studies may complicate the interpretation of data. Aim: We aim to assess the impact of different analysis procedures on the morphology, detection, and lateralization of auditory responses in fNIRS. Specifically, we determine whether averaging or generalized linear model (GLM)-based analysis generates different experimental conclusions when applied to a block-protocol design. The impact of parameter selection of GLMs on detecting auditory-evoked responses was also quantified. Approach: 17 listeners were exposed to three commonly employed auditory stimuli: noise, speech, and silence. A block design, comprising sounds of 5 s duration and 10 to 20 s silent intervals, was employed. Results: Both analysis procedures generated similar response morphologies and amplitude estimates, and both indicated that responses to speech were significantly greater than to noise or silence. Neither approach indicated a significant effect of brain hemisphere on responses to speech. Methods to correct for systemic hemodynamic responses using short channels improved detection at the individual level. Conclusions: Consistent with theoretical considerations, simulations, and other experimental domains, GLM and averaging analyses generate the same group-level experimental conclusions. We release this dataset publicly for use in future development and optimization of algorithms.

Prevalence and patterns of sensory processing behaviors in a large clinical sample of children with prenatal alcohol exposure.

BACKGROUND: Atypical behavioral responses to sensation are reported in a large proportion of children affected by prenatal alcohol exposure (PAE). Systematic examination of symptoms across the fetal alcohol spectrum in a large clinical sample is needed to inform diagnosis and intervention. AIMS: To describe the prevalence and patterns of atypical sensory processing symptoms in a clinical sample of children with PAE. METHODS: Retrospective analysis of diagnostic clinical data from the University of Washington Fetal Alcohol Syndrome Diagnostic and Prevention Network (FASDPN). Participants were ages 3 through 11 years, had a diagnosis on the fetal alcohol spectrum, and Short Sensory Profile (SSP) assessment. The proportions of children categorized with definite differences on the SSP across selected clinical and demographic features were examined with chi-square analyses. OUTCOMES: The sample consisted of 325 children; 73.2 % had SSP total scores in the definite difference range. Atypical sensory processing symptoms were significantly more prevalent among children with higher reported levels of PAE. The prevalence of atypical symptoms was comparably high across age, levels of diagnostic severity, and other prenatal/postnatal risks. CONCLUSIONS: Results lend support for altered sensory processing as another domain of brain function affected by the teratogenic impact of PAE, guiding clinical work and research.

Listening Difficulties in Children With Fetal Alcohol Spectrum Disorders: More Than a Problem of Audibility.

Purpose Data from standardized caregiver questionnaires indicate that children with fetal alcohol spectrum disorders (FASDs) frequently exhibit atypical auditory behaviors, including reduced responsivity to spoken stimuli. Another body of evidence suggests that prenatal alcohol exposure may result in auditory dysfunction involving loss of audibility (i.e., hearing loss) and/or impaired processing of clearly audible, "suprathreshold" sounds necessary for sound-in-noise listening. Yet, the nexus between atypical auditory behavior and underlying auditory dysfunction in children with FASDs remains largely unexplored. Method To investigate atypical auditory behaviors in FASDs and explore their potential physiological bases, we examined clinical data from 325 children diagnosed with FASDs at the University of Washington using the FASD 4-Digit Diagnostic Code. Atypical behaviors reported on the "auditory filtering" domain of the Short Sensory Profile were assessed to document their prevalence across FASD diagnoses and explore their relationship to reported hearing loss and/or central nervous system measures of cognition, attention, and language function that may indicate suprathreshold processing deficits. Results Atypical auditory behavior was reported among 80% of children with FASDs, a prevalence that did not vary by FASD diagnostic severity or hearing status but was positively correlated with attention-deficit/hyperactivity disorder. In contrast, hearing loss was documented in the clinical records of 40% of children with fetal alcohol syndrome (FAS; a diagnosis on the fetal alcohol spectrum characterized by central nervous system dysfunction, facial dysmorphia, and growth deficiency), 16-fold more prevalent than for those with less severe FASDs (2.4%). Reported hearing loss was significantly associated with physical features characteristic of FAS. Conclusion Children with FAS but not other FASDs may be at a particular risk for hearing loss. However, listening difficulties in the absence of hearing loss-presumably related to suprathreshold processing deficits-are prevalent across the entire fetal alcohol spectrum. The nature and impact of both listening difficulties and hearing loss in FASDs warrant further investigation.

Neural Switch Asymmetry in Feature-Based Auditory Attention Tasks.

Active listening involves dynamically switching attention between competing talkers and is essential to following conversations in everyday environments. Previous investigations in human listeners have examined the neural mechanisms that support switching auditory attention within the acoustic featural cues of pitch and auditory space. Here, we explored the cortical circuitry underlying endogenous switching of auditory attention between pitch and spatial cues necessary to discern target from masker words. Because these tasks are of unequal difficulty, we expected an asymmetry in behavioral switch costs for hard-to-easy versus easy-to-hard switches, mirroring prior evidence from vision-based cognitive task-switching paradigms. We investigated the neural correlates of this behavioral switch asymmetry and associated cognitive control operations in the present auditory paradigm. Behaviorally, we observed no switch-cost asymmetry, i.e., no performance difference for switching from the more difficult attend-pitch to the easier attend-space condition (P→S) versus switching from easy-to-hard (S→P). However, left lateral prefrontal cortex activity, correlated with improved performance, was observed during a silent gap period when listeners switched attention from P→S, relative to switching within pitch cues. No such differential activity was seen for the analogous easy-to-hard switch. We hypothesize that this neural switch asymmetry reflects proactive cognitive control mechanisms that successfully reconfigured neurally-specified task parameters and resolved competition from other such "task sets," thereby obviating the expected behavioral switch-cost asymmetry. The neural switch activity observed was generally consistent with that seen in cognitive paradigms, suggesting that established cognitive models of attention switching may be productively applied to better understand similar processes in audition.

Investigating the fit between phonological feature systems and brain responses to speech using EEG.

This paper describes a technique to assess the correspondence between patterns of similarity in the brain's response to speech sounds and the patterns of similarity encoded in phonological feature systems, by quantifying the recoverability of phonological features from the neural data using supervised learning. The technique is applied to EEG recordings collected during passive listening to consonant-vowel syllables. Three published phonological feature systems are compared, and are shown to differ in their ability to recover certain speech sound contrasts from the neural data. For the phonological feature system that best reflects patterns of similarity in the neural data, a leave-one-out analysis indicates some consistency across subjects in which features have greatest impact on the fit, but considerable across-subject heterogeneity remains in the rank ordering of features in this regard.

Auditory attention switching with listening difficulty: Behavioral and pupillometric measures.

Pupillometry has emerged as a useful tool for studying listening effort. Past work involving listeners with normal audiological thresholds has shown that switching attention between competing talker streams evokes pupil dilation indicative of listening effort [McCloy, Lau, Larson, Pratt, and Lee (2017). J. Acoust. Soc. Am. 141(4), 2440-2451]. The current experiment examines behavioral and pupillometric data from a two-stream target detection task requiring attention-switching between auditory streams, in two participant groups: audiometrically normal listeners who self-report difficulty localizing sound sources and/or understanding speech in reverberant or acoustically crowded environments, and their age-matched controls who do not report such problems. Three experimental conditions varied the number and type of stream segregation cues available. Participants who reported listening difficulty showed both behavioral and pupillometric signs of increased effort compared to controls, especially in trials where listeners had to switch attention between streams, or trials where only a single stream segregation cue was available.

Auditory Brainstem Responses to Continuous Natural Speech in Human Listeners.

Speech is an ecologically essential signal, whose processing crucially involves the subcortical nuclei of the auditory brainstem, but there are few experimental options for studying these early responses in human listeners under natural conditions. While encoding of continuous natural speech has been successfully probed in the cortex with neurophysiological tools such as electroencephalography (EEG) and magnetoencephalography, the rapidity of subcortical response components combined with unfavorable signal-to-noise ratios signal-to-noise ratio has prevented application of those methods to the brainstem. Instead, experiments have used thousands of repetitions of simple stimuli such as clicks, tone-bursts, or brief spoken syllables, with deviations from those paradigms leading to ambiguity in the neural origins of measured responses. In this study we developed and tested a new way to measure the auditory brainstem response (ABR) to ongoing, naturally uttered speech, using EEG to record from human listeners. We found a high degree of morphological similarity between the speech-derived ABRs and the standard click-evoked ABR, in particular, a preserved Wave V, the most prominent voltage peak in the standard click-evoked ABR. Because this method yields distinct peaks that recapitulate the canonical ABR, at latencies too short to originate from the cortex, the responses measured can be unambiguously determined to be subcortical in origin. The use of naturally uttered speech to measure the ABR allows the design of engaging behavioral tasks, facilitating new investigations of the potential effects of cognitive processes like language and attention on brainstem processing.

Integration of Visual Information in Auditory Cortex Promotes Auditory Scene Analysis through Multisensory Binding.

How and where in the brain audio-visual signals are bound to create multimodal objects remains unknown. One hypothesis is that temporal coherence between dynamic multisensory signals provides a mechanism for binding stimulus features across sensory modalities. Here, we report that when the luminance of a visual stimulus is temporally coherent with the amplitude fluctuations of one sound in a mixture, the representation of that sound is enhanced in auditory cortex. Critically, this enhancement extends to include both binding and non-binding features of the sound. We demonstrate that visual information conveyed from visual cortex via the phase of the local field potential is combined with auditory information within auditory cortex. These data provide evidence that early cross-sensory binding provides a bottom-up mechanism for the formation of cross-sensory objects and that one role for multisensory binding in auditory cortex is to support auditory scene analysis.

Pupillometry shows the effort of auditory attention switching.

Successful speech communication often requires selective attention to a target stream amidst competing sounds, as well as the ability to switch attention among multiple interlocutors. However, auditory attention switching negatively affects both target detection accuracy and reaction time, suggesting that attention switches carry a cognitive cost. Pupillometry is one method of assessing mental effort or cognitive load. Two experiments were conducted to determine whether the effort associated with attention switches is detectable in the pupillary response. In both experiments, pupil dilation, target detection sensitivity, and reaction time were measured; the task required listeners to either maintain or switch attention between two concurrent speech streams. Secondary manipulations explored whether switch-related effort would increase when auditory streaming was harder. In experiment 1, spatially distinct stimuli were degraded by simulating reverberation (compromising across-time streaming cues), and target-masker talker gender match was also varied. In experiment 2, diotic streams separable by talker voice quality and pitch were degraded by noise vocoding, and the time alloted for mid-trial attention switching was varied. All trial manipulations had some effect on target detection sensitivity and/or reaction time; however, only the attention-switching manipulation affected the pupillary response: greater dilation was observed in trials requiring switching attention between talkers.

Temporal alignment of pupillary response with stimulus events via deconvolution.

Analysis of pupil dilation has been used as an index of attentional effort in the auditory domain. Previous work has modeled the pupillary response to attentional effort as a linear time-invariant system with a characteristic impulse response, and used deconvolution to estimate the attentional effort that gives rise to changes in pupil size. Here it is argued that one parameter of the impulse response (the latency of response maximum, t(max)) has been mis-estimated in the literature; a different estimate is presented, and it is shown how deconvolution with this value of t(max) yields more intuitively plausible and informative results.

Defining Auditory-Visual Objects: Behavioral Tests and Physiological Mechanisms.

Crossmodal integration is a term applicable to many phenomena in which one sensory modality influences task performance or perception in another sensory modality. We distinguish the term binding as one that should be reserved specifically for the process that underpins perceptual object formation. To unambiguously differentiate binding form other types of integration, behavioral and neural studies must investigate perception of a feature orthogonal to the features that link the auditory and visual stimuli. We argue that supporting true perceptual binding (as opposed to other processes such as decision-making) is one role for cross-sensory influences in early sensory cortex. These early multisensory interactions may therefore form a physiological substrate for the bottom-up grouping of auditory and visual stimuli into auditory-visual (AV) objects.

Auditory attention strategy depends on target linguistic properties and spatial configuration.

Whether crossing a busy intersection or attending a large dinner party, listeners sometimes need to attend to multiple spatially distributed sound sources or streams concurrently. How they achieve this is not clear-some studies suggest that listeners cannot truly simultaneously attend to separate streams, but instead combine attention switching with short-term memory to achieve something resembling divided attention. This paper presents two oddball detection experiments designed to investigate whether directing attention to phonetic versus semantic properties of the attended speech impacts listeners' ability to divide their auditory attention across spatial locations. Each experiment uses four spatially distinct streams of monosyllabic words, variation in cue type (providing phonetic or semantic information), and requiring attention to one or two locations. A rapid button-press response paradigm is employed to minimize the role of short-term memory in performing the task. Results show that differences in the spatial configuration of attended and unattended streams interact with linguistic properties of the speech streams to impact performance. Additionally, listeners may leverage phonetic information to make oddball detection judgments even when oddballs are semantically defined. Both of these effects appear to be mediated by the overall complexity of the acoustic scene.

Leveraging anatomical information to improve transfer learning in brain-computer interfaces.

OBJECTIVE: Brain-computer interfaces (BCIs) represent a technology with the potential to rehabilitate a range of traumatic and degenerative nervous system conditions but require a time-consuming training process to calibrate. An area of BCI research known as transfer learning is aimed at accelerating training by recycling previously recorded training data across sessions or subjects. Training data, however, is typically transferred from one electrode configuration to another without taking individual head anatomy or electrode positioning into account, which may underutilize the recycled data. APPROACH: We explore transfer learning with the use of source imaging, which estimates neural activity in the cortex. Transferring estimates of cortical activity, in contrast to scalp recordings, provides a way to compensate for variability in electrode positioning and head morphologies across subjects and sessions. MAIN RESULTS: Based on simulated and measured electroencephalography activity, we trained a classifier using data transferred exclusively from other subjects and achieved accuracies that were comparable to or surpassed a benchmark classifier (representative of a real-world BCI). Our results indicate that classification improvements depend on the number of trials transferred and the cortical region of interest. SIGNIFICANCE: These findings suggest that cortical source-based transfer learning is a principled method to transfer data that improves BCI classification performance and provides a path to reduce BCI calibration time.