EEG-based auditory attention decoding with audiovisual speech for hearing-impaired listeners.

Auditory attention decoding (AAD) was used to determine the attended speaker during an auditory selective attention task. However, the auditory factors modulating AAD remained unclear for hearing-impaired (HI) listeners. In this study, scalp electroencephalogram (EEG) was recorded with an auditory selective attention paradigm, in which HI listeners were instructed to attend one of the two simultaneous speech streams with or without congruent visual input (articulation movements), and at a high or low target-to-masker ratio (TMR). Meanwhile, behavioral hearing tests (i.e. audiogram, speech reception threshold, temporal modulation transfer function) were used to assess listeners' individual auditory abilities. The results showed that both visual input and increasing TMR could significantly enhance the cortical tracking of the attended speech and AAD accuracy. Further analysis revealed that the audiovisual (AV) gain in attended speech cortical tracking was significantly correlated with listeners' auditory amplitude modulation (AM) sensitivity, and the TMR gain in attended speech cortical tracking was significantly correlated with listeners' hearing thresholds. Temporal response function analysis revealed that subjects with higher AM sensitivity demonstrated more AV gain over the right occipitotemporal and bilateral frontocentral scalp electrodes.

The Mandarin Chinese auditory emotions stimulus database: A validated set of Chinese pseudo-sentences.

Emotional prosody is fully embedded in language and can be influenced by the linguistic properties of a specific language. Considering the limitations of existing Chinese auditory stimulus database studies, we developed and validated an emotional auditory stimuli database composed of Chinese pseudo-sentences, recorded by six professional actors in Mandarin Chinese. Emotional expressions included happiness, sadness, anger, fear, disgust, pleasant surprise, and neutrality. All emotional categories were vocalized into two types of sentence patterns, declarative and interrogative. In addition, all emotional pseudo-sentences, except for neutral, were vocalized at two levels of emotional intensity: normal and strong. Each recording was validated with 40 native Chinese listeners in terms of the recognition accuracy of the intended emotion portrayal; finally, 4361 pseudo-sentence stimuli were included in the database. Validation of the database using a forced-choice recognition paradigm revealed high rates of emotional recognition accuracy. The detailed acoustic attributes of vocalization were provided and connected to the emotion recognition rates. This corpus could be a valuable resource for researchers and clinicians to explore the behavioral and neural mechanisms underlying emotion processing of the general population and emotional disturbances in neurological, psychiatric, and developmental disorders. The Mandarin Chinese auditory emotion stimulus database is available at the Open Science Framework ( https://osf.io/sfbm6/?view_only=e22a521e2a7d44c6b3343e11b88f39e3 ).

The effect of speech material on the band importance function for Mandarin Chinese.

Speech material influences the relative contributions of different frequency regions to intelligibility for English. In the current study, whether a similar effect of speech material is present for Mandarin Chinese was investigated. Speech recognition was measured using three speech materials in Mandarin, including disyllabic words, nonsense sentences, and meaningful sentences. These materials differed from one another in terms of the amount of contextual information and word frequency. The band importance function (BIF), as defined under the Speech Intelligibility Index (SII) framework, was used to quantify the contributions across frequency regions. The BIFs for the three speech materials were estimated from 16 adults who were native speakers of Mandarin. A Bayesian adaptive procedure was used to efficiently estimate the octave-frequency BIFs for the three materials for each listener. As the amount of contextual information increased, low-frequency bands (e.g., 250 and 500 Hz) became more important for speech recognition, consistent with English. The BIF was flatter for Mandarin than for comparable English speech materials. Introducing the language- and material-specific BIFs to the SII model led to improved predictions of Mandarin speech-recognition performance. Results suggested the necessity of developing material-specific BIFs for Mandarin.

The effect of F0 contour on the intelligibility of speech in the presence of interfering sounds for Mandarin Chinese.

In Mandarin Chinese, the fundamental frequency (F0) contour defines lexical "Tones" that differ in meaning despite being phonetically identical. Flattening the F0 contour impairs the intelligibility of Mandarin Chinese in background sounds. This might occur because the flattening introduces misleading lexical information. To avoid this effect, two types of speech were used: single-Tone speech contained Tones 1 and 0 only, which have a flat F0 contour; multi-Tone speech contained all Tones and had a varying F0 contour. The intelligibility of speech in steady noise was slightly better for single-Tone speech than for multi-Tone speech. The intelligibility of speech in a two-talker masker, with the difference in mean F0 between the target and masker matched across conditions, was worse for the multi-Tone target in the multi-Tone masker than for any other combination of target and masker, probably because informational masking was maximal for this combination. The introduction of a perceived spatial separation between the target and masker, via the precedence effect, led to better performance for all target-masker combinations, especially the multi-Tone target in the multi-Tone masker. In summary, a flat F0 contour does not reduce the intelligibility of Mandarin Chinese when the introduction of misleading lexical cues is avoided.

Individually tailored spectral-change enhancement for the hearing impaired.

An algorithm for enhancing spectral changes over time was previously shown to improve the intelligibility of speech in steady speech-spectrum noise (SSN) for hearing-impaired subjects but tended to impair intelligibility for speech in a background of two-talker speech. Large individual differences were found and the application of a genetic algorithm for selecting the "best" parameter values for each listener was found to be beneficial. In the present study, the spectral-change enhancement (SCE) processing was modified by individually tailoring the degree of SCE based on the frequency-dependent hearing loss of the subjects, and by using finer frequency resolution. The effect of the modified SCE processing on the intelligibility and quality of speech in SSN and babble noise (BBN) was evaluated. Ten subjects with mild to moderate hearing loss were tested twice for all tests. The SCE processing led to small but significant improvements in the intelligibility of speech in both SSN and BBN, while the effect of the SCE processing on speech quality was small.

Speaking rhythmically improves speech recognition under "cocktail-party" conditions.

This study examines whether speech rhythm affects speech recognition under "cocktail-party" conditions. Against a two-talker masker, but not a speech-spectrum noise masker, recognition of the last (third) keyword in a normal rhythmic sentence was significantly better than that of the first keyword. However, this word-position-related speech-recognition improvement disappeared for rhythmically hybrid target sentences that were constructed by grouping parts from different sentences with different artificially modulated rhythms (rates) (fast, normal, or slow). Thus, the normal rhythm with a constant rate plays a role in improving speech recognition against informational speech masking, probably through a build-up of temporal prediction for target words.

Rapid tuning of auditory "what" and "where" pathways by training.

Behavioral improvement within the first hour of training is commonly explained as procedural learning (i.e., strategy changes resulting from task familiarization). However, it may additionally reflect a rapid adjustment of the perceptual and/or attentional system in a goal-directed task. In support of this latter hypothesis, we show feature-specific gains in performance for groups of participants briefly trained to use either a spectral or spatial difference between 2 vowels presented simultaneously during a vowel identification task. In both groups, the neuromagnetic activity measured during the vowel identification task following training revealed source activity in auditory cortices, prefrontal, inferior parietal, and motor areas. More importantly, the contrast between the 2 groups revealed a striking double dissociation in which listeners trained on spectral or spatial cues showed higher source activity in ventral ("what") and dorsal ("where") brain areas, respectively. These feature-specific effects indicate that brief training can implicitly bias top-down processing to a trained acoustic cue and induce a rapid recalibration of the ventral and dorsal auditory streams during speech segregation and identification.

[The progress and prospect of prepulse inhibition in autism].

Prepulse inhibition (PPI) is suppression of the startle reflex when an intense startling stimulus is preceded by a weaker sensory stimulus (the prepulse). It is an operational measurement of sensorimotor gating mechanism to help human adapt to complex environment. This weak prepulse protect central cognitive processing by damping the effect of intense stimuli. Autistics cannot select out behaviorally important information from a lot of irrelevant resources and reflect abnormal gating mechanism and attentional abnormalities. Previous studies have not made agreement on whether autistic patients demonstrated deficits in PPI, because the results depend on age, sex, severity of the disease as well as the experimental parameters used. Moreover, these studies have not covered whether autistics have suffered deficits in higher-order processing. In this review, the "top-down" modulation of selective attention and subjective emotion are introduced into the PPI experiment. We also introduce fear conditioning and perceived spatial separation paradigm to further explore the interaction between autistic cognitive process and gating mechanism.

Aging effects on detection of spectral changes induced by a break in sound correlation.

OBJECTIVES: Previous studies have shown that both younger adults and older adults with clinically normal hearing are able to detect a break in correlation (BIC) between interaurally correlated sounds presented over headphones. This ability to detect a BIC improved when the correlated sounds were presented over left and right loudspeakers rather than over left and right headphones, suggesting that additional spectral cues provided by comb filtering (caused by interference between the two channels) facilitate detection of the BIC. However, older adults receive significantly less benefit than younger adults from a switch to loudspeaker presentation. It is hypothesized that this is a result of an age-related reduction in the sensitivity to the monaural spectral cues provided by comb filtering. DESIGN: Two experiments were conducted in this study. Correlated white noises with a BIC in the temporal middle were presented from two spatially separated loudspeakers (positioned at ±45-degree azimuth) and recorded at the right ear of a Knowles Electronic Manikin for Acoustic Research (KEMAR). In Experiment 1, the waveforms recorded at the KEMAR's right ear were presented to the participant's right ear over a headphone in 14 younger adults and 24 older adults with clinically normal hearing. In Experiment 2, 8 of the 14 younger participants participated. Under the monaurally cueing condition, the waveforms recorded at the KEMAR's right ear were presented to the participant's right ear as Experiment 1. Under the binaurally cueing condition, waveforms delivered from the left loudspeaker and those from the right loudspeaker were recorded at the KEMAR's left and right ear, respectively, thereby eliminating the spectral ripple cue, and were presented to the participant's left and right ears, respectively. For each of the two experiments, the break duration threshold for detecting the BIC was examined when the interloudspeaker interval (delay) (ILI) was 0, 1, 2, or 4 msec (left loudspeaker leading). RESULTS: In Experiment 1, both younger participants and older participants detected the BIC in the waveforms recorded by the right ear of KEMAR, but older participants had higher detection thresholds than younger participants when the ILI was 0, 2, or 4 msec without an effect of SPL shift between 59 and 71 dB. In Experiment 2, each of the eight younger participants was able to detect the occurrence of the BIC in either the monaurally cueing or binaural-cueing condition. In addition, the detection threshold under the monaurally cueing condition was substantially the same as that under the binaurally cueing condition at each of the four ILIs. CONCLUSIONS: Younger adults and older adults with clinically normal hearing are able to detect the monaural spectral changes arising from comb filtering when a sudden drop in intersound correlation is introduced. However, younger adults are more sensitive than older adults are, at detecting the BIC. The findings suggest that older adults are less able than younger adults to detect a periodic ripple in the sound spectrum. This age-related ability reduction may contribute to older adults' difficulties in hearing under noisy, reverberant conditions.

Temporally pre-presented lipreading cues release speech from informational masking.

Listeners can use temporally pre-presented content cues and concurrently presented lipreading cues to improve speech recognition under masking conditions. This study investigated whether temporally pre-presented lipreading cues also unmask speech. In a test trial, before the target sentence was co-presented with the masker, either target-matched (priming) lipreading video or static face (priming-control) video was presented in quiet. Participants' target-recognition performance was improved by a shift from the priming-control condition to the priming condition when the masker was speech but not noise. This release from informational masking suggests a combined effect of working memory and cross-modal integration on selective attention to target speech.

[Behavioral and neurobiological abnormalities induced by social isolation as a useful animal model of schizophrenia].

Social isolation influences the development of the brain, causing dysfunctions at behavioral, cellular and molecular levels. The present paper summarizes the abnormalities induced by social isolation in behaviors, neurotransmitters and cell apoptosis. At the behavioral level, social isolation induces hyperlocomotion, abnormalities in startle reflex and prepulse inhibition (PPI), and dysfunctions in conditioned learning, reversal learning and memory. Moreover, social isolation causes changes of neurotransmitters, such as the increase of dopamine in the nucleus accumbens, the amygdala and other brain regions in the limbic system, the decrease of dopamine in medial prefrontal cortex, the decrease of 5-HT in the nucleus accumbens and the hippocampus, and changes of glutamine in the prefrontal cortex. Finally, social isolation affects cell apoptosis in different brain areas, such as the medial prefrontal cortex, amygdala, nucleus accumbens, and hippocampus. Both the changes in neurotransmitters and cell apoptosis may contribute to the behavioral dysfunctions in social isolated rats. Since schizophrenic patients have similar abnormalities in behaviors and neurotransmitters, isolation rearing can be used as a useful animal model of schizophrenia.

Differentially organized top-down modulation of prepulse inhibition of startle.

Prepulse inhibition (PPI) of startle is the suppression of the startle reflex when a weaker sensory stimulus (the prepulse) shortly precedes the startling stimulus. PPI can be attentionally enhanced in both humans and laboratory animals. This study investigated whether the following three forebrain structures, which are critical for initial cortical processing of auditory signals, auditory fear conditioning/memories, and spatial attention, respectively, play a role in the top-down modulation of PPI in rats: the primary auditory cortex (A1), lateral nucleus of the amygdala (LA), and posterior parietal cortex (PPC). The results show that, under the noise-masking condition, PPI was enhanced by fear conditioning of the prepulse in a prepulse-specific manner, and the conditioning-induced PPI enhancement was further increased by perceptual separation between the conditioned prepulse and the noise masker. Reversibly blocking glutamate receptors in the A1 with 2 mm kynurenic acid eliminated both the conditioning-induced and perceptual separation-induced PPI enhancements. Blocking the LA eliminated the conditioning-induced but not the perceptual separation-induced PPI enhancement, and blocking the PPC specifically eliminated the perceptual separation-induced PPI enhancement. The two types of PPI enhancements were also eliminated by the extinction manipulation. Thus, the top-down modulation of PPI is differentially organized and depends on operations of various forebrain structures. Due to the fine-tuned modulation by higher-order cognitive processes, functions of PPI can be more flexible to complex environments. The top-down enhancements of PPI in rats are also useful for modeling some mental disorders, such as schizophrenia, attention deficit/hyperactivity disorder, and posttraumatic stress disorder.

Human auditory cortex activity shows additive effects of spectral and spatial cues during speech segregation.

In noisy social gatherings, listeners perceptually integrate sounds originating from one person's voice (e.g., fundamental frequency (f(0)) and harmonics) at a particular location and segregate these from concurrent sounds of other talkers. Though increasing the spectral or the spatial distance between talkers promotes speech segregation, synergetic effects of spatial and spectral distances are less well understood. We studied how spectral and/or spatial distances between 2 simultaneously presented steady-state vowels contribute to perception and activation in auditory cortex using magnetoencephalography. Participants were more accurate in identifying both vowels when they differed in f(0) and location than when they differed in a single cue only or when they shared the same f(0) and location. The combined effect of f(0) and location differences closely matched the sum of single effects. The improvement in concurrent vowel identification coincided with an object-related negativity that peaked at about 140 ms after vowel onset. The combined effect of f(0) and location closely matched the sum of the single effects even though vowels with different f(0), location, or both generated different time courses of neuromagnetic activity. We propose that during auditory scene analysis, acoustic differences among the various sources are combined linearly to increase the perceptual distance between the co-occurring sound objects.

Sensitivity to a break in interaural correlation is co-modulated by intensity level and interaural delay.

This study investigated whether sound intensity affects listeners' sensitivity to a break in interaural correlation (BIC) embedded in wideband noise at different interaural delays. The results show that the detection duration threshold remained stable at the intensity between 60 and 70 dB SPL, but increased in accelerating fashion as the intensity decreased toward 40 dB SPL. Moreover, the threshold elevated linearly as the interaural delay increased from 0 to 4 ms, and the elevation slope became larger as the intensity decreased from 50 to 40 dB SPL. Thus, detecting the BIC is co-modulated by both intensity and interaural delay.

Informational masking of speech produced by speech-like sounds without linguistic content.

This study investigated whether speech-like maskers without linguistic content produce informational masking of speech. The target stimuli were nonsense Chinese Mandarin sentences. In experiment I, the masker contained harmonics the fundamental frequency (F0) of which was sinusoidally modulated and the mean F0 of which was varied. The magnitude of informational masking was evaluated by measuring the change in intelligibility (releasing effect) produced by inducing a perceived spatial separation of the target speech and masker via the precedence effect. The releasing effect was small and was only clear when the target and masker had the same mean F0, suggesting that informational masking was small. Performance with the harmonic maskers was better than with a steady speech-shaped noise (SSN) masker. In experiments II and III, the maskers were speech-like synthesized signals, alternating between segments with harmonic structure and segments composed of SSN. Performance was much worse than for experiment I, and worse than when an SSN masker was used, suggesting that substantial informational masking occurred. The similarity of the F0 contours of the target and masker had little effect. The informational masking effect was not influenced by whether or not the noise-like segments of the masker were synchronous with the unvoiced segments of the target speech.

Perceptual fusion tendency of speech sounds.

To discriminate and to recognize sound sources in a noisy, reverberant environment, listeners need to perceptually integrate the direct wave with the reflections of each sound source. It has been confirmed that perceptual fusion between direct and reflected waves of a speech sound helps listeners recognize this speech sound in a simulated reverberant environment with disrupting sound sources. When the delay between a direct sound wave and its reflected wave is sufficiently short, the two waves are perceptually fused into a single sound image as coming from the source location. Interestingly, compared with nonspeech sounds such as clicks and noise bursts, speech sounds have a much larger perceptual fusion tendency. This study investigated why the fusion tendency for speech sounds is so large. Here we show that when the temporal amplitude fluctuation of speech was artificially time reversed, a large perceptual fusion tendency of speech sounds disappeared, regardless of whether the speech acoustic carrier was in normal or reversed temporal order. Moreover, perceptual fusion of normal-order speech, but not that of time-reversed speech, was accompanied by increased coactivation of the attention-control-related, spatial-processing-related, and speech-processing-related cortical areas. Thus, speech-like acoustic carriers modulated by speech amplitude fluctuation selectively activate a cortical network for top-down modulations of speech processing, leading to an enhancement of perceptual fusion of speech sounds. This mechanism represents a perceptual-grouping strategy for unmasking speech under adverse conditions.

Improvement of intelligibility of ideal binary-masked noisy speech by adding background noise.

When a target-speech/masker mixture is processed with the signal-separation technique, ideal binary mask (IBM), intelligibility of target speech is remarkably improved in both normal-hearing listeners and hearing-impaired listeners. Intelligibility of speech can also be improved by filling in speech gaps with un-modulated broadband noise. This study investigated whether intelligibility of target speech in the IBM-treated target-speech/masker mixture can be further improved by adding a broadband-noise background. The results of this study show that following the IBM manipulation, which remarkably released target speech from speech-spectrum noise, foreign-speech, or native-speech masking (experiment 1), adding a broadband-noise background with the signal-to-noise ratio no less than 4 dB significantly improved intelligibility of target speech when the masker was either noise (experiment 2) or speech (experiment 3). The results suggest that since adding the noise background shallows the areas of silence in the time-frequency domain of the IBM-treated target-speech/masker mixture, the abruption of transient changes in the mixture is smoothed and the perceived continuity of target-speech components becomes enhanced, leading to improved target-speech intelligibility. The findings are useful for advancing computational auditory scene analysis, hearing-aid/cochlear-implant designs, and understanding of speech perception under "cocktail-party" conditions.

Categorical perception of lexical tones based on acoustic-electric stimulation.

The effect of low-frequency acoustic input on the categorical perception of lexical tones was investigated with simulated electric-acoustic hearing. A synthesized T1-T2 (flat-rising) tone continuum of Mandarin monosyllables /i/ was used, and they were manipulated as five conditions: unprocessed, low-frequency acoustic-only, electric-only, electric-acoustic stimulation, and bimodal stimulation. Results showed the performance under electric-only condition was the significantly lowest, and the difference of other pairwise comparisons between conditions was quite small. These findings suggest that the low-frequency acoustic input can shape the categorical perception, and the combinations of acoustic and electric hearing within or across ears have no significant effect.

The Associations between Sleep Duration, Academic Pressure, and Depressive Symptoms among Chinese Adolescents: Results from China Family Panel Studies.

Depressive symptoms are a common mental health problem among adolescents, which may affect their physical and mental health development and impose heavy burdens on individual families and society. This study aimed to examine the associations between sleep duration, academic pressure, and depressive symptoms among Chinese adolescents and to construct the mediation model to explore the mediating effect of sleep duration. The data are from the China Family Panel Studies (CFPS). Methodologically, the aforementioned associations were explored by constructing a structural equation model and applying multivariate multilevel logistic regression. In this study, we found that approximately 6.49% of the 3724 Chinese adolescents had depressive symptoms. Sleep duration of <6 h/night (OR = 2.39, 95%CI = 1.33-4.32) and high/maximum academic pressure (high: OR = 1.43, 95%CI = 1.02-1.99; maximum: OR = 2.43, 95%CI = 1.58-3.73) were both associated with an increased risk of depressive symptoms in adolescents. Meanwhile, the multiplicative interaction between sleep duration and academic pressure was significantly associated with depressive symptoms in adolescents (p < 0.001). The sleep duration played a partial mediating role in the relationship between academic pressure and depressive symptoms (a*b = 0.006, 95%BootCI = 0.001-0.012). Our study highlights that it is essential to mitigate the academic pressure of adolescents to increase their sleep duration and further reduce the occurrence of depressive symptoms by adopting corresponding preventive measures.

Trajectory patterns of blood pressure change up to six years and the risk of dementia: a nationwide cohort study.

The present study aimed to investigate the associations between the trajectory of blood pressure (BP) change and the risk of subsequent dementia and to explore the differences in age, gender, and hypertension subgroups. We included 10,660 participants aged ≥ 60 years from 1998 to 2018 waves of the Chinese Longitudinal Healthy Longevity Survey. Latent growth mixture models were used to estimate BP trajectories. Cox-proportional hazard models were used to analyze the effects of BP trajectories on the risk of dementia. According to the results, stabilized systolic BP (SBP) was found to be associated with a higher risk of dementia compared with normal SBP [adjusted hazard ratio (aHR): 1.62; 95% confidence interval (CI): 1.27-2.07] and elevated SBP (aHR: 2.22; 95% CI: 1.51-3.28) in and only in the subgroups of the oldest-old, women, and subjects without hypertension at baseline. Similarly, stabilized pulse pressure (PP) was associated with a higher risk of dementia compared with normal PP (aHR: 1.52; 95% CI: 1.24-1.88) and elevated PP (aHR: 2.12; 95% CI: 1.48-3.04) in and only in the subgroups of the oldest-old, women, and subjects with hypertension at baseline. These findings suggest that stabilized SBP and PP have predictive significance for the occurrence of dementia in late life, and the factors of age, gender, and late-life hypertension should be considered when estimating the risk of BP decline on dementia.