Common and distinct neural representations of imagined and perceived speech.

Humans excel at constructing mental representations of speech streams in the absence of external auditory input: the internal experience of speech imagery. Elucidating the neural processes underlying speech imagery is critical to understanding this higher-order brain function in humans. Here, using functional magnetic resonance imaging, we investigated the shared and distinct neural correlates of imagined and perceived speech by asking participants to listen to poems articulated by a male voice (perception condition) and to imagine hearing poems spoken by that same voice (imagery condition). We found that compared to baseline, speech imagery and perception activated overlapping brain regions, including the bilateral superior temporal gyri and supplementary motor areas. The left inferior frontal gyrus was more strongly activated by speech imagery than by speech perception, suggesting functional specialization for generating speech imagery. Although more research with a larger sample size and a direct behavioral indicator is needed to clarify the neural systems underlying the construction of complex speech imagery, this study provides valuable insights into the neural mechanisms of the closely associated but functionally distinct processes of speech imagery and perception.

Age differentially modulates the cortical tracking of the lower and higher level linguistic structures during speech comprehension.

Speech comprehension requires listeners to rapidly parse continuous speech into hierarchically-organized linguistic structures (i.e. syllable, word, phrase, and sentence) and entrain the neural activities to the rhythm of different linguistic levels. Aging is accompanied by changes in speech processing, but it remains unclear how aging affects different levels of linguistic representation. Here, we recorded magnetoencephalography signals in older and younger groups when subjects actively and passively listened to the continuous speech in which hierarchical linguistic structures of word, phrase, and sentence were tagged at 4, 2, and 1 Hz, respectively. A newly-developed parameterization algorithm was applied to separate the periodically linguistic tracking from the aperiodic component. We found enhanced lower-level (word-level) tracking, reduced higher-level (phrasal- and sentential-level) tracking, and reduced aperiodic offset in older compared with younger adults. Furthermore, we observed the attentional modulation on the sentential-level tracking being larger for younger than for older ones. Notably, the neuro-behavior analyses showed that subjects' behavioral accuracy was positively correlated with the higher-level linguistic tracking, reversely correlated with the lower-level linguistic tracking. Overall, these results suggest that the enhanced lower-level linguistic tracking, reduced higher-level linguistic tracking and less flexibility of attentional modulation may underpin aging-related decline in speech comprehension.

Disrupted neural tracking of sound localization during non-rapid eye movement sleep.

Spatial hearing in humans is a high-level auditory process that is crucial to rapid sound localization in the environment. Both neurophysiological models with animals and neuroimaging evidence from human subjects in the wakefulness stage suggest that the localization of auditory objects is mainly located in the posterior auditory cortex. However, whether this cognitive process is preserved during sleep remains unclear. To fill this research gap, we investigated the sleeping brain's capacity to identify sound locations by recording simultaneous electroencephalographic (EEG) and magnetoencephalographic (MEG) signals during wakefulness and non-rapid eye movement (NREM) sleep in human subjects. Using the frequency-tagging paradigm, the subjects were presented with a basic syllable sequence at 5 Hz and a location change that occurred every three syllables, resulting in a sound localization shift at 1.67 Hz. The EEG and MEG signals were used for sleep scoring and neural tracking analyses, respectively. Neural tracking responses at 5 Hz reflecting basic auditory processing were observed during both wakefulness and NREM sleep, although the responses during sleep were weaker than those during wakefulness. Cortical responses at 1.67 Hz, which correspond to the sound location change, were observed during wakefulness regardless of attention to the stimuli but vanished during NREM sleep. These results for the first time indicate that sleep preserves basic auditory processing but disrupts the higher-order brain function of sound localization.

Neural representations of imagined speech revealed by frequency-tagged magnetoencephalography responses.

Speech mental imagery is a quasi-perceptual experience that occurs in the absence of real speech stimulation. How imagined speech with higher-order structures such as words, phrases and sentences is rapidly organized and internally constructed remains elusive. To address this issue, subjects were tasked with imagining and perceiving poems along with a sequence of reference sounds with a presentation rate of 4 Hz while magnetoencephalography (MEG) recording was conducted. Giving that a sentence in a traditional Chinese poem is five syllables, a sentential rhythm was generated at a distinctive frequency of 0.8 Hz. Using the frequency tagging we concurrently tracked the neural processing timescale to the top-down generation of rhythmic constructs embedded in speech mental imagery and the bottom-up sensory-driven activity that were precisely tagged at the sentence-level rate of 0.8 Hz and a stimulus-level rate of 4 Hz, respectively. We found similar neural responses induced by the internal construction of sentences from syllables with both imagined and perceived poems and further revealed shared and distinct cohorts of cortical areas corresponding to the sentence-level rhythm in imagery and perception. This study supports the view of a common mechanism between imagery and perception by illustrating the neural representations of higher-order rhythmic structures embedded in imagined and perceived speech.

Studying rhythm processing in speech through the lens of auditory-motor synchronization.

Continuous speech is organized into a hierarchy of rhythms. Accurate processing of this rhythmic hierarchy through the interactions of auditory and motor systems is fundamental to speech perception and production. In this mini-review, we aim to evaluate the implementation of behavioral auditory-motor synchronization paradigms when studying rhythm processing in speech. First, we present an overview of the classic finger-tapping paradigm and its application in revealing differences in auditory-motor synchronization between the typical and clinical populations. Next, we highlight key findings on rhythm hierarchy processing in speech and non-speech stimuli from finger-tapping studies. Following this, we discuss the potential caveats of the finger-tapping paradigm and propose the speech-speech synchronization (SSS) task as a promising tool for future studies. Overall, we seek to raise interest in developing new methods to shed light on the neural mechanisms of speech processing.

Reduced neural tracking of speech linguistic structures in children.

The adult brain can efficiently track both lower-level (i.e., syllable) and higher-level (i.e., phrase) linguistic structures to comprehend speech. When children actively or passively listened to speech, we found robust neural tracking of syllabic structure but marginally significant tracking of phrasal structure.

Neural Signatures of Hierarchical Linguistic Structures in Second Language Listening Comprehension.

Native speakers excel at parsing continuous speech into smaller elements and entraining their neural activities to the linguistic hierarchy at different levels (e.g., syllables, phrases, and sentences) to achieve speech comprehension. However, how a nonnative brain tracks hierarchical linguistic structures in second language (L2) speech comprehension and whether it relates to top-down attention and language proficiency remains elusive. Here, we applied a frequency-tagging paradigm in human adults and investigated the neural tracking responses to hierarchically organized linguistic structures (i.e., the syllabic rate of 4 Hz, the phrasal rate of 2 Hz, and the sentential rate of 1 Hz) in both first language (L1) and L2 listeners when they attended to a speech stream or ignored it. We revealed disrupted neural responses to higher-order linguistic structures (i.e., phrases and sentences) for L2 listeners in which the phrasal-level tracking was functionally related to an L2 subject's language proficiency. We also observed less efficient top-down modulation of attention in L2 speech comprehension than in L1 speech comprehension. Our results indicate that the reduced δ-band neuronal oscillations that subserve the internal construction of higher-order linguistic structures may compromise listening comprehension in a nonnative language.

An artificial intelligence-based pipeline for automated detection and localisation of epileptic sources from magnetoencephalography.

Objective.Magnetoencephalography (MEG) is a powerful non-invasive diagnostic modality for presurgical epilepsy evaluation. However, the clinical utility of MEG mapping for localising epileptic foci is limited by its low efficiency, high labour requirements, and considerable interoperator variability. To address these obstacles, we proposed a novel artificial intelligence-based automated magnetic source imaging (AMSI) pipeline for automated detection and localisation of epileptic sources from MEG data.Approach.To expedite the analysis of clinical MEG data from patients with epilepsy and reduce human bias, we developed an autolabelling method, a deep-learning model based on convolutional neural networks and a hierarchical clustering method based on a perceptual hash algorithm, to enable the coregistration of MEG and magnetic resonance imaging, the detection and clustering of epileptic activity, and the localisation of epileptic sources in a highly automated manner. We tested the capability of the AMSI pipeline by assessing MEG data from 48 epilepsy patients.Main results.The AMSI pipeline was able to rapidly detect interictal epileptiform discharges with 93.31% ± 3.87% precision based on a 35-patient dataset (with sevenfold patientwise cross-validation) and robustly rendered accurate localisation of epileptic activity with a lobar concordance of 87.18% against interictal and ictal stereo-electroencephalography findings in a 13-patient dataset. We also showed that the AMSI pipeline accomplishes the necessary processes and delivers objective results within a much shorter time frame (∼12 min) than traditional manual processes (∼4 h).Significance.The AMSI pipeline promises to facilitate increased utilisation of MEG data in the clinical analysis of patients with epilepsy.

Sensitivity to a break in interaural correlation is co-modulated by intensity level and interaural delay.

This study investigated whether sound intensity affects listeners' sensitivity to a break in interaural correlation (BIC) embedded in wideband noise at different interaural delays. The results show that the detection duration threshold remained stable at the intensity between 60 and 70 dB SPL, but increased in accelerating fashion as the intensity decreased toward 40 dB SPL. Moreover, the threshold elevated linearly as the interaural delay increased from 0 to 4 ms, and the elevation slope became larger as the intensity decreased from 50 to 40 dB SPL. Thus, detecting the BIC is co-modulated by both intensity and interaural delay.

Negative emotions in the target speaker's voice enhance speech recognition under "cocktail-party" environments.

Under a "cocktail-party" environment with simultaneous multiple talkers, recognition of target speech is effectively improved by a number of perceptually unmasking cues. It remains unclear whether emotions embedded in the target-speaker's voice can either improve speech perception alone or interact with other cues facilitating speech perception against a masker background. This study used two target-speaker voices with different emotional valences to examine whether recognition of target speech is modulated by the emotional valence when the target speech and the maskers were perceptually co-located or separated. The results showed that both the speech recognition against the masker background and the separation-induced unmasking effect were higher for the target speaker with a negatively emotional voice than for the target speaker with a positively emotional voice. Moreover, when the negative voice was fear conditioned, the target-speech recognition was further improved against speech informational masking. These results suggested that the emotionally vocal unmasking cue interacts significantly with the perceived spatial-separation unmasking cue, facilitating the unmasking effect against a masking background. Thus, emotional features embedded in the target-speaker's vocal timbre are also useful for unmasking the target speech in "cocktail-party" environments.

Common Brain Substrates Underlying Auditory Speech Priming and Perceived Spatial Separation.

Under a "cocktail party" environment, listeners can utilize prior knowledge of the content and voice of the target speech [i.e., auditory speech priming (ASP)] and perceived spatial separation to improve recognition of the target speech among masking speech. Previous studies suggest that these two unmasking cues are not processed independently. However, it is unclear whether the unmasking effects of these two cues are supported by common neural bases. In the current study, we aimed to first confirm that ASP and perceived spatial separation contribute to the improvement of speech recognition interactively in a multitalker condition and further investigate whether there exist intersectant brain substrates underlying both unmasking effects, by introducing these two unmasking cues in a unified paradigm and using functional magnetic resonance imaging. The results showed that neural activations by the unmasking effects of ASP and perceived separation partly overlapped in brain areas: the left pars triangularis (TriIFG) and orbitalis of the inferior frontal gyrus, left inferior parietal lobule, left supramarginal gyrus, and bilateral putamen, all of which are involved in the sensorimotor integration and the speech production. The activations of the left TriIFG were correlated with behavioral improvements caused by ASP and perceived separation. Meanwhile, ASP and perceived separation also enhanced the functional connectivity between the left IFG and brain areas related to the suppression of distractive speech signals: the anterior cingulate cortex and the left middle frontal gyrus, respectively. Therefore, these findings suggest that the motor representation of speech is important for both the unmasking effects of ASP and perceived separation and highlight the critical role of the left IFG in these unmasking effects in "cocktail party" environments.

Spatiotemporal dynamics of predictive brain mechanisms during speech processing: an MEG study.

Rapid and efficient speech processing benefits from the prediction derived from prior expectations based on the identification of individual words. It is known that speech processing is carried out within a distributed frontotemporal network. However, the spatiotemporal causal dynamics of predictive brain mechanisms in sound-to-meaning mapping within this network remain unclear. Using magnetoencephalography, we adopted a semantic anomaly paradigm which consists of expected, unexpected and time-reversed Mandarin Chinese speech, and localized the effects of violated expectation in frontotemporal brain regions, the sensorimotor cortex and the supramarginal gyrus from 250 ms relative to the target words. By further investigating the causal cortical dynamics, we provided the description of the causal dynamic network within the framework of the dual stream model, and highlighted the importance of the connections within the ventral pathway, the top-down modulation from the left inferior frontal gyrus and the cross-stream integration during the speech processing of violated expectation.

Neural tracking of speech mental imagery during rhythmic inner counting.

The subjective inner experience of mental imagery is among the most ubiquitous human experiences in daily life. Elucidating the neural implementation underpinning the dynamic construction of mental imagery is critical to understanding high-order cognitive function in the human brain. Here, we applied a frequency-tagging method to isolate the top-down process of speech mental imagery from bottom-up sensory-driven activities and concurrently tracked the neural processing time scales corresponding to the two processes in human subjects. Notably, by estimating the source of the magnetoencephalography (MEG) signals, we identified isolated brain networks activated at the imagery-rate frequency. In contrast, more extensive brain regions in the auditory temporal cortex were activated at the stimulus-rate frequency. Furthermore, intracranial stereotactic electroencephalogram (sEEG) evidence confirmed the participation of the inferior frontal gyrus in generating speech mental imagery. Our results indicate that a disassociated neural network underlies the dynamic construction of speech mental imagery independent of auditory perception.

Both Val158Met Polymorphism of Catechol-O-Methyltransferase Gene and Menstrual Cycle Affect Prepulse Inhibition but Not Attentional Modulation of Prepulse Inhibition in Younger-Adult Females.

Prepulse inhibition (PPI) can be modulated by both the Val158Met (rs4680) polymorphism of the Catechol-O-Methyltransferase (COMT) gene and the menstrual-cycle-related hormone fluctuations, each of which affects the subcortical/cortical dopamine metabolism. PPI can also be modulated by attention. The attentional modulation of PPI (AMPPI) is sensitive to psychoses. Whether the Val158Met polymorphism affects the AMPPI in female adults at different menstrual-cycle phases is unknown. This study examined whether AMPPI and/or PPI are affected by the Val158Met polymorphism in 177 younger-adult females whose menstrual cycles were mutually different across the menstruation, proliferative, or secretory phases. The AMPPI was evaluated by comparing PPI under the condition of the auditory precedence-effect-induced perceptual spatial separation between the prepulse stimulus and a masking noise (PPIPSS) against that under the condition of the precedence-effect-induced perceptual spatial co-location (PPIPSC). The results showed that both the menstrual cycle and the COMT Val158Met polymorphism affected both PPIPSC and PPIPSS, but not the AMPPI (difference between PPIPSS and PPIPSC). Moreover, throughout the menstrual cycle, both PPIPSC and PPIPSS decreased monotonously in Val/Val-carrier participants. However, the decreasing pattern was not overserved in either Met/Met-carrier or Met/Val-carrier participants. Thus, in healthy younger-adult females, PPIPSC and PPIPSS, but not the AMPPI, is vulnerable to changes of ovarian hormones, and the COMT Val158Met polymorphism also has a modulating effect on this menstrual-cycle-dependent PPI variation. In contrast, the AMPPI seems to be more steadily trait-based, less vulnerable to ovarian hormone fluctuations, and may be useful in assisting the diagnosis of schizophrenia in female adults.

Emotionally conditioning the target-speech voice enhances recognition of the target speech under "cocktail-party" listening conditions.

Under a noisy "cocktail-party" listening condition with multiple people talking, listeners can use various perceptual/cognitive unmasking cues to improve recognition of the target speech against informational speech-on-speech masking. One potential unmasking cue is the emotion expressed in a speech voice, by means of certain acoustical features. However, it was unclear whether emotionally conditioning a target-speech voice that has none of the typical acoustical features of emotions (i.e., an emotionally neutral voice) can be used by listeners for enhancing target-speech recognition under speech-on-speech masking conditions. In this study we examined the recognition of target speech against a two-talker speech masker both before and after the emotionally neutral target voice was paired with a loud female screaming sound that has a marked negative emotional valence. The results showed that recognition of the target speech (especially the first keyword in a target sentence) was significantly improved by emotionally conditioning the target speaker's voice. Moreover, the emotional unmasking effect was independent of the unmasking effect of the perceived spatial separation between the target speech and the masker. Also, (skin conductance) electrodermal responses became stronger after emotional learning when the target speech and masker were perceptually co-located, suggesting an increase of listening efforts when the target speech was informationally masked. These results indicate that emotionally conditioning the target speaker's voice does not change the acoustical parameters of the target-speech stimuli, but the emotionally conditioned vocal features can be used as cues for unmasking target speech.

Mental imagery of face enhances face-sensitive event-related potentials to ambiguous visual stimuli.

Visual mental imagery forms mental representations of visual objects when correspondent stimuli are absent and shares some characters with visual perception. Both the vertex-positive-potential (VPP) and N170 components of event-related potentials (ERPs) to visual stimuli have a remarkable preference to faces. This study investigated whether visual mental imagery modulates the face-sensitive VPP and/or N170 components. The results showed that with significantly larger amplitudes under the face-imagery condition than the house-imagery condition, the VPP and P2 responses, but not the N170 component, were elicited by phase-randomized ambiguous stimuli. Thus, the brain substrates underlying VPP are not completely identical to those underlying N170, and the VPP/P2 manifestation of the category selectivity in imagery probably reflects an integration of top-down mental imagery signals (from the prefrontal cortex) and bottom-up perception signals (from the early visual cortex) in the occipito-temporal cortex where VPP and P2 originate.

Similar Impacts of the Interaural Delay and Interaural Correlation on Binaural Gap Detection.

The subjective representation of the sounds delivered to the two ears of a human listener is closely associated with the interaural delay and correlation of these two-ear sounds. When the two-ear sounds, e.g., arbitrary noises, arrive simultaneously, the single auditory image of the binaurally identical noises becomes increasingly diffuse, and eventually separates into two auditory images as the interaural correlation decreases. When the interaural delay increases from zero to several milliseconds, the auditory image of the binaurally identical noises also changes from a single image to two distinct images. However, measuring the effect of these two factors on an identical group of participants has not been investigated. This study examined the impacts of interaural correlation and delay on detecting a binaurally uncorrelated fragment (interaural correlation = 0) embedded in the binaurally correlated noises (i.e., binaural gap or break in interaural correlation). We found that the minimum duration of the binaural gap for its detection (i.e., duration threshold) increased exponentially as the interaural delay between the binaurally identical noises increased linearly from 0 to 8 ms. When no interaural delay was introduced, the duration threshold also increased exponentially as the interaural correlation of the binaurally correlated noises decreased linearly from 1 to 0.4. A linear relationship between the effect of interaural delay and that of interaural correlation was described for listeners participating in this study: a 1 ms increase in interaural delay appeared to correspond to a 0.07 decrease in interaural correlation specific to raising the duration threshold. Our results imply that a tradeoff may exist between the impacts of interaural correlation and interaural delay on the subjective representation of sounds delivered to two human ears.

Attentional modulation of the early cortical representation of speech signals in informational or energetic masking.

It is easier to recognize a masked speech when the speech and its masker are perceived as spatially segregated. Using event-related potentials, this study examined how the early cortical representation of speech is affected by different masker types and perceptual locations, when the listener is either passively or actively listening to the target speech syllable. The results showed that the two-talker-speech masker induced a much larger masking effect on the N1/P2 complex than either the steady-state-noise masker or the amplitude-modulated speech-spectrum-noise masker did. Also, a switch from the passive- to active-listening condition enhanced the N1/P2 complex only when the masker was speech. Moreover, under the active-listening condition, perceived separation between target and masker enhanced the N1/P2 complex only when the masker was speech. Thus, when a masker is present, the effect of selective attention to the target-speech signal on the early cortical representation of the speech signal is masker-type dependent.