Neural correlates of audiovisual narrative speech perception in children and adults on the autism spectrum: A functional magnetic resonance imaging study.

Autistic individuals show substantially reduced benefit from observing visual articulations during audiovisual speech perception, a multisensory integration deficit that is particularly relevant to social communication. This has mostly been studied using simple syllabic or word-level stimuli and it remains unclear how altered lower-level multisensory integration translates to the processing of more complex natural multisensory stimulus environments in autism. Here, functional neuroimaging was used to examine neural correlates of audiovisual gain (AV-gain) in 41 autistic individuals to those of 41 age-matched non-autistic controls when presented with a complex audiovisual narrative. Participants were presented with continuous narration of a story in auditory-alone, visual-alone, and both synchronous and asynchronous audiovisual speech conditions. We hypothesized that previously identified differences in audiovisual speech processing in autism would be characterized by activation differences in brain regions well known to be associated with audiovisual enhancement in neurotypicals. However, our results did not provide evidence for altered processing of auditory alone, visual alone, audiovisual conditions or AV- gain in regions associated with the respective task when comparing activation patterns between groups. Instead, we found that autistic individuals responded with higher activations in mostly frontal regions where the activation to the experimental conditions was below baseline (de-activations) in the control group. These frontal effects were observed in both unisensory and audiovisual conditions, suggesting that these altered activations were not specific to multisensory processing but reflective of more general mechanisms such as an altered disengagement of Default Mode Network processes during the observation of the language stimulus across conditions.

Exploring the neural underpinnings of chord prediction uncertainty: an electroencephalography (EEG) study.

Predictive processing in the brain, involving interaction between interoceptive (bodily signal) and exteroceptive (sensory) processing, is essential for understanding music as it encompasses musical temporality dynamics and affective responses. This study explores the relationship between neural correlates and subjective certainty of chord prediction, focusing on the alignment between predicted and actual chord progressions in both musically appropriate chord sequences and random chord sequences. Participants were asked to predict the final chord in sequences while their brain activity was measured using electroencephalography (EEG). We found that the stimulus preceding negativity (SPN), an EEG component associated with predictive processing of sensory stimuli, was larger for non-harmonic chord sequences than for harmonic chord progressions. Additionally, the heartbeat evoked potential (HEP), an EEG component related to interoceptive processing, was larger for random chord sequences and correlated with prediction certainty ratings. HEP also correlated with the N5 component, found while listening to the final chord. Our findings suggest that HEP more directly reflects the subjective prediction certainty than SPN. These findings offer new insights into the neural mechanisms underlying music perception and prediction, emphasizing the importance of considering auditory prediction certainty when examining the neural basis of music cognition.

Altered hierarchical auditory predictive processing after lesions to the orbitofrontal cortex.

Orbitofrontal cortex (OFC) is classically linked to inhibitory control, emotion regulation, and reward processing. Recent perspectives propose that the OFC also generates predictions about perceptual events, actions, and their outcomes. We tested the role of the OFC in detecting violations of prediction at two levels of abstraction (i.e., hierarchical predictive processing) by studying the event-related potentials (ERPs) of patients with focal OFC lesions (n = 12) and healthy controls (n = 14) while they detected deviant sequences of tones in a local-global paradigm. The structural regularities of the tones were controlled at two hierarchical levels by rules defined at a local (i.e., between tones within sequences) and at a global (i.e., between sequences) level. In OFC patients, ERPs elicited by standard tones were unaffected at both local and global levels compared to controls. However, patients showed an attenuated mismatch negativity (MMN) and P3a to local prediction violation, as well as a diminished MMN followed by a delayed P3a to the combined local and global level prediction violation. The subsequent P3b component to conditions involving violations of prediction at the level of global rules was preserved in the OFC group. Comparable effects were absent in patients with lesions restricted to the lateral PFC, which lends a degree of anatomical specificity to the altered predictive processing resulting from OFC lesion. Overall, the altered magnitudes and time courses of MMN/P3a responses after lesions to the OFC indicate that the neural correlates of detection of auditory regularity violation are impacted at two hierarchical levels of rule abstraction.

Older adults preserve audiovisual integration through enhanced cortical activations, not by recruiting new regions.

Effective interactions with the environment rely on the integration of multisensory signals: Our brains must efficiently combine signals that share a common source, and segregate those that do not. Healthy ageing can change or impair this process. This functional magnetic resonance imaging study assessed the neural mechanisms underlying age differences in the integration of auditory and visual spatial cues. Participants were presented with synchronous audiovisual signals at various degrees of spatial disparity and indicated their perceived sound location. Behaviourally, older adults were able to maintain localisation accuracy. At the neural level, they integrated auditory and visual cues into spatial representations along dorsal auditory and visual processing pathways similarly to their younger counterparts but showed greater activations in a widespread system of frontal, temporal, and parietal areas. According to multivariate Bayesian decoding, these areas encoded critical stimulus information beyond that which was encoded in the brain areas commonly activated by both groups. Surprisingly, however, the boost in information provided by these areas with age-related activation increases was comparable across the 2 age groups. This dissociation-between comparable information encoded in brain activation patterns across the 2 age groups, but age-related increases in regional blood-oxygen-level-dependent responses-contradicts the widespread notion that older adults recruit new regions as a compensatory mechanism to encode task-relevant information. Instead, our findings suggest that activation increases in older adults reflect nonspecific or modulatory mechanisms related to less efficient or slower processing, or greater demands on attentional resources.

Auditory influence on stickiness perception: an fMRI study of multisensory integration.

This study explored how the human brain perceives stickiness through tactile and auditory channels, especially when presented with congruent or incongruent intensity cues. In our behavioral and functional MRI (fMRI) experiments, we presented participants with adhesive tape stimuli at two different intensities. The congruent condition involved providing stickiness stimuli with matching intensity cues in both auditory and tactile channels, whereas the incongruent condition involved cues of different intensities. Behavioral results showed that participants were able to distinguish between the congruent and incongruent conditions with high accuracy. Through fMRI searchlight analysis, we tested which brain regions could distinguish between congruent and incongruent conditions, and as a result, we identified the superior temporal gyrus, known primarily for auditory processing. Interestingly, we did not observe any significant activation in regions associated with somatosensory or motor functions. This indicates that the brain dedicates more attention to auditory cues than to tactile cues, possibly due to the unfamiliarity of conveying the sensation of stickiness through sound. Our results could provide new perspectives on the complexities of multisensory integration, highlighting the subtle yet significant role of auditory processing in understanding tactile properties such as stickiness.

Auditory prediction and prediction error responses evoked through a novel cascade roving paradigm: a human ECoG study.

Auditory sensory processing is assumed to occur in a hierarchical structure including the primary auditory cortex (A1), superior temporal gyrus, and frontal areas. These areas are postulated to generate predictions for incoming stimuli, creating an internal model of the surrounding environment. Previous studies on mismatch negativity have indicated the involvement of the superior temporal gyrus in this processing, whereas reports have been mixed regarding the contribution of the frontal cortex. We designed a novel auditory paradigm, the "cascade roving" paradigm, which incorporated complex structures (cascade sequences) into a roving paradigm. We analyzed electrocorticography data from six patients with refractory epilepsy who passively listened to this novel auditory paradigm and detected responses to deviants mainly in the superior temporal gyrus and inferior frontal gyrus. Notably, the inferior frontal gyrus exhibited broader distribution and sustained duration of deviant-elicited responses, seemingly differing in spatio-temporal characteristics from the prediction error responses observed in the superior temporal gyrus, compared with conventional oddball paradigms performed on the same participants. Moreover, we observed that the deviant responses were enhanced through stimulus repetition in the high-gamma range mainly in the superior temporal gyrus. These features of the novel paradigm may aid in our understanding of auditory predictive coding.

Effects of Stimulus Rate and Periodicity on Auditory Cortical Entrainment to Continuous Sounds.

The neural mechanisms underlying the exogenous coding and neural entrainment to repetitive auditory stimuli have seen a recent surge of interest. However, few studies have characterized how parametric changes in stimulus presentation alter entrained responses. We examined the degree to which the brain entrains to repeated speech (i.e., /ba/) and nonspeech (i.e., click) sounds using phase-locking value (PLV) analysis applied to multichannel human electroencephalogram (EEG) data. Passive cortico-acoustic tracking was investigated in N = 24 normal young adults utilizing EEG source analyses that isolated neural activity stemming from both auditory temporal cortices. We parametrically manipulated the rate and periodicity of repetitive, continuous speech and click stimuli to investigate how speed and jitter in ongoing sound streams affect oscillatory entrainment. Neuronal synchronization to speech was enhanced at 4.5 Hz (the putative universal rate of speech) and showed a differential pattern to that of clicks, particularly at higher rates. PLV to speech decreased with increasing jitter but remained superior to clicks. Surprisingly, PLV entrainment to clicks was invariant to periodicity manipulations. Our findings provide evidence that the brain's neural entrainment to complex sounds is enhanced and more sensitized when processing speech-like stimuli, even at the syllable level, relative to nonspeech sounds. The fact that this specialization is apparent even under passive listening suggests a priority of the auditory system for synchronizing to behaviorally relevant signals.

Spontaneous emergence of rudimentary music detectors in deep neural networks.

Music exists in almost every society, has universal acoustic features, and is processed by distinct neural circuits in humans even with no experience of musical training. However, it remains unclear how these innate characteristics emerge and what functions they serve. Here, using an artificial deep neural network that models the auditory information processing of the brain, we show that units tuned to music can spontaneously emerge by learning natural sound detection, even without learning music. The music-selective units encoded the temporal structure of music in multiple timescales, following the population-level response characteristics observed in the brain. We found that the process of generalization is critical for the emergence of music-selectivity and that music-selectivity can work as a functional basis for the generalization of natural sound, thereby elucidating its origin. These findings suggest that evolutionary adaptation to process natural sounds can provide an initial blueprint for our sense of music.

Neural encoding of musical expectations in a non-human primate.

The appreciation of music is a universal trait of humankind.1,2,3 Evidence supporting this notion includes the ubiquity of music across cultures4,5,6,7 and the natural predisposition toward music that humans display early in development.8,9,10 Are we musical animals because of species-specific predispositions? This question cannot be answered by relying on cross-cultural or developmental studies alone, as these cannot rule out enculturation.11 Instead, it calls for cross-species experiments testing whether homologous neural mechanisms underlying music perception are present in non-human primates. We present music to two rhesus monkeys, reared without musical exposure, while recording electroencephalography (EEG) and pupillometry. Monkeys exhibit higher engagement and neural encoding of expectations based on the previously seeded musical context when passively listening to real music as opposed to shuffled controls. We then compare human and monkey neural responses to the same stimuli and find a species-dependent contribution of two fundamental musical features-pitch and timing12-in generating expectations: while timing- and pitch-based expectations13 are similarly weighted in humans, monkeys rely on timing rather than pitch. Together, these results shed light on the phylogeny of music perception. They highlight monkeys' capacity for processing temporal structures beyond plain acoustic processing, and they identify a species-dependent contribution of time- and pitch-related features to the neural encoding of musical expectations.

Making sense of music: Insights from neurophysiology and connectivity analyses in naturalistic listening conditions.

According to the latest frameworks, auditory perception and memory involve the constant prediction of future sound events by the brain, based on the continuous extraction of feature regularities from the environment. The neural hierarchical mechanisms for predictive processes in perception and memory for sounds are typically studied in relation to simple acoustic features in isolated sounds or sound patterns inserted in highly certain contexts. Such studies have identified reliable prediction formation and error signals, e.g., the N100 or the mismatch negativity (MMN) evoked responses. In real life, though, individuals often face situations in which uncertainty prevails and where making sense of sounds becomes a hard challenge. In music, not only deviations from predictions are masterly set up by composers to induce emotions but sometimes the sheer uncertainty of sound scenes is exploited for aesthetic purposes, especially in compositional styles such as Western atonal classical music. In very recent magnetoencephalography (MEG) and electroencephalography (EEG) studies, experimental and technical advances in stimulation paradigms and analysis approaches have permitted the identification of prediction-error responses from highly uncertain, atonal contexts and the extraction of prediction-related responses from real, continuous music. Moreover, functional connectivity analyses revealed the emergence of cortico-hippocampal interactions during the formation of auditory memories for more predictable vs. less predictable patterns. These findings contribute to understanding the general brain mechanisms that enable us to predict even highly uncertain sound environments and to possibly make sense of and appreciate even atonal music.

Different hemispheric lateralization for periodicity and formant structure of vowels in the auditory cortex and its changes between childhood and adulthood.

The spectral formant structure and periodicity pitch are the major features that determine the identity of vowels and the characteristics of the speaker. However, very little is known about how the processing of these features in the auditory cortex changes during development. To address this question, we independently manipulated the periodicity and formant structure of vowels while measuring auditory cortex responses using magnetoencephalography (MEG) in children aged 7-12 years and adults. We analyzed the sustained negative shift of source current associated with these vowel properties, which was present in the auditory cortex in both age groups despite differences in the transient components of the auditory response. In adults, the sustained activation associated with formant structure was lateralized to the left hemisphere early in the auditory processing stream requiring neither attention nor semantic mapping. This lateralization was not yet established in children, in whom the right hemisphere contribution to formant processing was strong and decreased during or after puberty. In contrast to the formant structure, periodicity was associated with a greater response in the right hemisphere in both children and adults. These findings suggest that left-lateralization for the automatic processing of vowel formant structure emerges relatively late in ontogenesis and pose a serious challenge to current theories of hemispheric specialization for speech processing.

That sounds awful! Does sound unpleasantness modulate the mismatch negativity and its habituation?

There are sounds that most people perceive as highly unpleasant, for instance, the sound of rubbing pieces of polystyrene together. Previous research showed larger physiological and neural responses for such aversive compared to neutral sounds. Hitherto, it remains unclear whether habituation, i.e., diminished responses to repeated stimulus presentation, which is typically reported for neutral sounds, occurs to the same extent for aversive stimuli. We measured the mismatch negativity (MMN) in response to rare occurrences of aversive or neutral deviant sounds within an auditory oddball sequence in 24 healthy participants, while they performed a demanding visual distractor task. Deviants occurred as single events (i.e., between two standards) or as double deviants (i.e., repeating the identical deviant sound in two consecutive trials). All deviants elicited a clear MMN, and amplitudes were larger for aversive than for neutral deviants (irrespective of their position within a deviant pair). This supports the claim of preattentive emotion evaluation during early auditory processing. In contrast to our expectations, MMN amplitudes did not show habituation, but increased in response to deviant repetition-similarly for aversive and neutral deviants. A more fine-grained analysis of individual MMN amplitudes in relation to individual arousal and valence ratings of each sound item revealed that stimulus-specific MMN amplitudes were best predicted by the interaction of deviant position and perceived arousal, but not by valence. Deviants with perceived higher arousal elicited larger MMN amplitudes only at the first deviant position, indicating that the MMN reflects preattentive processing of the emotional content of sounds.

Visual Stimuli Modulate Local Field Potentials But Drive No High-Frequency Activity in Human Auditory Cortex.

Neuroimaging studies suggest cross-sensory visual influences in human auditory cortices (ACs). Whether these influences reflect active visual processing in human ACs, which drives neuronal firing and concurrent broadband high-frequency activity (BHFA; >70 Hz), or whether they merely modulate sound processing is still debatable. Here, we presented auditory, visual, and audiovisual stimuli to 16 participants (7 women, 9 men) with stereo-EEG depth electrodes implanted near ACs for presurgical monitoring. Anatomically normalized group analyses were facilitated by inverse modeling of intracranial source currents. Analyses of intracranial event-related potentials (iERPs) suggested cross-sensory responses to visual stimuli in ACs, which lagged the earliest auditory responses by several tens of milliseconds. Visual stimuli also modulated the phase of intrinsic low-frequency oscillations and triggered 15-30 Hz event-related desynchronization in ACs. However, BHFA, a putative correlate of neuronal firing, was not significantly increased in ACs after visual stimuli, not even when they coincided with auditory stimuli. Intracranial recordings demonstrate cross-sensory modulations, but no indication of active visual processing in human ACs.

Relationship between schematic and dynamic expectations of melodic patterns in music perception.

Prediction is fundamental in music listening. Two types of expectations have been proposed: schematic expectations, which arise from knowledge of tonal regularities (e.g., harmony and key) acquired through long-term plasticity and learning, and dynamic expectations, which arise from short-term regularity representations (e.g., rhythmic patterns and melodic contours) extracted from ongoing musical contexts. Although both expectations are indispensable in music listening, how they interact with each other in music prediction remains unclear. The present study examined the relationship between schematic and dynamic expectations in music processing using event-related potentials (ERPs). At the ending note of the melodies, the schematic expectation was violated by presenting a note with music-syntactic irregular (i.e., outof- key note), while the dynamic expectation was violated by presenting a contour deviant based on online statistical learning of melodic patterns. Schematic and dynamic expectations were manipulated to predict the same note. ERPs were recorded for the music-syntactic irregularity and the contour deviant, which occurred independently or simultaneously. The results showed that the music-syntactic irregularity elicited an early right anterior negativity (ERAN), reflecting the prediction error in the schematic expectation, while the contour deviant elicited a mismatch negativity (MMN), reflecting the prediction error in the dynamic expectation. Both components occurred within a similar latency range. Moreover, the ERP amplitude was multiplicatively increased when the irregularity and deviance occurred simultaneously. These findings suggest that schematic and dynamic expectations function concurrently in an interactive manner when both expectations predict the same note.

Voice categorization in the four-month-old human brain.

Voices are the most relevant social sounds for humans and therefore have crucial adaptive value in development. Neuroimaging studies in adults have demonstrated the existence of regions in the superior temporal sulcus that respond preferentially to voices. Yet, whether voices represent a functionally specific category in the young infant's mind is largely unknown. We developed a highly sensitive paradigm relying on fast periodic auditory stimulation (FPAS) combined with scalp electroencephalography (EEG) to demonstrate that the infant brain implements a reliable preferential response to voices early in life. Twenty-three 4-month-old infants listened to sequences containing non-vocal sounds from different categories presented at 3.33 Hz, with highly heterogeneous vocal sounds appearing every third stimulus (1.11 Hz). We were able to isolate a voice-selective response over temporal regions, and individual voice-selective responses were found in most infants within only a few minutes of stimulation. This selective response was significantly reduced for the same frequency-scrambled sounds, indicating that voice selectivity is not simply driven by the envelope and the spectral content of the sounds. Such a robust selective response to voices as early as 4 months of age suggests that the infant brain is endowed with the ability to rapidly develop a functional selectivity to this socially relevant category of sounds.

Culture influences conscious appraisal of, but not automatic aversion to, acoustically rough musical intervals.

There is debate whether the foundations of consonance and dissonance are rooted in culture or in psychoacoustics. In order to disentangle the contribution of culture and psychoacoustics, we considered automatic responses to the perfect fifth and the major second (flattened by 25 cents) intervals alongside conscious evaluations of the same intervals across two cultures and two levels of musical expertise. Four groups of participants completed the tasks: expert performers of Lithuanian Sutartines, English speaking musicians in Western diatonic genres, Lithuanian non-musicians and English-speaking non-musicians. Sutartines singers were chosen as this style of singing is an example of 'beat diaphony' where intervals of parts form predominantly rough sonorities and audible beats. There was no difference in automatic responses to intervals, suggesting that an aversion to acoustically rough intervals is not governed by cultural familiarity but may have a physical basis in how the human auditory system works. However, conscious evaluations resulted in group differences with Sutartines singers rating both the flattened major as more positive than did other groups. The results are discussed in the context of recent developments in consonance and dissonance research.

The influence of musical parameters and subjective musical ratings on perceptions of culture.

Recent research suggests that music can affect evaluations of other groups and cultures. However, little is known about the objective and subjective musical parameters that influence these evaluations. We aimed to fill this gap through two studies. Study 1 collected responses from 52 American participants who listened to 30 folk-song melodies from different parts of the world. Linear mixed-effects models tested the influence of objective and subjective musical parameters of these melodies on evaluations of the cultures from which they originated. Musical parameters consistently predicted cultural evaluations. The most prominent musical parameter was musical velocity, a measure of number of pitch onsets, predicting more cultural warmth, competence and evolvedness and less cultural threat. Next, with a sample of 212 American participants, Study 2 used a within-subjects experiment to alter the tempo and dissonance for a subset of six melody excerpts from Study 1, testing for causal effects. Linear mixed-effects models revealed that both dissonance and slow tempo predicted more negative cultural evaluations. Together, both studies demonstrate how musical parameters can influence cultural perceptions. Avenues for future research are discussed.

Competing influence of visual speech on auditory neural adaptation.

Visual information from a speaker's face enhances auditory neural processing and speech recognition. To determine whether auditory memory can be influenced by visual speech, the degree of auditory neural adaptation of an auditory syllable preceded by an auditory, visual, or audiovisual syllable was examined using EEG. Consistent with previous findings and additional adaptation of auditory neurons tuned to acoustic features, stronger adaptation of N1, P2 and N2 auditory evoked responses was observed when the auditory syllable was preceded by an auditory compared to a visual syllable. However, although stronger than when preceded by a visual syllable, lower adaptation was observed when the auditory syllable was preceded by an audiovisual compared to an auditory syllable. In addition, longer N1 and P2 latencies were then observed. These results further demonstrate that visual speech acts on auditory memory but suggest competing visual influences in the case of audiovisual stimulation.

Neural signatures of automatic repetition detection in temporally regular and jittered acoustic sequences.

Detection of repeating patterns within continuous sound streams is crucial for efficient auditory perception. Previous studies demonstrated a remarkable sensitivity of the human auditory system to periodic repetitions in unfamiliar, meaningless sounds. Automatic repetition detection was reflected in different EEG markers, including sustained activity, neural synchronisation, and event-related responses to pattern occurrences. The current study investigated how listeners' attention and the temporal regularity of a sound modulate repetition perception, and how this influence is reflected in different EEG markers that were previously suggested to subserve dissociable functions. We reanalysed data of a previous study in which listeners were presented with sequences of unfamiliar artificial sounds that either contained repetitions of a certain sound segment or not. Repeating patterns occurred either regularly or with a temporal jitter within the sequences, and participants' attention was directed either towards the pattern repetitions or away from the auditory stimulation. Across both regular and jittered sequences during both attention and in-attention, pattern repetitions led to increased sustained activity throughout the sequence, evoked a characteristic positivity-negativity complex in the event-related potential, and enhanced inter-trial phase coherence of low-frequency oscillatory activity time-locked to repeating pattern onsets. While regularity only had a minor (if any) influence, attention significantly strengthened pattern repetition perception, which was consistently reflected in all three EEG markers. These findings suggest that the detection of pattern repetitions within continuous sounds relies on a flexible mechanism that is robust against in-attention and temporal irregularity, both of which typically occur in naturalistic listening situations. Yet, attention to the auditory input can enhance processing of repeating patterns and improve repetition detection.