Spatiotemporal whole-brain activity and functional connectivity of melodies recognition.

Music is a non-verbal human language, built on logical, hierarchical structures, that offers excellent opportunities to explore how the brain processes complex spatiotemporal auditory sequences. Using the high temporal resolution of magnetoencephalography, we investigated the unfolding brain dynamics of 70 participants during the recognition of previously memorized musical sequences compared to novel sequences matched in terms of entropy and information content. Measures of both whole-brain activity and functional connectivity revealed a widespread brain network underlying the recognition of the memorized auditory sequences, which comprised primary auditory cortex, superior temporal gyrus, insula, frontal operculum, cingulate gyrus, orbitofrontal cortex, basal ganglia, thalamus, and hippocampus. Furthermore, while the auditory cortex responded mainly to the first tones of the sequences, the activity of higher-order brain areas such as the cingulate gyrus, frontal operculum, hippocampus, and orbitofrontal cortex largely increased over time during the recognition of the memorized versus novel musical sequences. In conclusion, using a wide range of analytical techniques spanning from decoding to functional connectivity and building on previous works, our study provided new insights into the spatiotemporal whole-brain mechanisms for conscious recognition of auditory sequences.

Modeling enculturated bias in entrainment to rhythmic patterns.

Long-term and culture-specific experience of music shapes rhythm perception, leading to enculturated expectations that make certain rhythms easier to track and more conducive to synchronized movement. However, the influence of enculturated bias on the moment-to-moment dynamics of rhythm tracking is not well understood. Recent modeling work has formulated entrainment to rhythms as a formal inference problem, where phase is continuously estimated based on precise event times and their correspondence to timing expectations: PIPPET (Phase Inference from Point Process Event Timing). Here we propose that the problem of optimally tracking a rhythm also requires an ongoing process of inferring which pattern of event timing expectations is most suitable to predict a stimulus rhythm. We formalize this insight as an extension of PIPPET called pPIPPET (PIPPET with pattern inference). The variational solution to this problem introduces terms representing the likelihood that a stimulus is based on a particular member of a set of event timing patterns, which we initialize according to culturally-learned prior expectations of a listener. We evaluate pPIPPET in three experiments. First, we demonstrate that pPIPPET can qualitatively reproduce enculturated bias observed in human tapping data for simple two-interval rhythms. Second, we simulate categorization of a continuous three-interval rhythm space by Western-trained musicians through derivation of a comprehensive set of priors for pPIPPET from metrical patterns in a sample of Western rhythms. Third, we simulate iterated reproduction of three-interval rhythms, and show that models configured with notated rhythms from different cultures exhibit both universal and enculturated biases as observed experimentally in listeners from those cultures. These results suggest the influence of enculturated timing expectations on human perceptual and motor entrainment can be understood as approximating optimal inference about the rhythmic stimulus, with respect to prototypical patterns in an empirical sample of rhythms that represent the music-cultural environment of the listener.

Correction: PPM-Decay: A computational model of auditory prediction with memory decay.

[This corrects the article DOI: 10.1371/journal.pcbi.1008304.].

Musicianship and melodic predictability enhance neural gain in auditory cortex during pitch deviance detection.

When listening to music, pitch deviations are more salient and elicit stronger prediction error responses when the melodic context is predictable and when the listener is a musician. Yet, the neuronal dynamics and changes in connectivity underlying such effects remain unclear. Here, we employed dynamic causal modeling (DCM) to investigate whether the magnetic mismatch negativity response (MMNm)-and its modulation by context predictability and musical expertise-are associated with enhanced neural gain of auditory areas, as a plausible mechanism for encoding precision-weighted prediction errors. Using Bayesian model comparison, we asked whether models with intrinsic connections within primary auditory cortex (A1) and superior temporal gyrus (STG)-typically related to gain control-or extrinsic connections between A1 and STG-typically related to propagation of prediction and error signals-better explained magnetoencephalography responses. We found that, compared to regular sounds, out-of-tune pitch deviations were associated with lower intrinsic (inhibitory) connectivity in A1 and STG, and lower backward (inhibitory) connectivity from STG to A1, consistent with disinhibition and enhanced neural gain in these auditory areas. More predictable melodies were associated with disinhibition in right A1, while musicianship was associated with disinhibition in left A1 and reduced connectivity from STG to left A1. These results indicate that musicianship and melodic predictability, as well as pitch deviations themselves, enhance neural gain in auditory cortex during deviance detection. Our findings are consistent with predictive processing theories suggesting that precise and informative error signals are selected by the brain for subsequent hierarchical processing.

Predictive Uncertainty Underlies Auditory Boundary Perception.

Anticipating the future is essential for efficient perception and action planning. Yet the role of anticipation in event segmentation is understudied because empirical research has focused on retrospective cues such as surprise. We address this concern in the context of perception of musical-phrase boundaries. A computational model of cognitive sequence processing was used to control the information-dynamic properties of tone sequences. In an implicit, self-paced listening task (N = 38), undergraduates dwelled longer on tones generating high entropy (i.e., high uncertainty) than on those generating low entropy (i.e., low uncertainty). Similarly, sequences that ended on tones generating high entropy were rated as sounding more complete (N = 31 undergraduates). These entropy effects were independent of both the surprise (i.e., information content) and phrase position of target tones in the original musical stimuli. Our results indicate that events generating high entropy prospectively contribute to segmentation processes in auditory sequence perception, independently of the properties of the subsequent event.

PPM-Decay: A computational model of auditory prediction with memory decay.

Statistical learning and probabilistic prediction are fundamental processes in auditory cognition. A prominent computational model of these processes is Prediction by Partial Matching (PPM), a variable-order Markov model that learns by internalizing n-grams from training sequences. However, PPM has limitations as a cognitive model: in particular, it has a perfect memory that weights all historic observations equally, which is inconsistent with memory capacity constraints and recency effects observed in human cognition. We address these limitations with PPM-Decay, a new variant of PPM that introduces a customizable memory decay kernel. In three studies-one with artificially generated sequences, one with chord sequences from Western music, and one with new behavioral data from an auditory pattern detection experiment-we show how this decay kernel improves the model's predictive performance for sequences whose underlying statistics change over time, and enables the model to capture effects of memory constraints on auditory pattern detection. The resulting model is available in our new open-source R package, ppm (https://github.com/pmcharrison/ppm).

Evaluative judgment across domains: Liking balance, contour, symmetry and complexity in melodies and visual designs.

Evaluative judgment-i.e., assessing to what degree a stimulus is liked or disliked-is a fundamental aspect of cognition, facilitating comparison and choosing among alternatives, deciding, and prioritizing actions. Neuroimaging studies have shown that evaluative judgment involves the projection of sensory information to the reward circuit. To investigate whether evaluative judgments are based on modality-specific or modality-general attributes, we compared the extent to which balance, contour, symmetry, and complexity affect liking responses in the auditory and visual modalities. We found no significant correlation for any of the four attributes across sensory modalities, except for contour. This suggests that evaluative judgments primarily rely on modality-specific sensory representations elaborated in the brain's sensory cortices and relayed to the reward circuit, rather than abstract modality-general representations. The individual traits art experience, openness to experience, and desire for aesthetics were associated with the extent to which design or compositional attributes influenced liking, but inconsistently across sensory modalities and attributes, also suggesting modality-specific influences.

Melodic expectations in 5- and 6-year-old children.

It has been argued that children implicitly acquire the rules relating to the structure of music in their environment using domain-general mechanisms such as statistical learning. Closely linked to statistical learning is the ability to form expectations about future events. Whether children as young as 5 years can make use of such internalized regularities to form expectations about the next note in a melody is still unclear. The possible effect of the home musical environment on the strength of musical expectations has also been under-explored. Using a newly developed melodic priming task that included melodies with either "expected" or "unexpected" endings according to rules of Western music theory, we tested 5- and 6-year-old children (N = 46). The stimuli in this task were constructed using the information dynamics of music (IDyOM) system, a probabilistic model estimating the level of "unexpectedness" of a note given the preceding context. Results showed that responses to expected versus unexpected tones were faster and more accurate, indicating that children have already formed robust melodic expectations at 5 years of age. Aspects of the home musical environment significantly predicted the strength of melodic expectations, suggesting that implicit musical learning may be influenced by the quantity of informal exposure to the surrounding musical environment.

Predictability and Uncertainty in the Pleasure of Music: A Reward for Learning?

Music ranks among the greatest human pleasures. It consistently engages the reward system, and converging evidence implies it exploits predictions to do so. Both prediction confirmations and errors are essential for understanding one's environment, and music offers many of each as it manipulates interacting patterns across multiple timescales. Learning models suggest that a balance of these outcomes (i.e., intermediate complexity) optimizes the reduction of uncertainty to rewarding and pleasurable effect. Yet evidence of a similar pattern in music is mixed, hampered by arbitrary measures of complexity. In the present studies, we applied a well-validated information-theoretic model of auditory expectation to systematically measure two key aspects of musical complexity: predictability (operationalized as information content [IC]), and uncertainty (entropy). In Study 1, we evaluated how these properties affect musical preferences in 43 male and female participants; in Study 2, we replicated Study 1 in an independent sample of 27 people and assessed the contribution of veridical predictability by presenting the same stimuli seven times. Both studies revealed significant quadratic effects of IC and entropy on liking that outperformed linear effects, indicating reliable preferences for music of intermediate complexity. An interaction between IC and entropy further suggested preferences for more predictability during more uncertain contexts, which would facilitate uncertainty reduction. Repeating stimuli decreased liking ratings but did not disrupt the preference for intermediate complexity. Together, these findings support long-hypothesized optimal zones of predictability and uncertainty in musical pleasure with formal modeling, relating the pleasure of music listening to the intrinsic reward of learning.SIGNIFICANCE STATEMENT Abstract pleasures, such as music, claim much of our time, energy, and money despite lacking any clear adaptive benefits like food or shelter. Yet as music manipulates patterns of melody, rhythm, and more, it proficiently exploits our expectations. Given the importance of anticipating and adapting to our ever-changing environments, making and evaluating uncertain predictions can have strong emotional effects. Accordingly, we present evidence that listeners consistently prefer music of intermediate predictive complexity, and that preferences shift toward expected musical outcomes in more uncertain contexts. These results are consistent with theories that emphasize the intrinsic reward of learning, both by updating inaccurate predictions and validating accurate ones, which is optimal in environments that present manageable predictive challenges (i.e., reducible uncertainty).

Auditory but Not Audiovisual Cues Lead to Higher Neural Sensitivity to the Statistical Regularities of an Unfamiliar Musical Style.

It is still a matter of debate whether visual aids improve learning of music. In a multisession study, we investigated the neural signatures of novel music sequence learning with or without aids (auditory-only: AO, audiovisual: AV). During three training sessions on three separate days, participants (nonmusicians) reproduced (note by note on a keyboard) melodic sequences generated by an artificial musical grammar. The AV group (n = 20) had each note color-coded on screen, whereas the AO group (n = 20) had no color indication. We evaluated learning of the statistical regularities of the novel music grammar before and after training by presenting melodies ending on correct or incorrect notes and by asking participants to judge the correctness and surprisal of the final note, while EEG was recorded. We found that participants successfully learned the new grammar. Although the AV group, as compared to the AO group, reproduced longer sequences during training, there was no significant difference in learning between groups. At the neural level, after training, the AO group showed a larger N100 response to low-probability compared with high-probability notes, suggesting an increased neural sensitivity to statistical properties of the grammar; this effect was not observed in the AV group. Our findings indicate that visual aids might improve sequence reproduction while not necessarily promoting better learning, indicating a potential dissociation between sequence reproduction and learning. We suggest that the difficulty induced by auditory-only input during music training might enhance cognitive engagement, thereby improving neural sensitivity to the underlying statistical properties of the learned material.

From learning to creativity: Identifying the behavioural and neural correlates of learning to predict human judgements of musical creativity.

Human creativity is intricately linked to acquired knowledge. However, to date learning a new musical style and subsequent musical creativity have largely been studied in isolation. We introduced a novel experimental paradigm combining behavioural, electrophysiological, and computational methods, to examine the neural correlates of unfamiliar music learning, and to investigate how neural and computational measures can predict human creativity. We investigated music learning by training non-musicians (N = 40) on an artificial music grammar. Participants' knowledge of the grammar was tested before and after three training sessions on separate days by assessing explicit recognition of the notes of the grammar, while additionally recording their EEG. After each training session, participants created their own musical compositions, which were later evaluated by human experts. A computational model of auditory expectation was used to quantify the statistical properties of both the grammar and the compositions. Results showed that participants successfully learned the new grammar. This was also reflected in the N100, P200, and P3a components, which were higher in response to incorrect than correct notes. The delta band (2.5-4.5 Hz) power in response to grammatical notes during first exposure to the grammar positively correlated with learning, suggesting a potential neural mechanism of encoding. On the other hand, better learning was associated with lower alpha and higher beta band power after training, potentially reflecting neural mechanisms of retrieval. Importantly, learning was a significant predictor of creativity, as judged by experts. There was also an inverted U-shaped relationship between percentage of correct intervals and creativity, as compositions with an intermediate proportion of correct intervals were associated with the highest creativity. Finally, the P200 in response to incorrect notes was predictive of creativity, suggesting a link between the neural correlates of learning, and creativity. Overall, our findings shed light on the neural mechanisms of learning an unfamiliar music grammar, and offer novel contributions to the associations between learning measures and creative compositions based on learned materials.

Musical prediction error responses similarly reduced by predictive uncertainty in musicians and non-musicians.

Auditory prediction error responses elicited by surprising sounds can be reliably recorded with musical stimuli that are more complex and realistic than those typically employed in EEG or MEG oddball paradigms. However, these responses are reduced as the predictive uncertainty of the stimuli increases. In this study, we investigate whether this effect is modulated by musical expertise. Magnetic mismatch negativity (MMNm) responses were recorded from 26 musicians and 24 non-musicians while they listened to low- and high-uncertainty melodic sequences in a musical multi-feature paradigm that included pitch, slide, intensity and timbre deviants. When compared to non-musicians, musically trained participants had significantly larger pitch and slide MMNm responses. However, both groups showed comparable reductions in pitch and slide MMNm amplitudes in the high-uncertainty condition compared with the low-uncertainty condition. In a separate, behavioural deviance detection experiment, musicians were more accurate and confident about their responses than non-musicians, but deviance detection in both groups was similarly affected by the uncertainty of the melodies. In both experiments, the interaction between uncertainty and expertise was not significant, suggesting that the effect is comparable in both groups. Consequently, our results replicate the modulatory effect of predictive uncertainty on prediction error; show that it is present across different types of listeners; and suggest that expertise-related and stimulus-driven modulations of predictive precision are dissociable and independent.

A Set of 200 Musical Stimuli Varying in Balance, Contour, Symmetry, and Complexity: Behavioral and Computational Assessments.

We present a novel set of 200 Western tonal musical stimuli (MUST) to be used in research on perception and appreciation of music. It consists of four subsets of 50 stimuli varying in balance, contour, symmetry, or complexity. All are 4 s long and designed to be musically appealing and experimentally controlled. We assessed them behaviorally and computationally. The behavioral assessment (Study 1) aimed to determine whether musically untrained participants could identify variations in each attribute. Forty-three participants rated the stimuli in each subset on the corresponding attribute. We found that inter-rater reliability was high and that the ratings mirrored the design features well. Participants' ratings also served to create an abridged set of 24 stimuli per subset. The computational assessment (Study 2) required the development of a specific battery of computational measures describing the structural properties of each stimulus. We distilled nonredundant composite measures for each attribute and examined whether they predicted participants' ratings. Our results show that the composite measures indeed predicted participants' ratings. Moreover, the composite complexity measure predicted complexity ratings as well as existing models of musical complexity. We conclude that the four subsets are suitable for use in studies that require presenting participants with short musical motifs varying in balance, contour, symmetry, or complexity, and that the stimuli and the computational measures are valuable resources for research in music psychology, empirical aesthetics, music information retrieval, and musicology. The MUST set and MATLAB toolbox codifying the computational measures are freely available at osf.io/bfxz7.

Intracranial Recordings and Computational Modeling of Music Reveal the Time Course of Prediction Error Signaling in Frontal and Temporal Cortices.

Prediction is held to be a fundamental process underpinning perception, action, and cognition. To examine the time course of prediction error signaling, we recorded intracranial EEG activity from nine presurgical epileptic patients while they listened to melodies whose information theoretical predictability had been characterized using a computational model. We examined oscillatory activity in the superior temporal gyrus (STG), the middle temporal gyrus (MTG), and the pars orbitalis of the inferior frontal gyrus, lateral cortical areas previously implicated in auditory predictive processing. We also examined activity in anterior cingulate gyrus (ACG), insula, and amygdala to determine whether signatures of prediction error signaling may also be observable in these subcortical areas. Our results demonstrate that the information content (a measure of unexpectedness) of musical notes modulates the amplitude of low-frequency oscillatory activity (theta to beta power) in bilateral STG and right MTG from within 100 and 200 msec of note onset, respectively. Our results also show this cortical activity to be accompanied by low-frequency oscillatory modulation in ACG and insula-areas previously associated with mediating physiological arousal. Finally, we showed that modulation of low-frequency activity is followed by that of high-frequency (gamma) power from approximately 200 msec in the STG, between 300 and 400 msec in the left insula, and between 400 and 500 msec in the ACG. We discuss these results with respect to models of neural processing that emphasize gamma activity as an index of prediction error signaling and highlight the usefulness of musical stimuli in revealing the wide-reaching neural consequences of predictive processing.

Neural entrainment is associated with subjective groove and complexity for performed but not mechanical musical rhythms.

Both movement and neural activity in humans can be entrained by the regularities of an external stimulus, such as the beat of musical rhythms. Neural entrainment to auditory rhythms supports temporal perception, and is enhanced by selective attention and by hierarchical temporal structure imposed on rhythms. However, it is not known how neural entrainment to rhythms is related to the subjective experience of groove (the desire to move along with music or rhythm), the perception of a regular beat, the perception of complexity, and the experience of pleasure. In two experiments, we used musical rhythms (from Steve Reich's Clapping Music) to investigate whether rhythms that are performed by humans (with naturally variable timing) and rhythms that are mechanical (with precise timing), elicit differences in (1) neural entrainment, as measured by inter-trial phase coherence, and (2) subjective ratings of the complexity, preference, groove, and beat strength of rhythms. We also combined results from the two experiments to investigate relationships between neural entrainment and subjective perception of musical rhythms. We found that mechanical rhythms elicited a greater degree of neural entrainment than performed rhythms, likely due to the greater temporal precision in the stimulus, and the two types only elicited different ratings for some individual rhythms. Neural entrainment to performed rhythms, but not to mechanical ones, correlated with subjective desire to move and subjective complexity. These data, therefore, suggest multiple interacting influences on neural entrainment to rhythms, from low-level stimulus properties to high-level cognition and perception.

Brain responses in humans reveal ideal observer-like sensitivity to complex acoustic patterns.

We use behavioral methods, magnetoencephalography, and functional MRI to investigate how human listeners discover temporal patterns and statistical regularities in complex sound sequences. Sensitivity to patterns is fundamental to sensory processing, in particular in the auditory system, because most auditory signals only have meaning as successions over time. Previous evidence suggests that the brain is tuned to the statistics of sensory stimulation. However, the process through which this arises has been elusive. We demonstrate that listeners are remarkably sensitive to the emergence of complex patterns within rapidly evolving sound sequences, performing on par with an ideal observer model. Brain responses reveal online processes of evidence accumulation--dynamic changes in tonic activity precisely correlate with the expected precision or predictability of ongoing auditory input--both in terms of deterministic (first-order) structure and the entropy of random sequences. Source analysis demonstrates an interaction between primary auditory cortex, hippocampus, and inferior frontal gyrus in the process of discovering the regularity within the ongoing sound sequence. The results are consistent with precision based predictive coding accounts of perceptual inference and provide compelling neurophysiological evidence of the brain's capacity to encode high-order temporal structure in sensory signals.

That note sounds wrong! Age-related effects in processing of musical expectation.

Part of musical understanding and enjoyment stems from the ability to accurately predict what note (or one of a small set of notes) is likely to follow after hearing the first part of a melody. Selective violation of expectations can add to aesthetic response but radical or frequent violations are likely to be disliked or not comprehended. In this study we investigated whether a lifetime of exposure to music among untrained older adults would enhance their reaction to unexpected endings of unfamiliar melodies. Older and younger adults listened to melodies that had expected or unexpected ending notes, according to Western music theory. Ratings of goodness-of-fit were similar in the groups, as was ERP response to the note onset (N1). However, in later time windows (P200 and Late Positive Component), the amplitude of a response to unexpected and expected endings was both larger in older adults, corresponding to greater sensitivity, and more widespread in locus, consistent with a dedifferentiation pattern. Lateralization patterns also differed. We conclude that older adults refine their understanding of this important aspect of music throughout life, with the ability supported by changing patterns of neural activity.

Trajectories and revolutions in popular melody based on U.S. charts from 1950 to 2023.

In the past century, the history of popular music has been analyzed from many different perspectives, with sociologists, musicologists and philosophers all offering distinct narratives characterizing the evolution of popular music. However, quantitative studies on this subject began only in the last decade and focused on features extracted from raw audio, which limits the scope to low-level components of music. The present study investigates the evolution of a more abstract dimension of popular music, specifically melody, using a new dataset of popular melodies spanning from 1950 to 2023. To identify "melodic revolutions", changepoint detection was applied to a multivariate time series comprising features related to the pitch and rhythmic structure of the melodies. Two major revolutions in 1975 and 2000 and one smaller revolution in 1996, characterized by significant decreases in complexity, were located. The revolutions divided the time series into three eras, which were modeled separately with autoregression, linear regression and vector autoregression. Linear regression of autoregression residuals underscored inter-feature relationships, which become stronger in post-2000 melodies. The overriding pattern emerging from these analyses shows decreasing complexity and increasing note density in popular melodies over time, especially since 2000.

Perceptual representations mediate effects of stimulus properties on liking for music.

Perceptual pleasure and its concomitant hedonic value play an essential role in everyday life, motivating behavior and thus influencing how individuals choose to spend their time and resources. However, how pleasure arises from perception of sensory information remains relatively poorly understood. In particular, research has neglected the question of how perceptual representations mediate the relationships between stimulus properties and liking (e.g., stimulus symmetry can only affect liking if it is perceived). The present research addresses this gap for the first time, analyzing perceptual and liking ratings of 96 nonmusicians (power of 0.99) and finding that perceptual representations mediate effects of feature-based and information-based stimulus properties on liking for a novel set of melodies varying in balance, contour, symmetry, or complexity. Moreover, variability due to individual differences and stimuli accounts for most of the variance in liking. These results have broad implications for psychological research on sensory valuation, advocating a more explicit account of random variability and the mediating role of perceptual representations of stimulus properties.