Laying the Foundation: Modern Transformers for Gold-Standard Sleep Analysis and Beyond.

Accurate sleep assessment is critical to the practice of sleep medicine and sleep research. The recent availability of large quantities of publicly available sleep data, alongside recent breakthroughs in AI like transformer architectures, present novel opportunities for data-driven discovery efforts. Transformers are flexible neural networks that not only excel at classification tasks, but also can enable data-driven discovery through un- or self-supervised learning, which requires no human annotations to the input data. While transformers have been extensively used in supervised learning scenarios for sleep stage classification, they have not been fully explored or optimized in forms designed from the ground up for use in un- or self-supervised learning tasks in sleep. A necessary first step will be to study these models on a canonical benchmark supervised learning task (5-class sleep stage classification). Hence, to lay the groundwork for future data-driven discovery efforts, we evaluated optimizations of a transformer-based architecture that has already demonstrated substantial success in self-supervised learning in another domain (audio speech recognition), and trained it to perform the canonical 5-class sleep stage classification task, to establish foundational baselines in the sleep domain. We found that small transformer models designed from the start for (later) self-supervised learning can match other state-of-the-art automated sleep scoring techniques, while also providing the basis for future data-driven discovery efforts using large sleep data sets.

Self-Supervised Transformer Model Training for a Sleep-EEG Foundation Model.

The American Academy of Sleep Medicine (AASM) recognizes five sleep/wake states (Wake, N1, N2, N3, REM), yet this classification schema provides only a high-level summary of sleep and likely overlooks important neurological or health information. New, data-driven approaches are needed to more deeply probe the information content of sleep signals. Here we present a self-supervised approach that learns the structure embedded in large quantities of neurophysiological sleep data. This masked transformer training procedure is inspired by high performing self-supervised methods developed for speech transcription. We show that self-supervised pre-training matches or outperforms supervised sleep stage classification, especially when labeled data or compute-power is limited. Perhaps more importantly, we also show that our pretrained model is flexible and can be fine-tuned to perform well on new tasks including distinguishing individuals and quantifying "brain age" (a potential health biomarker). This suggests that modern methods can automatically learn information that is potentially overlooked by the 5-class sleep staging schema, laying the groundwork for new schemas and further data-driven exploration of sleep.

The Rapid Emergence of Auditory Object Representations in Cortex Reflect Central Acoustic Attributes.

Human listeners are bombarded by acoustic information that the brain rapidly organizes into coherent percepts of objects and events in the environment, which aids speech and music perception. The efficiency of auditory object recognition belies the critical constraint that acoustic stimuli necessarily require time to unfold. Using magnetoencephalography, we studied the time course of the neural processes that transform dynamic acoustic information into auditory object representations. Participants listened to a diverse set of 36 tokens comprising everyday sounds from a typical human environment. Multivariate pattern analysis was used to decode the sound tokens from the magnetoencephalographic recordings. We show that sound tokens can be decoded from brain activity beginning 90 msec after stimulus onset with peak decoding performance occurring at 155 msec poststimulus onset. Decoding performance was primarily driven by differences between category representations (e.g., environmental vs. instrument sounds), although within-category decoding was better than chance. Representational similarity analysis revealed that these emerging neural representations were related to harmonic and spectrotemporal differences among the stimuli, which correspond to canonical acoustic features processed by the auditory pathway. Our findings begin to link the processing of physical sound properties with the perception of auditory objects and events in cortex.

Acoustic Correlates of Auditory Object and Event Perception: Speakers, Musical Timbres, and Environmental Sounds.

Human listeners must identify and orient themselves to auditory objects and events in their environment. What acoustic features support a listener's ability to differentiate the great variety of natural sounds they might encounter? Studies of auditory object perception typically examine identification (and confusion) responses or dissimilarity ratings between pairs of objects and events. However, the majority of this prior work has been conducted within single categories of sound. This separation has precluded a broader understanding of the general acoustic attributes that govern auditory object and event perception within and across different behaviorally relevant sound classes. The present experiments take a broader approach by examining multiple categories of sound relative to one another. This approach bridges critical gaps in the literature and allows us to identify (and assess the relative importance of) features that are useful for distinguishing sounds within, between and across behaviorally relevant sound categories. To do this, we conducted behavioral sound identification (Experiment 1) and dissimilarity rating (Experiment 2) studies using a broad set of stimuli that leveraged the acoustic variability within and between different sound categories via a diverse set of 36 sound tokens (12 utterances from different speakers, 12 instrument timbres, and 12 everyday objects from a typical human environment). Multidimensional scaling solutions as well as analyses of item-pair-level responses as a function of different acoustic qualities were used to understand what acoustic features informed participants' responses. In addition to the spectral and temporal envelope qualities noted in previous work, listeners' dissimilarity ratings were associated with spectrotemporal variability and aperiodicity. Subsets of these features (along with fundamental frequency variability) were also useful for making specific within or between sound category judgments. Dissimilarity ratings largely paralleled sound identification performance, however the results of these tasks did not completely mirror one another. In addition, musical training was related to improved sound identification performance.

Separable neural representations of sound sources: Speaker identity and musical timbre.

Human listeners can quickly and easily recognize different sound sources (objects and events) in their environment. Understanding how this impressive ability is accomplished can improve signal processing and machine intelligence applications along with assistive listening technologies. However, it is not clear how the brain represents the many sounds that humans can recognize (such as speech and music) at the level of individual sources, categories and acoustic features. To examine the cortical organization of these representations, we used patterns of fMRI responses to decode 1) four individual speakers and instruments from one another (separately, within each category), 2) the superordinate category labels associated with each stimulus (speech or instrument), and 3) a set of simple synthesized sounds that could be differentiated entirely on their acoustic features. Data were collected using an interleaved silent steady state sequence to increase the temporal signal-to-noise ratio, and mitigate issues with auditory stimulus presentation in fMRI. Largely separable clusters of voxels in the temporal lobes supported the decoding of individual speakers and instruments from other stimuli in the same category. Decoding the superordinate category of each sound was more accurate and involved a larger portion of the temporal lobes. However, these clusters all overlapped with areas that could decode simple, acoustically separable stimuli. Thus, individual sound sources from different sound categories are represented in separate regions of the temporal lobes that are situated within regions implicated in more general acoustic processes. These results bridge an important gap in our understanding of cortical representations of sounds and their acoustics.

The time course of sound category identification: Insights from acoustic features.

Humans have an impressive, automatic capacity for identifying and organizing sounds in their environment. However, little is known about the timescales that sound identification functions on, or the acoustic features that listeners use to identify auditory objects. To better understand the temporal and acoustic dynamics of sound category identification, two go/no-go perceptual gating studies were conducted. Participants heard speech, musical instrument, and human-environmental sounds ranging from 12.5 to 200 ms in duration. Listeners could reliably identify sound categories with just 25 ms of duration. In experiment 1, participants' performance on instrument sounds showed a distinct processing advantage at shorter durations. Experiment 2 revealed that this advantage was largely dependent on regularities in instrument onset characteristics relative to the spectrotemporal complexity of environmental sounds and speech. Models of participant responses indicated that listeners used spectral, temporal, noise, and pitch cues in the task. Aspects of spectral centroid were associated with responses for all categories, while noisiness and spectral flatness were associated with environmental and instrument responses, respectively. Responses for speech and environmental sounds were also associated with spectral features that varied over time. Experiment 2 indicated that variability in fundamental frequency was useful in identifying steady state speech and instrument stimuli.

Comparison of Methods for Collecting and Modeling Dissimilarity Data: Applications to Complex Sound Stimuli.

Sorting procedures are frequently adopted as an alternative to dissimilarity ratings to measure the dissimilarity of large sets of stimuli in a comparatively short time. However, systematic empirical research on the consequences of this experiment-design choice is lacking. We carried out a behavioral experiment to assess the extent to which sorting procedures compare to dissimilarity ratings in terms of efficiency, reliability, and accuracy, and the extent to which data from different data-collection methods are redundant and are better fit by different distance models. Participants estimated the dissimilarity of either semantically charged environmental sounds or semantically neutral synthetic sounds. We considered free and hierarchical sorting and derived indications concerning the properties of constrained and truncated hierarchical sorting methods from hierarchical sorting data. Results show that the higher efficiency of sorting methods comes at a considerable cost in terms of data reliability and accuracy. This loss appears to be minimized with truncated hierarchical sorting methods that start from a relatively low number of groups of stimuli. Finally, variations in data-collection method differentially affect the fit of various distance models at the group-average and individual levels. On the basis of these results, we suggest adopting sorting as an alternative to dissimilarity-rating methods only when strictly necessary. We also suggest analyzing the raw behavioral dissimilarities, and avoiding modeling them with one single distance model.