Ecology & computer audition: Applications of audio technology to monitor organisms and environment.

Among the 17 Sustainable Development Goals (SDGs) proposed within the 2030 Agenda and adopted by all the United Nations member states, the 13th SDG is a call for action to combat climate change. Moreover, SDGs 14 and 15 claim the protection and conservation of life below water and life on land, respectively. In this work, we provide a literature-founded overview of application areas, in which computer audition - a powerful but in this context so far hardly considered technology, combining audio signal processing and machine intelligence - is employed to monitor our ecosystem with the potential to identify ecologically critical processes or states. We distinguish between applications related to organisms, such as species richness analysis and plant health monitoring, and applications related to the environment, such as melting ice monitoring or wildfire detection. This work positions computer audition in relation to alternative approaches by discussing methodological strengths and limitations, as well as ethical aspects. We conclude with an urgent call to action to the research community for a greater involvement of audio intelligence methodology in future ecosystem monitoring approaches.

A wearable device perspective on the standard definitions of disability progression in multiple sclerosis.

INTRODUCTION: Multiple sclerosis (MS) is a leading cause of disability among young adults, but standard clinical scales may not accurately detect subtle changes in disability occurring between visits. This study aims to explore whether wearable device data provides more granular and objective measures of disability progression in MS. METHODS: Remote Assessment of Disease and Relapse in Central Nervous System Disorders (RADAR-CNS) is a longitudinal multicenter observational study in which 400 MS patients have been recruited since June 2018 and prospectively followed up for 24 months. Monitoring of patients included standard clinical visits with assessment of disability through use of the Expanded Disability Status Scale (EDSS), 6-minute walking test (6MWT) and timed 25-foot walk (T25FW), as well as remote monitoring through the use of a Fitbit. RESULTS: Among the 306 patients who completed the study (mean age, 45.6 years; females 67%), confirmed disability progression defined by the EDSS was observed in 74 patients, who had approximately 1392 fewer daily steps than patients without disability progression. However, the decrease in the number of steps experienced over time by patients with EDSS progression and stable patients was not significantly different. Similar results were obtained with disability progression defined by the 6MWT and the T25FW. CONCLUSION: The use of continuous activity monitoring holds great promise as a sensitive and ecologically valid measure of disability progression in MS.

HEAR4Health: a blueprint for making computer audition a staple of modern healthcare.

Recent years have seen a rapid increase in digital medicine research in an attempt to transform traditional healthcare systems to their modern, intelligent, and versatile equivalents that are adequately equipped to tackle contemporary challenges. This has led to a wave of applications that utilise AI technologies; first and foremost in the fields of medical imaging, but also in the use of wearables and other intelligent sensors. In comparison, computer audition can be seen to be lagging behind, at least in terms of commercial interest. Yet, audition has long been a staple assistant for medical practitioners, with the stethoscope being the quintessential sign of doctors around the world. Transforming this traditional technology with the use of AI entails a set of unique challenges. We categorise the advances needed in four key pillars: Hear, corresponding to the cornerstone technologies needed to analyse auditory signals in real-life conditions; Earlier, for the advances needed in computational and data efficiency; Attentively, for accounting to individual differences and handling the longitudinal nature of medical data; and, finally, Responsibly, for ensuring compliance to the ethical standards accorded to the field of medicine. Thus, we provide an overview and perspective of HEAR4Health: the sketch of a modern, ubiquitous sensing system that can bring computer audition on par with other AI technologies in the strive for improved healthcare systems.

Vocalisation Repertoire at the End of the First Year of Life: An Exploratory Comparison of Rett Syndrome and Typical Development.

Rett syndrome (RTT) is a rare, late detected developmental disorder associated with severe deficits in the speech-language domain. Despite a few reports about atypicalities in the speech-language development of infants and toddlers with RTT, a detailed analysis of the pre-linguistic vocalisation repertoire of infants with RTT is yet missing. Based on home video recordings, we analysed the vocalisations between 9 and 11 months of age of three female infants with typical RTT and compared them to three age-matched typically developing (TD) female controls. The video material of the infants had a total duration of 424 min with 1655 infant vocalisations. For each month, we (1) calculated the infants' canonical babbling ratios with CBRUTTER, i.e., the ratio of number of utterances containing canonical syllables to total number of utterances, and (2) classified their pre-linguistic vocalisations in three non-canonical and four canonical vocalisation subtypes. All infants achieved the milestone of canonical babbling at 9 months of age according to their canonical babbling ratios, i.e. CBRUTTER ≥ 0.15. We revealed overall lower CBRsUTTER and a lower proportion of canonical pre-linguistic vocalisations consisting of well-formed sounds that could serve as parts of target-language words for the RTT group compared to the TD group. Further studies with more data from individuals with RTT are needed to study the atypicalities in the pre-linguistic vocalisation repertoire which may portend the later deficits in spoken language that are characteristic features of RTT.

Triplet Loss-Based Models for COVID-19 Detection from Vocal Sounds.

This work focuses on the automatic detection of COVID-19 from the analysis of vocal sounds, including sustained vowels, coughs, and speech while reading a short text. Specifically, we use the Mel-spectrogram representations of these acoustic signals to train neural network-based models for the task at hand. The extraction of deep learnt representations from the Mel-spectrograms is performed with Convolutional Neural Networks (CNNs). In an attempt to guide the training of the embedded representations towards more separable and robust inter-class representations, we explore the use of a triplet loss function. The experiments performed are conducted using the Your Voice Counts dataset, a new dataset containing German speakers collected using smartphones. The results obtained support the suitability of using triplet loss-based models to detect COVID-19 from vocal sounds. The best Unweighted Average Recall (UAR) of 66.5 % is obtained using a triplet loss-based model exploiting vocal sounds recorded while reading.

Automatic vocalisation-based detection of fragile X syndrome and Rett syndrome.

Fragile X syndrome (FXS) and Rett syndrome (RTT) are developmental disorders currently not diagnosed before toddlerhood. Even though speech-language deficits are among the key symptoms of both conditions, little is known about infant vocalisation acoustics for an automatic earlier identification of affected individuals. To bridge this gap, we applied intelligent audio analysis methodology to a compact dataset of 4454 home-recorded vocalisations of 3 individuals with FXS and 3 individuals with RTT aged 6 to 11 months, as well as 6 age- and gender-matched typically developing controls (TD). On the basis of a standardised set of 88 acoustic features, we trained linear kernel support vector machines to evaluate the feasibility of automatic classification of (a) FXS vs TD, (b) RTT vs TD, (c) atypical development (FXS+RTT) vs TD, and (d) FXS vs RTT vs TD. In paradigms (a)-(c), all infants were correctly classified; in paradigm (d), 9 of 12 were so. Spectral/cepstral and energy-related features were most relevant for classification across all paradigms. Despite the small sample size, this study reveals new insights into early vocalisation characteristics in FXS and RTT, and provides technical underpinnings for a future earlier identification of affected individuals, enabling earlier intervention and family counselling.

The Acoustic Dissection of Cough: Diving Into Machine Listening-based COVID-19 Analysis and Detection.

OBJECTIVES: The coronavirus disease 2019 (COVID-19) has caused a crisis worldwide. Amounts of efforts have been made to prevent and control COVID-19's transmission, from early screenings to vaccinations and treatments. Recently, due to the spring up of many automatic disease recognition applications based on machine listening techniques, it would be fast and cheap to detect COVID-19 from recordings of cough, a key symptom of COVID-19. To date, knowledge of the acoustic characteristics of COVID-19 cough sounds is limited but would be essential for structuring effective and robust machine learning models. The present study aims to explore acoustic features for distinguishing COVID-19 positive individuals from COVID-19 negative ones based on their cough sounds. METHODS: By applying conventional inferential statistics, we analyze the acoustic correlates of COVID-19 cough sounds based on the ComParE feature set, i.e., a standardized set of 6,373 acoustic higher-level features. Furthermore, we train automatic COVID-19 detection models with machine learning methods and explore the latent features by evaluating the contribution of all features to the COVID-19 status predictions. RESULTS: The experimental results demonstrate that a set of acoustic parameters of cough sounds, e.g., statistical functionals of the root mean square energy and Mel-frequency cepstral coefficients, bear essential acoustic information in terms of effect sizes for the differentiation between COVID-19 positive and COVID-19 negative cough samples. Our general automatic COVID-19 detection model performs significantly above chance level, i.e., at an unweighted average recall (UAR) of 0.632, on a data set consisting of 1,411 cough samples (COVID-19 positive/negative: 210/1,201). CONCLUSIONS: Based on the acoustic correlates analysis on the ComParE feature set and the feature analysis in the effective COVID-19 detection approach, we find that several acoustic features that show higher effects in conventional group difference testing are also higher weighted in the machine learning models.

Is Speech the New Blood? Recent Progress in AI-Based Disease Detection From Audio in a Nutshell.

In recent years, advancements in the field of artificial intelligence (AI) have impacted several areas of research and application. Besides more prominent examples like self-driving cars or media consumption algorithms, AI-based systems have further started to gain more and more popularity in the health care sector, however whilst being restrained by high requirements for accuracy, robustness, and explainability. Health-oriented AI research as a sub-field of digital health investigates a plethora of human-centered modalities. In this article, we address recent advances in the so far understudied but highly promising audio domain with a particular focus on speech data and present corresponding state-of-the-art technologies. Moreover, we give an excerpt of recent studies on the automatic audio-based detection of diseases ranging from acute and chronic respiratory diseases via psychiatric disorders to developmental disorders and neurodegenerative disorders. Our selection of presented literature shows that the recent success of deep learning methods in other fields of AI also more and more translates to the field of digital health, albeit expert-designed feature extractors and classical ML methodologies are still prominently used. Limiting factors, especially for speech-based disease detection systems, are related to the amount and diversity of available data, e. g., the number of patients and healthy controls as well as the underlying distribution of age, languages, and cultures. Finally, we contextualize and outline application scenarios of speech-based disease detection systems as supportive tools for health-care professionals under ethical consideration of privacy protection and faulty prediction.

The development of visual attention in early infancy: Insights from a free-viewing paradigm.

Theories of visual attention suggest a cascading development of subfunctions such as alertness, spatial orientation, attention to object features, and endogenous control. Here, we aimed to track infants' visual developmental steps from a primarily exogenously to more endogenously controlled processing style during their first months of life. In this repeated measures study, 51 infants participated in seven fortnightly assessments at postterm ages of 4-16 weeks. Infants were presented with the same set of static and dynamic paired comparison stimuli in each assessment. Visual behavior was evaluated by a newly introduced scoring scheme. Our results confirmed the suggested visual developmental hierarchy and clearly demonstrated the suitability of our scoring scheme for documenting developmental changes in visual attention during early infancy. Besides the general ontogenetic course of development, we also discuss intra- and interindividual differences which may affect single assessments, and highlight the importance of repeated measurements for reliable evaluation of developmental changes.

Fitbeat: COVID-19 estimation based on wristband heart rate using a contrastive convolutional auto-encoder.

This study proposes a contrastive convolutional auto-encoder (contrastive CAE), a combined architecture of an auto-encoder and contrastive loss, to identify individuals with suspected COVID-19 infection using heart-rate data from participants with multiple sclerosis (MS) in the ongoing RADAR-CNS mHealth research project. Heart-rate data was remotely collected using a Fitbit wristband. COVID-19 infection was either confirmed through a positive swab test, or inferred through a self-reported set of recognised symptoms of the virus. The contrastive CAE outperforms a conventional convolutional neural network (CNN), a long short-term memory (LSTM) model, and a convolutional auto-encoder without contrastive loss (CAE). On a test set of 19 participants with MS with reported symptoms of COVID-19, each one paired with a participant with MS with no COVID-19 symptoms, the contrastive CAE achieves an unweighted average recall of 95.3 % , a sensitivity of 100 % and a specificity of 90.6 % , an area under the receiver operating characteristic curve (AUC-ROC) of 0.944, indicating a maximum successful detection of symptoms in the given heart rate measurement period, whilst at the same time keeping a low false alarm rate.

The voice of COVID-19: Acoustic correlates of infection in sustained vowels.

COVID-19 is a global health crisis that has been affecting our daily lives throughout the past year. The symptomatology of COVID-19 is heterogeneous with a severity continuum. Many symptoms are related to pathological changes in the vocal system, leading to the assumption that COVID-19 may also affect voice production. For the first time, the present study investigates voice acoustic correlates of a COVID-19 infection based on a comprehensive acoustic parameter set. We compare 88 acoustic features extracted from recordings of the vowels /i:/, /e:/, /u:/, /o:/, and /a:/ produced by 11 symptomatic COVID-19 positive and 11 COVID-19 negative German-speaking participants. We employ the Mann-Whitney U test and calculate effect sizes to identify features with prominent group differences. The mean voiced segment length and the number of voiced segments per second yield the most important differences across all vowels indicating discontinuities in the pulmonic airstream during phonation in COVID-19 positive participants. Group differences in front vowels are additionally reflected in fundamental frequency variation and the harmonics-to-noise ratio, group differences in back vowels in statistics of the Mel-frequency cepstral coefficients and the spectral slope. Our findings represent an important proof-of-concept contribution for a potential voice-based identification of individuals infected with COVID-19.

Novel AI driven approach to classify infant motor functions.

The past decade has evinced a boom of computer-based approaches to aid movement assessment in early infancy. Increasing interests have been dedicated to develop AI driven approaches to complement the classic Prechtl general movements assessment (GMA). This study proposes a novel machine learning algorithm to detect an age-specific movement pattern, the fidgety movements (FMs), in a prospectively collected sample of typically developing infants. Participants were recorded using a passive, single camera RGB video stream. The dataset of 2800 five-second snippets was annotated by two well-trained and experienced GMA assessors, with excellent inter- and intra-rater reliabilities. Using OpenPose, the infant full pose was recovered from the video stream in the form of a 25-points skeleton. This skeleton was used as input vector for a shallow multilayer neural network (SMNN). An ablation study was performed to justify the network's architecture and hyperparameters. We show for the first time that the SMNN is sufficient to discriminate fidgety from non-fidgety movements in a sample of age-specific typical movements with a classification accuracy of 88%. The computer-based solutions will complement original GMA to consistently perform accurate and efficient screening and diagnosis that may become universally accessible in daily clinical practice in the future.

Efficient Collection and Representation of Preverbal Data in Typical and Atypical Development.

Human preverbal development refers to the period of steadily increasing vocal capacities until the emergence of a child's first meaningful words. Over the last decades, research has intensively focused on preverbal behavior in typical development. Preverbal vocal patterns have been phonetically classified and acoustically characterized. More recently, specific preverbal phenomena were discussed to play a role as early indicators of atypical development. Recent advancements in audio signal processing and machine learning have allowed for novel approaches in preverbal behavior analysis including automatic vocalization-based differentiation of typically and atypically developing individuals. In this paper, we give a methodological overview of current strategies for collecting and acoustically representing preverbal data for intelligent audio analysis paradigms. Efficiency in the context of data collection and data representation is discussed. Following current research trends, we set a special focus on challenges that arise when dealing with preverbal data of individuals with late detected developmental disorders, such as autism spectrum disorder or Rett syndrome.

Cerebral Palsy: Early Markers of Clinical Phenotype and Functional Outcome.

The Prechtl General Movement Assessment (GMA) has become a cornerstone assessment in early identification of cerebral palsy (CP), particularly during the fidgety movement period at 3-5 months of age. Additionally, assessment of motor repertoire, such as antigravity movements and postural patterns, which form the Motor Optimality Score (MOS), may provide insight into an infant's later motor function. This study aimed to identify early specific markers for ambulation, gross motor function (using the Gross Motor Function Classification System, GMFCS), topography (unilateral, bilateral), and type (spastic, dyskinetic, ataxic, and hypotonic) of CP in a large worldwide cohort of 468 infants. We found that 95% of children with CP did not have fidgety movements, with 100% having non-optimal MOS. GMFCS level was strongly correlated to MOS. An MOS > 14 was most likely associated with GMFCS outcomes I or II, whereas GMFCS outcomes IV or V were hardly ever associated with an MOS > 8. A number of different movement patterns were associated with more severe functional impairment (GMFCS III-V), including atypical arching and persistent cramped-synchronized movements. Asymmetrical segmental movements were strongly associated with unilateral CP. Circular arm movements were associated with dyskinetic CP. This study demonstrated that use of the MOS contributes to understanding later CP prognosis, including early markers for type and severity.

Early motor and pre-linguistic verbal development in Prader-Willi syndrome - A case report.

BACKGROUND: Prader-Willi syndrome (PWS) is a rare genetic disorder. Infants with PWS show a neurodevelopmental dysfunction which entails a delayed motor and language development, but studies on their spontaneous movements (i.e. general movements) or pre-linguistic speech-language development before 6 months of age are missing so far. AIM: To describe early motor and pre-linguistic verbal development in an infant with PWS. METHODS AND PROCEDURES: Prospective case report; in addition to the assessment of general movements and the concurrent movement repertoire, we report on early verbal forms, applying the Stark Assessment of Early Vocal Development-Revised. OUTCOMES AND RESULTS: General movements were abnormal on days 8 and 15. No fidgety movements were observed at 11 weeks; they only emerged at 17 weeks and lasted until at least 27 weeks post-term. The movement character was monotonous, and early motor milestones were only achieved with a delay. At 27 weeks the infant produced age-adequate types of vocalisations. However, none of the canonical-syllable vocalisations that typically emerge at that age were observed. Early vocalisations appeared monotonous and with a peculiarly harmonic structure. CONCLUSIONS AND IMPLICATIONS: Early motor and pre-linguistic verbal behaviours were monotonous in an infant with PWS throughout his first 6 months of life. This suggests that early signs of neurodevelopmental dysfunction (i.e. abnormal general movements) might already be diagnosed in infants with PWS during their first weeks of life, potentially enabling us to diagnose and intervene at an early stage.

Canonical Babbling: A Marker for Earlier Identification of Late Detected Developmental Disorders?

PURPOSE OF REVIEW: To summarize findings about the emergence and characteristics of canonical babbling in children with late detected developmental disorders (LDDDs), such as autism spectrum disorder, Rett syndrome, and fragile X syndrome. In particular, we ask whether infants' vocal development in the first year of life contains any markers that may contribute to earlier detection of these disorders. RECENT FINDINGS: Only a handful studies have investigated canonical babbling in infants with LDDDs. With divergent research paradigms and definitions applied, findings on the onset and characteristics of canonical babbling are inconsistent and difficult to compare. Infants with LDDDs showed reduced likelihood to produce canonical babbling vocalizations. If achieved, this milestone was more likely to be reached beyond the critical time window of 5-10 months. SUMMARY: Canonical babbling appears promising as a potential marker for early detection of infants at risk for developmental disorders. In-depth studies on babbling characteristics in LDDDs are warranted.

On the lighter side: Medicine or etiquette? Rethinking a lecturer's teaching assignment.

Purpose: To investigate the extent to which medical students demonstrate politeness. With respect to patient-physician interactions, politeness appears to be a factor in therapeutic success, perhaps because it might induce greater patient compliance. Method: We assessed 354 third-semester medical students on one type of politeness, that is the percentage of students who greeted the teacher upon entering the lecture room. Results: Overall, 47% of the students initiated a greeting and this percentage did not change when the lecturers wore white coats. Females were less likely to initiate a greeting (35%) than males (55%). Conclusion: The results lead us to question whether university lecturers should strictly stick to their content of the curriculum or should they also teach their students about etiquette related to good clinician-patient relationships?

Early Vocal Development in Autism Spectrum Disorder, Rett Syndrome, and Fragile X Syndrome: Insights from Studies using Retrospective Video Analysis.

This article provides an overview of studies assessing the early vocalisations of children with autism spectrum disorder (ASD), Rett syndrome (RTT), and fragile X syndrome (FXS) using retrospective video analysis (RVA) during the first two years of life. Electronic databases were systematically searched and a total of 23 studies were selected. These studies were then categorised according to whether children were later diagnosed with ASD (13 studies), RTT (8 studies), or FXS (2 studies), and then described in terms of (a) participant characteristics, (b) control group characteristics, (c) video footage, (d) behaviours analysed, and (e) main findings. This overview supports the use of RVA in analysing the early development of vocalisations in children later diagnosed with ASD, RTT or FXS, and provides an in-depth analysis of vocalisation presentation, complex vocalisation production, and the rate and/or frequency of vocalisation production across the three disorders. Implications are discussed in terms of extending crude vocal analyses to more precise methods that might provide more powerful means by which to discriminate between disorders during early development. A greater understanding of the early manifestation of these disorders may then lead to improvements in earlier detection.

Typical vs. atypical: Combining auditory Gestalt perception and acoustic analysis of early vocalisations in Rett syndrome.

BACKGROUND: Early speech-language development of individuals with Rett syndrome (RTT) has been repeatedly characterised by a co-occurrence of apparently typical and atypical vocalisations. AIMS: To describe specific features of this intermittent character of typical versus atypical early RTT-associated vocalisations by combining auditory Gestalt perception and acoustic vocalisation analysis. METHODS AND PROCEDURES: We extracted N = 363 (pre-)linguistic vocalisations from home video recordings of an infant later diagnosed with RTT. In a listening experiment, all vocalisations were assessed for (a)typicality by five experts on early human development. Listeners' auditory concepts of (a)typicality were investigated in context of a comprehensive set of acoustic time-, spectral- and/or energy-related higher-order features extracted from the vocalisations. OUTCOMES AND RESULTS: More than half of the vocalisations were rated as 'atypical' by at least one listener. Atypicality was mainly related to the auditory attribute 'timbre', and to prosodic, spectral, and voice quality features in the acoustic domain. CONCLUSIONS AND IMPLICATIONS: Knowledge gained in our study shall contribute to the generation of an objective model of early vocalisation atypicality. Such a model might be used for increasing caregivers' and healthcare professionals' sensitivity to identify atypical vocalisation patterns, or even for a probabilistic approach to automatically detect RTT based on early vocalisations.

A Novel Way to Measure and Predict Development: A Heuristic Approach to Facilitate the Early Detection of Neurodevelopmental Disorders.

PURPOSE OF REVIEW: Substantial research exists focusing on the various aspects and domains of early human development. However, there is a clear blind spot in early postnatal development when dealing with neurodevelopmental disorders, especially those that manifest themselves clinically only in late infancy or even in childhood. RECENT FINDINGS: This early developmental period may represent an important timeframe to study these disorders but has historically received far less research attention. We believe that only a comprehensive interdisciplinary approach will enable us to detect and delineate specific parameters for specific neurodevelopmental disorders at a very early age to improve early detection/diagnosis, enable prospective studies and eventually facilitate randomised trials of early intervention. In this article, we propose a dynamic framework for characterising neurofunctional biomarkers associated with specific disorders in the development of infants and children. We have named this automated detection 'Fingerprint Model', suggesting one possible approach to accurately and early identify neurodevelopmental disorders.