Intake monitoring in free-living conditions: Overview and lessons we have learned.

The progress in artificial intelligence and machine learning algorithms over the past decade has enabled the development of new methods for the objective measurement of eating, including both the measurement of eating episodes as well as the measurement of in-meal eating behavior. These allow the study of eating behavior outside the laboratory in free-living conditions, without the need for video recordings and laborious manual annotations. In this paper, we present a high-level overview of our recent work on intake monitoring using a smartwatch, as well as methods using an in-ear microphone. We also present evaluation results of these methods in challenging, real-world datasets. Furthermore, we discuss use-cases of such intake monitoring tools for advancing research in eating behavior, for improving dietary monitoring, as well as for developing evidence-based health policies. Our goal is to inform researchers and users of intake monitoring methods regarding (i) the development of new methods based on commercially available devices, (ii) what to expect in terms of effectiveness, and (iii) how these methods can be used in research as well as in practical applications.

Toward Systems Models for Obesity Prevention: A Big Role for Big Data.

The relation among the various causal factors of obesity is not well understood, and there remains a lack of viable data to advance integrated, systems models of its etiology. The collection of big data has begun to allow the exploration of causal associations between behavior, built environment, and obesity-relevant health outcomes. Here, the traditional epidemiologic and emerging big data approaches used in obesity research are compared, describing the research questions, needs, and outcomes of 3 broad research domains: eating behavior, social food environments, and the built environment. Taking tangible steps at the intersection of these domains, the recent European Union project "BigO: Big data against childhood obesity" used a mobile health tool to link objective measurements of health, physical activity, and the built environment. BigO provided learning on the limitations of big data, such as privacy concerns, study sampling, and the balancing of epidemiologic domain expertise with the required technical expertise. Adopting big data approaches will facilitate the exploitation of data concerning obesity-relevant behaviors of a greater variety, which are also processed at speed, facilitated by mobile-based data collection and monitoring systems, citizen science, and artificial intelligence. These approaches will allow the field to expand from causal inference to more complex, systems-level predictive models, stimulating ambitious and effective policy interventions.

Bite-Weight Estimation Using Commercial Ear Buds.

While automatic tracking and measuring of our physical activity is a well established domain, not only in research but also in commercial products and every-day lifestyle, automatic measurement of eating behavior is significantly more limited. Despite the abundance of methods and algorithms that are available in bibliography, commercial solutions are mostly limited to digital logging applications for smart-phones. One factor that limits the adoption of such solutions is that they usually require specialized hardware or sensors. Based on this, we evaluate the potential for estimating the weight of consumed food (per bite) based only on the audio signal that is captured by commercial ear buds (Samsung Galaxy Buds). Specifically, we examine a combination of features (both audio and non-audio features) and trainable estimators (linear regression, support vector regression, and neural-network based estimators) and evaluate on an in-house dataset of 8 participants and 4 food types. Results indicate good potential for this approach: our best results yield mean absolute error of less than 1 g for 3 out of 4 food types when training food-specific models, and 2.1 g when training on all food types together, both of which improve over an existing literature approach.

Self-Supervised Feature Learning of 1D Convolutional Neural Networks with Contrastive Loss for Eating Detection Using an In-Ear Microphone.

The importance of automated and objective monitoring of dietary behavior is becoming increasingly accepted. The advancements in sensor technology along with recent achievements in machine-learning-based signal-processing algorithms have enabled the development of dietary monitoring solutions that yield highly accurate results. A common bottleneck for developing and training machine learning algorithms is obtaining labeled data for training supervised algorithms, and in particular ground truth annotations. Manual ground truth annotation is laborious, cumbersome, can sometimes introduce errors, and is sometimes impossible in free-living data collection. As a result, there is a need to decrease the labeled data required for training. Additionally, unlabeled data, gathered in-the-wild from existing wearables (such as Bluetooth earbuds) can be used to train and fine-tune eating-detection models. In this work, we focus on training a feature extractor for audio signals captured by an in-ear microphone for the task of eating detection in a self-supervised way. We base our approach on the SimCLR method for image classification, proposed by Chen et al. from the domain of computer vision. Results are promising as our self-supervised method achieves similar results to supervised training alternatives, and its overall effectiveness is comparable to current state-of-the-art methods. Code is available at https://github.com/mug-auth/ssl-chewing.

Fast Eating Is Associated with Increased BMI among High-School Students.

Fast self-reported eating rate (SRER) has been associated with increased adiposity in children and adults. No studies have been conducted among high-school students, and SRER has not been validated vs. objective eating rate (OBER) in such populations. The objectives were to investigate (among high-school student populations) the association between OBER and BMI z-scores (BMIz), the validity of SRER vs. OBER, and potential differences in BMIz between SRER categories. Three studies were conducted. Study 1 included 116 Swedish students (mean ± SD age: 16.5 ± 0.8, 59% females) who were eating school lunch. Food intake and meal duration were objectively recorded, and OBER was calculated. Additionally, students provided SRER. Study 2 included students (n = 50, mean ± SD age: 16.7 ± 0.6, 58% females) from Study 1 who ate another objectively recorded school lunch. Study 3 included 1832 high-school students (mean ± SD age: 15.8 ± 0.9, 51% females) from Sweden (n = 748) and Greece (n = 1084) who provided SRER. In Study 1, students with BMIz ≥ 0 had faster OBER vs. students with BMIz < 0 (mean difference: +7.7 g/min or +27%, p = 0.012), while students with fast SRER had higher OBER vs. students with slow SRER (mean difference: +13.7 g/min or +56%, p = 0.001). However, there was "minimal" agreement between SRER and OBER categories (κ = 0.31, p < 0.001). In Study 2, OBER during lunch 1 had a "large" correlation with OBER during lunch 2 (r = 0.75, p < 0.001). In Study 3, fast SRER students had higher BMIz vs. slow SRER students (mean difference: 0.37, p < 0.001). Similar observations were found among both Swedish and Greek students. For the first time in high-school students, we confirm the association between fast eating and increased adiposity. Our validation analysis suggests that SRER could be used as a proxy for OBER in studies with large sample sizes on a group level. With smaller samples, OBER should be used instead. To assess eating rate on an individual level, OBER can be used while SRER should be avoided.

Exploring Associations Between Children's Obesogenic Behaviors and the Local Environment Using Big Data: Development and Evaluation of the Obesity Prevention Dashboard.

BACKGROUND: Obesity is a major public health problem globally and in Europe. The prevalence of childhood obesity is also soaring. Several parameters of the living environment are contributing to this increase, such as the density of fast food retailers, and thus, preventive health policies against childhood obesity must focus on the environment to which children are exposed. Currently, there are no systems in place to objectively measure the effect of living environment parameters on obesogenic behaviors and obesity. The H2020 project "BigO: Big Data Against Childhood Obesity" aims to tackle childhood obesity by creating new sources of evidence based on big data. OBJECTIVE: This paper introduces the Obesity Prevention dashboard (OPdashboard), implemented in the context of BigO, which offers an interactive data platform for the exploration of objective obesity-related behaviors and local environments based on the data recorded using the BigO mHealth (mobile health) app. METHODS: The OPdashboard, which can be accessed on the web, allows for (1) the real-time monitoring of children's obesogenic behaviors in a city area, (2) the extraction of associations between these behaviors and the local environment, and (3) the evaluation of interventions over time. More than 3700 children from 33 schools and 2 clinics in 5 European cities have been monitored using a custom-made mobile app created to extract behavioral patterns by capturing accelerometer and geolocation data. Online databases were assessed in order to obtain a description of the environment. The dashboard's functionality was evaluated during a focus group discussion with public health experts. RESULTS: The preliminary association outcomes in 2 European cities, namely Thessaloniki, Greece, and Stockholm, Sweden, indicated a correlation between children's eating and physical activity behaviors and the availability of food-related places or sports facilities close to schools. In addition, the OPdashboard was used to assess changes to children's physical activity levels as a result of the health policies implemented to decelerate the COVID-19 outbreak. The preliminary outcomes of the analysis revealed that in urban areas the decrease in physical activity was statistically significant, while a slight increase was observed in the suburbs. These findings indicate the importance of the availability of open spaces for behavioral change in children. Discussions with public health experts outlined the dashboard's potential to aid in a better understanding of the interplay between children's obesogenic behaviors and the environment, and improvements were suggested. CONCLUSIONS: Our analyses serve as an initial investigation using the OPdashboard. Additional factors must be incorporated in order to optimize its use and obtain a clearer understanding of the results. The unique big data that are available through the OPdashboard can lead to the implementation of models that are able to predict population behavior. The OPdashboard can be considered as a tool that will increase our understanding of the underlying factors in childhood obesity and inform the design of regional interventions both for prevention and treatment.

Inferring the Spatial Distribution of Physical Activity in Children Population from Characteristics of the Environment.

Obesity affects a rising percentage of the children and adolescent population, contributing to decreased quality of life and increased risk for comorbidities. Although the major causes of obesity are known, the obesogenic behaviors manifest as a result of complex interactions of the individual with the living environment. For this reason, addressing childhood obesity remains a challenging problem for public health authorities. The BigO project (https://bigoprogram.eu) relies on large-scale behavioral and environmental data collection to create tools that support policy making and intervention design. In this work, we propose a novel analysis approach for modeling the expected population behavior as a function of the local environment. We experimentally evaluate this approach in predicting the expected physical activity level in small geographic regions using urban environment characteristics. Experiments on data collected from 156 children and adolescents verify the potential of the proposed approach. Specifically, we train models that predict the physical activity level in a region, achieving 81% leave-one-out accuracy. In addition, we exploit the model predictions to automatically visualize heatmaps of the expected population behavior in areas of interest, from which we draw useful insights. Overall, the predictive models and the automatic heatmaps are promising tools in gaining direct perception for the spatial distribution of the population's behavior, with potential uses by public health authorities.

Collecting big behavioral data for measuring behavior against obesity.

Obesity is currently affecting very large portions of the global population. Effective prevention and treatment starts at the early age and requires objective knowledge of population-level behavior on the region/neighborhood scale. To this end, we present a system for extracting and collecting behavioral information on the individual-level objectively and automatically. The behavioral information is related to physical activity, types of visited places, and transportation mode used between them. The system employs indicator-extraction algorithms from the literature which we evaluate on publicly available datasets. The system has been developed and integrated in the context of the EU-funded BigO project that aims at preventing obesity in young populations.

A smartphone application for semi-controlled collection of objective eating behavior data from multiple subjects.

BACKGROUND & OBJECTIVE: The study of eating behavior has made significant progress towards understanding the association of specific eating behavioral patterns with medical problems, such as obesity and eating disorders. Smartphones have shown promise in monitoring and modifying unhealthy eating behavior patterns, often with the help of sensors for behavior data recording. However, when it comes to semi-controlled deployment settings, smartphone apps that facilitate eating behavior data collection are missing. To fill this gap, the present work introduces ASApp, one of the first smartphone apps to support researchers in the collection of heterogeneous objective (sensor-acquired) and subjective (self-reported) eating behavior data in an integrated manner from large-scale, naturalistic human subject research (HSR) studies. METHODS: This work presents the overarching and deployment-specific requirements that have driven the design of ASApp, followed by the heterogeneous eating behavior dataset that is collected and the employed data collection protocol. The collected dataset combines objective and subjective behavior information, namely (a) dietary self-assessment information, (b) the food weight timeseries throughout an entire meal (using a portable weight scale connected wirelessly), (c) a photograph of the meal, and (d) a series of quantitative eating behavior indicators, mainly calculated from the food weight timeseries. The designed data collection protocol is quick, straightforward, robust and capable of satisfying the requirement of semi-controlled HSR deployment. RESULTS: The implemented functionalities of ASApp for research assistants and study participants are presented in detail along with the corresponding user interfaces. ASApp has been successfully deployed for data collection in an in-house testing study and the SPLENDID study, i.e., a real-life semi-controlled HSR study conducted in the cafeteria of a Swedish high-school in the context of an EC-funded research project. The two deployment studies are described and the promising results from the evaluation of the app with respect to attractiveness, usability, and technical soundness are discussed. Access details for ASApp are also provided. CONCLUSIONS: This work presents the requirement elucidation, design, implementation and evaluation of a novel smartphone application that supports researchers in the integrated collection of a concise yet rich set of heterogeneous eating behavior data for semi-controlled HSR.

BigO: A public health decision support system for measuring obesogenic behaviors of children in relation to their local environment.

Obesity is a complex disease and its prevalence depends on multiple factors related to the local socioeconomic, cultural and urban context of individuals. Many obesity prevention strategies and policies, however, are horizontal measures that do not depend on context-specific evidence. In this paper we present an overview of BigO (http://bigoprogram.eu), a system designed to collect objective behavioral data from children and adolescent populations as well as their environment in order to support public health authorities in formulating effective, context-specific policies and interventions addressing childhood obesity. We present an overview of the data acquisition, indicator extraction, data exploration and analysis components of the BigO system, as well as an account of its preliminary pilot application in 33 schools and 2 clinics in four European countries, involving over 4,200 participants.

Automatic Analysis of Food Intake and Meal Microstructure Based on Continuous Weight Measurements.

The structure of the cumulative food intake (CFI) curve has been associated with obesity and eating disorders. Scales that record the weight loss of a plate from which a subject eats food are used for capturing this curve; however, their measurements are contaminated by additive noise and are distorted by certain types of artifacts. This paper presents an algorithm for automatically processing continuous in-meal weight measurements in order to extract the clean CFI curve and in-meal eating indicators, such as total food intake and food intake rate. The algorithm relies on the representation of the weight-time series by a string of symbols that correspond to events such as bites or food additions. A context-free grammar is next used to model a meal as a sequence of such events. The selection of the most likely parse tree is finally used to determine the predicted eating sequence. The algorithm is evaluated on a dataset of 113 meals collected using the Mandometer, a scale that continuously samples plate weight during eating. We evaluate the effectiveness for seven indicators and for bite-instance detection. We compare our approach with three state-of-the-art algorithms, and achieve the lowest error rates for most indicators (24 g for total meal weight). The proposed algorithm extracts the parameters of the CFI curve automatically, eliminating the need for manual data processing, and thus facilitating large-scale studies of eating behavior.

Predicting Real-Life Eating Behaviours Using Single School Lunches in Adolescents.

Large portion sizes and a high eating rate are associated with high energy intake and obesity. Most individuals maintain their food intake weight (g) and eating rate (g/min) rank in relation to their peers, despite food and environmental manipulations. Single meal measures may enable identification of "large portion eaters" and "fast eaters," finding individuals at risk of developing obesity. The aim of this study was to predict real-life food intake weight and eating rate based on one school lunch. Twenty-four high-school students with a mean (±SD) age of 16.8 yr (±0.7) and body mass index of 21.9 (±4.1) were recruited, using no exclusion criteria. Food intake weight and eating rate was first self-rated ("Less," "Average" or "More than peers"), then objectively recorded during one school lunch (absolute weight of consumed food in grams). Afterwards, subjects recorded as many main meals (breakfasts, lunches and dinners) as possible in real-life for a period of at least two weeks, using a Bluetooth connected weight scale and a smartphone application. On average participants recorded 18.9 (7.3) meals during the study. Real-life food intake weight was 327.4 g (±110.6), which was significantly lower (p = 0.027) than the single school lunch, at 367.4 g (±167.2). When the intra-class correlation of food weight intake between the objectively recorded real-life and school lunch meals was compared, the correlation was excellent (R = 0.91). Real-life eating rate was 33.5 g/min (±14.8), which was significantly higher (p = 0.010) than the single school lunch, at 27.7 g/min (±13.3). The intra-class correlation of the recorded eating rate between real-life and school lunch meals was very large (R = 0.74). The participants' recorded food intake weights and eating rates were divided into terciles and compared between school lunches and real-life, with moderate or higher agreement (κ = 0.75 and κ = 0.54, respectively). In contrast, almost no agreement was observed between self-rated and real-life recorded rankings of food intake weight and eating rate (κ = 0.09 and κ = 0.08, respectively). The current study provides evidence that both food intake weight and eating rates per meal vary considerably in real-life per individual. However, based on these behaviours, most students can be correctly classified in regard to their peers based on single school lunches. In contrast, self-reported food intake weight and eating rate are poor predictors of real-life measures. Finally, based on the recorded individual variability of real-life food intake weight and eating rate, it is not advised to rank individuals based on single recordings collected in real-life settings.

The SPLENDID Eating Detection Sensor: Development and Feasibility Study.

BACKGROUND: The available methods for monitoring food intake-which for a great part rely on self-report-often provide biased and incomplete data. Currently, no good technological solutions are available. Hence, the SPLENDID eating detection sensor (an ear-worn device with an air microphone and a photoplethysmogram [PPG] sensor) was developed to enable complete and objective measurements of eating events. The technical performance of this device has been described before. To date, literature is lacking a description of how such a device is perceived and experienced by potential users. OBJECTIVE: The objective of our study was to explore how potential users perceive and experience the SPLENDID eating detection sensor. METHODS: Potential users evaluated the eating detection sensor at different stages of its development: (1) At the start, 12 health professionals (eg, dieticians, personal trainers) were interviewed and a focus group was held with 5 potential end users to find out their thoughts on the concept of the eating detection sensor. (2) Then, preliminary prototypes of the eating detection sensor were tested in a laboratory setting where 23 young adults reported their experiences. (3) Next, the first wearable version of the eating detection sensor was tested in a semicontrolled study where 22 young, overweight adults used the sensor on 2 separate days (from lunch till dinner) and reported their experiences. (4) The final version of the sensor was tested in a 4-week feasibility study by 20 young, overweight adults who reported their experiences. RESULTS: Throughout all the development stages, most individuals were enthusiastic about the eating detection sensor. However, it was stressed multiple times that it was critical that the device be discreet and comfortable to wear for a longer period. In the final study, the eating detection sensor received an average grade of 3.7 for wearer comfort on a scale of 1 to 10. Moreover, experienced discomfort was the main reason for wearing the eating detection sensor <2 hours a day. The participants reported having used the eating detection sensor on 19/28 instructed days on average. CONCLUSIONS: The SPLENDID eating detection sensor, which uses an air microphone and a PPG sensor, is a promising new device that can facilitate the collection of reliable food intake data, as shown by its technical potential. Potential users are enthusiastic, but to be successful wearer comfort and discreetness of the device need to be improved.

Control of Eating Behavior Using a Novel Feedback System.

Subjects eat food from a plate that sits on a scale connected to a computer that records the weight loss of the plate during the meal and makes up a curve of food intake, meal duration and rate of eating modeled by a quadratic equation. The purpose of the method is to change eating behavior by providing visual feedback on the computer screen that the subject can adapt to because her/his own rate of eating appears on the screen during the meal. The data generated by the method is automatically analyzed and fitted to the quadratic equation using a custom made algorithm. The method has the advantage of recording eating behavior objectively and offers the possibility of changing eating behavior both in experiments and in clinical practice. A limitation may be that experimental subjects are affected by the method. The same limitation may be an advantage in clinical practice, as eating behavior is more easily stabilized by the method. A treatment that uses this method has normalized body weight and restored the health of several hundred patients with anorexia nervosa and other eating disorders and has reduced the weight and improved the health of severely overweight patients.

Chewing detection from an in-ear microphone using convolutional neural networks.

Detecting chewing sounds from a microphone placed inside the outer ear for eating behaviour monitoring still remains a challenging task. This is mainly due the difficulty in discriminating non-chewing sounds (e.g. speech or sounds caused by walking) from chews, as well as due to to the high variability of the chewing sounds of different food types. Most approaches rely on detecting distictive structures on the sound wave, or on extracting a set of features and using a classifier to detect chews. In this work, we propose to use feature-learning in the time domain with 1-dimensional convolutional neural networks for for chewing detection. We apply a network of convolutional layers followed by fully connected layers directly on windows of the audio samples to detect chewing activity, and then aggregate individual chews to eating events. Experimental results on a large, semi-free living dataset collected in the context of the SPLENDID project indicate high effectiveness, with an accuracy of 0.980 and F1 score of 0.883.

The SPLENDID chewing detection challenge.

Monitoring of eating behavior using wearable technology is receiving increased attention, driven by the recent advances in wearable devices and mobile phones. One particularly interesting aspect of eating behavior is the monitoring of chewing activity and eating occurrences. There are several chewing sensor types and chewing detection algorithms proposed in the bibliography, however no datasets are publicly available to facilitate evaluation and further research. In this paper, we present a multi-modal dataset of over 60 hours of recordings from 14 participants in semi-free living conditions, collected in the context of the SPLENDID project. The dataset includes raw signals from a photoplethysmography (PPG) sensor and a 3D accelerometer, and a set of extracted features from audio recordings; detailed annotations and ground truth are also provided both at eating event level and at individual chew level. We also provide a baseline evaluation method, and introduce the "challenge" of improving the baseline chewing detection algorithms. The dataset can be downloaded from http: //dx.doi.org/10.17026/dans-zxw-v8gy, and supplementary code can be downloaded from https://github. com/mug-auth/chewing-detection-challenge.git.

A Novel Chewing Detection System Based on PPG, Audio, and Accelerometry.

In the context of dietary management, accurate monitoring of eating habits is receiving increased attention. Wearable sensors, combined with the connectivity and processing of modern smartphones, can be used to robustly extract objective and real-time measurements of human behavior. In particular, for the task of chewing detection, several approaches based on an in-ear microphone can be found in the literature, while other types of sensors have also been reported, such as strain sensors. In this paper, performed in the context of the SPLENDID project, we propose to combine an in-ear microphone with a photoplethysmography (PPG) sensor placed in the ear concha, in a new high accuracy and low sampling rate prototype chewing detection system. We propose a pipeline that initially processes each sensor signal separately, and then fuses both to perform the final detection. Features are extracted from each modality, and support vector machine (SVM) classifiers are used separately to perform snacking detection. Finally, we combine the SVM scores from both signals in a late-fusion scheme, which leads to increased eating detection accuracy. We evaluate the proposed eating monitoring system on a challenging, semifree living dataset of 14 subjects, which includes more than 60 h of audio and PPG signal recordings. Results show that fusing the audio and PPG signals significantly improves the effectiveness of eating event detection, achieving accuracy up to 0.938 and class-weighted accuracy up to 0.892.

A novel approach for chewing detection based on a wearable PPG sensor.

Monitoring of human eating behaviour has been attracting interest over the last few years, as a means to a healthy lifestyle, but also due to its association with serious health conditions, such as eating disorders and obesity. Use of self-reports and other non-automated means of monitoring have been found to be unreliable, compared to the use of wearable sensors. Various modalities have been reported, such as acoustic signal from ear-worn microphones, or signal from wearable strain sensors. In this work, we introduce a new sensor for the task of chewing detection, based on a novel photoplethysmography (PPG) sensor placed on the outer earlobe to perform the task. We also present a processing pipeline that includes two chewing detection algorithms from literature and one new algorithm, to process the captured PPG signal, and present their effectiveness. Experiments are performed on an annotated dataset recorded from 21 individuals, including more than 10 hours of eating and non-eating activities. Results show that the PPG sensor can be successfully used to support dietary monitoring.

A parametric Probabilistic Context-Free Grammar for food intake analysis based on continuous meal weight measurements.

Monitoring and modification of eating behaviour through continuous meal weight measurements has been successfully applied in clinical practice to treat obesity and eating disorders. For this purpose, the Mandometer, a plate scale, along with video recordings of subjects during the course of single meals, has been used to assist clinicians in measuring relevant food intake parameters. In this work, we present a novel algorithm for automatically constructing a subject's food intake curve using only the Mandometer weight measurements. This eliminates the need for direct clinical observation or video recordings, thus significantly reducing the manual effort required for analysis. The proposed algorithm aims at identifying specific meal related events (e.g. bites, food additions, artifacts), by applying an adaptive pre-processing stage using Delta coefficients, followed by event detection based on a parametric Probabilistic Context-Free Grammar on the derivative of the recorded sequence. Experimental results on a dataset of 114 meals from individuals suffering from obesity or eating disorders, as well as from individuals with normal BMI, demonstrate the effectiveness of the proposed approach.