Synthetic Data in Human Analysis: A Survey.

Deep neural networks have become prevalent in human analysis, boosting the performance of applications, such as biometric recognition, action recognition, as well as person re-identification. However, the performance of such networks scales with the available training data. In human analysis, the demand for large-scale datasets poses a severe challenge, as data collection is tedious, time-expensive, costly and must comply with data protection laws. Current research investigates the generation of synthetic data as an efficient and privacy-ensuring alternative to collecting real data in the field. This survey introduces the basic definitions and methodologies, essential when generating and employing synthetic data for human analysis. We summarise current state-of-the-art methods and the main benefits of using synthetic data. We also provide an overview of publicly available synthetic datasets and generation models. Finally, we discuss limitations, as well as open research problems in this field. This survey is intended for researchers and practitioners in the field of human analysis.

Learning Invariance From Generated Variance for Unsupervised Person Re-Identification.

This work focuses on unsupervised representation learning in person re-identification (ReID). Recent self-supervised contrastive learning methods learn invariance by maximizing the representation similarity between two augmented views of a same image. However, traditional data augmentation may bring to the fore undesirable distortions on identity features, which is not always favorable in id-sensitive ReID tasks. In this article, we propose to replace traditional data augmentation with a generative adversarial network (GAN) that is targeted to generate augmented views for contrastive learning. A 3D mesh guided person image generator is proposed to disentangle a person image into id-related and id-unrelated features. Deviating from previous GAN-based ReID methods that only work in id-unrelated space (pose and camera style), we conduct GAN-based augmentation on both id-unrelated and id-related features. We further propose specific contrastive losses to help our network learn invariance from id-unrelated and id-related augmentations. By jointly training the generative and the contrastive modules, our method achieves new state-of-the-art unsupervised person ReID performance on mainstream large-scale benchmarks.

Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection.

Designing activity detection systems that can be successfully deployed in daily-living environments requires datasets that pose the challenges typical of real-world scenarios. In this paper, we introduce a new untrimmed daily-living dataset that features several real-world challenges: Toyota Smarthome Untrimmed (TSU). TSU contains a wide variety of activities performed in a spontaneous manner. The dataset contains dense annotations including elementary, composite activities and activities involving interactions with objects. We provide an analysis of the real-world challenges featured by our dataset, highlighting the open issues for detection algorithms. We show that current state-of-the-art methods fail to achieve satisfactory performance on the TSU dataset. Therefore, we propose a new baseline method for activity detection to tackle the novel challenges provided by our dataset. This method leverages one modality (i.e. optic flow) to generate the attention weights to guide another modality (i.e RGB) to better detect the activity boundaries. This is particularly beneficial to detect activities characterized by high temporal variance. We show that the method we propose outperforms state-of-the-art methods on TSU and on another popular challenging dataset, Charades.

Feasibility Study of an Internet-Based Platform for Tele-Neuropsychological Assessment of Elderly in Remote Areas.

Today, in rural isolated areas or so-called 'medical deserts', access to diagnosis and care is very limited. With the current pandemic crisis, now even more than ever, telemedicine platforms are gradually more employed for remote medical assessment. Only a few are tailored to comprehensive teleneuropsychological assessment of older adults. Hence, our study focuses on evaluating the feasibility of performing a remote neuropsychological assessment of older adults suffering from a cognitive complaint. 50 participants (aged 55 and older) were recruited at the local hospital of Digne-les-Bains, France. A brief neuropsychological assessment including a short clinical interview and several validated neuropsychological tests was administered in two conditions, once by Teleneuropsychology (TNP) and once by Face-to-Face (FTF) in a crossover design. Acceptability and user experience was assessed through questionnaires. Results show high agreement in most tests between the FTF and TNP conditions. The TNP was overall well accepted by the participants. However, differences in test performances were observed, which urges the need to validate TNP tests with broader samples with normative data.

VPN++: Rethinking Video-Pose Embeddings for Understanding Activities of Daily Living.

Many attempts have been made towards combining RGB and 3D poses for the recognition of Activities of Daily Living (ADL). ADL may look very similar and often necessitate to model fine-grained details to distinguish them. Because the recent 3D ConvNets are too rigid to capture the subtle visual patterns across an action, this research direction is dominated by methods combining RGB and 3D Poses. But the cost of computing 3D poses from RGB stream is high in the absence of appropriate sensors. This limits the usage of aforementioned approaches in real-world applications requiring low latency. Then, how to best take advantage of 3D Poses for recognizing ADL? To this end, we propose an extension of a pose driven attention mechanism: Video-Pose Network (VPN), exploring two distinct directions. One is to transfer the Pose knowledge into RGB through a feature-level distillation and the other towards mimicking pose driven attention through an attention-level distillation. Finally, these two approaches are integrated into a single model, we call VPN++. It is worth noting that VPN++ exploits the pose embeddings at training via distillation but not at inference. We show that VPN++ is not only effective but also provides a high speed up and high resilience to noisy Poses. VPN++, with or without 3D Poses, outperforms the representative baselines on 4 public datasets. Code is available at https://github.com/srijandas07/vpnplusplus.

One-class autoencoder approach for optimal electrode set identification in wearable EEG event monitoring.

A limiting factor towards the wide use of wearable devices for continuous healthcare monitoring is their cumbersome and obtrusive nature. This is particularly true in electroencephalography (EEG), where numerous electrodes are placed in contact with the scalp to perform brain activity recordings. In this work, we propose to identify the optimal wearable EEG electrode set, in terms of minimal number of electrodes, comfortable location and performance, for EEG-based event detection and monitoring. By relying on the demonstrated power of autoencoder (AE) networks to learn latent representations from high-dimensional data, our proposed strategy trains an AE architecture in a one-class classification setup with different electrode combinations as input data. The model performance is assessed using the F-score. Alpha waves detection is the use case through which we demonstrate that the proposed method allows to detect a brain state from an optimal set of electrodes. The so-called wearable configuration, consisting of electrodes in the forehead and behind the ear, is the chosen optimal set, with an average F-score of 0.78. This study highlights the beneficial impact of a learning-based approach in the design of wearable devices for real-life event-related monitoring.

Remote cognitive assessment of older adults in rural areas by telemedicine and automatic speech and video analysis: protocol for a cross-over feasibility study.

INTRODUCTION: Early detection of cognitive impairments is crucial for the successful implementation of preventive strategies. However, in rural isolated areas or so-called 'medical deserts', access to diagnosis and care is very limited. With the current pandemic crisis, now even more than ever, remote solutions such as telemedicine platforms represent great potential and can help to overcome this barrier. Moreover, current advances made in voice and image analysis can help overcome the barrier of physical distance by providing additional information on a patients' emotional and cognitive state. Therefore, the aim of this study is to evaluate the feasibility and reliability of a videoconference system for remote cognitive testing empowered by automatic speech and video analysis. METHODS AND ANALYSIS: 60 participants (aged 55 and older) with and without cognitive impairment will be recruited. A complete neuropsychological assessment including a short clinical interview will be administered in two conditions, once by telemedicine and once by face-to-face. The order of administration procedure will be counterbalanced so half of the sample starts with the videoconference condition and the other half with the face-to-face condition. Acceptability and user experience will be assessed among participants and clinicians in a qualitative and quantitative manner. Speech and video features will be extracted and analysed to obtain additional information on mood and engagement levels. In a subgroup, measurements of stress indicators such as heart rate and skin conductance will be compared. ETHICS AND DISSEMINATION: The procedures are not invasive and there are no expected risks or burdens to participants. All participants will be informed that this is an observational study and their consent taken prior to the experiment. Demonstration of the effectiveness of such technology makes it possible to diffuse its use across all rural areas ('medical deserts') and thus, to improve the early diagnosis of neurodegenerative pathologies, while providing data crucial for basic research. Results from this study will be published in peer-reviewed journals.

Pilot Study to Assess the Feasibility of a Mobile Unit for Remote Cognitive Screening of Isolated Elderly in Rural Areas.

BACKGROUND: Given the current COVID-19 pandemic situation, now more than ever, remote solutions for assessing and monitoring individuals with cognitive impairment are urgently needed. Older adults in particular, living in isolated rural areas or so-called 'medical deserts', are facing major difficulties in getting access to diagnosis and care. Telemedical approaches to assessments are promising and seem well accepted, reducing the burden of bringing patients to specialized clinics. However, many older adults are not yet adequately equipped to allow for proper implementation of this technology. A potential solution could be a mobile unit in the form of a van, equipped with the telemedical system which comes to the patients' home. The aim of this proof-of-concept study is to evaluate the feasibility and reliability of such mobile unit settings for remote cognitive testing. Methods and analysis: eight participants (aged between 69 and 86 years old) from the city of Digne-Les-Bains volunteered for this study. A basic neuropsychological assessment, including a short clinical interview, is administered in two conditions, by telemedicine in a mobile clinic (equipped van) at a participants' home and face to face in a specialized clinic. The administration procedure order is randomized, and the results are compared with each other. Acceptability and user experience are assessed among participants and clinicians in a qualitative and quantitative manner. Measurements of stress indicators were collected for comparison. RESULTS: The analysis revealed no significant differences in test results between the two administration procedures. Participants were, overall, very satisfied with the mobile clinic experience and found the use of the telemedical system relatively easy. CONCLUSION: A mobile unit equipped with a telemedical service could represent a solution for remote cognitive testing overcoming barriers in rural areas to access specialized diagnosis and care.

Efficacy of serious exergames in improving neuropsychiatric symptoms in neurocognitive disorders: Results of the X-TORP cluster randomized trial.

INTRODUCTION: The aim of this study was to evaluate the efficacy of a serious exergame in improving the neuropsychiatric symptoms of patients with neurocognitive disorders. METHODS: X-Torp is a serious exergame combining motor and cognitive activities. Ninety-one subjects (mean age = 81.7 years, mean Mini-Mental State Examination = 18.3) were recruited in 16 centers. Centers were randomized into intervention and control centers. Subjects underwent assessment for cognitive and behavioral symptoms at baseline (BL), the end of the intervention (W12), and 12 weeks after the end of the intervention (W24). RESULTS: The comparison of neuropsychiatric symptoms between BL and W12 and W24 showed that subjects of the intervention group improved in apathy between BL and W12. Mixed analysis (time BL, W12, W24 x group) indicated a significant increase in apathy and neuropsychiatric symptoms in the control subjects. DISCUSSION: The use of X-Torp improved neuropsychiatric symptoms, particularly apathy. Future studies should more consistently use behavioral and neuropsychiatric symptoms as outcome measures.

A French-Greek Cross-Site Comparison Study of the Use of Automatic Video Analyses for the Assessment of Autonomy in Dementia Patients.

BACKGROUND: At present, the assessment of autonomy in daily living activities, one of the key symptoms in Alzheimer's disease (AD), involves clinical rating scales. METHODS: In total, 109 participants were included. In particular, 11 participants during a pre-test in Nice, France, and 98 participants (27 AD, 38 mild cognitive impairment-MCI-and 33 healthy controls-HC) in Thessaloniki, Greece, carried out a standardized scenario consisting of several instrumental activities of daily living (IADLs), such as making a phone call or preparing a pillbox while being recorded. Data were processed by a platform of video signal analysis in order to extract kinematic parameters, detecting activities undertaken by the participant. RESULTS: The video analysis data can be used to assess IADL task quality and provide clinicians with objective measurements of the patients' performance. Furthermore, it reveals that the HC statistically significantly outperformed the MCI, which had better performance compared to the AD participants. CONCLUSIONS: Accurate activity recognition data for the analyses of the performance on IADL activities were obtained.

Recommendations for the Nonpharmacological Treatment of Apathy in Brain Disorders.

Apathy is a common neuropsychiatric syndrome observed across many neurocognitive and psychiatric disorders. Although there are currently no definitive standard therapies for the treatment of apathy, nonpharmacological treatment (NPT) is often considered to be at the forefront of clinical management. However, guidelines on how to select, prescribe, and administer NPT in clinical practice are lacking. Furthermore, although new Information and Communication Technologies (ICT) are beginning to be employed in NPT, their role is still unclear. The objective of the present work is to provide recommendations for the use of NPT for apathy, and to discuss the role of ICT in this domain, based on opinions gathered from experts in the field. The expert panel included 20 researchers and healthcare professionals working on brain disorders and apathy. Following a standard Delphi methodology, experts answered questions via several rounds of web-surveys, and then discussed the results in a plenary meeting. The experts suggested that NPT are useful to consider as therapy for people presenting with different neurocognitive and psychiatric diseases at all stages, with evidence of apathy across domains. The presence of a therapist and/or a caregiver is important in delivering NPT effectively, but parts of the treatment may be performed by the patient alone. NPT can be delivered both in clinical settings and at home. However, while remote treatment delivery may be cost and time-effective, it should be considered with caution, and tailored based on the patient's cognitive and physical profile and living conditions.

An Unsupervised Framework for Online Spatiotemporal Detection of Activities of Daily Living by Hierarchical Activity Models.

Automatic detection and analysis of human activities captured by various sensors (e.g., sequences of images captured by RGB camera) play an essential role in various research fields in order to understand the semantic content of a captured scene. The main focus of the earlier studies has been widely on supervised classification problem, where a label is assigned to a given short clip. Nevertheless, in real-world scenarios, such as in Activities of Daily Living (ADL), the challenge is to automatically browse long-term (days and weeks) stream of videos to identify segments with semantics corresponding to the model activities and their temporal boundaries. This paper proposes an unsupervised solution to address this problem by generating hierarchical models that combine global trajectory information with local dynamics of the human body. Global information helps in modeling the spatiotemporal evolution of long-term activities, hence, their spatial and temporal localization. Moreover, the local dynamic information incorporates complex local motion patterns of daily activities into the models. Our proposed method is evaluated using realistic datasets captured from observation rooms in hospitals and nursing homes. The experimental data on a variety of monitoring scenarios in hospital settings reveals how this framework can be exploited to provide timely diagnose and medical interventions for cognitive disorders, such as Alzheimer's disease. The obtained results show that our framework is a promising attempt capable of generating activity models without any supervision.

Assistive Technologies to Address Capabilities of People with Dementia: From Research to Practice.

Assistive technologies became pervasive and virtually present in all our life domains. They can be either an enabler or an obstacle leading to social exclusion. The Fondation Médéric Alzheimer gathered international experts of dementia care, with backgrounds in biomedical, human and social sciences, to analyze how assistive technologies can address the capabilities of people with dementia, on the basis of their needs. Discussion covered the unmet needs of people with dementia, the domains of daily life activities where assistive technologies can provide help to people with dementia, the enabling and empowering impact of technology to improve their safety and wellbeing, barriers and limits of use, technology assessment, ethical and legal issues. The capability approach (possible freedom) appears particularly relevant in person-centered dementia care and technology development. The focus is not on the solution, rather on what the person can do with it: seeing dementia as disability, with technology as an enabler to promote capabilities of the person, provides a useful framework for both research and practice. This article summarizes how these concepts took momentum in professional practice and public policies in the past 15 years (2000-2015), discusses current issues in the design, development and economic model of assistive technologies for people with dementia, and covers how these technologies are being used and assessed.

Recommendations for the Use of Serious Games in Neurodegenerative Disorders: 2016 Delphi Panel.

The use of Serious Games (SG) in the health domain is expanding. In the field of neurodegenerative disorders (ND) such as Alzheimer's disease, SG are currently employed both to support and improve the assessment of different functional and cognitive abilities, and to provide alternative solutions for patients' treatment, stimulation, and rehabilitation. As the field is quite young, recommendations on the use of SG in people with ND are still rare. In 2014 we proposed some initial recommendations (Robert et al., 2014). The aim of the present work was to update them, thanks to opinions gathered by experts in the field during an expert Delphi panel. Results confirmed that SG are adapted to elderly people with mild cognitive impairment (MCI) and dementia, and can be employed for several purposes, including assessment, stimulation, and improving wellbeing, with some differences depending on the population (e.g., physical stimulation may be better suited for people with MCI). SG are more adapted for use with trained caregivers (both at home and in clinical settings), with a frequency ranging from 2 to 4 times a week. Importantly, the target of SG, their frequency of use and the context in which they are played depend on the SG typology (e.g., Exergame, cognitive game), and should be personalized with the help of a clinician.

Online Recognition of Daily Activities by Color-Depth Sensing and Knowledge Models.

Visual activity recognition plays a fundamental role in several research fields as a way to extract semantic meaning of images and videos. Prior work has mostly focused on classification tasks, where a label is given for a video clip. However, real life scenarios require a method to browse a continuous video flow, automatically identify relevant temporal segments and classify them accordingly to target activities. This paper proposes a knowledge-driven event recognition framework to address this problem. The novelty of the method lies in the combination of a constraint-based ontology language for event modeling with robust algorithms to detect, track and re-identify people using color-depth sensing (Kinect® sensor). This combination enables to model and recognize longer and more complex events and to incorporate domain knowledge and 3D information into the same models. Moreover, the ontology-driven approach enables human understanding of system decisions and facilitates knowledge transfer across different scenes. The proposed framework is evaluated with real-world recordings of seniors carrying out unscripted, daily activities at hospital observation rooms and nursing homes. Results demonstrated that the proposed framework outperforms state-of-the-art methods in a variety of activities and datasets, and it is robust to variable and low-frame rate recordings. Further work will investigate how to extend the proposed framework with uncertainty management techniques to handle strong occlusion and ambiguous semantics, and how to exploit it to further support medicine on the timely diagnosis of cognitive disorders, such as Alzheimer's disease.

Physical and Cognitive Stimulation Using an Exergame in Subjects with Normal Aging, Mild and Moderate Cognitive Impairment.

BACKGROUND: The use of Serious exerGames (SeG) as enriched environments (EE), which promotes cognitive simulation with physical activity in a positive emotional context, has been proposed to represent a powerful method to slow down the decline due to neurodegenerative diseases (ND), such as Alzheimer's disease (AD). However, so far, no SeG targeting EE has been tested in ND subjects. OBJECTIVE: This study aimed at evaluating the usability and short-term training effects of X-Torp, an action SeG designed for elderly ND subjects with mild cognitive impairment (MCI) and AD. METHODS: X-Torp is a SeG played using the Microsoft® Kinect™. 10 ND subjects and 8 healthy elderly controls (HEC) were enrolled in a 1-month program with three training sessions per week. Usability was evaluated through game time, game performance, the aerobic intensity level reached, perceived emotions, and perceived usability. RESULTS: All participants successfully completed the training program. ND subjects played less and had a lower game performance compared to HEC. During the sessions, ND subjects maintained a light intensity of aerobic activity, while HEC maintained a moderate intensity. Both groups experienced only positive emotions, and reported a 'moderate' to 'high' perceived competence, a 'moderate' game difficulty, and a 'high' interest in the game. CONCLUSION: Usability results suggest that X-Torp represents a usable EE for healthy subjects and persons with MCI and AD. However, in order to reach moderate or high intensity of aerobic activity, X-Torp control modes should be adapted to become more physically stimulating.

Semantic Event Fusion of Different Visual Modality Concepts for Activity Recognition.

Combining multimodal concept streams from heterogeneous sensors is a problem superficially explored for activity recognition. Most studies explore simple sensors in nearly perfect conditions, where temporal synchronization is guaranteed. Sophisticated fusion schemes adopt problem-specific graphical representations of events that are generally deeply linked with their training data and focused on a single sensor. This paper proposes a hybrid framework between knowledge-driven and probabilistic-driven methods for event representation and recognition. It separates semantic modeling from raw sensor data by using an intermediate semantic representation, namely concepts. It introduces an algorithm for sensor alignment that uses concept similarity as a surrogate for the inaccurate temporal information of real life scenarios. Finally, it proposes the combined use of an ontology language, to overcome the rigidity of previous approaches at model definition, and a probabilistic interpretation for ontological models, which equips the framework with a mechanism to handle noisy and ambiguous concept observations, an ability that most knowledge-driven methods lack. We evaluate our contributions in multimodal recordings of elderly people carrying out IADLs. Results demonstrated that the proposed framework outperforms baseline methods both in event recognition performance and in delimiting the temporal boundaries of event instances.

Dementia beyond 2025: Knowledge and uncertainties.

Given that there may well be no significant advances in drug development before 2025, prevention of dementia-Alzheimer's disease through the management of vascular and lifestyle-related risk factors may be a more realistic goal than treatment. Level of education and cognitive reserve assessment in neuropsychological testing deserve attention, as well as cultural, social, and economic aspects of caregiving. Assistive technologies for dementia care remain complex. Serious games are emerging as virtual educational and pleasurable tools, designed for individual and cooperative skill building. Public policies are likely to pursue improving awareness and understanding of dementia; providing good quality early diagnosis and intervention for all; improving quality of care from diagnosis to the end of life, using clinical and economic end points; delivering dementia strategies quicker, with an impact on more people. Dementia should remain presented as a stand-alone concept, distinct from frailty or loss of autonomy. The basic science of sensory impairment and social engagement in people with dementia needs to be developed. E-learning and serious games programs may enhance public and professional education. Faced with funding shortage, new professional dynamics and economic models may emerge through coordinated, flexible research networks. Psychosocial research could be viewed as an investment in quality of care, rather than an academic achievement in a few centers of excellence. This would help provide a competitive advantage to the best operators. Stemming from care needs, a logical, systems approach to dementia care environment through organizational, architectural, and psychosocial interventions may be developed, to help reduce symptoms in people with dementia and enhance quality of life. Dementia-friendly environments, culture, and domesticity are key factors for such interventions.

Ecological Assessment of Autonomy in Instrumental Activities of Daily Living in Dementia Patients by the Means of an Automatic Video Monitoring System.

Currently, the assessment of autonomy and functional ability involves clinical rating scales. However, scales are often limited in their ability to provide objective and sensitive information. By contrast, information and communication technologies may overcome these limitations by capturing more fully functional as well as cognitive disturbances associated with Alzheimer disease (AD). We investigated the quantitative assessment of autonomy in dementia patients based not only on gait analysis but also on the participant performance on instrumental activities of daily living (IADL) automatically recognized by a video event monitoring system (EMS). Three groups of participants (healthy controls, mild cognitive impairment, and AD patients) had to carry out a standardized scenario consisting of physical tasks (single and dual task) and several IADL such as preparing a pillbox or making a phone call while being recorded. After, video sensor data were processed by an EMS that automatically extracts kinematic parameters of the participants' gait and recognizes their carried out activities. These parameters were then used for the assessment of the participants' performance levels, here referred as autonomy. Autonomy assessment was approached as classification task using artificial intelligence methods that takes as input the parameters extracted by the EMS, here referred as behavioral profile. Activities were accurately recognized by the EMS with high precision. The most accurately recognized activities were "prepare medication" with 93% and "using phone" with 89% precision. The diagnostic group classifier obtained a precision of 73.46% when combining the analyses of physical tasks with IADL. In a further analysis, the created autonomy group classifier which obtained a precision of 83.67% when combining physical tasks and IADL. Results suggest that it is possible to quantitatively assess IADL functioning supported by an EMS and that even based on the extracted data the groups could be classified with high accuracy. This means that the use of such technologies may provide clinicians with diagnostic relevant information to improve autonomy assessment in real time decreasing observer biases.