Predicting daily cognition and lifestyle behaviors for older adults using smart home data and ecological momentary assessment.

OBJECTIVE: Extraction of digital markers from passive sensors placed in homes is a promising method for understanding real-world behaviors. In this study, machine learning (ML) and multilevel modeling (MLM) are used to examine types of digital markers and whether smart home sensors can predict cognitive functioning, lifestyle behaviors, and contextual factors measured through ecological momentary assessment (EMA). METHOD: Smart home sensors were installed in the homes of 44 community-dwelling midlife and older adults for 3-4 months. Sensor data were categorized into eight digital markers. Participants responded to iPad-delivered EMA prompts 4×/day for 2 wk. Prompts included an n-back task and survey on recent (past 2 h) lifestyle and contextual factors. RESULTS: ML marker rankings revealed that sensor counts (indicating increased activity) and time outside the home were among the most influential markers for all survey questions. Additionally, MLM revealed for every 1000 sensor counts, mental sharpness, social, physical, and cognitive EMA responses increased by 0.134-0.155 points on a 5-point scale. For every additional 30-minutes spent outside home, social, physical, and cognitive EMA responses increased by 0.596, 0.472, and 0.157 points. Advanced ML joint classification/regression significantly predicted EMA responses from smart home digital markers with error of 0.370 on a 5-point scale, and n-back performance with a normalized error of 0.040. CONCLUSION: Results from ML and MLM were complimentary and comparable, suggesting that machine learning may be used to develop generalized models to predict everyday cognition and track lifestyle behaviors and contextual factors that impact health outcomes using smart home sensor data.

CALDA: Improving Multi-Source Time Series Domain Adaptation With Contrastive Adversarial Learning.

Unsupervised domain adaptation (UDA) provides a strategy for improving machine learning performance in data-rich (target) domains where ground truth labels are inaccessible but can be found in related (source) domains. In cases where meta-domain information such as label distributions is available, weak supervision can further boost performance. We propose a novel framework, CALDA, to tackle these two problems. CALDA synergistically combines the principles of contrastive learning and adversarial learning to robustly support multi-source UDA (MS-UDA) for time series data. Similar to prior methods, CALDA utilizes adversarial learning to align source and target feature representations. Unlike prior approaches, CALDA additionally leverages cross-source label information across domains. CALDA pulls examples with the same label close to each other, while pushing apart examples with different labels, reshaping the space through contrastive learning. Unlike prior contrastive adaptation methods, CALDA requires neither data augmentation nor pseudo labeling, which may be more challenging for time series. We empirically validate our proposed approach. Based on results from human activity recognition, electromyography, and synthetic datasets, we find utilizing cross-source information improves performance over prior time series and contrastive methods. Weak supervision further improves performance, even in the presence of noise, allowing CALDA to offer generalizable strategies for MS-UDA.

The Indoor Predictability of Human Mobility: Estimating Mobility with Smart Home Sensors.

Analyzing human mobility patterns is valuable for understanding human behavior and providing location-anticipating services. In this work, we theoretically estimate the predictability of human movement for indoor settings, a problem that has not yet been tackled by the community. To validate the model, we utilize location data collected by ambient sensors in residential settings. The data support the model and allow us to contrast the predictability of various groups, including single-resident homes, homes with multiple residents, and homes with pets.

But will they use it? Predictors of adoption of an electronic memory aid in individuals with amnestic mild cognitive impairment.

OBJECTIVE: Electronic memory aids are being researched and developed widely to assist the everyday functioning of individuals experiencing cognitive decline. Although development studies show promise in the initial use of electronic memory aids, little is known about the factors that influence adoption of these aids after training ends. METHOD: We analyzed the baseline characteristics (e.g., demographics, cognitive performance) and training usage (e.g., frequency and pattern of use) of 32 older adults experiencing amnestic mild cognitive impairment who participated in a pilot clinical trial with an electronic memory and management aid (EMMA) tablet application. Sixteen participants who were still using EMMA at 3-months posttraining were defined as "adopters," whereas the 16 participants who were not using EMMA at 3-months posttraining were defined as "nonadopters." RESULTS: Adopters scored higher on baseline delayed memory (Cohen's d = .87) and language (Cohen's d = .82) index scores than nonadopters. Adopters also interacted with EMMA more frequently (Cohen's d = 1.34) and in greater quantities (Cohen's d > .87) than nonadopters by Week 2 of training. Stepwise logistic regression revealed that higher baseline language score and increased frequency of use during training significantly predicted classification of adopters at 3-months posttraining. CONCLUSIONS: Adoption of this electronic memory aid was enhanced by teaching the aid to individuals who demonstrated average-level language abilities and who used the aid on average eight times per day during training. Encouraging individuals to use the aid early and often during training can increase adoption of electronic memory aids. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Piloting Smart Home Sensors to Detect Overnight Respiratory and Withdrawal Symptoms in Adults Prescribed Opioids.

BACKGROUND: Novel strategies are needed to curb the opioid overdose epidemic. Smart home sensors have been successfully deployed as digital biomarkers to monitor health conditions, yet they have not been used to assess symptoms important to opioid use and overdose risks. AIM: This study piloted smart home sensors and investigated their ability to accurately detect clinically pertinent symptoms indicative of opioid withdrawal or respiratory depression in adults prescribed methadone. METHODS: Participants (n = 4; 3 completed) were adults with opioid use disorder exhibiting moderate levels of pain intensity, withdrawal symptoms, and sleep disturbance. Participants were invited to two 8-hour nighttime sleep opportunities to be recorded in a sleep research laboratory, using observed polysomnography and ambient smart home sensors attached to lab bedroom walls. Measures of feasibility included completeness of data captured. Accuracy was determined by comparing polysomnographic data of sleep/wake and respiratory status assessments with time and event sensor data. RESULTS: Smart home sensors captured overnight data on 48 out of 64 hours (75% completeness). Sensors detected sleep/wake patterns in alignment with observed sleep episodes captured by polysomnography 89.4% of the time. Apnea events (n = 118) were only detected with smart home sensors in two episodes where oxygen desaturations were less severe (>80%). CONCLUSIONS: Smart home technology could serve as a less invasive substitute for biologic monitoring for adults with pain, sleep disturbances, and opioid withdrawal symptoms. Supplemental sensors should be added to detect apnea events. Such innovations could provide a step forward in assessing overnight symptoms important to populations taking opioids.

Multimodal Fusion of Smart Home and Text-based Behavior Markers for Clinical Assessment Prediction.

New modes of technology are offering unprecedented opportunities to unobtrusively collect data about people's behavior. While there are many use cases for such information, we explore its utility for predicting multiple clinical assessment scores. Because clinical assessments are typically used as screening tools for impairment and disease, such as mild cognitive impairment (MCI), automatically mapping behavioral data to assessment scores can help detect changes in health and behavior across time. In this paper, we aim to extract behavior markers from two modalities, a smart home environment and a custom digital memory notebook app, for mapping to ten clinical assessments that are relevant for monitoring MCI onset and changes in cognitive health. Smart home-based behavior markers reflect hourly, daily, and weekly activity patterns, while app-based behavior markers reflect app usage and writing content/style derived from free-form journal entries. We describe machine learning techniques for fusing these multimodal behavior markers and utilizing joint prediction. We evaluate our approach using three regression algorithms and data from 14 participants with MCI living in a smart home environment. We observed moderate to large correlations between predicted and ground-truth assessment scores, ranging from r = 0.601 to r = 0.871 for each clinical assessment.

Automated Cognitive Health Assessment Using Partially Complete Time Series Sensor Data.

BACKGROUND: Behavior and health are inextricably linked. As a result, continuous wearable sensor data offer the potential to predict clinical measures. However, interruptions in the data collection occur, which create a need for strategic data imputation. OBJECTIVE: The objective of this work is to adapt a data generation algorithm to impute multivariate time series data. This will allow us to create digital behavior markers that can predict clinical health measures. METHODS: We created a bidirectional time series generative adversarial network to impute missing sensor readings. Values are imputed based on relationships between multiple fields and multiple points in time, for single time points or larger time gaps. From the complete data, digital behavior markers are extracted and are mapped to predicted clinical measures. RESULTS: We validate our approach using continuous smartwatch data for n = 14 participants. When reconstructing omitted data, we observe an average normalized mean absolute error of 0.0197. We then create machine learning models to predict clinical measures from the reconstructed, complete data with correlations ranging from r = 0.1230 to r = 0.7623. This work indicates that wearable sensor data collected in the wild can be used to offer insights on a person's health in natural settings.

Detecting Smartwatch-based Behavior Change in Response to a Multi-domain Brain Health Intervention.

In this study, we introduce and validate a computational method to detect lifestyle change that occurs in response to a multi-domain healthy brain aging intervention. To detect behavior change, digital behavior markers (DM) are extracted from smartwatch sensor data and a Permutation-based Change Detection (PCD) algorithm quantifies the change in marker-based behavior from a pre-intervention, one-week baseline. To validate the method, we verify that changes are successfully detected from synthetic data with known pattern differences. Next, we employ this method to detect overall behavior change for n=28 BHI subjects and n=17 age-matched control subjects. For these individuals, we observe a monotonic increase in behavior change from the baseline week with a slope of 0.7460 for the intervention group and a slope of 0.0230 for the control group. Finally, we utilize a random forest algorithm to perform leave-one-subject-out prediction of intervention versus control subjects based on digital marker delta values. The random forest predicts whether the subject is in the intervention or control group with an accuracy of 0.87. This work has implications for capturing objective, continuous data to inform our understanding of intervention adoption and impact.

PIP: Pictorial Interpretable Prototype Learning for Time Series Classification.

Time series classifiers are not only challenging to design, but they are also notoriously difficult to deploy for critical applications because end users may not understand or trust black-box models. Despite new efforts, explanations generated by other interpretable time series models are complicated for non-engineers to understand. The goal of PIP is to provide time series explanations that are tailored toward specific end users. To address the challenge, this paper introduces PIP, a novel deep learning architecture that jointly learns classification models and meaningful visual class prototypes. PIP allows users to train the model on their choice of class illustrations. Thus, PIP can create a user-friendly explanation by leaning on end-users definitions. We hypothesize that a pictorial description is an effective way to communicate a learned concept to non-expert users. Based on an end-user experiment with participants from multiple backgrounds, PIP offers an improved combination of accuracy and interpretability over baseline methods for time series classification.

Multi-Person Activity Recognition in Continuously Monitored Smart Homes.

Activity recognizers are challenging to design for continuous, in-home settings. However, they are notoriously difficult to create when there is more than one resident in the home. Despite recent efforts, there remains a need for an algorithm that can estimate the number of residents in the house, split a time series stream into separate substreams, and accurately identify each resident's activities. To address this challenge, we introduce Gamut. This novel unsupervised method jointly estimates the number of residents and associates sensor readings with those residents, based on a multi-target Gaussian mixture probability hypothesis density filter. We hypothesize that the proposed method will offer robust recognition for homes with two or more residents. In experiments with labeled data collected from 50 single-resident and 11 multi-resident homes, we observe that Gamut outperforms previous unsupervised and supervised methods, offering a robust strategy to track behavioral routines in complex settings.

Nurse-in-the-loop smart home detection of health events associated with diagnosed chronic conditions: A case-event series.

Background: Telehealth and home-based care options significantly expanded during the SARS-CoV2 pandemic. Sophisticated, remote monitoring technologies now exist that support at-home care. Advances in the research of smart homes for health monitoring have shown these technologies are capable of recognizing and predicting health changes in near-real time. However, few nurses are familiar enough with this technology to use smart homes for optimizing patient care or expanding their reach into the home between healthcare touch points. Objective: The objective of this work is to explore a partnership between nurses and smart homes for automated remote monitoring and assessing of patient health. We present a series of health event cases to demonstrate how this partnership may be harnessed to effectively detect and report on clinically relevant health events that can be automatically detected by smart homes. Participants: 25 participants with multiple chronic health conditions. Methods: Ambient sensors were installed in the homes of 25 participants with multiple chronic health conditions. Motion, light, temperature, and door usage data were continuously collected from participants' homes. Descriptions of health events and participants' associated behaviors were captured via weekly nursing telehealth visits with study participants and used to analyze sensor data representing health events. Two cases of participants with congestive heart failure exacerbations, one case of urinary tract infection, two cases of bowel inflammation flares, and four cases of participants with sleep interruption were explored. Results: For each case, clinically relevant health events aligned with changes from baseline in behavior data patterns derived from sensors installed in the participant's home. In some cases, the detected event was precipitated by additional behavior patterns that could be used to predict the event. Conclusions: We found evidence in this case series that continuous sensor-based monitoring of patient behavior in home settings may be used to provide automated detection of health events. Nursing insights into smart home sensor data could be used to initiate preventive strategies and provide timely intervention. Tweetable abstract: Nurses partnered with smart homes could detect exacerbations of health conditions at home leading to early intervention.

Memory-Aware Active Learning in Mobile Sensing Systems.

We propose a novel active learning framework for activity recognition using wearable sensors. Our work is unique in that it takes limitations of the oracle into account when selecting sensor data for annotation by the oracle. Our approach is inspired by human-beings' limited capacity to respond to prompts on their mobile device. This capacity constraint is manifested not only in the number of queries that a person can respond to in a given time-frame but also in the time lag between the query issuance and the oracle response. We introduce the notion of mindful active learning and propose a computational framework, called EMMA, to maximize the active learning performance taking informativeness of sensor data, query budget, and human memory into account. We formulate this optimization problem, propose an approach to model memory retention, discuss the complexity of the problem, and propose a greedy heuristic to solve the optimization problem. Additionally, we design an approach to perform mindful active learning in batch where multiple sensor observations are selected simultaneously for querying the oracle. We demonstrate the effectiveness of our approach using three publicly available activity datasets and by simulating oracles with various memory strengths. We show that the activity recognition accuracy ranges from 21% to 97% depending on memory strength, query budget, and difficulty of the machine learning task. Our results also indicate that EMMA achieves an accuracy level that is, on average, 13.5% higher than the case when only informativeness of the sensor data is considered for active learning. Moreover, we show that the performance of our approach is at most 20% less than the experimental upper-bound and up to 80% higher than the experimental lower-bound. To evaluate the performance of EMMA for batch active learning, we design two instantiations of EMMA to perform active learning in batch mode. We show that these algorithms improve the algorithm training time at the cost of a reduced accuracy in performance. Another finding in our work is that integrating clustering into the process of selecting sensor observations for batch active learning improves the activity learning performance by 11.1% on average, mainly due to reducing the redundancy among the selected sensor observations. We observe that mindful active learning is most beneficial when the query budget is small and/or the oracle's memory is weak. This observation emphasizes advantages of utilizing mindful active learning strategies in mobile health settings that involve interaction with older adults and other populations with cognitive impairments.

A Survey of Deep Network Techniques All Classifiers Can Adopt.

Deep neural networks (DNNs) have introduced novel and useful tools to the machine learning community. Other types of classifiers can potentially make use of these tools as well to improve their performance and generality. This paper reviews the current state of the art for deep learning classifier technologies that are being used outside of deep neural networks. Non-neural network classifiers can employ many components found in DNN architectures. In this paper, we review the feature learning, optimization, and regularization methods that form a core of deep network technologies. We then survey non-neural network learning algorithms that make innovative use of these methods to improve classification performance. Because many opportunities and challenges still exist, we discuss directions that can be pursued to expand the area of deep learning for a variety of classification algorithms.

Indirectly-Supervised Anomaly Detection of Clinically-Meaningful Health Events from Smart Home Data.

Anomaly detection techniques can extract a wealth of information about unusual events. Unfortunately, these methods yield an abundance of findings that are not of interest, obscuring relevant anomalies. In this work, we improve upon traditional anomaly detection methods by introducing Isudra, an Indirectly-Supervised Detector of Relevant Anomalies from time series data. Isudra employs Bayesian optimization to select time scales, features, base detector algorithms, and algorithm hyperparameters that increase true positive and decrease false positive detection. This optimization is driven by a small amount of example anomalies, driving an indirectly-supervised approach to anomaly detection. Additionally, we enhance the approach by introducing a warm start method that reduces optimization time between similar problems. We validate the feasibility of Isudra to detect clinically-relevant behavior anomalies from over 2 million sensor readings collected in 5 smart homes, reflecting 26 health events. Results indicate that indirectly-supervised anomaly detection outperforms both supervised and unsupervised algorithms at detecting instances of health-related anomalies such as falls, nocturia, depression, and weakness.

Fusing Ambient and Mobile Sensor Features Into a Behaviorome for Predicting Clinical Health Scores.

Advances in machine learning and low-cost, ubiquitous sensors offer a practical method for understanding the predictive relationship between behavior and health. In this study, we analyze this relationship by building a behaviorome, or set of digital behavior markers, from a fusion of data collected from ambient and wearable sensors. We then use the behaviorome to predict clinical scores for a sample of n = 21 participants based on continuous data collected from smart homes and smartwatches and automatically labeled with corresponding activity and location types. To further investigate the relationship between domains, including participant demographics, self-report and external observation-based health scores, and behavior markers, we propose a joint inference technique that improves predictive performance for these types of high-dimensional spaces. For our participant sample, we observe correlations ranging from small to large for the clinical scores. We also observe an improvement in predictive performance when multiple sensor modalities are used and when joint inference is employed.

sMRT: Multi-Resident Tracking in Smart Homes With Sensor Vectorization.

Smart homes equipped with anonymous binary sensors offer a low-cost, unobtrusive solution that powers activity-aware applications, such as building automation, health monitoring, behavioral intervention, and home security. However, when multiple residents are living in a smart home, associating sensor events with the corresponding residents can pose a major challenge. Previous approaches to multi-resident tracking in smart homes rely on extra information, such as sensor layouts, floor plans, and annotated data, which may not be available or inconvenient to obtain in practice. To address those challenges in real-life deployment, we introduce the sMRT algorithm that simultaneously tracks the location of each resident and estimates the number of residents in the smart home, without relying on ground-truth annotated sensor data or other additional information. We evaluate the performance of our approach using two smart home datasets recorded in real-life settings and compare sMRT with two other methods that rely on sensor layout and ground truth-labeled sensor data.

Recent Developments in Privacy-Preserving Mining of Clinical Data.

With the dramatic increases in both the capability to collect personal data and the capability to analyze large amounts of data, increasingly sophisticated and personal insights are being drawn. These insights are valuable for clinical applications but also open up possibilities for identification and abuse of personal information. In this paper, we survey recent research on classical methods of privacy-preserving data mining. Looking at dominant techniques and recent innovations to them, we examine the applicability of these methods to the privacy-preserving analysis of clinical data. We also discuss promising directions for future research in this area.

Behavioral Differences Between Subject Groups Identified Using Smart Homes and Change Point Detection.

With the arrival of the internet of things, smart environments are becoming increasingly ubiquitous in our everyday lives. Sensor data collected from smart home environments can provide unobtrusive, longitudinal time series data that are representative of the smart home resident's routine behavior and how this behavior changes over time. When longitudinal behavioral data are available from multiple smart home residents, differences between groups of subjects can be investigated. Group-level discrepancies may help isolate behaviors that manifest in daily routines due to a health concern or major lifestyle change. To acquire such insights, we propose an algorithmic framework based on change point detection called Behavior Change Detection for Groups (BCD-G). We hypothesize that, using BCD-G, we can quantify and characterize differences in behavior between groups of individual smart home residents. We evaluate our BCD-G framework using one month of continuous sensor data for each of fourteen smart home residents, divided into two groups. All subjects in the first group are diagnosed with cognitive impairment. The second group consists of cognitively healthy, age-matched controls. Using BCD-G, we identify differences between these two groups, such as how impairment affects patterns of performing activities of daily living and how clinically-relevant behavioral features, such as in-home walking speed, differ for cognitively-impaired individuals. With the unobtrusive monitoring of smart home environments, clinicians can use BCD-G for remote identification of behavior changes that are early indicators of health concerns.

Study of Effectiveness of Prior Knowledge for Smart Home Kit Installation.

Smart-Home in a Box (SHiB) is a ubiquitous system that intends to improve older adults' life quality. SHiB requires self-installation before use. Our previous study found that it is not easy for seniors to install SHiB correctly. SHiB CBLE is a computer-based learning environment that is designed to help individuals install a SHiB kit. This article presents an experiment examining how smart home sensor installation was affected by knowledge gained from two methods, SHiB CBLE, and a written document. Results show that participants who were trained by the CBLE took significantly (p<0.05) less time in the installation session than those in the control group. The accuracy rate of SHiB kit installation is 78% for the group trained by the CBLE and 77% for the control group. Participants trained by the CBLE showed significantly (p<0.01) higher confidence in the actual installation than those in the control group. These results suggest that having a training before the actual installation will help installers avoid unnecessary work, shorten the installation time, and increase installers' confidence.

Automated Smart Home Assessment to Support Pain Management: Multiple Methods Analysis.

BACKGROUND: Poorly managed pain can lead to substance use disorders, depression, suicide, worsening health, and increased use of health services. Most pain assessments occur in clinical settings away from patients' natural environments. Advances in smart home technology may allow observation of pain in the home setting. Smart homes recognizing human behaviors may be useful for quantifying functional pain interference, thereby creating new ways of assessing pain and supporting people living with pain. OBJECTIVE: This study aimed to determine if a smart home can detect pain-related behaviors to perform automated assessment and support intervention for persons with chronic pain. METHODS: A multiple methods, secondary data analysis was conducted using historic ambient sensor data and weekly nursing assessment data from 11 independent older adults reporting pain across 1-2 years of smart home monitoring. A qualitative approach was used to interpret sensor-based data of 27 unique pain events to support clinician-guided training of a machine learning model. A periodogram was used to calculate circadian rhythm strength, and a random forest containing 100 trees was employed to train a machine learning model to recognize pain-related behaviors. The model extracted 550 behavioral markers for each sensor-based data segment. These were treated as both a binary classification problem (event, control) and a regression problem. RESULTS: We found 13 clinically relevant behaviors, revealing 6 pain-related behavioral qualitative themes. Quantitative results were classified using a clinician-guided random forest technique that yielded a classification accuracy of 0.70, sensitivity of 0.72, specificity of 0.69, area under the receiver operating characteristic curve of 0.756, and area under the precision-recall curve of 0.777 in comparison to using standard anomaly detection techniques without clinician guidance (0.16 accuracy achieved; P<.001). The regression formulation achieved moderate correlation, with r=0.42. CONCLUSIONS: Findings of this secondary data analysis reveal that a pain-assessing smart home may recognize pain-related behaviors. Utilizing clinicians' real-world knowledge when developing pain-assessing machine learning models improves the model's performance. A larger study focusing on pain-related behaviors is warranted to improve and test model performance.