IntiVisor: A Visual Analytics System for Interaction Log Analysis.

Application developers frequently augment their code to produce event logs of specific operations performed by their users. Subsequent analysis of these event logs can help provide insight about the users' behavior relative to its intended use. The analysis process typically includes both event organization and pattern discovery activities. However, most existing visual analytics systems for interaction log analysis excel at supporting pattern discovery and overlook the importance of flexible event organization. This omission limits the practical application of these systems. Therefore, we developed a novel visual analytics system called IntiVisor that implements the entire end-to-end interaction analysis approach. An evaluation of the system with interaction data from four visualization applications showed the value and importance of supporting event organization in interaction log analysis.

Empirical networks for localized COVID-19 interventions using WiFi infrastructure at university campuses.

Infectious diseases, like COVID-19, pose serious challenges to university campuses, which typically adopt closure as a non-pharmaceutical intervention to control spread and ensure a gradual return to normalcy. Intervention policies, such as remote instruction (RI) where large classes are offered online, reduce potential contact but also have broad side-effects on campus by hampering the local economy, students' learning outcomes, and community wellbeing. In this paper, we demonstrate that university policymakers can mitigate these tradeoffs by leveraging anonymized data from their WiFi infrastructure to learn community mobility-a methodology we refer to as WiFi mobility models (WiMob). This approach enables policymakers to explore more granular policies like localized closures (LC). WiMob can construct contact networks that capture behavior in various spaces, highlighting new potential transmission pathways and temporal variation in contact behavior. Additionally, WiMob enables us to design LC policies that close super-spreader locations on campus. By simulating disease spread with contact networks from WiMob, we find that LC maintains the same reduction in cumulative infections as RI while showing greater reduction in peak infections and internal transmission. Moreover, LC reduces campus burden by closing fewer locations, forcing fewer students into completely online schedules, and requiring no additional isolation. WiMob can empower universities to conceive and assess a variety of closure policies to prevent future outbreaks.

Development of an interactive tool of early social responsiveness to track autism risk in infants and toddlers.

AIM: To evaluate the psychometric properties of a 4-minute assessment designed to identify early autism spectrum disorder (ASD) status through evaluation of early social responsiveness (ESR). METHOD: This retrospective, preliminary study included children between 13 and 24 months (78 males, 79 females mean age 19.4mo, SD 3.1) from two independent data sets (an experimental/training sample [n=120] and a validation/test sample [n=37]). The ESR assessment examined social behaviors (e.g. eye contact, smiling, ease-of-social-engagement) across five common play activities (e.g. rolling a ball, looking at a book). Data analyses examined reliability and accuracy of the assessment in identifying ESR abilities and in discriminating children with and without ASD. RESULTS: Results indicated adequate internal consistency and test-retest reliability of the ESR assessment. Receiver operator curve analysis identified a cutoff score that discriminated infants with ASD-risk from peers in the training sample. This score yielded moderate sensitivity and high specificity for best-estimate ASD diagnosis in the validation sample. INTERPRETATION: Preliminary findings indicated that brief, systematic observation of ESR may assist in discriminating infants with and without ASD, providing concrete evidence to validate or supplement parents', pediatricians', or clinicians' concerns. Future studies could examine the utility of ESR 'growth curves'.

Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables.

Supervised training of human activity recognition (HAR) systems based on body-worn inertial measurement units (IMUs) is often constrained by the typically rather small amounts of labeled sample data. Systems like IMUTube have been introduced that employ cross-modality transfer approaches to convert videos of activities of interest into virtual IMU data. We demonstrate for the first time how such large-scale virtual IMU datasets can be used to train HAR systems that are substantially more complex than the state-of-the-art. Complexity is thereby represented by the number of model parameters that can be trained robustly. Our models contain components that are dedicated to capture the essentials of IMU data as they are of relevance for activity recognition, which increased the number of trainable parameters by a factor of 1100 compared to state-of-the-art model architectures. We evaluate the new model architecture on the challenging task of analyzing free-weight gym exercises, specifically on classifying 13 dumbbell execises. We have collected around 41 h of virtual IMU data using IMUTube from exercise videos available from YouTube. The proposed model is trained with the large amount of virtual IMU data and calibrated with a mere 36 min of real IMU data. The trained model was evaluated on a real IMU dataset and we demonstrate the substantial performance improvements of 20% absolute F1 score compared to the state-of-the-art convolutional models in HAR.

Clinician Perspectives on Using Computational Mental Health Insights From Patients' Social Media Activities: Design and Qualitative Evaluation of a Prototype.

BACKGROUND: Previous studies have suggested that social media data, along with machine learning algorithms, can be used to generate computational mental health insights. These computational insights have the potential to support clinician-patient communication during psychotherapy consultations. However, how clinicians perceive and envision using computational insights during consultations has been underexplored. OBJECTIVE: The aim of this study is to understand clinician perspectives regarding computational mental health insights from patients' social media activities. We focus on the opportunities and challenges of using these insights during psychotherapy consultations. METHODS: We developed a prototype that can analyze consented patients' Facebook data and visually represent these computational insights. We incorporated the insights into existing clinician-facing assessment tools, the Hamilton Depression Rating Scale and Global Functioning: Social Scale. The design intent is that a clinician will verbally interview a patient (eg, How was your mood in the past week?) while they reviewed relevant insights from the patient's social media activities (eg, number of depression-indicative posts). Using the prototype, we conducted interviews (n=15) and 3 focus groups (n=13) with mental health clinicians: psychiatrists, clinical psychologists, and licensed clinical social workers. The transcribed qualitative data were analyzed using thematic analysis. RESULTS: Clinicians reported that the prototype can support clinician-patient collaboration in agenda-setting, communicating symptoms, and navigating patients' verbal reports. They suggested potential use scenarios, such as reviewing the prototype before consultations and using the prototype when patients missed their consultations. They also speculated potential negative consequences: patients may feel like they are being monitored, which may yield negative effects, and the use of the prototype may increase the workload of clinicians, which is already difficult to manage. Finally, our participants expressed concerns regarding the prototype: they were unsure whether patients' social media accounts represented their actual behaviors; they wanted to learn how and when the machine learning algorithm can fail to meet their expectations of trust; and they were worried about situations where they could not properly respond to the insights, especially emergency situations outside of clinical settings. CONCLUSIONS: Our findings support the touted potential of computational mental health insights from patients' social media account data, especially in the context of psychotherapy consultations. However, sociotechnical issues, such as transparent algorithmic information and institutional support, should be addressed in future endeavors to design implementable and sustainable technology.

A Real-Time Eating Detection System for Capturing Eating Moments and Triggering Ecological Momentary Assessments to Obtain Further Context: System Development and Validation Study.

BACKGROUND: Eating behavior has a high impact on the well-being of an individual. Such behavior involves not only when an individual is eating, but also various contextual factors such as with whom and where an individual is eating and what kind of food the individual is eating. Despite the relevance of such factors, most automated eating detection systems are not designed to capture contextual factors. OBJECTIVE: The aims of this study were to (1) design and build a smartwatch-based eating detection system that can detect meal episodes based on dominant hand movements, (2) design ecological momentary assessment (EMA) questions to capture meal contexts upon detection of a meal by the eating detection system, and (3) validate the meal detection system that triggers EMA questions upon passive detection of meal episodes. METHODS: The meal detection system was deployed among 28 college students at a US institution over a period of 3 weeks. The participants reported various contextual data through EMAs triggered when the eating detection system correctly detected a meal episode. The EMA questions were designed after conducting a survey study with 162 students from the same campus. Responses from EMAs were used to define exclusion criteria. RESULTS: Among the total consumed meals, 89.8% (264/294) of breakfast, 99.0% (406/410) of lunch, and 98.0% (589/601) of dinner episodes were detected by our novel meal detection system. The eating detection system showed a high accuracy by capturing 96.48% (1259/1305) of the meals consumed by the participants. The meal detection classifier showed a precision of 80%, recall of 96%, and F1 of 87.3%. We found that over 99% (1248/1259) of the detected meals were consumed with distractions. Such eating behavior is considered "unhealthy" and can lead to overeating and uncontrolled weight gain. A high proportion of meals was consumed alone (680/1259, 54.01%). Our participants self-reported 62.98% (793/1259) of their meals as healthy. Together, these results have implications for designing technologies to encourage healthy eating behavior. CONCLUSIONS: The presented eating detection system is the first of its kind to leverage EMAs to capture the eating context, which has strong implications for well-being research. We reflected on the contextual data gathered by our system and discussed how these insights can be used to design individual-specific interventions.

Designing a Clinician-Facing Tool for Using Insights From Patients' Social Media Activity: Iterative Co-Design Approach.

BACKGROUND: Recent research has emphasized the need for accessing information about patients to augment mental health patients' verbal reports in clinical settings. Although it has not been introduced in clinical settings, computational linguistic analysis on social media has proved it can infer mental health attributes, implying a potential use as collateral information at the point of care. To realize this potential and make social media insights actionable to clinical decision making, the gaps between computational linguistic analysis on social media and the current work practices of mental health clinicians must be bridged. OBJECTIVE: This study aimed to identify information derived from patients' social media data that can benefit clinicians and to develop a set of design implications, via a series of low-fidelity (lo-fi) prototypes, on how to deliver the information at the point of care. METHODS: A team of clinical researchers and human-computer interaction (HCI) researchers conducted a long-term co-design activity for over 6 months. The needs-affordances analysis framework was used to refine the clinicians' potential needs, which can be supported by patients' social media data. On the basis of those identified needs, the HCI researchers iteratively created 3 different lo-fi prototypes. The prototypes were shared with both groups of researchers via a videoconferencing software for discussion and feedback. During the remote meetings, potential clinical utility, potential use of the different prototypes in a treatment setting, and areas of improvement were discussed. RESULTS: Our first prototype was a card-type interface that supported treatment goal tracking. Each card included attribute levels: depression, anxiety, social activities, alcohol, and drug use. This version confirmed what types of information are helpful but revealed the need for a glanceable dashboard that highlights the trends of these information. As a result, we then developed the second prototype, an interface that shows the clinical state and trend. We found that focusing more on the changes since the last visit without visual representation can be more compatible with clinicians' work practices. In addition, the second phase of needs-affordances analysis identified 3 categories of information relevant to patients with schizophrenia: symptoms related to psychosis, symptoms related to mood and anxiety, and social functioning. Finally, we developed the third prototype, a clinical summary dashboard that showed changes from the last visit in plain texts and contrasting colors. CONCLUSIONS: This exploratory co-design research confirmed that mental health attributes inferred from patients' social media data can be useful for clinicians, although it also revealed a gap between computational social media analyses and clinicians' expectations and conceptualizations of patients' mental health states. In summary, the iterative co-design process crystallized design directions for the future interface, including how we can organize and provide symptom-related information in a way that minimizes the clinicians' workloads.

Measuring Self-Esteem with Passive Sensing.

Self-esteem encompasses how individuals evaluate themselves and is an important contributor to their success. Self-esteem has been traditionally measured using survey-based methodologies. However, surveys suffer from limitations such as retrospective recall and reporting biases, leading to a need for proactive measurement approaches. Our work uses smartphone sensors to predict self-esteem and is situated in a multimodal sensing study on college students for five weeks. We use theory-driven features, such as phone communications and physical activity to predict three dimensions, performance, social, and appearance self-esteem. We conduct statistical modeling including linear, ensemble, and neural network regression to measure self-esteem. Our best model predicts self-esteem with a high correlation (r) of 0.60 and low SMAPE of 7.26% indicating high predictive accuracy. We inspect the top features finding theoretical alignment; for example, social interaction significantly contributes to performance and appearance-based self-esteem, whereas, and physical activity is the most significant contributor towards social self-esteem. Our work reveals the efficacy of passive sensors for predicting self-esteem, and we situate our observations with literature and discuss the implications of our work for tailored interventions and improving wellbeing.

NotifiVR: Exploring Interruptions and Notifications in Virtual Reality.

The proliferation of high resolution and affordable virtual reality (VR) headsets is quickly making room-scale VR experiences available in our homes. Most VR experiences strive to achieve complete immersion by creating a disconnect from the real world. However, due to the lack of a standardized notification management system and minimal context awareness in VR, an immersed user may face certain situations such as missing an important phone call (digital scenario), tripping over wandering pets (physical scenario), or losing track of time (temporal scenario). In this paper, we present the results of 1) a survey across 61 VR users to understand common interruptions and scenarios that would benefit from some form of notifications; 2) a design exercise with VR professionals to explore possible notification methods; and 3) an empirical study on the noticeability and perception of 5 different VR interruption scenarios across 6 modality combinations (e.g., audio, visual, haptic, audio + haptic, visual + haptic, and audio + visual) implemented in Unity and presented using the HTC Vive headset. Finally, we combine key learnings from each of these steps along with participant feedback to present a set of observations and recommendations for notification design in VR.

Exploring Symmetric and Asymmetric Bimanual Eating Detection with Inertial Sensors on the Wrist.

Motivated by health applications, eating detection with off-the-shelf devices has been an active area of research. A common approach has been to recognize and model individual intake gestures with wrist-mounted inertial sensors. Despite promising results, this approach is limiting as it requires the sensing device to be worn on the hand performing the intake gesture, which cannot be guaranteed in practice. Through a study with 14 participants comparing eating detection performance when gestural data is recorded with a wrist-mounted device on (1) both hands, (2) only the dominant hand, and (3) only the non-dominant hand, we provide evidence that a larger set of arm and hand movement patterns beyond food intake gestures are predictive of eating activities when L1 or L2 normalization is applied to the data. Our results are supported by the theory of asymmetric bimanual action and contribute to the field of automated dietary monitoring. In particular, it shines light on a new direction for eating activity recognition with consumer wearables in realistic settings.

EarBit: Using Wearable Sensors to Detect Eating Episodes in Unconstrained Environments.

Chronic and widespread diseases such as obesity, diabetes, and hypercholesterolemia require patients to monitor their food intake, and food journaling is currently the most common method for doing so. However, food journaling is subject to self-bias and recall errors, and is poorly adhered to by patients. In this paper, we propose an alternative by introducing EarBit, a wearable system that detects eating moments. We evaluate the performance of inertial, optical, and acoustic sensing modalities and focus on inertial sensing, by virtue of its recognition and usability performance. Using data collected in a simulated home setting with minimum restrictions on participants' behavior, we build our models and evaluate them with an unconstrained outside-the-lab study. For both studies, we obtained video footage as ground truth for participants activities. Using leave-one-user-out validation, EarBit recognized all the eating episodes in the semi-controlled lab study, and achieved an accuracy of 90.1% and an F1-score of 90.9% in detecting chewing instances. In the unconstrained, outside-the-lab evaluation, EarBit obtained an accuracy of 93% and an F1-score of 80.1% in detecting chewing instances. It also accurately recognized all but one recorded eating episodes. These episodes ranged from a 2 minute snack to a 30 minute meal.

ewrapper: Operationalizing engagement strategies in mHealth.

The advancement of digital technologies particularly in the domain of mobile health (mHealth) holds great promise in the promotion of health behavior. However, keeping users engaged remains a central challenge. This paper proposes a novel approach to address this issue by supplementing existing and future mHealth applications with an engagement wrapper - a collection of engagement strategies integrated into a single, coherent model. The engagement wrapper is operationalized within the format of an ambient display on the lock screen of mobile devices.

Investigating the accuracy of a novel telehealth diagnostic approach for autism spectrum disorder.

Research indicates that a substantial amount of time elapses between parents' first concerns about their child's development and a formal diagnosis of autism spectrum disorder (ASD). Telehealth presents an opportunity to expedite the diagnostic process. This project compared a novel telehealth diagnostic approach that utilizes clinically guided in-home video recordings to the gold standard in-person diagnostic assessment. Participants included 40 families seeking an ASD evaluation for their child and 11 families of typically developing children. Children were between the ages of 18 months and 6 years 11 months; mean adaptive behavior composite = 75.47 (SD = 15.94). All parent participants spoke English fluently. Families completed the Naturalistic Observation Diagnostic Assessment (NODA) for ASD, which was compared to an in-person assessment (IPA). Agreement between the 2 methods, as well as sensitivity, specificity, and interrater reliability, were calculated for the full sample and the subsample of families seeking an ASD evaluation. Diagnostic agreement between NODA and the IPA was 88.2% (κ = 0.75) in the full sample and 85% (κ = 0.58) in the subsample. Sensitivity was 84.9% in both, whereas specificity was 94.4% in the full sample and 85.7% in the subsample. Kappa coefficients for interrater reliability indicated 85% to 90% accuracy between raters. NODA utilizes telehealth technology for families to share information with professionals and provides a method to inform clinical judgment for a diagnosis of ASD. Due to the high level of agreement with the IPA in this sample, NODA has potential to improve the efficiency of the diagnostic process for ASD. (PsycINFO Database Record

Center of excellence for mobile sensor data-to-knowledge (MD2K).

Mobile sensor data-to-knowledge (MD2K) was chosen as one of 11 Big Data Centers of Excellence by the National Institutes of Health, as part of its Big Data-to-Knowledge initiative. MD2K is developing innovative tools to streamline the collection, integration, management, visualization, analysis, and interpretation of health data generated by mobile and wearable sensors. The goal of the big data solutions being developed by MD2K is to reliably quantify physical, biological, behavioral, social, and environmental factors that contribute to health and disease risk. The research conducted by MD2K is targeted at improving health through early detection of adverse health events and by facilitating prevention. MD2K will make its tools, software, and training materials widely available and will also organize workshops and seminars to encourage their use by researchers and clinicians.

A Novel System for Supporting Autism Diagnosis Using Home Videos: Iterative Development and Evaluation of System Design.

BACKGROUND: Observing behavior in the natural environment is valuable to obtain an accurate and comprehensive assessment of a child's behavior, but in practice it is limited to in-clinic observation. Research shows significant time lag between when parents first become concerned and when the child is finally diagnosed with autism. This lag can delay early interventions that have been shown to improve developmental outcomes. OBJECTIVE: To develop and evaluate the design of an asynchronous system that allows parents to easily collect clinically valid in-home videos of their child's behavior and supports diagnosticians in completing diagnostic assessment of autism. METHODS: First, interviews were conducted with 11 clinicians and 6 families to solicit feedback from stakeholders about the system concept. Next, the system was iteratively designed, informed by experiences of families using it in a controlled home-like experimental setting and a participatory design process involving domain experts. Finally, in-field evaluation of the system design was conducted with 5 families of children (4 with previous autism diagnosis and 1 child typically developing) and 3 diagnosticians. For each family, 2 diagnosticians, blind to the child's previous diagnostic status, independently completed an autism diagnosis via our system. We compared the outcome of the assessment between the 2 diagnosticians, and between each diagnostician and the child's previous diagnostic status. RESULTS: The system that resulted through the iterative design process includes (1) NODA smartCapture, a mobile phone-based application for parents to record prescribed video evidence at home; and (2) NODA Connect, a Web portal for diagnosticians to direct in-home video collection, access developmental history, and conduct an assessment by linking evidence of behaviors tagged in the videos to the Diagnostic and Statistical Manual of Mental Disorders criteria. Applying clinical judgment, the diagnostician concludes a diagnostic outcome. During field evaluation, without prior training, parents easily (average rating of 4 on a 5-point scale) used the system to record video evidence. Across all in-home video evidence recorded during field evaluation, 96% (26/27) were judged as clinically useful, for performing an autism diagnosis. For 4 children (3 with autism and 1 typically developing), both diagnosticians independently arrived at the correct diagnostic status (autism versus typical). Overall, in 91% of assessments (10/11) via NODA Connect, diagnosticians confidently (average rating 4.5 on a 5-point scale) concluded a diagnostic outcome that matched with the child's previous diagnostic status. CONCLUSIONS: The in-field evaluation demonstrated that the system's design enabled parents to easily record clinically valid evidence of their child's behavior, and diagnosticians to complete a diagnostic assessment. These results shed light on the potential for appropriately designed telehealth technology to support clinical assessments using in-home video captured by families. This assessment model can be readily generalized to other conditions where direct observation of behavior plays a central role in the assessment process.

Inferring Meal Eating Activities in Real World Settings from Ambient Sounds: A Feasibility Study.

Dietary self-monitoring has been shown to be an effective method for weight-loss, but it remains an onerous task despite recent advances in food journaling systems. Semi-automated food journaling can reduce the effort of logging, but often requires that eating activities be detected automatically. In this work we describe results from a feasibility study conducted in-the-wild where eating activities were inferred from ambient sounds captured with a wrist-mounted device; twenty participants wore the device during one day for an average of 5 hours while performing normal everyday activities. Our system was able to identify meal eating with an F-score of 79.8% in a person-dependent evaluation, and with 86.6% accuracy in a person-independent evaluation. Our approach is intended to be practical, leveraging off-the-shelf devices with audio sensing capabilities in contrast to systems for automated dietary assessment based on specialized sensors.

A Pilot Randomized Trial of Text-Messaging for Symptom Awareness and Diabetes Knowledge in Adolescents With Type 1 Diabetes.

Adolescents with type 1 diabetes typically receive clinical care every 3 months. Between visits, diabetes-related issues may not be frequently reflected, learned, and documented by the patients, limiting their self-awareness and knowledge about their condition. We designed a text-messaging system to help resolve this problem. In a pilot, randomized controlled trial with 30 adolescents, we examined the effect of text messages about symptom awareness and diabetes knowledge on glucose control and quality of life. The intervention group that received more text messages between visits had significant improvements in quality of life.

A Practical Approach for Recognizing Eating Moments with Wrist-Mounted Inertial Sensing.

Recognizing when eating activities take place is one of the key challenges in automated food intake monitoring. Despite progress over the years, most proposed approaches have been largely impractical for everyday usage, requiring multiple on-body sensors or specialized devices such as neck collars for swallow detection. In this paper, we describe the implementation and evaluation of an approach for inferring eating moments based on 3-axis accelerometry collected with a popular off-the-shelf smartwatch. Trained with data collected in a semi-controlled laboratory setting with 20 subjects, our system recognized eating moments in two free-living condition studies (7 participants, 1 day; 1 participant, 31 days), with F-scores of 76.1% (66.7% Precision, 88.8% Recall), and 71.3% (65.2% Precision, 78.6% Recall). This work represents a contribution towards the implementation of a practical, automated system for everyday food intake monitoring, with applicability in areas ranging from health research and food journaling.

Predicting Daily Activities From Egocentric Images Using Deep Learning.

We present a method to analyze images taken from a passive egocentric wearable camera along with the contextual information, such as time and day of week, to learn and predict everyday activities of an individual. We collected a dataset of 40,103 egocentric images over a 6 month period with 19 activity classes and demonstrate the benefit of state-of-the-art deep learning techniques for learning and predicting daily activities. Classification is conducted using a Convolutional Neural Network (CNN) with a classification method we introduce called a late fusion ensemble. This late fusion ensemble incorporates relevant contextual information and increases our classification accuracy. Our technique achieves an overall accuracy of 83.07% in predicting a person's activity across the 19 activity classes. We also demonstrate some promising results from two additional users by fine-tuning the classifier with one day of training data.