Ki-Cook: clustering multimodal cooking representations through knowledge-infused learning.

Cross-modal recipe retrieval has gained prominence due to its ability to retrieve a text representation given an image representation and vice versa. Clustering these recipe representations based on similarity is essential to retrieve relevant information about unknown food images. Existing studies cluster similar recipe representations in the latent space based on class names. Due to inter-class similarity and intraclass variation, associating a recipe with a class name does not provide sufficient knowledge about recipes to determine similarity. However, recipe title, ingredients, and cooking actions provide detailed knowledge about recipes and are a better determinant of similar recipes. In this study, we utilized this additional knowledge of recipes, such as ingredients and recipe title, to identify similar recipes, emphasizing attention especially on rare ingredients. To incorporate this knowledge, we propose a knowledge-infused multimodal cooking representation learning network, Ki-Cook, built on the procedural attribute of the cooking process. To the best of our knowledge, this is the first study to adopt a comprehensive recipe similarity determinant to identify and cluster similar recipe representations. The proposed network also incorporates ingredient images to learn multimodal cooking representation. Since the motivation for clustering similar recipes is to retrieve relevant information for an unknown food image, we evaluated the ingredient retrieval task. We performed an empirical analysis to establish that our proposed model improves the Coverage of Ground Truth by 12% and the Intersection Over Union by 10% compared to the baseline models. On average, the representations learned by our model contain an additional 15.33% of rare ingredients compared to the baseline models. Owing to this difference, our qualitative evaluation shows a 39% improvement in clustering similar recipes in the latent space compared to the baseline models, with an inter-annotator agreement of the Fleiss kappa score of 0.35.

Determination of Personalized Asthma Triggers From Multimodal Sensing and a Mobile App: Observational Study.

BACKGROUND: Asthma is a chronic pulmonary disease with multiple triggers. It can be managed by strict adherence to an asthma care plan and by avoiding these triggers. Clinicians cannot continuously monitor their patients' environment and their adherence to an asthma care plan, which poses a significant challenge for asthma management. OBJECTIVE: In this study, pediatric patients were continuously monitored using low-cost sensors to collect asthma-relevant information. The objective of this study was to assess whether kHealth kit, which contains low-cost sensors, can identify personalized triggers and provide actionable insights to clinicians for the development of a tailored asthma care plan. METHODS: The kHealth asthma kit was developed to continuously track the symptoms of asthma in pediatric patients and monitor the patients' environment and adherence to their care plan for either 1 or 3 months. The kit consists of an Android app-based questionnaire to collect information on asthma symptoms and medication intake, Fitbit to track sleep and activity, the Peak Flow meter to monitor lung functions, and Foobot to monitor indoor air quality. The data on the patient's outdoor environment were collected using third-party Web services based on the patient's zip code. To date, 107 patients consented to participate in the study and were recruited from the Dayton Children's Hospital, of which 83 patients completed the study as instructed. RESULTS: Patient-generated health data from the 83 patients who completed the study were included in the cohort-level analysis. Of the 19% (16/83) of patients deployed in spring, the symptoms of 63% (10/16) and 19% (3/16) of patients suggested pollen and particulate matter (PM2.5), respectively, to be their major asthma triggers. Of the 17% (14/83) of patients deployed in fall, symptoms of 29% (4/17) and 21% (3/17) of patients suggested pollen and PM2.5, respectively, to be their major triggers. Among the 28% (23/83) of patients deployed in winter, PM2.5 was identified as the major trigger for 83% (19/23) of patients. Similar correlations were not observed between asthma symptoms and factors such as ozone level, temperature, and humidity. Furthermore, 1 patient from each season was chosen to explain, in detail, his or her personalized triggers by observing temporal associations between triggers and asthma symptoms gathered using the kHealth asthma kit. CONCLUSIONS: The continuous monitoring of pediatric asthma patients using the kHealth asthma kit generates insights on the relationship between their asthma symptoms and triggers across different seasons. This can ultimately inform personalized asthma management and intervention plans.

kBot: Knowledge-enabled Personalized Chatbot for Asthma Self-Management.

There is a well-recognized need for a shift to proactive asthma care given the impact asthma has on overall healthcare costs. The demand for continuous monitoring of patient's adherence to the medication care plan, assessment of environmental triggers, and management of asthma can be challenging in traditional clinical settings and taxing on clinical professionals. Recent years have seen a robust growth of general purpose conversational systems. However, they lack the capabilities to support applications such an individual's health, which requires the ability to contextualize, learn interactively, and provide the proper hyper-personalization needed to hold meaningful conversations. In this paper, we present kBot, a knowledge-enabled personalized chatbot system designed for health applications and adapted to help pediatric asthmatic patients (age 8 to 15) to better control their asthma. Its core functionalities include continuous monitoring of the patient's medication adherence and tracking of relevant health signals and environment data. kBot takes the form of an Android application with a frontend chat interface capable of conversing in both text and voice, and a backend cloud-based server application that handles data collection, processing, and dialogue management. It achieves contextualization by piecing together domain knowledge from online sources and inputs from our clinical partners. The personalization aspect is derived from patient answering questionnaires and day-to-day conversations. kBOT's preliminary evaluation focused on chatbot quality, technology acceptance, and system usability involved eight asthma clinicians and eight researchers. For both groups, kBot achieved an overall technology acceptance value of greater than 8 on the 11-point Likert scale and a mean System Usability Score (SUS) greater than 80.