Evaluating language models for mathematics through interactions.

There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of evaluating LLMs relies on static pairs of inputs and outputs; this is insufficient for making an informed decision about which LLMs are best to use in an interactive setting, and how that varies by setting. Static assessment therefore limits how we understand language model capabilities. We introduce CheckMate, an adaptable prototype platform for humans to interact with and evaluate LLMs. We conduct a study with CheckMate to evaluate three language models (InstructGPT, ChatGPT, and GPT-4) as assistants in proving undergraduate-level mathematics, with a mixed cohort of participants from undergraduate students to professors of mathematics. We release the resulting interaction and rating dataset, MathConverse. By analyzing MathConverse, we derive a taxonomy of human query behaviors and uncover that despite a generally positive correlation, there are notable instances of divergence between correctness and perceived helpfulness in LLM generations, among other findings. Further, we garner a more granular understanding of GPT-4 mathematical problem-solving through a series of case studies, contributed by experienced mathematicians. We conclude with actionable takeaways for ML practitioners and mathematicians: models that communicate uncertainty, respond well to user corrections, and can provide a concise rationale for their recommendations, may constitute better assistants. Humans should inspect LLM output carefully given their current shortcomings and potential for surprising fallibility.

HCI-modelling for improving the clinical usability of digital health technologies.

INTRODUCTION: Digital Health Technologies (DHTs) have been shown to have variable usability as measured by efficiency, effectiveness and user satisfaction despite large-scale government projects to regulate and standardise user interface (UI) design. We hypothesised that Human-Computer Interaction (HCI) modelling could improve the methodology for DHT design and regulation, and support the creation of future evidence-based UI standards and guidelines for DHTs. METHODOLOGY: Using a Design Science Research (DSR) framework, we developed novel UI components that adhered to existing standards and guidelines (combining the NHS Common User Interface (CUI) standard and the NHS Design System). We firstly evaluated the Patient Banner UI component for compliance with the two guidelines and then used HCI-modelling to evaluate the "Add New Patient" workflow to measure time to task completion and cognitive load. RESULTS: Combining the two guidelines to produce new UI elements is technically feasible for the Patient Banner and the Patient Name Input components. There are some inconsistencies between the NHS Design System and the NHS CUI when implementing the Patient Banner. HCI-modelling successfully quantified challenges adhering to the NHS CUI and the NHS Design system for the "Add New Patient" workflow. DISCUSSION: We successfully developed new design artefacts combing two major design guidelines for DHTs. By quantifying usability issues using HCI-modelling, we have demonstrated the feasibility of a methodology that combines HCI-modelling into a human-centred design (HCD) process could enable the development of standardised UI elements for DHTs that is more scientifically robust than HCD alone. CONCLUSION: Combining HCI-modelling and Human-Centred Design could improve scientific progress towards developing safer and more user-friendly DHTs.

Socially situated artificial intelligence enables learning from human interaction.

Regardless of how much data artificial intelligence agents have available, agents will inevitably encounter previously unseen situations in real-world deployments. Reacting to novel situations by acquiring new information from other people-socially situated learning-is a core faculty of human development. Unfortunately, socially situated learning remains an open challenge for artificial intelligence agents because they must learn how to interact with people to seek out the information that they lack. In this article, we formalize the task of socially situated artificial intelligence-agents that seek out new information through social interactions with people-as a reinforcement learning problem where the agent learns to identify meaningful and informative questions via rewards observed through social interaction. We manifest our framework as an interactive agent that learns how to ask natural language questions about photos as it broadens its visual intelligence on a large photo-sharing social network. Unlike active-learning methods, which implicitly assume that humans are oracles willing to answer any question, our agent adapts its behavior based on observed norms of which questions people are or are not interested to answer. Through an 8-mo deployment where our agent interacted with 236,000 social media users, our agent improved its performance at recognizing new visual information by 112%. A controlled field experiment confirmed that our agent outperformed an active-learning baseline by 25.6%. This work advances opportunities for continuously improving artificial intelligence (AI) agents that better respect norms in open social environments.

Humidity Adaptive Antifreeze Hydrogel Sensor for Intelligent Control and Human-Computer Interaction.

Conductive hydrogels have emerged as ideal candidate materials for strain sensors due to their signal transduction capability and tissue-like flexibility, resembling human tissues. However, due to the presence of water molecules, hydrogels can experience dehydration and low-temperature freezing, which greatly limits the application scope as sensors. In this study, an ionic co-hybrid hydrogel called PBLL is proposed, which utilizes the amphoteric ion betaine hydrochloride (BH) in conjunction with hydrated lithium chloride (LiCl) thereby achieving the function of humidity adaptive. PBLL hydrogel retains water at low humidity (<50%) and absorbs water from air at high humidity (>50%) over the 17 days of testing. Remarkably, the PBLL hydrogel also exhibits strong anti-freezing properties (-80 °C), high conductivity (8.18 S m-1 at room temperature, 1.9 S m-1 at -80 °C), high gauge factor (GF approaching 5.1). Additionally, PBLL hydrogels exhibit strong inhibitory effects against Escherichia coli (E. coli) and Staphylococcus aureus (S. aureus), as well as biocompatibility. By synergistically integrating PBLL hydrogel with wireless transmission and Internet of Things (IoT) technologies, this study has accomplished real-time human-computer interaction systems for sports training and rehabilitation evaluation. PBLL hydrogel exhibits significant potential in the fields of medical rehabilitation, artificial intelligence (AI), and the Internet of Things (IoT).

Visual Sensor with Host-Guest Specific Recognition and Light-Electrical Co-Controlled Switch.

Perception of UV radiation has important applications in medical health, industrial production, electronic communication, etc. In numerous application scenarios, there is an increasing demand for the intuitive and low-cost detection of UV radiation through colorimetric visual behavior, as well as the efficient and multi-functional utilization of UV radiation. However, photodetectors based on photoconductive modes or photosensitive colorimetric materials are not conducive to portable or multi-scene applications owing to their complex and expensive photosensitive components, potential photobleaching, and single-stimulus response behavior. Here, a multifunctional visual sensor based on the "host-guest photo-controlled permutation" strategy and the "lock and key" model is developed. The host-guest specific molecular recognition and electrochromic sensing platform is integrated at the micro-molecular scale, enabling multi-functional and multi-scene applications in the convenient and fast perception of UV radiation, military camouflage, and information erasure at the macro level of human-computer interaction through light-electrical co-controlled visual switching characteristics. This light-electrical co-controlled visual sensor based on an optoelectronic multi-mode sensing system is expected to provide new ideas and paradigms for healthcare, microelectronics manufacturing, and wearable electronic devices owing to its advantages of signal visualization, low energy consumption, low cost, and versatility.

Examining the Popularity, Content, and Intersections With the Substance Abuse and Mental Health Services Administration's Definition of Recovery in a Nonclinical Online Cannabis Cessation Community: Infodemiology Study of Reddit Posts.

BACKGROUND: Cannabis consumption has increased in recent years, as has cannabis use disorder. While researchers have explored public online community discussions of active cannabis use, less is known about the popularity and content of publicly available online communities intended to support cannabis cessation. OBJECTIVE: This study aims to examine the level of engagement and dominant content of an online community for cannabis cessation through 3 specific aims. First, we examine the use of a subreddit cannabis cessation community (r/leaves) over time to evaluate the popularity of this type of resource for individuals who want to stop using cannabis. Second, we examine the content of posts in the community to identify popular topics related to cessation. Third, we compare the thematic findings relative to the 4 domains of recovery defined by the Substance Abuse and Mental Health Services Administration (SAMHSA). By examining these 3 gaps, we take the initial steps toward understanding the experiences being shared online among individuals interested in cannabis cessation and compare them with the principles outlined in the SAMHSA definition of recovery. METHODS: Using the Pushshift application programming interface, we collected the count of posts by year between 2011 and 2021 and the narrative of the 100 posts with the most comments per year in a popular cannabis cessation-focused subreddit (r/leaves). A linear model and a nonlinear model were compared to evaluate change in the number of posts by year. Mixed natural language processing and qualitative analyses were applied to identify top terms, phrases, and themes present in posts over time. Overlap between themes and the 4 SAMHSA domains of recovery (health, purpose, community, and home) were examined. RESULTS: The number of annual posts in r/leaves increased from 420 in 2011 to 34,841 in 2021 (83-fold increase), with exponential growth since 2018. The term that was the most common across posts was "smoke" (2019 posts). Five major themes were identified, and a narrative arc was represented, from motivations and perceived benefits of cannabis use to the negative consequences of use, strategies to change behaviors, and the positive and negative consequences of change. There was substantial overlap between these 5 themes and 3 of SAMHSA's 4 domains of recovery: health, purpose, and community. However, the domain of home was less commonly identified. CONCLUSIONS: Engagement in this online cannabis support community appears to be increasing. Individuals using this forum discussed several topics, including multiple aspects of recovery defined by the SAMHSA. Online communities, such as this one may, serve as an important pathway for individuals seeking to reduce or cease their consumption of cannabis.

Timing, Indicators, and Approaches to Digital Patient Experience Evaluation: Umbrella Systematic Review.

BACKGROUND: The increasing prevalence of DH applications has outpaced research and practice in digital health (DH) evaluations. Patient experience (PEx) was reported as one of the challenges facing the health system by the World Health Organization. To generate evidence on DH and promote the appropriate integration and use of technologies, a standard evaluation of PEx in DH is required. OBJECTIVE: This study aims to systematically identify evaluation timing considerations (ie, when to measure), evaluation indicators (ie, what to measure), and evaluation approaches (ie, how to measure) with regard to digital PEx. The overall aim of this study is to generate an evaluation guide for further improving digital PEx evaluation. METHODS: This is a 2-phase study parallel to our previous study. In phase 1, literature reviews related to PEx in DH were systematically searched from Scopus, PubMed, and Web of Science databases. Two independent raters conducted 2 rounds of paper screening, including title and abstract screening and full-text screening, and assessed the interrater reliability for 20% (round 1: 23/115 and round 2: 12/58) random samples using the Fleiss-Cohen coefficient (round 1: k1=0.88 and round 2: k2=0.80). When reaching interrater reliability (k>0.60), TW conducted the rest of the screening process, leaving any uncertainties for group discussions. Overall, 38% (45/119) of the articles were considered eligible for further thematic analysis. In phase 2, to check if there were any meaningful novel insights that would change our conclusions, we performed an updated literature search in which we collected 294 newly published reviews, of which 102 (34.7%) were identified as eligible articles. We considered them to have no important changes to our original results on the research objectives. Therefore, they were not integrated into the synthesis of this review and were used as supplementary materials. RESULTS: Our review highlights 5 typical evaluation objectives that serve 5 stakeholder groups separately. We identified a set of key evaluation timing considerations and classified them into 3 categories: intervention maturity stages, timing of the evaluation, and timing of data collection. Information on evaluation indicators of digital PEx was identified and summarized into 3 categories (intervention outputs, patient outcomes, and health care system impact), 9 themes, and 22 subthemes. A set of evaluation theories, common study designs, data collection methods and instruments, and data analysis approaches was captured, which can be used or adapted to evaluate digital PEx. CONCLUSIONS: Our findings enabled us to generate an evaluation guide to help DH intervention researchers, designers, developers, and program evaluators evaluate digital PEx. Finally, we propose 6 directions for encouraging further digital PEx evaluation research and practice to address the challenge of poor PEx.

Intelligent Gesture Recognition Based on Screen Reflectance Multi-Band Spectral Features.

Human-computer interaction (HCI) with screens through gestures is a pivotal method amidst the digitalization trend. In this work, a gesture recognition method is proposed that combines multi-band spectral features with spatial characteristics of screen-reflected light. Based on the method, a red-green-blue (RGB) three-channel spectral gesture recognition system has been developed, composed of a display screen integrated with narrowband spectral receivers as the hardware setup. During system operation, emitted light from the screen is reflected by gestures and received by the narrowband spectral receivers. These receivers at various locations are tasked with capturing multiple narrowband spectra and converting them into light-intensity series. The availability of multi-narrowband spectral data integrates multidimensional features from frequency and spatial domains, enhancing classification capabilities. Based on the RGB three-channel spectral features, this work formulates an RGB multi-channel convolutional neural network long short-term memory (CNN-LSTM) gesture recognition model. It achieves accuracies of 99.93% in darkness and 99.89% in illuminated conditions. This indicates the system's capability for stable operation across different lighting conditions and accurate interaction. The intelligent gesture recognition method can be widely applied for interactive purposes on various screens such as computers and mobile phones, facilitating more convenient and precise HCI.

Fault Diagnosis for Reducers Based on a Digital Twin.

A new method based on a digital twin is proposed for fault diagnosis, in order to compensate for the shortcomings of the existing methods for fault diagnosis modeling, including the single fault type, low similarity, and poor visual effect of state monitoring. First, a fault diagnosis test platform is established to analyze faults under constant and variable speed conditions. Then, the obtained data are integrated into the Unity3D platform to realize online diagnosis and updated with real-time working status data. Finally, an industrial test of the digital twin model is conducted, allowing for its comparison with other advanced methods in order to verify its accuracy and application feasibility. It was found that the accuracy of the proposed method for the entire reducer was 99.5%, higher than that of other methods based on individual components (e.g., 93.5% for bearings, 96.3% for gear shafts, and 92.6% for shells).

Early Eye Disengagement Is Regulated by Task Complexity and Task Repetition in Visual Tracking Task.

Understanding human actions often requires in-depth detection and interpretation of bio-signals. Early eye disengagement from the target (EEDT) represents a significant eye behavior that involves the proactive disengagement of the gazes from the target to gather information on the anticipated pathway, thereby enabling rapid reactions to the environment. It remains unknown how task difficulty and task repetition affect EEDT. We aim to provide direct evidence of how these factors influence EEDT. We developed a visual tracking task in which participants viewed arrow movement videos while their eye movements were tracked. The task complexity was increased by increasing movement steps. Every movement pattern was performed twice to assess the effect of repetition on eye movement. Participants were required to recall the movement patterns for recall accuracy evaluation and complete cognitive load assessment. EEDT was quantified by the fixation duration and frequency within the areas of eye before arrow. When task difficulty increased, we found the recall accuracy score decreased, the cognitive load increased, and EEDT decreased significantly. The EEDT was higher in the second trial, but significance only existed in tasks with lower complexity. EEDT was positively correlated with recall accuracy and negatively correlated with cognitive load. Performing EEDT was reduced by task complexity and increased by task repetition. EEDT may be a promising sensory measure for assessing task performance and cognitive load and can be used for the future development of eye-tracking-based sensors.

Flexible Self-Powered Low-Decibel Voice Recognition Mask.

In environments where silent communication is essential, such as libraries and conference rooms, the need for a discreet means of interaction is paramount. Here, we present a single-electrode, contact-separated triboelectric nanogenerator (CS-TENG) characterized by robust high-frequency sensing capabilities and long-term stability. Integrating this TENG onto the inner surface of a mask allows for the capture of conversational speech signals through airflow vibrations, generating a comprehensive dataset. Employing advanced signal processing techniques, including short-time Fourier transform (STFT), Mel-frequency cepstral coefficients (MFCC), and deep learning neural networks, facilitates the accurate identification of speaker content and verification of their identity. The accuracy rates for each category of vocabulary and identity recognition exceed 92% and 90%, respectively. This system represents a pivotal advancement in facilitating secure and efficient unobtrusive communication in quiet settings, with promising implications for smart home applications, virtual assistant technology, and potential deployment in security and confidentiality-sensitive contexts.

Real-Time Sensor-Based Human Activity Recognition for eFitness and eHealth Platforms.

Human Activity Recognition (HAR) plays an important role in the automation of various tasks related to activity tracking in such areas as healthcare and eldercare (telerehabilitation, telemonitoring), security, ergonomics, entertainment (fitness, sports promotion, human-computer interaction, video games), and intelligent environments. This paper tackles the problem of real-time recognition and repetition counting of 12 types of exercises performed during athletic workouts. Our approach is based on the deep neural network model fed by the signal from a 9-axis motion sensor (IMU) placed on the chest. The model can be run on mobile platforms (iOS, Android). We discuss design requirements for the system and their impact on data collection protocols. We present architecture based on an encoder pretrained with contrastive learning. Compared to end-to-end training, the presented approach significantly improves the developed model's quality in terms of accuracy (F1 score, MAPE) and robustness (false-positive rate) during background activity. We make the AIDLAB-HAR dataset publicly available to encourage further research.

Sensing the Intentions to Speak in VR Group Discussions.

While virtual reality (VR) technologies enable remote communication through the use of 3D avatars, it is often difficult to foster engaging group discussions without addressing the limitations to the non-verbal communication among distributed participants. In this paper, we discuss a technique to detect the intentions to speak in group discussions by tapping into intricate sensor data streams from VR headsets and hand-controllers. To this end, we developed a prototype VR group discussion app equipped with comprehensive sensor data-logging functions and conducted an experiment of VR group discussions (N = 24). We used the quantitative and qualitative experimental data to analyze participants' experiences of group discussions in relation to the temporal patterns of their different speaking intentions. We then propose a sensor-based mechanism for detecting speaking intentions by employing a sampling strategy that considers the temporal patterns of speaking intentions, and we verify the feasibility of our approach in group discussion settings.

Automated Monitoring of Human-Computer Interaction for Assessing Teachers' Digital Competence Based on LMS Data Extraction.

The fast-paced evolution of technology has compelled the digitalization of education, requiring educators to interact with computers and develop digital competencies relevant to the teaching-learning process. This need has prompted various organizations to define frameworks for assessing digital competency emphasizing teachers' interaction with computer technologies in education. Different authors have presented assessment methods for teachers' digital competence based on the video analysis of recorded classes using sensors such as cameras, microphones, or electroencephalograms. The main limitation of these solutions is the large number of resources they require, making it difficult to assess large numbers of teachers in resource-constrained environments. This article proposes the automation of teachers' digital competence evaluation process based on monitoring metrics obtained from teachers' interaction with a Learning Management System (LMS). Based on the Digital Competence Framework for Educators (DigCompEdu), indicators were defined and extracted that allow automatic measurement of a teacher's competency level. A tool was designed and implemented to conduct a successful proof of concept capable of automating the evaluation process of all university faculty, including 987 lecturers from different fields of knowledge. Results obtained allow for drawing conclusions on technological adoption according to the teacher's profile and planning educational actions to improve these competencies.

A Wearable Visually Impaired Assistive System Based on Semantic Vision SLAM for Grasping Operation.

Because of the absence of visual perception, visually impaired individuals encounter various difficulties in their daily lives. This paper proposes a visual aid system designed specifically for visually impaired individuals, aiming to assist and guide them in grasping target objects within a tabletop environment. The system employs a visual perception module that incorporates a semantic visual SLAM algorithm, achieved through the fusion of ORB-SLAM2 and YOLO V5s, enabling the construction of a semantic map of the environment. In the human-machine cooperation module, a depth camera is integrated into a wearable device worn on the hand, while a vibration array feedback device conveys directional information of the target to visually impaired individuals for tactile interaction. To enhance the system's versatility, a Dobot Magician manipulator is also employed to aid visually impaired individuals in grasping tasks. The performance of the semantic visual SLAM algorithm in terms of localization and semantic mapping was thoroughly tested. Additionally, several experiments were conducted to simulate visually impaired individuals' interactions in grasping target objects, effectively verifying the feasibility and effectiveness of the proposed system. Overall, this system demonstrates its capability to assist and guide visually impaired individuals in perceiving and acquiring target objects.

Detecting Leadership Opportunities in Group Discussions Using Off-the-Shelf VR Headsets.

The absence of some forms of non-verbal communication in virtual reality (VR) can make VR-based group discussions difficult even when a leader is assigned to each group to facilitate discussions. In this paper, we discuss if the sensor data from off-the-shelf VR devices can be used to detect opportunities for facilitating engaging discussions and support leaders in VR-based group discussions. To this end, we focus on the detection of suppressed speaking intention in VR-based group discussions by using personalized and general models. Our extensive analysis of experimental data reveals some factors that should be considered to enable effective feedback to leaders. In particular, our results show the benefits of combining the sensor data from leaders and low-engagement participants, and the usefulness of specific HMD sensor features.

Hand Gesture Recognition Using FSK Radar Sensors.

Hand gesture recognition, which is one of the fields of human-computer interaction (HCI) research, extracts the user's pattern using sensors. Radio detection and ranging (RADAR) sensors are robust under severe environments and convenient to use for hand gestures. The existing studies mostly adopted continuous-wave (CW) radar, which only shows a good performance at a fixed distance, which is due to its limitation of not seeing the distance. This paper proposes a hand gesture recognition system that utilizes frequency-shift keying (FSK) radar, allowing for a recognition method that can work at the various distances between a radar sensor and a user. The proposed system adopts a convolutional neural network (CNN) model for the recognition. From the experimental results, the proposed recognition system covers the range from 30 cm to 180 cm and shows an accuracy of 93.67% over the entire range.

A Multilayer Architecture towards the Development and Distribution of Multimodal Interface Applications on the Edge.

Today, Smart Assistants (SAs) are supported by significantly improved Natural Language Processing (NLP) and Natural Language Understanding (NLU) engines as well as AI-enabled decision support, enabling efficient information communication, easy appliance/device control, and seamless access to entertainment services, among others. In fact, an increasing number of modern households are being equipped with SAs, which promise to enhance user experience in the context of smart environments through verbal interaction. Currently, the market in SAs is dominated by products manufactured by technology giants that provide well designed off-the-shelf solutions. However, their simple setup and ease of use come with trade-offs, as these SAs abide by proprietary and/or closed-source architectures and offer limited functionality. Their enforced vendor lock-in does not provide (power) users with the ability to build custom conversational applications through their SAs. On the other hand, employing an open-source approach for building and deploying an SA (which comes with a significant overhead) necessitates expertise in multiple domains and fluency in the multimodal technologies used to build the envisioned applications. In this context, this paper proposes a methodology for developing and deploying conversational applications on the edge on top of an open-source software and hardware infrastructure via a multilayer architecture that simplifies low-level complexity and reduces learning overhead. The proposed approach facilitates the rapid development of applications by third-party developers, thereby enabling the establishment of a marketplace of customized applications aimed at the smart assisted living domain, among others. The supporting framework supports application developers, device owners, and ecosystem administrators in building, testing, uploading, and deploying applications, remotely controlling devices, and monitoring device performance. A demonstration of this methodology is presented and discussed focusing on health and assisted living applications for the elderly.

Novel Wearable System to Recognize Sign Language in Real Time.

The aim of this study is to develop a practical software solution for real-time recognition of sign language words using two arms. This will facilitate communication between hearing-impaired individuals and those who can hear. We are aware of several sign language recognition systems developed using different technologies, including cameras, armbands, and gloves. However, the system we propose in this study stands out for its practicality, utilizing surface electromyography (muscle activity) and inertial measurement unit (motion dynamics) data from both arms. We address the drawbacks of other methods, such as high costs, low accuracy due to ambient light and obstacles, and complex hardware requirements, which have limited their practical application. Our software can run on different operating systems using digital signal processing and machine learning methods specific to this study. For the test, we created a dataset of 80 words based on their frequency of use in daily life and performed a thorough feature extraction process. We tested the recognition performance using various classifiers and parameters and compared the results. The random forest algorithm emerged as the most successful, achieving a remarkable 99.875% accuracy, while the naïve Bayes algorithm had the lowest success rate with 87.625% accuracy. The new system promises to significantly improve communication for people with hearing disabilities and ensures seamless integration into daily life without compromising user comfort or lifestyle quality.

Study on Gesture Recognition Method with Two-Stream Residual Network Fusing sEMG Signals and Acceleration Signals.

Currently, surface EMG signals have a wide range of applications in human-computer interaction systems. However, selecting features for gesture recognition models based on traditional machine learning can be challenging and may not yield satisfactory results. Considering the strong nonlinear generalization ability of neural networks, this paper proposes a two-stream residual network model with an attention mechanism for gesture recognition. One branch processes surface EMG signals, while the other processes hand acceleration signals. Segmented networks are utilized to fully extract the physiological and kinematic features of the hand. To enhance the model's capacity to learn crucial information, we introduce an attention mechanism after global average pooling. This mechanism strengthens relevant features and weakens irrelevant ones. Finally, the deep features obtained from the two branches of learning are fused to further improve the accuracy of multi-gesture recognition. The experiments conducted on the NinaPro DB2 public dataset resulted in a recognition accuracy of 88.25% for 49 gestures. This demonstrates that our network model can effectively capture gesture features, enhancing accuracy and robustness across various gestures. This approach to multi-source information fusion is expected to provide more accurate and real-time commands for exoskeleton robots and myoelectric prosthetic control systems, thereby enhancing the user experience and the naturalness of robot operation.