SleepFC: Feature Pyramid and Cross-Scale Context Learning for Sleep Staging.

Automated sleep staging is essential to assess sleep quality and treat sleep disorders, so the issue of electroencephalography (EEG)-based sleep staging has gained extensive research interests. However, the following difficulties exist in this issue: 1) how to effectively learn the intrinsic features of salient waves from single-channel EEG signals; 2) how to learn and capture the useful information of sleep stage transition rules; 3) how to address the class imbalance problem of sleep stages. To handle these problems in sleep staging, we propose a novel method named SleepFC. This method comprises convolutional feature pyramid network (CFPN), cross-scale temporal context learning (CSTCL), and class adaptive fine-tuning loss function (CAFTLF) based classification network. CFPN learns the multi-scale features from salient waves of EEG signals. CSTCL extracts the informative multi-scale transition rules between sleep stages. CAFTLF-based classification network handles the class imbalance problem. Extensive experiments on three public benchmark datasets demonstrate the superiority of SleepFC over the state-of-the-art approaches. Particularly, SleepFC has a significant performance advantage in recognizing the N1 sleep stage, which is challenging to distinguish.

Review of robotic systems for thoracoabdominal puncture interventional surgery.

Cancer, with high morbidity and high mortality, is one of the major burdens threatening human health globally. Intervention procedures via percutaneous puncture have been widely used by physicians due to its minimally invasive surgical approach. However, traditional manual puncture intervention depends on personal experience and faces challenges in terms of precisely puncture, learning-curve, safety and efficacy. The development of puncture interventional surgery robotic (PISR) systems could alleviate the aforementioned problems to a certain extent. This paper attempts to review the current status and prospective of PISR systems for thoracic and abdominal application. In this review, the key technologies related to the robotics, including spatial registration, positioning navigation, puncture guidance feedback, respiratory motion compensation, and motion control, are discussed in detail.

MaskCAE: Masked Convolutional AutoEncoder via Sensor Data Reconstruction for Self-Supervised Human Activity Recognition.

Self-supervised Human Activity Recognition (HAR) has been gradually gaining a lot of attention in ubiquitous computing community. Its current focus primarily lies in how to overcome the challenge of manually labeling complicated and intricate sensor data from wearable devices, which is often hard to interpret. However, current self-supervised algorithms encounter three main challenges: performance variability caused by data augmentations in contrastive learning paradigm, limitations imposed by traditional self-supervised models, and the computational load deployed on wearable devices by current mainstream transformer encoders. To comprehensively tackle these challenges, this paper proposes a powerful self-supervised approach for HAR from a novel perspective of denoising autoencoder, the first of its kind to explore how to reconstruct masked sensor data built on a commonly employed, well-designed, and computationally efficient fully convolutional network. Extensive experiments demonstrate that our proposed Masked Convolutional AutoEncoder (MaskCAE) outperforms current state-of-the-art algorithms in self-supervised, fully supervised, and semi-supervised situations without relying on any data augmentations, which fills the gap of masked sensor data modeling in HAR area. Visualization analyses show that our MaskCAE could effectively capture temporal semantics in time series sensor data, indicating its great potential in modeling abstracted sensor data. An actual implementation is evaluated on an embedded platform.

Development and Evaluation of Refreshable Braille Display and Active Touch-Reading System for Digital Reading of the Visually Impaired.

The traditional way of reading through Braille books is constraining the reading experience of blind or visually impaired (BVI) in the digital age. In order to improve the reading convenience of BVI, this paper proposes a low-cost and refreshable Braille display device, and solves the problems of high energy consumption and low latching force existing in existing devices. Further, the Braille display device was combined with the 3D Systems Touch device to develop an active Braille touch-reading system for digital reading of BVI with the help of the CHAI3D virtual environment. Firstly, according to the actual needs of BVI to touch and read the Braille dots, this paper utilizes the beam structure to provide a full latching function for the raised Braille dot without energy consumption. Through theoretical derivation and finite element analysis, the performance of the Braille dot actuator is optimized to provide sufficient feedback force and latching force for finger's touch-reading. Then, this paper designs a virtual Braille interactive environment based on the CHAI3D, and combines the sense of touch with audio to effectively improve the recognition accuracy and reading efficiency of BVI for Braille through the multi-modal presentation of Braille information. The performance test results of the device show that the average lifting force of the Braille dot actuator is 101.67 mN, the latching force is over 5 N, and the average refresh frequency is 17.1 Hz, which meets the touch-reading needs of BVI. User experiments show that the average accuracy rate of BVI subjects in identifying digitized Braille is 95.5%, and subjects have a high subjective evaluation of the system.

Fractal Spiking Neural Network Scheme for EEG-Based Emotion Recognition.

Electroencephalogram (EEG)-based emotion recognition is of great significance for aiding in clinical diagnosis, treatment, nursing and rehabilitation. Current research on this issue mainly focuses on utilizing various network architectures with different types of neurons to exploit the temporal, spectral, or spatial information from EEG for classification. However, most studies fail to take full advantage of the useful Temporal-Spectral-Spatial (TSS) information of EEG signals. In this paper, we propose a novel and effective Fractal Spike Neural Network (Fractal-SNN) scheme, which can exploit the multi-scale TSS information from EEG, for emotion recognition. Our designed Fractal-SNN block in the proposed scheme approximately simulates the biological neural connection structures based on spiking neurons and a new fractal rule, allowing for the extraction of discriminative multi-scale TSS features from the signals. Our designed training technique, inverted drop-path, can enhance the generalization ability of the Fractal-SNN scheme. Sufficient experiments on four public benchmark databases, DREAMER, DEAP, SEED-IV and MPED, under the subject-dependent protocols demonstrate the superiority of the proposed scheme over the related advanced methods. In summary, the proposed scheme provides a promising solution for EEG-based emotion recognition.

Decoding Natural Grasping Behaviors: Insights Into MRCP Source Features and Coupling Dynamics.

The effective decoding of natural grasping behaviors is crucial for the natural control of neural prosthetics. This study aims to investigate the decoding performance of movement-related cortical potential (MRCP) source features between complex grasping actions and explore the temporal and frequency differences in inter-muscular and cortical-muscular coupling strength during movement. Based on the human grasping taxonomy and their frequency, five natural grasping motions-medium wrap, adducted thumb, adduction grip, tip pinch, and writing tripod-were chosen. We collected 64-channel electroencephalogram (EEG) and 5-channel surface electromyogram (sEMG) data from 17 healthy participants, and projected six EEG frequency bands into source space for further analysis. Results from multi-classification and binary classification demonstrated that MRCP source features could not only distinguish between power grasp and precision grasp, but also detect subtle action differences such as thumb adduction and abduction during the execution phase. Besides, we found that during natural reach-and-grasp movement, the coupling strength from cortical to muscle is lower than that from muscle to cortical, except in the hold phase of γ frequency band. Furthermore, a 12-Hz peak of inter-muscular coupling strength was found in movement execution, which might be related to movement planning and execution. We believe that this research will enhance our comprehension of the control and feedback mechanisms of human hand grasping and contributes to a natural and intuitive control for brain-computer interface.

Quantitative Assessment via Multi-Domain Fusion of Muscle Synergy Associated with Upper-Limb Motor Function for Stroke Rehabilitation.

Quantitative assessment of upper limb motor function can assist therapists in providing appropriate rehabilitation strategies, which plays an essential role in post-stroke rehabilitation. Conventionally, the most frequently used assessments are based on clinical scales or kinematic metrics, which rely on subjective scores or may be masked at the kinematic level by compensatory strategies. Recently, muscle synergies which encodes the simplified neuromuscular control strategy deployed by the central nervous system have been gradually used to assess post-stroke impairment. In general, muscle synergies are decomposed into two components: synergy vectors and synergy activation. Synergy vectors represent the relative weighting of each muscle within each synergy, that is muscle coordination; synergy activation represents the recruitment of the muscle synergy over time, that is muscle activation strength. Both the characteristics of synergy vectors and synergy activation are crucial for adequately assessing patients' motor function. Therefore, we integrate the spatial domain and temporal domain features extracted from synergy vectors and synergy activation for constructing a multi-domain assessment system based on Random Forest classifier, which may provide great qualitative classification accuracy. Furthermore, a novel functional score is generated from the probabilities belonging to the pathological group. Finally, we conduct a study with ten healthy subjects and ten post-stroke patients to verify the effectiveness of the proposed method. The experimental results show that the classification accuracy was enhanced to 98.56% by fusing the characteristics derived from different domains, which was higher than that based on spatial domain (94.90%) and temporal domain (91.08%), respectively. Furthermore, the assessment score generated by multi-domain fusion framework exhibited a significant correlation with the clinical score. These promising results show the potential of applying the proposed method to clinical assessments for post-stroke patients.

A high-fidelity virtual liver model incorporating biological characteristics.

Flexible tissue modeling plays an important role in the field of telemedicine. It is related to whether the soft tissue deformation process can be accurately, real-time and vividly simulated during surgery. However, most existing models lack the unique biological characteristics. To solve this problem, we proposed a high-fidelity virtual liver model incorporating biological characteristics, such as the viscoelastic, anisotropic and nonlinear biological characteristics. Besides, to the best of our knowledge, our study is the first to introduce the viscoplasticity of biological tissues to improve the fidelity of the liver model. This mothod was proposed to describe the viscoplastic characteristics of the diseased liver resection process, when the liver is in a state of excessive deformation and loss of elasticity, however, there are few works focusing on this problem. The 3DMax2020 and OpenGL4.6 were used to build a liver surgery simulation platform, and the PHANTOM OMNI manual controller was used to sense the feedback force during the operation. The proposed model was verified from three aspects of accuracy, fidelity and real-time performance. The experimental results show that the proposed virtual liver model can enhance visual perception ability, improve deformation accuracy and fidelity.

Bionic Snail Robot Enhanced by Poroelastic Foams Crawls Using Direct and Retrograde Waves.

Snails employ a distinctive crawling mechanism in which the pedal waves travel along the foot and interact with the mucus to promote efficient movement on various substrates. Inspired by the concavities on the pedal wave, we develop a new bionic snail robot that introduces transverse patterns in a longitudinal wave to periodically change the friction. The poroelastic foam serves as flexible constraint and fills the robot's internal cavity. It contributes to the bending action, and maintains the thinness and softness of the robot. Then, the model of the robot's single segment is built utilizing the Euler-Bernoulli beam theory. The model aligns well with the experimental data, thereby confirming the effectiveness of soft constraints. The evaluation of pedal wave is conducted, which further guides the optimization of the control sequence. The experiments demonstrated the robot performing retrograde wave locomotion on dry substrates. Notably, shear-thickening fluids were found to be suitable for this particular crawling pattern compared with other mucus simulants, resulting in direct wave locomotion with a 49% increase in speed and a 33% reduction in energy usage. The load capacity of the soft snail robot was also enhanced, enabling it to carry loads up to 2.84 times its own weight. The use of mucus in crawling also brings valuable insights for the enhancement of other biomimetic robots.

Cross-subject emotion recognition using hierarchical feature optimization and support vector machine with multi-kernel collaboration.

Objective. Due to individual differences, it is greatly challenging to realize the multiple types of emotion identification across subjects.Approach. In this research, a hierarchical feature optimization method is proposed in order to represent emotional states effectively based on peripheral physiological signals. Firstly, sparse learning combined with binary search is employed to achieve feature selection of single signals. Then an improved fast correlation-based filter is proposed to implement fusion optimization of multi-channel signal features. Aiming at overcoming the limitations of the support vector machine (SVM), which uses a single kernel function to make decisions, the multi-kernel function collaboration strategy is proposed to improve the classification performance of SVM.Main results. The effectiveness of the proposed method is verified on the DEAP dataset. Experimental results show that the proposed method presents a competitive performance for four cross-subject  types of emotion identification with an accuracy of 84% (group 1) and 85.07% (group 2). Significance. The proposed model with hierarchical feature optimization and SVM with multi-kernel function collaboration demonstrates superior emotion recognition accuracy compared to state-of-the-art techniques. In addition, the analysis based on DEAP dataset composition characteristics presents a novel perspective to explore the emotion recognition issue more objectively and comprehensively.

FreqSense: Adaptive Sampling Rates for Sensor-Based Human Activity Recognition Under Tunable Computational Budgets.

Recent years have witnessed great success of deep convolutional networks in sensor-based human activity recognition (HAR), yet their practical deployment remains a challenge due to the varying computational budgets required to obtain a reliable prediction. This article focuses on adaptive inference from a novel perspective of signal frequency, which is motivated by an intuition that low-frequency features are enough for recognizing "easy" activity samples, while only "hard" activity samples need temporally detailed information. We propose an adaptive resolution network by combining a simple subsampling strategy with conditional early-exit. Specifically, it is comprised of multiple subnetworks with different resolutions, where "easy" activity samples are first classified by lightweight subnetwork using the lowest sampling rate, while the subsequent subnetworks in higher resolution would be sequentially applied once the former one fails to reach a confidence threshold. Such dynamical decision process could adaptively select a proper sampling rate for each activity sample conditioned on an input if the budget varies, which will be terminated until enough confidence is obtained, hence avoiding excessive computations. Comprehensive experiments on four diverse HAR benchmark datasets demonstrate the effectiveness of our method in terms of accuracy-cost tradeoff. We benchmark the average latency on a real hardware.

SmartSpring: A Low-Cost Wearable Haptic VR Display with Controllable Passive Feedback.

With the development of virtual reality, the practical requirements of the wearable haptic interface have been greatly emphasized. While passive haptic devices are commonly used in virtual reality, they lack generality and are difficult to precisely generate continuous force feedback to users. In this work, we present SmartSpring, a new solution for passive haptics, which is inexpensive, lightweight and capable of providing controllable force feedback in virtual reality. We propose a hybrid spring-linkage structure as the proxy and flexibly control the mechanism for adjustable system stiffness. By analyzing the structure and force model, we enable a smart transform of the structure for producing continuous force signals. We quantitatively examine the real-world performance of SmartSpring to verify our model. By asymmetrically moving or actively pressing the end-effector, we show that our design can further support rendering torque and stiffness. Finally, we demonstrate the SmartSpring in a series of scenarios with user studies and a just noticeable difference analysis. Experimental results show the potential of the developed haptic display in virtual reality.

Criticality and clinical department prediction of ED patients using machine learning based on heterogeneous medical data.

PROBLEM: Emergency triage faces multiple challenges, including limited medical resources and inadequate manual triage nurses, which cause incorrect triage, overcrowding in the emergency department (ED), and long patient waiting time. OBJECTIVE: This paper aims to propose and validate an accurate and efficient artificial intelligence-based method for effectively ED triage and alleviating the pressure on medical resources. METHODS: We propose two novel machine learning models, TransNet and TextRNN, for predicting patient severity levels and clinical departments using heterogeneous medical data in ED triage. Our models employ a parallel structure for feature extraction and incorporate an attention mechanism to extract essential information from the fused features, enabling accurate predictions. The models analyze the triage data (2020-2022) from the ED of Beijing University People's Hospital, incorporating variables (demographics, triage vital signs, and chief complaints) to identify patient severity levels and clinical departments. We performed data cleaning, categorization, and encoding first. Then, we divided the available data into a training set (56%), a validation set (24%), and a test set (20%) by random sampling. Finally, our models underwent 5-fold cross-validation and were compared with other state-of-the-art models. RESULTS: We comprehensively evaluated the proposed models against various Recurrent Neural Networks (RNN), Convolutional Neural Networks (CNN), Traditional Machine Learning (TML), and Transformer-based (TF) models, achieving excellent performance in predicting triage outcomes. Specifically, TextRNN achieved a prediction success rate of 86.23% [85.86-86.70] for severity levels and 94.30% [94.00-94.46] for clinical departments among 161,198 ED visits. Moreover, TransNet demonstrated higher sensitivities of 84.08% and 90.05% for severity levels and clinical departments, respectively, with specificities of 76.48% and 95.16%. The accuracy of our model is 0.87%, 0.18%, 4.29%, and 1.96%, higher than that of the above four family models on average. Furthermore, our method significantly reduced under-triage by 12.06% and over-triage by 17.92% compared to manual triage. CONCLUSIONS: Experimental results demonstrated that the proposed models fuse heterogeneous medical data in the triage process, successfully predicting patients' triage outcomes. Our models can improve triage efficiency, reduce the under/over-triage rate, and provide physicians with valuable decision-making support.

Design and Evaluation of a Wearable Fingertip Device for Three-Dimensional Skin-Slip Display.

Skin-slip provides crucial cues about the interaction state and surface properties. Currently, most skin-slip devices focus on two-dimensional tactile slip display and have limitations when displaying surface properties like bumps and contours. In this paper, a wearable fingertip device with a simple, effective, and low-cost design for three-dimensional skin-slip display is proposed. Continuous multi-directional skin-slip and normal indentation are combined to convey the sensation of three-dimensional geometric properties in virtual reality during active finger exploration. The device has a tactile belt, a five-bar mechanism, and four motors. Cooperating with the angle-mapping strategy, two micro DC motors are used to transmit continuous multi-directional skin-slip. Two servo motors are used to drive the five-bar mechanism to provide normal indentation. The characteristics of the device were obtained through the bench tests. Three experiments were designed and sequentially conducted to evaluate the performance of the device in three-dimensional surface exploration. The experimental results suggested that this device could effectively transmit continuous multi-directional skin-slip sensations, convey different bumps, and display surface contours.

MS-FRAN: A Novel Multi-Source Domain Adaptation Method for EEG-Based Emotion Recognition.

Electroencephalogram (EEG)-based emotion recognition has gradually become a research hotspot. However, the large distribution differences of EEG signals across subjects make the current research stuck in a dilemma. To resolve this problem, in this article, we propose a novel and effective method, Multi-Source Feature Representation and Alignment Network (MS-FRAN). The effectiveness of proposed method mainly comes from three new modules: Wide Feature Extractor (WFE) for feature learning, Random Matching Operation (RMO) for model training, and Top- h ranked domain classifier selection (TOP) for emotion classification. MS-FRAN is not only effective in aligning the distributions of each pair of source and target domains, but also capable of reducing the distributional differences among the multiple source domains. Experimental results on the public benchmark datasets SEED and DEAP have demonstrated the advantage of our method over the related competitive approaches for cross-subject EEG-based emotion recognition.

Synthesize Personalized Training for Robot-assisted Upper Limb Rehabilitation with Diversity Enhancement.

For upper limb rehabilitation, the robot-assisted technique in combination with serious games requires well-specified training plans. For the best quality of the rehabilitation process, customized game levels for each user are desired, while it is labor-intensive to design and adjust game levels for different individuals. We work on generating training content for a desktop end-effector rehabilitation robot and propose a method to automatically generate individualized training plans. By modeling the search of the training motions as finding optimal hand paths and trajectories, we introduce solving the design problem with a multi-objective optimization (MO) solver. We further improve the MO solver to enhance the diversity of the solutions. With the proposed approach, our system is capable of automatically generating various training plans considering the training intensity and dexterity of each joint in the upper limb. In addition, the enhanced diversity avoids repeated training plans, which helps motivate the user in the rehabilitation. We test our method with different requirements on the training plans and validate the solutions.

A Novel Soft Glove Utilizing Honeycomb Pneumatic Actuators (HPAs) for Assisting Activities of Daily Living.

Fabric-based pneumatic actuators (FPAs) are extensively employed in the design of lightweight and compliant soft wearable assistive gloves. However, conventional FPAs typically exhibit limited output force, thereby restricting the applications of such gloves. This paper presents the development of a novel honeycomb pneumatic actuator (HPA) constructed using flexible thermoplastic polyurethane (TPU) coating through hot pressing or ultrasonic welding techniques. Compared to the previously utilized double-layer fabric-based pneumatic actuators (DLFPAs), the HPAs yields a remarkable 862% increase in end output force. It can produce a tip force of 13.57 N at a pressure of 150 kPa. The integration of HPAs onto a soft pneumatic glove enables the facilitation of various activities of daily living. A series of trials involving nine patients were conducted to assess the effectiveness of the soft glove. The experimental results indicate that when assisted by the glove, the patients' finger metacarpophalangeal (MCP) and proximal interphalangeal (PIP) joints achieved angles of 87.67 ± 19.27° and 64.2 ± 30.66°, respectively. Additionally, the average fingertip force reached 10.16 ± 4.24 N, the average grip force reached 26.04 ± 15.08 N, and the completion rate of daily functions for the patients increased from 39% to 76%. These outcomes demonstrate that the soft glove effectively aids in finger movements and significantly enhances the patients' daily functioning.

Continuous shared control of a mobile robot with brain-computer interface and autonomous navigation for daily assistance.

Although the electroencephalography (EEG) based brain-computer interface (BCI) has been successfully developed for rehabilitation and assistance, it is still challenging to achieve continuous control of a brain-actuated mobile robot system. In this study, we propose a continuous shared control strategy combining continuous BCI and autonomous navigation for a mobile robot system. The weight of shared control is designed to dynamically adjust the fusion of continuous BCI control and autonomous navigation. During this process, the system uses the visual-based simultaneous localization and mapping (SLAM) method to construct environmental maps. After obtaining the global optimal path, the system utilizes the brain-based shared control dynamic window approach (BSC-DWA) to evaluate safe and reachable trajectories while considering shared control velocity. Eight subjects participated in two-stage training, and six of these eight subjects participated in online shared control experiments. The training results demonstrated that naïve subjects could achieve continuous control performance with an average percent valid correct rate of approximately 97 % and an average total correct rate of over 80 %. The results of online shared control experiments showed that all of the subjects could complete navigation tasks in an unknown corridor with continuous shared control. Therefore, our experiments verified the feasibility and effectiveness of the proposed system combining continuous BCI, shared control, autonomous navigation, and visual SLAM. The proposed continuous shared control framework shows great promise in BCI-driven tasks, especially navigation tasks for brain-driven assistive mobile robots and wheelchairs in daily applications.

Online view enhancement for exploration inside medical volumetric data using virtual reality.

BACKGROUND AND OBJECTIVE: Medical image visualization is an essential tool for conveying anatomical information. Ray-casting-based volume rendering is commonly used for generating visualizations of raw medical images. However, exposing a target area inside the skin often requires manual tuning of transfer functions or segmentation of original images, as preset parameters in volume rendering may not work well for arbitrary scanned data. This process is tedious and unnatural. To address this issue, we propose a volume visualization system that enhances the view inside the skin, enabling flexible exploration of medical volumetric data using virtual reality. METHODS: In our proposed system, we design a virtual reality interface that allows users to walk inside the data. We introduce a view-dependent occlusion weakening method based on geodesic distance transform to support this interaction. By combining these methods, we develop a virtual reality system with intuitive interactions, facilitating online view enhancement for medical data exploration and annotation inside the volume. RESULTS: Our rendering results demonstrate that the proposed occlusion weakening method effectively weakens obstacles while preserving the target area. Furthermore, comparative analysis with other alternative solutions highlights the advantages of our method in virtual reality. We conducted user studies to evaluate our system, including area annotation and line drawing tasks. The results showed that our method with enhanced views achieved 47.73% and 35.29% higher accuracy compared to the group with traditional volume rendering. Additionally, subjective feedback from medical experts further supported the effectiveness of the designed interactions in virtual reality. CONCLUSIONS: We successfully address the occlusion problems in the exploration of medical volumetric data within a virtual reality environment. Our system allows for flexible integration of scanned medical volumes without requiring extensive manual preprocessing. The results of our user studies demonstrate the feasibility and effectiveness of walk-in interaction for medical data exploration.

A multimodal domestic service robot interaction system for people with declined abilities to express themselves.

Driven by the shortage of qualified nurses and the increasing average age of the population, the ambient assisted living style using intelligent service robots and smart home systems has become an excellent choice to free up caregiver time and energy and provide users with a sense of independence. However, users' unique environments and differences in abilities to express themselves through different interaction modalities make intention recognition and interaction between user and service system very difficult, limiting the use of these new nursing technologies. This paper presents a multimodal domestic service robot interaction system and proposes a multimodal fusion algorithm for intention recognition to deal with these problems. The impacts of short-term and long-term changes were taken into account. Implemented interaction modalities include touch, voice, myoelectricity gesture, visual gesture, and haptics. Users could freely choose one or more modalities through which to express themselves. Virtual games and virtual activities of independent living were designed for pre-training and evaluating users' abilities to use different interaction modalities in their unique environments. A domestic service robot interaction system was built, on which a set of experiments were carried out to test the system's stability and intention recognition ability in different scenarios. The experiment results show that the system is stable and effective and can adapt to different scenarios. In addition, the intention recognition rate in the experiments was 93.62%. Older adults could master the system quickly and use it to provide some assistance for their independent living.