Deep autoencoder-based behavioral pattern recognition outperforms standard statistical methods in high-dimensional zebrafish studies.

Zebrafish have become an essential model organism in screening for developmental neurotoxic chemicals and their molecular targets. The success of zebrafish as a screening model is partially due to their physical characteristics including their relatively simple nervous system, rapid development, experimental tractability, and genetic diversity combined with technical advantages that allow for the generation of large amounts of high-dimensional behavioral data. These data are complex and require advanced machine learning and statistical techniques to comprehensively analyze and capture spatiotemporal responses. To accomplish this goal, we have trained semi-supervised deep autoencoders using behavior data from unexposed larval zebrafish to extract quintessential "normal" behavior. Following training, our network was evaluated using data from larvae shown to have significant changes in behavior (using a traditional statistical framework) following exposure to toxicants that include nanomaterials, aromatics, per- and polyfluoroalkyl substances (PFAS), and other environmental contaminants. Further, our model identified new chemicals (Perfluoro-n-octadecanoic acid, 8-Chloroperfluorooctylphosphonic acid, and Nonafluoropentanamide) as capable of inducing abnormal behavior at multiple chemical-concentrations pairs not captured using distance moved alone. Leveraging this deep learning model will allow for better characterization of the different exposure-induced behavioral phenotypes, facilitate improved genetic and neurobehavioral analysis in mechanistic determination studies and provide a robust framework for analyzing complex behaviors found in higher-order model systems.

MFCF-Gait: Small Silhouette-Sensitive Gait Recognition Algorithm Based on Multi-Scale Feature Cross-Fusion.

Gait recognition based on gait silhouette profiles is currently a major approach in the field of gait recognition. In previous studies, models typically used gait silhouette images sized at 64 × 64 pixels as input data. However, in practical applications, cases may arise where silhouette images are smaller than 64 × 64, leading to a loss in detail information and significantly affecting model accuracy. To address these challenges, we propose a gait recognition system named Multi-scale Feature Cross-Fusion Gait (MFCF-Gait). At the input stage of the model, we employ super-resolution algorithms to preprocess the data. During this process, we observed that different super-resolution algorithms applied to larger silhouette images also affect training outcomes. Improved super-resolution algorithms contribute to enhancing model performance. In terms of model architecture, we introduce a multi-scale feature cross-fusion network model. By integrating low-level feature information from higher-resolution images with high-level feature information from lower-resolution images, the model emphasizes smaller-scale details, thereby improving recognition accuracy for smaller silhouette images. The experimental results on the CASIA-B dataset demonstrate significant improvements. On 64 × 64 silhouette images, the accuracies for NM, BG, and CL states reached 96.49%, 91.42%, and 78.24%, respectively. On 32 × 32 silhouette images, the accuracies were 94.23%, 87.68%, and 71.57%, respectively, showing notable enhancements.

Fusion of PCA and ICA in Statistical Subset Analysis for Speech Emotion Recognition.

Speech emotion recognition is key to many fields, including human-computer interaction, healthcare, and intelligent assistance. While acoustic features extracted from human speech are essential for this task, not all of them contribute to emotion recognition effectively. Thus, reduced numbers of features are required within successful emotion recognition models. This work aimed to investigate whether splitting the features into two subsets based on their distribution and then applying commonly used feature reduction methods would impact accuracy. Filter reduction was employed using the Kruskal-Wallis test, followed by principal component analysis (PCA) and independent component analysis (ICA). A set of features was investigated to determine whether the indiscriminate use of parametric feature reduction techniques affects the accuracy of emotion recognition. For this investigation, data from three databases-Berlin EmoDB, SAVEE, and RAVDES-were organized into subsets according to their distribution in applying both PCA and ICA. The results showed a reduction from 6373 features to 170 for the Berlin EmoDB database with an accuracy of 84.3%; a final size of 130 features for SAVEE, with a corresponding accuracy of 75.4%; and 150 features for RAVDESS, with an accuracy of 59.9%.

Enhancing Facial Expression Recognition through Light Field Cameras.

In this paper, we study facial expression recognition (FER) using three modalities obtained from a light field camera: sub-aperture (SA), depth map, and all-in-focus (AiF) images. Our objective is to construct a more comprehensive and effective FER system by investigating multimodal fusion strategies. For this purpose, we employ EfficientNetV2-S, pre-trained on AffectNet, as our primary convolutional neural network. This model, combined with a BiGRU, is used to process SA images. We evaluate various fusion techniques at both decision and feature levels to assess their effectiveness in enhancing FER accuracy. Our findings show that the model using SA images surpasses state-of-the-art performance, achieving 88.13% ± 7.42% accuracy under the subject-specific evaluation protocol and 91.88% ± 3.25% under the subject-independent evaluation protocol. These results highlight our model's potential in enhancing FER accuracy and robustness, outperforming existing methods. Furthermore, our multimodal fusion approach, integrating SA, AiF, and depth images, demonstrates substantial improvements over unimodal models. The decision-level fusion strategy, particularly using average weights, proved most effective, achieving 90.13% ± 4.95% accuracy under the subject-specific evaluation protocol and 93.33% ± 4.92% under the subject-independent evaluation protocol. This approach leverages the complementary strengths of each modality, resulting in a more comprehensive and accurate FER system.

A Combined CNN Architecture for Speech Emotion Recognition.

Emotion recognition through speech is a technique employed in various scenarios of Human-Computer Interaction (HCI). Existing approaches have achieved significant results; however, limitations persist, with the quantity and diversity of data being more notable when deep learning techniques are used. The lack of a standard in feature selection leads to continuous development and experimentation. Choosing and designing the appropriate network architecture constitutes another challenge. This study addresses the challenge of recognizing emotions in the human voice using deep learning techniques, proposing a comprehensive approach, and developing preprocessing and feature selection stages while constructing a dataset called EmoDSc as a result of combining several available databases. The synergy between spectral features and spectrogram images is investigated. Independently, the weighted accuracy obtained using only spectral features was 89%, while using only spectrogram images, the weighted accuracy reached 90%. These results, although surpassing previous research, highlight the strengths and limitations when operating in isolation. Based on this exploration, a neural network architecture composed of a CNN1D, a CNN2D, and an MLP that fuses spectral features and spectogram images is proposed. The model, supported by the unified dataset EmoDSc, demonstrates a remarkable accuracy of 96%.

Improving Human Activity Recognition With Wearable Sensors Through BEE: Leveraging Early Exit and Gradient Boosting.

Early-exiting has recently provided an ideal solution for accelerating activity inference by attaching internal classifiers to deep neural networks. It allows easy activity samples to be predicted at shallower layers, without executing deeper layers, hence leading to notable adaptiveness in terms of accuracy-speed trade-off under varying resource demands. However, prior most works typically optimize all the classifiers equally on all types of activity data. As a result, deeper classifiers will only see hard samples during test phase, which renders the model suboptimal due to the training-test data distribution mismatch. Such issue has been rarely explored in the context of activity recognition. In this paper, to close the gap, we propose to organize all these classifiers as a dynamic-depth network and jointly optimize them in a similar gradient-boosting manner. Specifically, a gradient-rescaling is employed to bound the gradients of parameters at different depths, that makes such training procedure more stable. Particularly, we perform a prediction reweighting to emphasize current deep classifier while weakening the ensemble of its previous classifiers, so as to relieve the shortage of training data at deeper classifiers. Comprehensive experiments on multiple HAR benchmarks including UCI-HAR, PAMAP2, UniMiB-SHAR, and USC-HAD verify that it is state-of-the-art in accuracy and speed. A real implementation is measured on an ARM-based mobile device.

LiteFer: An Approach Based on MobileViT Expression Recognition.

Facial expression recognition using convolutional neural networks (CNNs) is a prevalent research area, and the network's complexity poses obstacles for deployment on devices with limited computational resources, such as mobile devices. To address these challenges, researchers have developed lightweight networks with the aim of reducing model size and minimizing parameters without compromising accuracy. The LiteFer method introduced in this study incorporates depth-separable convolution and a lightweight attention mechanism, effectively reducing network parameters. Moreover, through comprehensive comparative experiments on the RAFDB and FERPlus datasets, its superior performance over various state-of-the-art lightweight expression-recognition methods is evident.

Gesture Recognition Framework for Teleoperation of Infrared (IR) Consumer Devices Using a Novel pFMG Soft Armband.

Wearable technologies represent a significant advancement in facilitating communication between humans and machines. Powered by artificial intelligence (AI), human gestures detected by wearable sensors can provide people with seamless interaction with physical, digital, and mixed environments. In this paper, the foundations of a gesture-recognition framework for the teleoperation of infrared consumer electronics are established. This framework is based on force myography data of the upper forearm, acquired from a prototype novel soft pressure-based force myography (pFMG) armband. Here, the sub-processes of the framework are detailed, including the acquisition of infrared and force myography data; pre-processing; feature construction/selection; classifier selection; post-processing; and interfacing/actuation. The gesture recognition system is evaluated using 12 subjects' force myography data obtained whilst performing five classes of gestures. Our results demonstrate an inter-session and inter-trial gesture average recognition accuracy of approximately 92.2% and 88.9%, respectively. The gesture recognition framework was successfully able to teleoperate several infrared consumer electronics as a wearable, safe and affordable human-machine interface system. The contribution of this study centres around proposing and demonstrating a user-centred design methodology to allow direct human-machine interaction and interface for applications where humans and devices are in the same loop or coexist, as typified between users and infrared-communicating devices in this study.

Machine Learning-Based Gesture Recognition Glove: Design and Implementation.

In the evolving field of human-computer interaction (HCI), gesture recognition has emerged as a critical focus, with smart gloves equipped with sensors playing one of the most important roles. Despite the significance of dynamic gesture recognition, most research on data gloves has concentrated on static gestures, with only a small percentage addressing dynamic gestures or both. This study explores the development of a low-cost smart glove prototype designed to capture and classify dynamic hand gestures for game control and presents a prototype of data gloves equipped with five flex sensors, five force sensors, and one inertial measurement unit (IMU) sensor. To classify dynamic gestures, we developed a neural network-based classifier, utilizing a convolutional neural network (CNN) with three two-dimensional convolutional layers and rectified linear unit (ReLU) activation where its accuracy was 90%. The developed glove effectively captures dynamic gestures for game control, achieving high classification accuracy, precision, and recall, as evidenced by the confusion matrix and training metrics. Despite limitations in the number of gestures and participants, the solution offers a cost-effective and accurate approach to gesture recognition, with potential applications in VR/AR environments.

Pattern recognition using spiking antiferromagnetic neurons.

Spintronic devices offer a promising avenue for the development of nanoscale, energy-efficient artificial neurons for neuromorphic computing. It has previously been shown that with antiferromagnetic (AFM) oscillators, ultra-fast spiking artificial neurons can be made that mimic many unique features of biological neurons. In this work, we train an artificial neural network of AFM neurons to perform pattern recognition. A simple machine learning algorithm called spike pattern association neuron (SPAN), which relies on the temporal position of neuron spikes, is used during training. In under a microsecond of physical time, the AFM neural network is trained to recognize symbols composed from a grid by producing a spike within a specified time window. We further achieve multi-symbol recognition with the addition of an output layer to suppress undesirable spikes. Through the utilization of AFM neurons and the SPAN algorithm, we create a neural network capable of high-accuracy recognition with overall power consumption on the order of picojoules.

Hand gesture recognition using sEMG signals with a multi-stream time-varying feature enhancement approach.

Hand gesture recognition based on sparse multichannel surface electromyography (sEMG) still poses a significant challenge to deployment as a muscle-computer interface. Many researchers have been working to develop an sEMG-based hand gesture recognition system. However, the existing system still faces challenges in achieving satisfactory performance due to ineffective feature enhancement, so the prediction is erratic and unstable. To comprehensively tackle these challenges, we introduce a novel approach: a lightweight sEMG-based hand gesture recognition system using a 4-stream deep learning architecture. Each stream strategically combines Temporal Convolutional Network (TCN)-based time-varying features with Convolutional Neural Network (CNN)-based frame-wise features. In the first stream, we harness the power of the TCN module to extract nuanced time-varying temporal features. The second stream integrates a hybrid Long short-term memory (LSTM)-TCN module. This stream extracts temporal features using LSTM and seamlessly enhances them with TCN to effectively capture intricate long-range temporal relations. The third stream adopts a spatio-temporal strategy, merging the CNN and TCN modules. This integration facilitates concurrent comprehension of both spatial and temporal features, enriching the model's understanding of the underlying dynamics of the data. The fourth stream uses a skip connection mechanism to alleviate potential problems of data loss, ensuring a robust information flow throughout the network and concatenating the 4 stream features, yielding a comprehensive and effective final feature representation. We employ a channel attention-based feature selection module to select the most effective features, aiming to reduce the computational complexity and feed them into the classification module. The proposed model achieves an average accuracy of 94.31% and 98.96% on the Ninapro DB1 and DB9 datasets, respectively. This high-performance accuracy proves the superiority of the proposed model, and its implications extend to enhancing the quality of life for individuals using prosthetic limbs and advancing control systems in the field of robotic human-machine interfaces.

Graph-based cell pattern recognition for merging the multi-modal optical microscopic image of neurons.

A deep understanding of neuron structure and function is crucial for elucidating brain mechanisms, diagnosing and treating diseases. Optical microscopy, pivotal in neuroscience, illuminates neuronal shapes, projections, and electrical activities. To explore the projection of specific functional neurons, scientists have been developing optical-based multimodal imaging strategies to simultaneously capture dynamic in vivo signals and static ex vivo structures from the same neuron. However, the original position of neurons is highly susceptible to displacement during ex vivo imaging, presenting a significant challenge for integrating multimodal information at the single-neuron level. This study introduces a graph-model-based approach for cell image matching, facilitating precise and automated pairing of sparsely labeled neurons across different optical microscopic images. It has been shown that utilizing neuron distribution as a matching feature can mitigate modal differences, the high-order graph model can address scale inconsistency, and the nonlinear iteration can resolve discrepancies in neuron density. This strategy was applied to the connectivity study of the mouse visual cortex, performing cell matching between the two-photon calcium image and the HD-fMOST brain-wide anatomical image sets. Experimental results demonstrate 96.67% precision, 85.29% recall rate, and 90.63% F1 Score, comparable to expert technicians. This study builds a bridge between functional and structural imaging, offering crucial technical support for neuron classification and circuitry analysis.

A novel procedure to automate the removal of PLI and motion artifacts using mode decomposition to enhance pattern recognition of sEMG signals for myoelectric control of prosthesis.

Hand Movement Recognition (HMR) with sEMG is crucial for artificial hand prostheses. HMR performance mostly depends on the feature information that is fed to the classifiers. However, sEMG often captures noise like power line interference (PLI) and motion artifacts. This may extract redundant and insignificant feature information, which can degrade HMR performance and increase computational complexity. This study aims to address these issues by proposing a novel procedure for automatically removing PLI and motion artifacts from experimental sEMG signals. This will make it possible to extract better features from the signal and improve the categorization of various hand movements. Empirical mode decomposition and energy entropy thresholding are utilized to select relevant mode components for artifact removal. Time domain features are then used to train classifiers (kNN, LDA, SVM) for hand movement categorization, achieving average accuracies of 92.36%, 93.63%, and 98.12%, respectively, across subjects. Additionally, muscle contraction efforts are classified into low, medium, and high categories using this technique. Validation is performed on data from ten subjects performing eight hand movement classes and three muscle contraction efforts with three surface electrode channels. Results indicate that the proposed preprocessing improves average accuracy by 9.55% with the SVM classifier, significantly reducing computational time.

Mitigating the Concurrent Interference of Electrode Shift and Loosening in Myoelectric Pattern Recognition Using Siamese Autoencoder Network.

The objective of this work is to develop a novel myoelectric pattern recognition (MPR) method to mitigate the concurrent interference of electrode shift and loosening, thereby improving the practicality of MPR-based gestural interfaces towards intelligent control. A Siamese auto-encoder network (SAEN) was established to learn robust feature representations against random occurrences of both electrode shift and loosening. The SAEN model was trained with a variety of shifted-view and masked-view feature maps, which were simulated through feature transformation operated on the original feature maps. Specifically, three mean square error (MSE) losses were devised to warrant the trained model's capability in adaptive recovery of any given interfered data. The SAEN was deployed as an independent feature extractor followed by a common support vector machine acting as the classifier. To evaluate the effectiveness of the proposed method, an eight-channel armband was adopted to collect surface electromyography (EMG) signals from nine subjects performing six gestures. Under the condition of concurrent interference, the proposed method achieved the highest classification accuracy in both offline and online testing compared to five common methods, with statistical significance (p <0.05). The proposed method was demonstrated to be effective in mitigating the electrode shift and loosening interferences. Our work offers a valuable solution for enhancing the robustness of myoelectric control systems.

Feature evaluation for myoelectric pattern recognition of multiple nearby reaching targets.

Intention detection of the reaching movement is considerable for myoelectric human and machine collaboration applications. A comprehensive set of handcrafted features was mined from windows of electromyogram (EMG) of the upper-limb muscles while reaching nine nearby targets like activities of daily living. The feature selection-based scoring method, neighborhood component analysis (NCA), selected the relevant feature subset. Finally, the target was recognized by the support vector machine (SVM) model. The classification performance was generalized by a nested cross-validation structure that selected the optimal feature subset in the inner loop. According to the low spatial resolution of the target location on display and following the slight discrimination of signals between targets, the best classification accuracy of 77.11 % was achieved for concatenating the features of two segments with a length of 2 and 0.25 s. Due to the lack of subtle variation in EMG, while reaching different targets, a wide range of features was applied to consider additional aspects of the knowledge contained in EMG signals. Furthermore, since NCA selected features that provided more discriminant power, it became achievable to employ various combinations of features and even concatenated features extracted from different movement parts to improve classification performance.

Improving Hand Gesture Recognition Robustness to Dynamic Posture Variations by Multimodal Deep Feature Fusion.

Surface electromyography (sEMG), a human-machine interface for gesture recognition, has shown promising potential for decoding motor intentions, but a variety of nonideal factors restrict its practical application in assistive robots. In this paper, we summarized the current mainstream gesture recognition strategies and proposed a gesture recognition method based on multimodal canonical correlation analysis feature fusion classification (MCAFC) for a nonideal condition that occurs in daily life, i.e., posture variations. The deep features of the sEMG and acceleration signals were first extracted via convolutional neural networks. A canonical correlation analysis was subsequently performed to associate the deep features of the two modalities. The transformed features were utilized as inputs to a linear discriminant analysis classifier to recognize the corresponding gestures. Both offline and real-time experiments were conducted on eight non-disabled subjects. The experimental results indicated that MCAFC achieved an average classification accuracy, average motion completion rate, and average motion completion time of 93.44%, 94.05%, and 1.38 s, respectively, with multiple dynamic postures, indicating significantly better performance than that of comparable methods. The results demonstrate the feasibility and superiority of the proposed multimodal signal feature fusion method for gesture recognition with posture variations, providing a new scheme for myoelectric control.

CDGT: Constructing diverse graph transformers for emotion recognition from facial videos.

Recognizing expressions from dynamic facial videos can find more natural affect states of humans, and it becomes a more challenging task in real-world scenes due to pose variations of face, partial occlusions and subtle dynamic changes of emotion sequences. Existing transformer-based methods often focus on self-attention to model the global relations among spatial features or temporal features, which cannot well focus on important expression-related locality structures from both spatial and temporal features for the in-the-wild expression videos. To this end, we incorporate diverse graph structures into transformers and propose a CDGT method to construct diverse graph transformers for efficient emotion recognition from in-the-wild videos. Specifically, our method contains a spatial dual-graphs transformer and a temporal hyperbolic-graph transformer. The former deploys a dual-graph constrained attention to capture latent emotion-related graph geometry structures among local spatial tokens for efficient feature representation, especially for the video frames with pose variations and partial occlusions. The latter adopts a hyperbolic-graph constrained self-attention that explores important temporal graph structure information under hyperbolic space to model more subtle changes of dynamic emotion. Extensive experimental results on in-the-wild video-based facial expression databases show that our proposed CDGT outperforms other state-of-the-art methods.

Dark-DSAR: Lightweight one-step pipeline for action recognition in dark videos.

Dark video human action recognition has a wide range of applications in the real world. General action recognition methods focus on the actor or the action itself, ignoring the dark scene where the action happens, resulting in unsatisfied accuracy in recognition. For dark scenes, the existing two-step action recognition methods are stage complex due to introducing additional augmentation steps, and the one-step pipeline method is not lightweight enough. To address these issues, a one-step Transformer-based method named Dark Domain Shift for Action Recognition (Dark-DSAR) is proposed in this paper, which integrates the tasks of domain migration and classification into a single step and enhances the model's functional coherence with respect to these two tasks, making our Dark-DSAR has low computation but high accuracy. Specifically, the domain shift module (DSM) achieves domain adaption from dark to bright to reduce the number of parameters and the computational cost. Besides, we explore the matching relationship between the input video size and the model, which can further optimize the inference efficiency by removing the redundant information in videos through spatial resolution dropping. Extensive experiments have been conducted on the datasets of ARID1.5, HMDB51-Dark, and UAV-human-night. Results show that the proposed Dark-DSAR obtains the best Top-1 accuracy on ARID1.5 with 89.49%, which is 2.56% higher than the state-of-the-art method, 67.13% and 61.9% on HMDB51-Dark and UAV-human-night, respectively. In addition, ablation experiments reveal that the action classifiers can gain ≥1% in accuracy compared to the original model when equipped with our DSM.

Aggregated Pattern Classification Method for improving neural disorder stage detection.

BACKGROUND: Neurological disorders pose a significant health challenge, and their early detection is critical for effective treatment planning and prognosis. Traditional classification of neural disorders based on causes, symptoms, developmental stage, severity, and nervous system effects has limitations. Leveraging artificial intelligence (AI) and machine learning (ML) for pattern recognition provides a potent solution to address these challenges. Therefore, this study focuses on proposing an innovative approach-the Aggregated Pattern Classification Method (APCM)-for precise identification of neural disorder stages. METHOD: The APCM was introduced to address prevalent issues in neural disorder detection, such as overfitting, robustness, and interoperability. This method utilizes aggregative patterns and classification learning functions to mitigate these challenges and enhance overall recognition accuracy, even in imbalanced data. The analysis involves neural images using observations from healthy individuals as a reference. Action response patterns from diverse inputs are mapped to identify similar features, establishing the disorder ratio. The stages are correlated based on available responses and associated neural data, with a preference for classification learning. This classification necessitates image and labeled data to prevent additional flaws in pattern recognition. Recognition and classification occur through multiple iterations, incorporating similar and diverse neural features. The learning process is finely tuned for minute classifications using labeled and unlabeled input data. RESULTS: The proposed APCM demonstrates notable achievements, with high pattern recognition (15.03%) and controlled classification errors (CEs) (10.61% less). The method effectively addresses overfitting, robustness, and interoperability issues, showcasing its potential as a powerful tool for detecting neural disorders at different stages. The ability to handle imbalanced data contributes to the overall success of the algorithm. CONCLUSION: The APCM emerges as a promising and effective approach for identifying precise neural disorder stages. By leveraging AI and ML, the method successfully resolves key challenges in pattern recognition. The high pattern recognition and reduced CEs underscore the method's potential for clinical applications. However, it is essential to acknowledge the reliance on high-quality neural image data, which may limit the generalizability of the approach. The proposed method allows future research to refine further and enhance its interpretability, providing valuable insights into neural disorder progression and underlying biological mechanisms.

Unravelling Influence Factors in Pattern Recognition Myoelectric Control Systems: The Impact of Limb Positions and Electrode Shifts.

Pattern recognition (PR)-based myoelectric control systems can naturally provide multifunctional and intuitive control of upper limb prostheses and restore lost limb function, but understanding their robustness remains an open scientific question. This study investigates how limb positions and electrode shifts-two factors that have been suggested to cause classification deterioration-affect classifiers' performance by quantifying changes in the class distribution using each factor as a class and computing the repeatability and modified separability indices. Ten intact-limb participants took part in the study. Linear discriminant analysis (LDA) was used as the classifier. The results confirmed previous studies that limb positions and electrode shifts deteriorate classification performance (14-21% decrease) with no difference between factors (p > 0.05). When considering limb positions and electrode shifts as classes, we could classify them with an accuracy of 96.13 ± 1.44% and 65.40 ± 8.23% for single and all motions, respectively. Testing on five amputees corroborated the above findings. We have demonstrated that each factor introduces changes in the feature space that are statistically new class instances. Thus, the feature space contains two statistically classifiable clusters when the same motion is collected in two different limb positions or electrode shifts. Our results are a step forward in understanding PR schemes' challenges for myoelectric control of prostheses and further validation needs be conducted on more amputee-related datasets.