Machine learning-empowered sleep staging classification using multi-modality signals.

The goal is to enhance an automated sleep staging system's performance by leveraging the diverse signals captured through multi-modal polysomnography recordings. Three modalities of PSG signals, namely electroencephalogram (EEG), electrooculogram (EOG), and electromyogram (EMG), were considered to obtain the optimal fusions of the PSG signals, where 63 features were extracted. These include frequency-based, time-based, statistical-based, entropy-based, and non-linear-based features. We adopted the ReliefF (ReF) feature selection algorithms to find the suitable parts for each signal and superposition of PSG signals. Twelve top features were selected while correlated with the extracted feature sets' sleep stages. The selected features were fed into the AdaBoost with Random Forest (ADB + RF) classifier to validate the chosen segments and classify the sleep stages. This study's experiments were investigated by obtaining two testing schemes: epoch-wise testing and subject-wise testing. The suggested research was conducted using three publicly available datasets: ISRUC-Sleep subgroup1 (ISRUC-SG1), sleep-EDF(S-EDF), Physio bank CAP sleep database (PB-CAPSDB), and S-EDF-78 respectively. This work demonstrated that the proposed fusion strategy overestimates the common individual usage of PSG signals.

Simplifying Multimodal With Single EOG Modality for Automatic Sleep Staging.

Polysomnography (PSG) recordings have been widely used for sleep staging in clinics, containing multiple modality signals (i.e., EEG and EOG). Recently, many studies have combined EEG and EOG modalities for sleep staging, since they are the most and the second most powerful modality for sleep staging among PSG recordings, respectively. However, EEG is complex to collect and sensitive to environment noise or other body activities, imbedding its use in clinical practice. Comparatively, EOG is much more easily to be obtained. In order to make full use of the powerful ability of EEG and the easy collection of EOG, we propose a novel framework to simplify multimodal sleep staging with a single EOG modality. It still performs well with only EOG modality in the absence of the EEG. Specifically, we first model the correlation between EEG and EOG, and then based on the correlation we generate multimodal features with time and frequency guided generators by adopting the idea of generative adversarial learning. We collected a real-world sleep dataset containing 67 recordings and used other four public datasets for evaluation. Compared with other existing sleep staging methods, our framework performs the best when solely using the EOG modality. Moreover, under our framework, EOG provides a comparable performance to EEG.

Association of Visual-Based Signals with Electroencephalography Patterns in Enhancing the Drowsiness Detection in Drivers with Obstructive Sleep Apnea.

Individuals with obstructive sleep apnea (OSA) face increased accident risks due to excessive daytime sleepiness. PERCLOS, a recognized drowsiness detection method, encounters challenges from image quality, eyewear interference, and lighting variations, impacting its performance, and requiring validation through physiological signals. We propose visual-based scoring using adaptive thresholding for eye aspect ratio with OpenCV for face detection and Dlib for eye detection from video recordings. This technique identified 453 drowsiness (PERCLOS ≥ 0.3 || CLOSDUR ≥ 2 s) and 474 wakefulness episodes (PERCLOS < 0.3 and CLOSDUR < 2 s) among fifty OSA drivers in a 50 min driving simulation while wearing six-channel EEG electrodes. Applying discrete wavelet transform, we derived ten EEG features, correlated them with visual-based episodes using various criteria, and assessed the sensitivity of brain regions and individual EEG channels. Among these features, theta-alpha-ratio exhibited robust mapping (94.7%) with visual-based scoring, followed by delta-alpha-ratio (87.2%) and delta-theta-ratio (86.7%). Frontal area (86.4%) and channel F4 (75.4%) aligned most episodes with theta-alpha-ratio, while frontal, and occipital regions, particularly channels F4 and O2, displayed superior alignment across multiple features. Adding frontal or occipital channels could correlate all episodes with EEG patterns, reducing hardware needs. Our work could potentially enhance real-time drowsiness detection reliability and assess fitness to drive in OSA drivers.

Single-channel EOG sleep staging on a heterogeneous cohort of subjects with sleep disorders.

Objective.Sleep staging based on full polysomnography is the gold standard in the diagnosis of many sleep disorders. It is however costly, complex, and obtrusive due to the use of multiple electrodes. Automatic sleep staging based on single-channel electro-oculography (EOG) is a promising alternative, requiring fewer electrodes which could be self-applied below the hairline. EOG sleep staging algorithms are however yet to be validated in clinical populations with sleep disorders.Approach.We utilized the SOMNIA dataset, comprising 774 recordings from subjects with various sleep disorders, including insomnia, sleep-disordered breathing, hypersomnolence, circadian rhythm disorders, parasomnias, and movement disorders. The recordings were divided into train (574), validation (100), and test (100) groups. We trained a neural network that integrated transformers within a U-Net backbone. This design facilitated learning of arbitrary-distance temporal relationships within and between the EOG and hypnogram.Main results.For 5-class sleep staging, we achieved median accuracies of 85.0% and 85.2% and Cohen's kappas of 0.781 and 0.796 for left and right EOG, respectively. The performance using the right EOG was significantly better than using the left EOG, possibly because in the recommended AASM setup, this electrode is located closer to the scalp. The proposed model is robust to the presence of a variety of sleep disorders, displaying no significant difference in performance for subjects with a certain sleep disorder compared to those without.Significance.The results show that accurate sleep staging using single-channel EOG can be done reliably for subjects with a variety of sleep disorders.

Comparing targeted memory reactivation during slow wave sleep and sleep stage 2.

Sleep facilitates declarative memory consolidation, which is assumed to rely on the reactivation of newly encoded memories orchestrated by the temporal interplay of slow oscillations (SO), fast spindles and ripples. SO as well as the number of spindles coupled to SO are more frequent during slow wave sleep (SWS) compared to lighter sleep stage 2 (S2). But, it is unclear whether memory reactivation is more effective during SWS than during S2. To test this question, we applied Targeted Memory Reactivation (TMR) in a declarative memory design by presenting learning-associated sound cues during SWS vs. S2 in a counterbalanced within-subject design. Contrary to our hypothesis, memory performance was not significantly better when cues were presented during SWS. Event-related potential (ERP) amplitudes were significantly higher for cues presented during SWS than S2, and the density of SO and SO-spindle complexes was generally higher during SWS than during S2. Whereas SO density increased during and after the TMR period, SO-spindle complexes decreased. None of the parameters were associated with memory performance. These findings suggest that the efficacy of TMR does not depend on whether it is administered during SWS or S2, despite differential processing of memory cues in these sleep stages.

Alpha anteriorization and theta posteriorization during deep sleep.

Brain states (wake, sleep, general anesthesia, etc.) are profoundly associated with the spatiotemporal dynamics of brain oscillations. Previous studies showed that the EEG alpha power shifted from the occipital cortex to the frontal cortex (alpha anteriorization) after being induced into a state of general anesthesia via propofol. The sleep research literature suggests that slow waves and sleep spindles are generated locally and propagated gradually to different brain regions. Since sleep and general anesthesia are conceptualized under the same framework of consciousness, the present study examines whether alpha anteriorization similarly occurs during sleep and how the EEG power in other frequency bands changes during different sleep stages. The results from the analysis of three polysomnography datasets of 234 participants show consistent alpha anteriorization during the sleep stages N2 and N3, beta anteriorization during stage REM, and theta posteriorization during stages N2 and N3. Although it is known that the neural circuits responsible for sleep are not exactly the same for general anesthesia, the findings of alpha anteriorization in this study suggest that, at macro level, the circuits for alpha oscillations are organized in the similar cortical areas. The spatial shifts of EEG power in different frequency bands during sleep may offer meaningful neurophysiological markers for the level of consciousness.

Sleep-stage dependence and co-existence of cardio-respiratory coordination and phase synchronization.

Interactions between the cardiac and respiratory systems play a pivotal role in physiological functioning. Nonetheless, the intricacies of cardio-respiratory couplings, such as cardio-respiratory phase synchronization (CRPS) and cardio-respiratory coordination (CRC), remain elusive, and an automated algorithm for CRC detection is lacking. This paper introduces an automated CRC detection algorithm, which allowed us to conduct a comprehensive comparison of CRPS and CRC during sleep for the first time using an extensive database. We found that CRPS is more sensitive to sleep-stage transitions, and intriguingly, there is a negative correlation between the degree of CRPS and CRC when fluctuations in breathing frequency are high. This comparative analysis holds promise in assisting researchers in gaining deeper insights into the mechanics of and distinctions between these two physiological phenomena. Additionally, the automated algorithms we devised have the potential to offer valuable insights into the clinical applications of CRC and CRPS.

ZleepAnlystNet: a novel deep learning model for automatic sleep stage scoring based on single-channel raw EEG data using separating training.

Numerous models for sleep stage scoring utilizing single-channel raw EEG signal have typically employed CNN and BiLSTM architectures. While these models, incorporating temporal information for sequence classification, demonstrate superior overall performance, they often exhibit low per-class performance for N1-stage, necessitating an adjustment of loss function. However, the efficacy of such adjustment is constrained by the training process. In this study, a pioneering training approach called separating training is introduced, alongside a novel model, to enhance performance. The developed model comprises 15 CNN models with varying loss function weights for feature extraction and 1 BiLSTM for sequence classification. Due to its architecture, this model cannot be trained using an end-to-end approach, necessitating separate training for each component using the Sleep-EDF dataset. Achieving an overall accuracy of 87.02%, MF1 of 82.09%, Kappa of 0.8221, and per-class F1-socres (W 90.34%, N1 54.23%, N2 89.53%, N3 88.96%, and REM 87.40%), our model demonstrates promising performance. Comparison with sleep technicians reveals a Kappa of 0.7015, indicating alignment with reference sleep stags. Additionally, cross-dataset validation and adaptation through training with the SHHS dataset yield an overall accuracy of 84.40%, MF1 of 74.96% and Kappa of 0.7785 when tested with the Sleep-EDF-13 dataset. These findings underscore the generalization potential in model architecture design facilitated by our novel training approach.

Fundamentals of sleep regulation: Model and benchmark values for fractal and oscillatory neurodynamics.

Homeostatic, circadian and ultradian mechanisms play crucial roles in the regulation of sleep. Evidence suggests that ratios of low-to-high frequency power in the electroencephalogram (EEG) spectrum indicate the instantaneous level of sleep pressure, influenced by factors such as individual sleep-wake history, current sleep stage, age-related differences and brain topography characteristics. These effects are well captured and reflected in the spectral exponent, a composite measure of the constant low-to-high frequency ratio in the periodogram, which is scale-free and exhibits lower interindividual variability compared to slow wave activity, potentially serving as a suitable standardization and reference measure. Here we propose an index of sleep homeostasis based on the spectral exponent, reflecting the level of membrane hyperpolarization and/or network bistability in the central nervous system in humans. In addition, we advance the idea that the U-shaped overnight deceleration of oscillatory slow and fast sleep spindle frequencies marks the biological night, providing somnologists with an EEG-index of circadian sleep regulation. Evidence supporting this assertion comes from studies based on sleep replacement, forced desynchrony protocols and high-resolution analyses of sleep spindles. Finally, ultradian sleep regulatory mechanisms are indicated by the recurrent, abrupt shifts in dominant oscillatory frequencies, with spindle ranges signifying non-rapid eye movement and non-spindle oscillations - rapid eye movement phases of the sleep cycles. Reconsidering the indicators of fundamental sleep regulatory processes in the framework of the new Fractal and Oscillatory Adjustment Model (FOAM) offers an appealing opportunity to bridge the gap between the two-process model of sleep regulation and clinical somnology.

Timing matters for accurate identification of the epileptogenic zone.

OBJECTIVE: Interictal biomarkers of the epileptogenic zone (EZ) and their use in machine learning models open promising avenues for improvement of epilepsy surgery evaluation. Currently, most studies restrict their analysis to short segments of intracranial EEG (iEEG). METHODS: We used 2381 hours of iEEG data from 25 patients to systematically select 5-minute segments across various interictal conditions. Then, we tested machine learning models for EZ localization using iEEG features calculated within these individual segments or across them and evaluated the performance by the area under the precision-recall curve (PRAUC). RESULTS: On average, models achieved a score of 0.421 (the result of the chance classifier was 0.062). However, the PRAUC varied significantly across the segments (0.323-0.493). Overall, NREM sleep achieved the highest scores, with the best results of 0.493 in N2. When using data from all segments, the model performed significantly better than single segments, except NREM sleep segments. CONCLUSIONS: The model based on a short segment of iEEG recording can achieve similar results as a model based on prolonged recordings. The analyzed segment should, however, be carefully and systematically selected, preferably from NREM sleep. SIGNIFICANCE: Random selection of short iEEG segments may give rise to inaccurate localization of the EZ.

Cardiac autonomic nervous activity during different sleep stages in individuals with spinal cord injury: The influence of physical training.

OBJECTIVE: The present study assessed the influence of physical training on cardiac autonomic activity in individuals with spinal cord injury (SCI) during different sleep stages. METHODS: Twenty-six volunteers were allocated into three groups: 9 sedentary individuals without SCI (control, CON); 8 sedentary tetraplegic individuals with chronic SCI (SED-SCI); 9 physically trained tetraplegic individuals with chronic SCI (TR-SCI). All participants underwent nocturnal polysomnography to monitor sleep stages: wakefulness, non-rapid eye movement (NREM) sleep (N1, N2, and N3 stages), and REM sleep. The electrocardiography data obtained during this exam were extracted to analyze the heart rate variability (HRV). RESULTS: Sleep stages influenced HRV in the time [RR interval and root mean square of successive RR interval differences (RMSSD)] and frequency [low-frequency (LF) and high-frequency (HF) powers and LF-to-HF ratio] domains (P < 0.05). SED-SCI individuals showed unchanged HRV compared to CON (P > 0.05). When comparing the TR-SCI and SED-SCI groups, no significant differences in HRV were reported in the time domain (P > 0.05). However, in the frequency domain, more accentuated HF power was observed in TR-SCI than in SED-SCI individuals during the N2 and N3 stages and REM sleep (P < 0.05). Moreover, TR-SCI had higher HF power than CON during the N3 stage (P < 0.05). CONCLUSIONS: TR-SCI individuals have greater HF power, indicative of parasympathetic modulation, than sedentary (injured or not injured) individuals during different sleep stages. Therefore, enhanced parasympathetic activity induced by physical training may improve cardiac autonomic modulation during sleep in individuals with chronic SCI.

Classification and automatic scoring of arousal intensity during sleep stages using machine learning.

Arousal during sleep can result in sleep fragmentation and various physiological effects, impairing cognitive function and raising blood pressure and heart rate. However, the current definition of arousal has limitations in assessing both amplitude and duration, making it challenging to measure sleep fragmentation accurately. Moreover, there is inconsistency among inter-raters in arousal scoring, which renders it susceptible to subjective variability. Therefore, this study aims to identify a highly accurate classifier for each sleep stage by employing optimized feature selection and machine learning models. According to electroencephalography (EEG) signals during the arousal phase, the intensity level was categorized into four levels. For control, the non-arousal cases were used as level 0 and referred as sham arousal, resulting in five arousal intensity levels. Wavelet transform was applied to analyze sleep arousal to extract features from EEG. Based on these features, we classified arousal intensity levels through machine learning algorithms. Due to the different characteristics of EEG in each sleep stage, the classification model was optimized for the four sleep stages. Excluding sham arousals, a total of 13,532 arousal events were used. The lowest intensity in the entire data, level 1, was computed to be 3107, level 2 was 3384, level 3 was 3472, and the highest intensity of level 4 was 3,569. The optimized classification model for each sleep stage achieved an average sensitivity of 82.68%, specificity of 95.68%, and AUROC of 96.30%. The sensitivity of the control, arousal intensity level 0, was 83.07%, a 1.25% increase over the unoptimized model and a 14.22% increase over previous research. This study used machine learning techniques to develop classifiers for each sleep stage, improving the accuracy of arousal intensity classification. The classifiers showed high sensitivity and specificity and revealed the unique characteristics of arousal intensity during different sleep stages. These findings represent a novel approach to arousal research and have implications for developing more accurate predictive models in sleep research.

Multivariate classification of multichannel long-term electrophysiology data identifies different sleep stages in fruit flies.

Identifying different sleep stages in humans and other mammals has traditionally relied on electroencephalograms. Such an approach is not feasible in certain animals such as invertebrates, although these animals could also be sleeping in stages. Here, we perform long-term multichannel local field potential recordings in the brains of behaving flies undergoing spontaneous sleep bouts. We acquired consistent spatial recordings of local field potentials across multiple flies, allowing us to compare brain activity across awake and sleep periods. Using machine learning, we uncover distinct temporal stages of sleep and explore the associated spatial and spectral features across the fly brain. Further, we analyze the electrophysiological correlates of microbehaviors associated with certain sleep stages. We confirm the existence of a distinct sleep stage associated with rhythmic proboscis extensions and show that spectral features of this sleep-related behavior differ significantly from those associated with the same behavior during wakefulness, indicating a dissociation between behavior and the brain states wherein these behaviors reside.

SlumberNet: deep learning classification of sleep stages using residual neural networks.

Sleep research is fundamental to understanding health and well-being, as proper sleep is essential for maintaining optimal physiological function. Here we present SlumberNet, a novel deep learning model based on residual network (ResNet) architecture, designed to classify sleep states in mice using electroencephalogram (EEG) and electromyogram (EMG) signals. Our model was trained and tested on data from mice undergoing baseline sleep, sleep deprivation, and recovery sleep, enabling it to handle a wide range of sleep conditions. Employing k-fold cross-validation and data augmentation techniques, SlumberNet achieved high levels of overall performance (accuracy = 97%; F1 score = 96%) in predicting sleep stages and showed robust performance even with a small and diverse training dataset. Comparison of SlumberNet's performance to manual sleep stage classification revealed a significant reduction in analysis time (~ 50 × faster), without sacrificing accuracy. Our study showcases the potential of deep learning to facilitate sleep research by providing a more efficient, accurate, and scalable method for sleep stage classification. Our work with SlumberNet further demonstrates the power of deep learning in mouse sleep research.

[Multi-modal physiological time-frequency feature extraction network for accurate sleep stage classification].

Sleep stage classification is essential for clinical disease diagnosis and sleep quality assessment. Most of the existing methods for sleep stage classification are based on single-channel or single-modal signal, and extract features using a single-branch, deep convolutional network, which not only hinders the capture of the diversity features related to sleep and increase the computational cost, but also has a certain impact on the accuracy of sleep stage classification. To solve this problem, this paper proposes an end-to-end multi-modal physiological time-frequency feature extraction network (MTFF-Net) for accurate sleep stage classification. First, multi-modal physiological signal containing electroencephalogram (EEG), electrocardiogram (ECG), electrooculogram (EOG) and electromyogram (EMG) are converted into two-dimensional time-frequency images containing time-frequency features by using short time Fourier transform (STFT). Then, the time-frequency feature extraction network combining multi-scale EEG compact convolution network (Ms-EEGNet) and bidirectional gated recurrent units (Bi-GRU) network is used to obtain multi-scale spectral features related to sleep feature waveforms and time series features related to sleep stage transition. According to the American Academy of Sleep Medicine (AASM) EEG sleep stage classification criterion, the model achieved 84.3% accuracy in the five-classification task on the third subgroup of the Institute of Systems and Robotics of the University of Coimbra Sleep Dataset (ISRUC-S3), with 83.1% macro F1 score value and 79.8% Cohen's Kappa coefficient. The experimental results show that the proposed model achieves higher classification accuracy and promotes the application of deep learning algorithms in assisting clinical decision-making.

Enhanced conductive body heat loss during sleep increases slow-wave sleep and calms the heart.

Substantial evidence suggests that the circadian decline of core body temperature (CBT) triggers the initiation of human sleep, with CBT continuing to decrease during sleep. Although the connection between habitual sleep and CBT patterns is established, the impact of external body cooling on sleep remains poorly understood. The main aim of the present study is to show whether a decline in body temperatures during sleep can be related to an increase in slow wave sleep (N3). This three-center study on 72 individuals of varying age, sex, and BMI used an identical type of a high-heat capacity mattress as a reproducible, non-disturbing way of body cooling, accompanied by measurements of CBT and proximal back skin temperatures, heart rate and sleep (polysomnography). The main findings were an increase in nocturnal sleep stage N3 (7.5 ± 21.6 min/7.5 h, mean ± SD; p = 0.0038) and a decrease in heart rate (- 2.36 ± 1.08 bpm, mean ± SD; p < 0.0001); sleep stage REM did not change (p = 0.3564). Subjects with a greater degree of body cooling exhibited a significant increase in nocturnal N3 and a decrease in REM sleep, mainly in the second part of the night. In addition, these subjects showed a phase advance in the NREM-REM sleep cycle distribution of N3 and REM. Both effects were significantly associated with increased conductive inner heat transfer, indicated by an increased CBT- proximal back skin temperature -gradient, rather than with changes in CBT itself. Our findings reveal a previously far disregarded mechanism in sleep research that has potential therapeutic implications: Conductive body cooling during sleep is a reliable method for promoting N3 and reducing heart rate.

Sleep Spindle Alterations in Children With Migraine.

BACKGROUND: The modulation of thalamocortical activity is the most important site of several levels of interference between sleep spindles and migraine. Thalamocortical circuits are responsible for the electrophysiological phenomenon of sleep spindles. Spindle alterations may be used as a beneficial marker in the diagnosis and follow-up of children with migraine. We aimed to formulate the hypothesis that there is a shared mechanism that underlies migraine and sleep spindle activity. METHODS: We analyzed the amplitude, frequency, duration, density, and activity of sleep spindles in non-rapid eye movement stage 2 sleep in patients with migraine without aura when compared with healthy control subjects. RESULTS: The amplitudes of average, slow, and fast sleep spindles were higher in children with migraine without aura (P = 0.020, 0.013, and 0.033, respectively). The frequency of fast spindles was lower in children with migraines without aura when compared with the control group (P = 0.03). Although not statistically significant, the fast sleep spindle duration in the migraine group was shorter (P = 0.055). Multivariate analysis revealed an increased risk of migraine associated with increased mean spindle amplitude and decreased fast spindle frequency and duration. CONCLUSIONS: Our data suggest that spindle alterations may correlate with the vulnerability to develop migraine and may be used as a model for future research about the association between the thalamocortical networks and migraine.

CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities.

Sleep abnormalities can have severe health consequences. Automated sleep staging, i.e. labelling the sequence of sleep stages from the patient's physiological recordings, could simplify the diagnostic process. Previous work on automated sleep staging has achieved great results, mainly relying on the EEG signal. However, often multiple sources of information are available beyond EEG. This can be particularly beneficial when the EEG recordings are noisy or even missing completely. In this paper, we propose CoRe-Sleep, a Coordinated Representation multimodal fusion network that is particularly focused on improving the robustness of signal analysis on imperfect data. We demonstrate how appropriately handling multimodal information can be the key to achieving such robustness. CoRe-Sleep tolerates noisy or missing modalities segments, allowing training on incomplete data. Additionally, it shows state-of-the-art performance when testing on both multimodal and unimodal data using a single model on SHHS-1, the largest publicly available study that includes sleep stage labels. The results indicate that training the model on multimodal data does positively influence performance when tested on unimodal data. This work aims at bridging the gap between automated analysis tools and their clinical utility.

MixSleepNet: A Multi-Type Convolution Combined Sleep Stage Classification Model.

BACKGROUND AND OBJECTIVE: Sleep staging is an essential step for sleep disorder diagnosis, which is time-intensive and laborious for experts to perform this work manually. Automatic sleep stage classification methods not only alleviate experts from these demanding tasks but also enhance the accuracy and efficiency of the classification process. METHODS: A novel multi-channel biosignal-based model constructed by the combination of a 3D convolutional operation and a graph convolutional operation is proposed for the automated sleep stages using various physiological signals. Both the 3D convolution and graph convolution can aggregate information from neighboring brain areas, which helps to learn intrinsic connections from the biosignals. Electroencephalogram (EEG), electromyogram (EMG), electrooculogram (EOG) and electrocardiogram (ECG) signals are employed to extract time domain and frequency domain features. Subsequently, these signals are input to the 3D convolutional and graph convolutional branches, respectively. The 3D convolution branch can explore the correlations between multi-channel signals and multi-band waves in each channel in the time series, while the graph convolution branch can explore the connections between each channel and each frequency band. In this work, we have developed the proposed multi-channel convolution combined sleep stage classification model (MixSleepNet) using ISRUC datasets (Subgroup 3 and 50 random samples from Subgroup 1). RESULTS: Based on the first expert's label, our generated MixSleepNet yielded an accuracy, F1-score and Cohen kappa scores of 0.830, 0.821 and 0.782, respectively for ISRUC-S3. It obtained accuracy, F1-score and Cohen kappa scores of 0.812, 0.786, and 0.756, respectively for the ISRUC-S1 dataset. In accordance with the evaluations conducted by the second expert, the comprehensive accuracies, F1-scores, and Cohen kappa coefficients for the ISRUC-S3 and ISRUC-S1 datasets are determined to be 0.837, 0.820, 0.789, and 0.829, 0.791, 0.775, respectively. CONCLUSION: The results of the performance metrics by the proposed method are much better than those from all the compared models. Additional experiments were carried out on the ISRUC-S3 sub-dataset to evaluate the contributions of each module towards the classification performance.