MEET: A Multi-Band EEG Transformer for Brain States Decoding.

OBJECTIVE: Electroencephalography (EEG) is among the most widely used and inexpensive neuroimaging techniques. Compared to the CNN or RNN based models, Transformer can better capture the temporal information in EEG signals and focus more on global features of the brain's functional activities. Importantly, according to the multiscale nature of EEG signals, it is crucial to consider the multi-band concept into the design of EEG Transformer architecture. METHODS: We propose a novel Multi-band EEG Transformer (MEET) to represent and analyze the multiscale temporal time series of human brain EEG signals. MEET mainly includes three parts: 1) transform the EEG signals into multi-band images, and preserve the 3D spatial information between electrodes; 2) design a Band Attention Block to compute the attention maps of the stacked multi-band images and infer the fused feature maps; 3) apply the Temporal Self-Attention and Spatial Self-Attention modules to extract the spatiotemporal features for the characterization and differentiation of multi-frame dynamic brain states. RESULTS: The experimental results show that: 1) MEET outperforms state-of-the-art methods on multiple open EEG datasets (SEED, SEED-IV, WM) for brain states classification; 2) MEET demonstrates that 5-bands fusion is the best integration strategy; and 3) MEET identifies interpretable brain attention regions. SIGNIFICANCE: MEET is an interpretable and universal model based on the multiband-multiscale characteristics of EEG. CONCLUSION: The innovative combination of band attention and temporal/spatial self-attention mechanisms in MEET achieves promising data-driven learning of the temporal dependencies and spatial relationships of EEG signals across the entire brain in a holistic and comprehensive fashion.

Differentiating brain states via multi-clip random fragment strategy-based interactive bidirectional recurrent neural network.

EEG is widely adopted to study the brain and brain computer interface (BCI) for its non-invasiveness and low costs. Specifically EEG can be applied to differentiate brain states, which is important for better understanding the working mechanisms of the brain. Recurrent neural network (RNN)-based learning strategy has been widely utilized to differentiate brain states, because its optimization architectures improve the classification performance for differentiating brain states at the group level. However, present classification performance is still far from satisfactory. We have identified two major focal points for improvements: one is about organizing the input EEG signals, and the other is related to the design of the RNN architecture. To optimize the above-mentioned issues and achieve better brain state classification performance, we propose a novel multi-clip random fragment strategy-based interactive bidirectional recurrent neural network (McRFS-IBiRNN) model in this work. This model has two advantages over previous methods. First, the McRFS component is designed to re-organize the input EEG signals to make them more suitable for the RNN architecture. Second, the IBiRNN component is an innovative design to model the RNN layers with interaction connections to enhance the fusion of bidirectional features. By adopting the proposed model, promising brain states classification performances are obtained. For example, 96.97% and 99.34% of individual and group level four-category classification accuracies are successfully obtained on the EEG motor/imagery dataset, respectively. A 99.01% accuracy can be observed for four-category classification tasks with new subjects not seen before, which demonstrates the generalization of our proposed method. Compared with existing methods, our model outperforms them with superior results. Overall, the proposed McRFS-IBiRNN model demonstrates great superiority in differentiating brain states on EEG signals.

MM-GLCM-CNN: A multi-scale and multi-level based GLCM-CNN for polyp classification.

Distinguishing malignant from benign lesions has significant clinical impacts on both early detection and optimal management of those early detections. Convolutional neural network (CNN) has shown great potential in medical imaging applications due to its powerful feature learning capability. However, it is very challenging to obtain pathological ground truth, addition to collected in vivo medical images, to construct objective training labels for feature learning, leading to the difficulty of performing lesion diagnosis. This is contrary to the requirement that CNN algorithms need a large number of datasets for the training. To explore the ability to learn features from small pathologically-proven datasets for differentiation of malignant from benign polyps, we propose a Multi-scale and Multi-level based Gray-level Co-occurrence Matrix CNN (MM-GLCM-CNN). Specifically, instead of inputting the lesions' medical images, the GLCM, which characterizes the lesion heterogeneity in terms of image texture characteristics, is fed into the MM-GLCN-CNN model for the training. This aims to improve feature extraction by introducing multi-scale and multi-level analysis into the construction of lesion texture characteristic descriptors (LTCDs). To learn and fuse multiple sets of LTCDs from small datasets for lesion diagnosis, we further propose an adaptive multi-input CNN learning framework. Furthermore, an Adaptive Weight Network is used to highlight important information and suppress redundant information after the fusion of the LTCDs. We evaluated the performance of MM-GLCM-CNN by the area under the receiver operating characteristic curve (AUC) merit on small private lesion datasets of colon polyps. The AUC score reaches 93.99% with a gain of 1.49% over current state-of-the-art lesion classification methods on the same dataset. This gain indicates the importance of incorporating lesion characteristic heterogeneity for the prediction of lesion malignancy using small pathologically-proven datasets.

An explainable deep learning framework for characterizing and interpreting human brain states.

Deep learning approaches have been widely adopted in the medical image analysis field. However, a most of existing deep learning approaches focus on achieving promising performances such as classification, detection, and segmentation, and much less effort is devoted to the explanation of the designed models. Similarly, in the brain imaging field, many deep learning approaches have been designed and applied to characterize and predict human brain states. However, these models lack interpretation. In response, we propose a novel domain knowledge informed self-attention graph pooling-based (SAGPool) graph convolutional neural network to study human brain states. Specifically, the dense individualized and common connectivity-based cortical landmarks system (DICCCOL, structural brain connectivity profiles) and holistic atlases of functional networks and interactions system (HAFNI, functional brain connectivity profiles) are integrated with the SAGPool model to better characterize and interpret the brain states. Extensive experiments are designed and carried out on the large-scale human connectome project (HCP) Q1 and S1200 dataset. Promising brain state classification performances are observed (e.g., an average of 93.7% for seven-task classification and 100% for binary classification). In addition, the importance of the brain regions, which contributes most to the accurate classification, is successfully quantified and visualized. A thorough neuroscientific interpretation suggests that these extracted brain regions and their importance calculated from self-attention graph pooling layer offer substantial explainability.

An Explainable and Generalizable Recurrent Neural Network Approach for Differentiating Human Brain States on EEG Dataset.

Electroencephalogram (EEG) is one of the most widely used brain computer interface (BCI) approaches. Despite the success of existing EEG approaches in brain state recognition studies, it is still challenging to differentiate brain states via explainable and generalizable deep learning approaches. In other words, how to explore meaningful and distinguishing features and how to overcome the huge variability and overfitting problem still need to be further studied. To alleviate these challenges, in this work, a multiple random fragment search-based multilayer recurrent neural network (MRFS-MRNN) is proposed to improve the differentiating performance and explore meaningful patterns. Specifically, an explainable MRNN module is proposed to capture the temporal dependences preserved in EEG time series. Besides, a MRFS module is designed to cut multiple random fragments from the entire EEG signal time course to improve the effectiveness of brain state differentiating ability. MRFS-MRNN is concatenatedto effectively overcome the huge variabilities and overfitting problems. Experiment results demonstrate that the proposed MRFS-MRNN model not only has excellent differentiating performance, but also has good explanation and generalization ability. The classification accuracies reach as high as 95.18% for binary classification and 89.19% for four-category classification on the individual level. Similarly, 95.53% and 85.84% classification accuracies are obtained for the binary and four-category classification on the group level. What's more, 94.28% and 85.43% classification accuracies of binary and four-category classifications are achieved for predicting brand new subjects. The experiment results showed that the proposed method outperformed other state-of-the-art (SOTA) models on the same underlying data and improved the explanation and generalization ability.

A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging.

Naturalistic stimuli, including movie, music, and speech, have been increasingly applied in the research of neuroimaging. Relative to a resting-state or single-task state, naturalistic stimuli can evoke more intense brain activities and have been proved to possess higher test-retest reliability, suggesting greater potential to study adaptive human brain function. In the current research, naturalistic functional magnetic resonance imaging (N-fMRI) has been a powerful tool to record brain states under naturalistic stimuli, and many efforts have been devoted to study the high-level semantic features from spatial or temporal representations via N-fMRI. However, integrating both spatial and temporal characteristics of brain activities for better interpreting the patterns under naturalistic stimuli is still underexplored. In this work, a novel hybrid learning framework that comprehensively investigates both the spatial (via Predictive Model) and the temporal [via convolutional neural network (CNN) model] characteristics of the brain is proposed. Specifically, to focus on certain relevant regions from the whole brain, regions of significance (ROS), which contain common spatial activation characteristics across individuals, are selected via the Predictive Model. Further, voxels of significance (VOS), whose signals contain significant temporal characteristics under naturalistic stimuli, are interpreted via one-dimensional CNN (1D-CNN) model. In this article, our proposed framework is applied onto the N-fMRI data during naturalistic classical/pop/speech audios stimuli. The promising performance is achieved via the Predictive Model to differentiate the different audio categories. Especially for distinguishing the classic and speech audios, the accuracy of classification is up to 92%. Moreover, spatial ROS and VOS are effectively obtained. Besides, temporal characteristics of the high-level semantic features are investigated on the frequency domain via convolution kernels of 1D-CNN model, and we effectively bridge the "semantic gap" between high-level semantic features of N-fMRI and low-level acoustic features of naturalistic audios in the frequency domain. Our results provide novel insights on characterizing spatiotemporal patterns of brain activities via N-fMRI and effectively explore the high-level semantic features under naturalistic stimuli, which will further benefit the understanding of the brain working mechanism and the advance of naturalistic stimuli clinical application.

Joint Analysis of Functional and Structural Connectomes Between Preterm and Term Infant Brains via Canonical Correlation Analysis With Locality Preserving Projection.

Preterm is a worldwide problem that affects infants' lives significantly. Moreover, the early impairment is more than limited to isolated brain regions but also to global and profound negative outcomes later, such as cognitive disorder. Therefore, seeking the differences of brain connectome between preterm and term infant brains is a vital step for understanding the developmental impairment caused by preterm. Existing studies revealed that studying the relationship between brain function and structure, and further investigating their differentiable connectomes between preterm and term infant brains is a way to comprehend and unveil the differences that occur in the preterm infant brains. Therefore, in this article, we proposed a novel canonical correlation analysis (CCA) with locality preserving projection (LPP) approach to investigate the relationship between brain functional and structural connectomes and how such a relationship differs between preterm and term infant brains. CCA is proposed to study the relationship between functional and structural connections, while LPP is adopted to identify the distinguishing features from the connections which can differentiate the preterm and term brains. After investigating the whole brain connections on a fine-scale connectome approach, we successfully identified 89 functional and 97 structural connections, which mostly contributed to differentiate preterm and term infant brains from the functional MRI (fMRI) and diffusion MRI (dMRI) of the public developing Human Connectome Project (dHCP) dataset. By further exploring those identified connections, the results innovatively revealed that the identified functional connections are short-range and within the functional network. On the contrary, the identified structural connections are usually remote connections across different functional networks. In addition, these connectome-level results show the new insights that longitudinal functional changes could deviate from longitudinal structural changes in the preterm infant brains, which help us better understand the brain-behavior changes in preterm infant brains.