RESUMO
Assessing communication abilities in patients with disorders of consciousness (DOCs) is challenging due to limitations in the behavioral scale. Electroencephalogram-based brain-computer interfaces (BCIs) and eye-tracking for detecting ocular changes can capture mental activities without requiring physical behaviors and thus may be a solution. This study proposes a hybrid BCI that integrates EEG and eye tracking to facilitate communication in patients with DOC. Specifically, the BCI presented a question and two randomly flashing answers (yes/no). The subjects were instructed to focus on an answer. A multimodal target recognition network (MTRN) is proposed to detect P300 potentials and eye-tracking responses (i.e., pupil constriction and gaze) and identify the target in real time. In the MTRN, the dual-stream feature extraction module with two independent multiscale convolutional neural networks extracts multiscale features from multimodal data. Then, the multimodal attention strategy adaptively extracts the most relevant information about the target from multimodal data. Finally, a prototype network is designed as a classifier to facilitate small-sample data classification. Ten healthy individuals, nine DOC patients and one LIS patient were included in this study. All healthy subjects achieved 100% accuracy. Five patients could communicate with our BCI, with 76.1±7.9% accuracy. Among them, two patients who were noncommunicative on the behavioral scale exhibited communication ability via our BCI. Additionally, we assessed the performance of unimodal BCIs and compared MTRNs with other methods. All the results suggested that our BCI can yield more sensitive outcomes than the CRS-R and can serve as a valuable communication tool.