Toward calibration-free motor imagery brain-computer interfaces: a VGG-based convolutional neural network and WGAN approach.

Objective.Motor imagery (MI) represents one major paradigm of Brain-computer interfaces (BCIs) in which users rely on their electroencephalogram (EEG) signals to control the movement of objects. However, due to the inter-subject variability, MI BCIs require recording subject-dependent data to train machine learning classifiers that are used to identify the intended motor action. This represents a challenge in developing MI BCIs as it complicates its calibration and hinders the wide adoption of such a technology.Approach.This study focuses on enhancing cross-subject (CS) MI EEG classification using EEG spectrum images. The proposed calibration-free approach employs deep learning techniques for MI classification and Wasserstein Generative Adversarial Networks (WGAN) for data augmentation. The proposed WGAN generates synthetic spectrum images from the recorded MI-EEG to expand the training dataset; aiming to enhance the classifier's performance. The proposed approach eliminates the need for any calibration data from the target subject, making it more suitable for real-world applications.Main results.To assess the robustness and efficacy of the proposed framework, we utilized the BCI competition IV-2B, IV-2 A, and IV-1 benchmark datasets, employing leave one-subject out validation. Our results demonstrate that using the proposed modified VGG-CNN classifier in addition to WGAN-generated data for augmentation leads to an enhancement in CS accuracy outperforming state-of-the-art methods.Significance.This approach could represent one step forward towards developing calibration-free BCI systems and hence broaden their applications.

On the role of generative artificial intelligence in the development of brain-computer interfaces.

Since their inception more than 50 years ago, Brain-Computer Interfaces (BCIs) have held promise to compensate for functions lost by people with disabilities through allowing direct communication between the brain and external devices. While research throughout the past decades has demonstrated the feasibility of BCI to act as a successful assistive technology, the widespread use of BCI outside the lab is still beyond reach. This can be attributed to a number of challenges that need to be addressed for BCI to be of practical use including limited data availability, limited temporal and spatial resolutions of brain signals recorded non-invasively and inter-subject variability. In addition, for a very long time, BCI development has been mainly confined to specific simple brain patterns, while developing other BCI applications relying on complex brain patterns has been proven infeasible. Generative Artificial Intelligence (GAI) has recently emerged as an artificial intelligence domain in which trained models can be used to generate new data with properties resembling that of available data. Given the enhancements observed in other domains that possess similar challenges to BCI development, GAI has been recently employed in a multitude of BCI development applications to generate synthetic brain activity; thereby, augmenting the recorded brain activity. Here, a brief review of the recent adoption of GAI techniques to overcome the aforementioned BCI challenges is provided demonstrating the enhancements achieved using GAI techniques in augmenting limited EEG data, enhancing the spatiotemporal resolution of recorded EEG data, enhancing cross-subject performance of BCI systems and implementing end-to-end BCI applications. GAI could represent the means by which BCI would be transformed into a prevalent assistive technology, thereby improving the quality of life of people with disabilities, and helping in adopting BCI as an emerging human-computer interaction technology for general use.

Generative adversarial networks in EEG analysis: an overview.

Electroencephalogram (EEG) signals have been utilized in a variety of medical as well as engineering applications. However, one of the challenges associated with recording EEG data is the difficulty of recording large amounts of data. Consequently, data augmentation is a potential solution to overcome this challenge in which the objective is to increase the amount of data. Inspired by the success of Generative Adversarial Networks (GANs) in image processing applications, generating artificial EEG data from the limited recorded data using GANs has seen recent success. This article provides an overview of various techniques and approaches of GANs for augmenting EEG signals. We focus on the utility of GANs in different applications including Brain-Computer Interface (BCI) paradigms such as motor imagery and P300-based systems, in addition to emotion recognition, epileptic seizures detection and prediction, and various other applications. We address in this article how GANs have been used in each study, the impact of using GANs on the model performance, the limitations of each algorithm, and future possibilities for developing new algorithms. We emphasize the utility of GANs in augmenting the limited EEG data typically available in the studied applications.

Object recognition and localization enhancement in visual prostheses: a real-time mixed reality simulation.

Blindness is a main threat that affects the daily life activities of any human. Visual prostheses have been introduced to provide artificial vision to the blind with the aim of allowing them to restore confidence and independence. In this article, we propose an approach that involves four image enhancement techniques to facilitate object recognition and localization for visual prostheses users. These techniques are clip art representation of the objects, edge sharpening, corner enhancement and electrode dropout handling. The proposed techniques are tested in a real-time mixed reality simulation environment that mimics vision perceived by visual prostheses users. Twelve experiments were conducted to measure the performance of the participants in object recognition and localization. The experiments involved single objects, multiple objects and navigation. To evaluate the performance of the participants in objects recognition, we measure their recognition time, recognition accuracy and confidence level. For object localization, two metrics were used to measure the performance of the participants which are the grasping attempt time and the grasping accuracy. The results demonstrate that using all enhancement techniques simultaneously gives higher accuracy, higher confidence level and less time for recognizing and grasping objects in comparison to not applying the enhancement techniques or applying pair-wise combinations of them. Visual prostheses could benefit from the proposed approach to provide users with an enhanced perception.

A multi-Kalman filter-based approach for decoding arm kinematics from EMG recordings.

BACKGROUND: Remarkable work has been recently introduced to enhance the usage of Electromyography (EMG) signals in operating prosthetic arms. Despite the rapid advancements in this field, providing a reliable, naturalistic myoelectric prosthesis remains a significant challenge. Other challenges include the limited number of allowed movements, lack of simultaneous, continuous control and the high computational power that could be needed for accurate decoding. In this study, we propose an EMG-based multi-Kalman filter approach to decode arm kinematics; specifically, the elbow angle (θ), wrist joint horizontal (X) and vertical (Y) positions in a continuous and simultaneous manner. RESULTS: Ten subjects were examined from which we recorded arm kinematics and EMG signals of the biceps, triceps, lateral and anterior deltoid muscles corresponding to a randomized set of movements. The performance of the proposed decoder is assessed using the correlation coefficient (CC) and the normalized root-mean-square error (NRMSE) computed between the actual and the decoded kinematic. Results demonstrate that when training and testing the decoder using same-subject data, an average CC of 0.68 ± 0.1, 0.67 ± 0.12 and 0.64 ± 0.11, and average NRMSE of 0.21 ± 0.06, 0.18 ± 0.03 and 0.24 ± 0.07 were achieved for θ, X, and Y, respectively. When training the decoder using the data of one subject and decoding the data of other subjects, an average CC of 0.61 ± 0.19, 0.61 ± 0.16 and 0.48 ± 0.17, and an average NRMSE of 0.23 ± 0.07, 0.2 ± 0.05 and 0.38 ± 0.15 were achieved for θ, X, and Y, respectively. CONCLUSIONS: These results suggest the efficacy of the proposed approach and indicates the possibility of obtaining a subject-independent decoder.

PVGAN: a generative adversarial network for object simplification in prosthetic vision.

Objective.By means of electrical stimulation of the visual system, visual prostheses provide promising solution for blind patients through partial restoration of their vision. Despite the great success achieved so far in this field, the limited resolution of the perceived vision using these devices hinders the ability of visual prostheses users to correctly recognize viewed objects. Accordingly, we propose a deep learning approach based on generative adversarial networks (GANs), termed prosthetic vision GAN (PVGAN), to enhance object recognition for the implanted patients by representing objects in the field of view based on a corresponding simplified clip art version.Approach.To assess the performance, an axon map model was used to simulate prosthetic vision in experiments involving normally-sighted participants. In these experiments, four types of image representation were examined. The first and second types comprised presenting phosphene simulation of real images containing the actual high-resolution object, and presenting phosphene simulation of the real image followed by the clip art image, respectively. The other two types were utilized to evaluate the performance in the case of electrode dropout, where the third type comprised presenting phosphene simulation of only clip art images without electrode dropout, while the fourth type involved clip art images with electrode dropout.Main results.The performance was measured through three evaluation metrics which are the accuracy of the participants in recognizing the objects, the time taken by the participants to correctly recognize the object, and the confidence level of the participants in the recognition process. Results demonstrate that representing the objects using clip art images generated by the PVGAN model results in a significant enhancement in the speed and confidence of the subjects in recognizing the objects.Significance.These results demonstrate the utility of using GANs in enhancing the quality of images perceived using prosthetic vision.

Electrode Dropout Compensation in Visual Prostheses: An Optimal Object Placement Approach.

Visual prostheses provide promising solution to the blind through partial restoration of their vision via electrical stimulation of the visual system. However, there are some challenges that hinder the ability of subjects implanted with visual prostheses to correctly identify an object. One of these challenges is electrode dropout; the malfunction of some electrodes resulting in consistently dark phosphenes. In this paper, we propose a dropout handling algorithm for better and faster identification of objects. In this algorithm, phosphenes representing the object are translated to another location within the same image that has the minimum number of dropouts. Using simulated prosthetic vision, experiments were conducted to test the efficacy of our proposed algorithm. Electrode dropout rates of 10%, 20% and 30% were examined. Our results demonstrate significant increase in the object recognition accuracy, reduction in the recognition time and increase in the recognition confidence level using the proposed approach compared to presenting the images without dropout handling.Clinical Relevance- These results demonstrate the utility of dropout handling in enhancing the perception of images in prosthetic vision.

A deep convolutional visual encoding model of neuronal responses in the LGN.

The Lateral Geniculate Nucleus (LGN) represents one of the major processing sites along the visual pathway. Despite its crucial role in processing visual information and its utility as one target for recently developed visual prostheses, it is much less studied compared to the retina and the visual cortex. In this paper, we introduce a deep learning encoder to predict LGN neuronal firing in response to different visual stimulation patterns. The encoder comprises a deep Convolutional Neural Network (CNN) that incorporates visual stimulus spatiotemporal representation in addition to LGN neuronal firing history to predict the response of LGN neurons. Extracellular activity was recorded in vivo using multi-electrode arrays from single units in the LGN in 12 anesthetized rats with a total neuronal population of 150 units. Neural activity was recorded in response to single-pixel, checkerboard and geometrical shapes visual stimulation patterns. Extracted firing rates and the corresponding stimulation patterns were used to train the model. The performance of the model was assessed using different testing data sets and different firing rate windows. An overall mean correlation coefficient between the actual and the predicted firing rates of 0.57 and 0.7 was achieved for the 10 ms and the 50 ms firing rate windows, respectively. Results demonstrate that the model is robust to variability in the spatiotemporal properties of the recorded neurons outperforming other examined models including the state-of-the-art Generalized Linear Model (GLM). The results indicate the potential of deep convolutional neural networks as viable models of LGN firing.

In-silico development and assessment of a Kalman filter motor decoder for prosthetic hand control.

Up to 50% of amputees abandon their prostheses, partly due to rapid degradation of the control systems, which require frequent recalibration. The goal of this study was to develop a Kalman filter-based approach to decoding motoneuron activity to identify movement kinematics and thereby provide stable, long-term, accurate, real-time decoding. The Kalman filter-based decoder was examined via biologically varied datasets generated from a high-fidelity computational model of the spinal motoneuron pool. The estimated movement kinematics controlled a simulated MuJoCo prosthetic hand. This clear-box approach showed successful estimation of hand movements under eight varied physiological conditions with no retraining. The mean correlation coefficient of 0.98 and mean normalized root mean square error of 0.06 over these eight datasets provide proof of concept that this decoder would improve long-term integrity of performance while performing new, untrained movements. Additionally, the decoder operated in real-time (~0.3 ms). Further results include robust performance of the Kalman filter when re-trained to more severe post-amputation limitations in the type and number of motoneurons remaining. An additional analysis shows that the decoder achieves better accuracy when using the firing of individual motoneurons as input, compared to using aggregate pool firing. Moreover, the decoder demonstrated robustness to noise affecting both the trained decoder parameters and the decoded motoneuron activity. These results demonstrate the utility of a proof of concept Kalman filter decoder that can support prosthetics' control systems to maintain accurate and stable real-time movement performance.

A Classification Approach to Recognize the Firing of Spinal Motoneurons in Amyotrophic Lateral Sclerosis.

Amyotrophic Lateral Sclerosis (ALS) is a fatal neurodegenerative disease that affects the nervous system causing muscle weakness, paralysis, leading to death. Given that abnormalities in spinal motoneuron (MN) excitability begin long before symptoms manifest, developing an approach that could recognize fluctuations in MN firing could help in early diagnosis of ALS. This paper introduces a machine learning approach to discriminate between ALS and normal MN firing. The approach is based on two electrophysiological markers; namely, spiking latency and the spike-triggered average signal. The method is examined using data generated from a computational model under systematic variation of MN properties. Such variations mimic the differential dynamic changes in cellular properties that different MN types experience during ALS progression. Our results demonstrate the ability of the approach to accurately recognize ALS firing patterns across the spectrum of examined variations in MN properties.Clinical Relevance- These results represent a proof of concept for using the proposed machine-learning approach in early diagnosis of ALS.

Modulating Lateral Geniculate Nucleus Neuronal Firing for Visual Prostheses: A Kalman Filter-Based Strategy.

Developing visual prostheses that target inner brain structures along the visual pathway represent a new hope for patients with completely damaged early visual pathway sites. One of the major challenges in the development of subcortical and cortical visual prostheses is tuning electrical stimulation that could optimally induce desired visual percepts. In this paper, we propose a Kalman filter-based strategy that could be used to identify electrical stimulation patterns that mimic a specific visual input for thalamic visual prostheses. We demonstrate the performance of the proposed strategy using a population of lateral geniculate nucleus neurons modeled using an adapted generalized non-linear model. A mean correlation of 0.69 is obtained between visually evoked and electrically evoked responses-driven using the proposed strategy-for an optimal electrode-placement setup. In addition, we demonstrate the performance for a random electrode-placement setup in which a mean correlation of 0.26 is obtained. For this latter setup, our analysis reveals an inversely proportional relationship between the obtained correlation and the distance between each neuron and the nearest electrode. The proposed strategy could be thus utilized to tune and enhance the performance of thalamic visual prostheses as well as other prosthesis systems.

MindEdit: A P300-based text editor for mobile devices.

Practical application of Brain-Computer Interfaces (BCIs) requires that the whole BCI system be portable. The mobility of BCI systems involves two aspects: making the electroencephalography (EEG) recording devices portable, and developing software applications with low computational complexity to be able to run on low computational-power devices such as tablets and smartphones. This paper addresses the development of MindEdit; a P300-based text editor for Android-based devices. Given the limited resources of mobile devices and their limited computational power, a novel ensemble classifier is utilized that uses Principal Component Analysis (PCA) features to identify P300 evoked potentials from EEG recordings. PCA computations in the proposed method are channel-based as opposed to concatenating all channels as in traditional feature extraction methods; thus, this method has less computational complexity compared to traditional P300 detection methods. The performance of the method is demonstrated on data recorded from MindEdit on an Android tablet using the Emotiv wireless neuroheadset. Results demonstrate the capability of the introduced PCA ensemble classifier to classify P300 data with maximum average accuracy of 78.37±16.09% for cross-validation data and 77.5±19.69% for online test data using only 10 trials per symbol and a 33-character training dataset. Our analysis indicates that the introduced method outperforms traditional feature extraction methods. For a faster operation of MindEdit, a variable number of trials scheme is introduced that resulted in an online average accuracy of 64.17±19.6% and a maximum bitrate of 6.25bit/min. These results demonstrate the efficacy of using the developed BCI application with mobile devices.

Moving object detection and background enhancement for thalamic visual prostheses.

Visual prostheses open the door of hope to restore functional vision for the blind. One of the main challenges facing their development is the limited number of electrodes used in the stimulation process which limits the resolution of the perceived images. To improve the perception, the useful features in the scene need to be enhanced while the other features should be suppressed to achieve better resolution. This paper introduces an image processing method to enhance three main features detectable by the natural visual pathway; namely the contrast, the motion and the edges. It then reduces the size of the image into an activity matrix used to generate the electric stimulation for the electrodes array. We compared the proposed method to four other image processing strategies in terms of the quality of the resulting image in addition to the perceived image using a simulation of prosthetic vision. Results demonstrate that the proposed method outperforms the other techniques in both aspects.

Tuning electrical stimulation for thalamic visual prosthesis: An autoencoder-based approach.

Visual prosthesis holds hope of vision restoration for millions with retinal degenerative diseases. Machine learning techniques such as artificial neural networks could help in improving prosthetic devices as they could learn how the brain encodes information and imitate that code. This paper introduces an autoencoder-based approach for tuning thalamic visual prostheses. The objective of the proposed approach is to estimate electrical stimuli that are equivalent to a given natural visual stimulus, in a way such that they both elicit responses that are as similar as possible when introduced to a Lateral Geniculate Nucleus (LGN) population. Applying the proposed method to a probabilistic model of LGN neurons, results demonstrate a significant similarity between both responses with a mean correlation of 0.672 for optimal electrodes placement and 0.354 for random electrodes placement. The results indicate the efficacy of the proposed approach in estimating an electrical stimulus equivalent to a specific visual stimulus.

Temporal precision in population-but not individual neuron-dynamics reveals rapid experience-dependent plasticity in the rat barrel cortex.

Cortical reorganization following sensory deprivation is characterized by alterations in the connectivity between neurons encoding spared and deprived cortical inputs. The extent to which this alteration depends on Spike Timing Dependent Plasticity (STDP), however, is largely unknown. We quantified changes in the functional connectivity between layer V neurons in the vibrissal primary somatosensory cortex (vSI) (barrel cortex) of rats following sensory deprivation. One week after chronic implantation of a microelectrode array in vSI, sensory-evoked activity resulting from mechanical deflections of individual whiskers was recorded (control data) after which two whiskers on the contralateral side were paired by sparing them while trimming all other whiskers on the rat's mystacial pad. The rats' environment was then enriched by placing novel objects in the cages to encourage exploratory behavior with the spared whiskers. Sensory-evoked activity in response to individual stimulation of spared whiskers and adjacent re-grown whiskers was then recorded under anesthesia 1-2 days and 6-7 days post-trimming (plasticity data). We analyzed spike trains within 100 ms of stimulus onset and confirmed previously published reports documenting changes in receptive field sizes in the spared whisker barrels. We analyzed the same data using Dynamic Bayesian Networks (DBNs) to infer the functional connectivity between the recorded neurons. We found that DBNs inferred from population responses to stimulation of each of the spared whiskers exhibited graded increase in similarity that was proportional to the pairing duration. A significant early increase in network similarity in the spared-whisker barrels was detected 1-2 days post pairing, but not when single neuron responses were examined during the same period. These results suggest that rapid reorganization of cortical neurons following sensory deprivation may be mediated by an STDP mechanism.

Performance analysis of a Principal Component Analysis ensemble classifier for Emotiv headset P300 spellers.

The current trend to use Brain-Computer Interfaces (BCIs) with mobile devices mandates the development of efficient EEG data processing methods. In this paper, we demonstrate the performance of a Principal Component Analysis (PCA) ensemble classifier for P300-based spellers. We recorded EEG data from multiple subjects using the Emotiv neuroheadset in the context of a classical oddball P300 speller paradigm. We compare the performance of the proposed ensemble classifier to the performance of traditional feature extraction and classifier methods. Our results demonstrate the capability of the PCA ensemble classifier to classify P300 data recorded using the Emotiv neuroheadset with an average accuracy of 86.29% on cross-validation data. In addition, offline testing of the recorded data reveals an average classification accuracy of 73.3% that is significantly higher than that achieved using traditional methods. Finally, we demonstrate the effect of the parameters of the P300 speller paradigm on the performance of the method.

NeuroQuest: a comprehensive analysis tool for extracellular neural ensemble recordings.

Analyzing the massive amounts of neural data collected using microelectrodes to extract biologically relevant information is a major challenge. Many scientific findings rest on the ability to overcome these challenges and to standardize experimental analysis across labs. This can be facilitated in part through comprehensive, efficient and practical software tools disseminated to the community at large. We have developed a comprehensive, MATLAB-based software package - entitled NeuroQuest - that bundles together a number of advanced neural signal processing algorithms in a user-friendly environment. Results demonstrate the efficiency and reliability of the software compared to other software packages, and versatility over a wide range of experimental conditions.

Millisecond-timescale local network coding in the rat primary somatosensory cortex.

Correlation among neocortical neurons is thought to play an indispensable role in mediating sensory processing of external stimuli. The role of temporal precision in this correlation has been hypothesized to enhance information flow along sensory pathways. Its role in mediating the integration of information at the output of these pathways, however, remains poorly understood. Here, we examined spike timing correlation between simultaneously recorded layer V neurons within and across columns of the primary somatosensory cortex of anesthetized rats during unilateral whisker stimulation. We used bayesian statistics and information theory to quantify the causal influence between the recorded cells with millisecond precision. For each stimulated whisker, we inferred stable, whisker-specific, dynamic bayesian networks over many repeated trials, with network similarity of 83.3±6% within whisker, compared to only 50.3±18% across whiskers. These networks further provided information about whisker identity that was approximately 6 times higher than what was provided by the latency to first spike and 13 times higher than what was provided by the spike count of individual neurons examined separately. Furthermore, prediction of individual neurons' precise firing conditioned on knowledge of putative pre-synaptic cell firing was 3 times higher than predictions conditioned on stimulus onset alone. Taken together, these results suggest the presence of a temporally precise network coding mechanism that integrates information across neighboring columns within layer V about vibrissa position and whisking kinetics to mediate whisker movement by motor areas innervated by layer V.

Causal neuronal networks provide functional signatures of stimulus encoding.

Graphical models are powerful tools to infer statistical relationships between simultaneously observed random variables. Here, we used Dynamic Bayesian Networks (DBN) to infer causal relationships between simultaneously recorded neurons in the rat somatosensory (barrel) cortex in response to whisker stimulation. DBNs attempt to explain the activity of the observed neurons by searching for the best network connectivity that explains the observed data. The results demonstrate that the networks inferred for the same whisker are stable across multiple repeated trials. In contrast to networks obtained using classical cross-correlograms, DBN was able to discriminate between direct and indirect connectivity. We also found strong consistency between the inferred connections and the sequence of neural firing relative to the stimulus onset.

Synergistic Coding by Cortical Neural Ensembles.

An essential step towards understanding how the brain orchestrates information processing at the cellular and population levels is to simultaneously observe the spiking activity of cortical neurons that mediate perception, learning, and motor processing. In this paper, we formulate an information theoretic approach to determine whether cooperation among neurons may constitute a governing mechanism of information processing when encoding external covariates. Specifically, we show that conditional independence between neuronal outputs may not provide an optimal encoding strategy when the firing probability of a neuron depends on the history of firing of other neurons connected to it. Rather, cooperation among neurons can provide a "message-passing" mechanism that preserves most of the information in the covariates under specific constraints governing their connectivity structure. Using a biologically plausible statistical learning model, we demonstrate the performance of the proposed approach in synergistically encoding a motor task using a subset of neurons drawn randomly from a large population. We demonstrate its superiority in approximating the joint density of the population from limited data compared to a statistically independent model and a maximum entropy (MaxEnt) model.