Subject-Independent Deep Architecture for EEG-Based Motor Imagery Classification.

Motor imagery (MI) classification based on electroencephalogram (EEG) is a widely-used technique in non-invasive brain-computer interface (BCI) systems. Since EEG recordings suffer from heterogeneity across subjects and labeled data insufficiency, designing a classifier that performs the MI independently from the subject with limited labeled samples would be desirable. To overcome these limitations, we propose a novel subject-independent semi-supervised deep architecture (SSDA). The proposed SSDA consists of two parts: an unsupervised and a supervised element. The training set contains both labeled and unlabeled data samples from multiple subjects. First, the unsupervised part, known as the columnar spatiotemporal auto-encoder (CST-AE), extracts latent features from all the training samples by maximizing the similarity between the original and reconstructed data. A dimensional scaling approach is employed to reduce the dimensionality of the representations while preserving their discriminability. Second, a supervised part learns a classifier based on the labeled training samples using the latent features acquired in the unsupervised part. Moreover, we employ center loss in the supervised part to minimize the embedding space distance of each point in a class to its center. The model optimizes both parts of the network in an end-to-end fashion. The performance of the proposed SSDA is evaluated on test subjects who were not seen by the model during the training phase. To assess the performance, we use two benchmark EEG-based MI task datasets. The results demonstrate that SSDA outperforms state-of-the-art methods and that a small number of labeled training samples can be sufficient for strong classification performance.

A computer vision approach for analyzing label free leukocyte trafficking dynamics on a microvascular mimetic.

High-content imaging techniques in conjunction with in vitro microphysiological systems (MPS) allow for novel explorations of physiological phenomena with a high degree of translational relevance due to the usage of human cell lines. MPS featuring ultrathin and nanoporous silicon nitride membranes (µSiM) have been utilized in the past to facilitate high magnification phase contrast microscopy recordings of leukocyte trafficking events in a living mimetic of the human vascular microenvironment. Notably, the imaging plane can be set directly at the endothelial interface in a µSiM device, resulting in a high-resolution capture of an endothelial cell (EC) and leukocyte coculture reacting to different stimulatory conditions. The abundance of data generated from recording observations at this interface can be used to elucidate disease mechanisms related to vascular barrier dysfunction, such as sepsis. The appearance of leukocytes in these recordings is dynamic, changing in character, location and time. Consequently, conventional image processing techniques are incapable of extracting the spatiotemporal profiles and bulk statistics of numerous leukocytes responding to a disease state, necessitating labor-intensive manual processing, a significant limitation of this approach. Here we describe a machine learning pipeline that uses a semantic segmentation algorithm and classification script that, in combination, is capable of automated and label-free leukocyte trafficking analysis in a coculture mimetic. The developed computational toolset has demonstrable parity with manually tabulated datasets when characterizing leukocyte spatiotemporal behavior, is computationally efficient and capable of managing large imaging datasets in a semi-automated manner.

An interactive time series image analysis software for dendritic spines.

Live fluorescence imaging has demonstrated the dynamic nature of dendritic spines, with changes in shape occurring both during development and in response to activity. The structure of a dendritic spine correlates with its functional efficacy. Learning and memory studies have shown that a great deal of the information stored by a neuron is contained in the synapses. High precision tracking of synaptic structures can give hints about the dynamic nature of memory and help us understand how memories evolve both in biological and artificial neural networks. Experiments that aim to investigate the dynamics behind the structural changes of dendritic spines require the collection and analysis of large time-series datasets. In this paper, we present an open-source software called SpineS for automatic longitudinal structural analysis of dendritic spines with additional features for manual intervention to ensure optimal analysis. We have tested the algorithm on in-vitro, in-vivo, and simulated datasets to demonstrate its performance in a wide range of possible experimental scenarios.

Changes in EEG complexity with neurofeedback and multi-sensory learning in children with dyslexia: A multiscale entropy analysis.

Multiscale entropy analysis (MSE) is a novel entropy-based approach for measuring dynamical complexity in physiological systems over a range of temporal scales. MSE has been successfully applied in the literature when measuring autism traits, Alzheimer's, and schizophrenia. However, until now, there has been no research on MSE applied to children with dyslexia. In this study, we have applied MSE analysis to the EEG data of an experimental group consisting of children with dyslexia as well as a control group consisting of typically developing children and compared the results. The experimental group comprised 16 participants with dyslexia who visited Ankara University Medical Faculty Child Neurology Department, and the control group comprised 20 age-matched typically developing children with no reading or writing problems. MSE was calculated for one continuous 60-s epoch for each experimental and control group's EEG session data. The experimental group showed significantly lower complexity at the lowest temporal scale and the medium temporal scales than the typically developing group. Moreover, the experimental group received 60 neurofeedback and multi-sensory learning sessions, each lasting 30 min, with Auto Train Brain. Post-treatment, the experimental group's lower complexity increased to the typically developing group's levels at lower and medium temporal scales in all channels.

A mobile app that uses neurofeedback and multi-sensory learning methods improves reading abilities in dyslexia: A pilot study.

Reading comprehension is difficult to improve for children with dyslexia because of the continuing demands of orthographic decoding in combination with limited working memory capacity. Children with dyslexia get special education that improves spelling, phonemic and vocabulary awareness, however the latest research indicated that special education does not improve reading comprehension. With the aim of improving reading comprehension, reading speed and all other reading abilities of children with dyslexia, Auto Train Brain that is a novel mobile app using neurofeedback and multi-sensory learning methods was developed. With a clinical study, we wanted to demonstrate the effectiveness of Auto Train Brain on reading abilities. We compared the cognitive improvements obtained with Auto Train Brain with the improvements obtained with special dyslexia training. Auto Train Brain was applied to 16 children with dyslexia 60 times for 30 minutes. The control group consisted of 14 children with dyslexia who did not have remedial training with Auto Train Brain, but who did continue special education. The TILLS test was applied to both the experimental and the control group at the beginning of the experiment and after a 6-month duration from the first TILLS test. Comparison of the pre- and post- TILLS test results indicated that applying neurofeedback and multi-sensory learning method improved reading comprehension of the experimental group more than that of the control group statistically significantly. Both Auto Train Brain and special education improved phonemic awareness and nonword spelling.

EEG Emotion Recognition via Graph-based Spatio-Temporal Attention Neural Networks.

Emotion recognition based on electroencephalography (EEG) signals has been receiving significant attention in the domains of affective computing and brain-computer interfaces (BCI). Although several deep learning methods have been proposed dealing with the emotion recognition task, developing methods that effectively extract and use discriminative features is still a challenge. In this work, we propose the novel spatio-temporal attention neural network (STANN) to extract discriminative spatial and temporal features of EEG signals by a parallel structure of multi-column convolutional neural network and attention-based bidirectional long-short term memory. Moreover, we explore the inter-channel relationships of EEG signals via graph signal processing (GSP) tools. Our experimental analysis demonstrates that the proposed network improves the state-of-the-art results in subject-wise, binary classification of valence and arousal levels as well as four-class classification in the valence-arousal emotion space when raw EEG signals or their graph representations, in an architecture coined as GFT-STANN, are used as model inputs.

Compressed Sensing Structured Illumination Microscopy.

We propose a new framework for super-resolution structured illumination microscopy (SR-SIM) based on compressed sensing (CS). Our framework addresses several key problems in SIM, including long readout time and photobleaching. CS has the potential to eliminate these problems because it allows the reduction of the number of measurements, can record an image faster, and excites fluorochromes with less excitation light. Key contribution of our proposed method is that sampling and down-modulation of an object scene are simultaneously performed. The impact of our contribution is demonstrated by simulation-based experiments involving computer-generated super-resolution microscopy images, considering reductions in both data quality and quantity.

Prediction of Motor Imagery Performance based on Pre-Trial Spatio-Spectral Alertness Features.

Electroencephalogram (EEG) based brain-computer interfaces (BCIs) enable communication by interpreting the user intent based on measured brain electrical activity. Such interpretation is usually performed by supervised classifiers constructed in training sessions. However, changes in cognitive states of the user, such as alertness and vigilance, during test sessions lead to variations in EEG patterns, causing classification performance decline in BCI systems. This research focuses on effects of alertness on the performance of motor imagery (MI) BCI as a common mental control paradigm. It proposes a new protocol to predict MI performance decline by alertness-related pre-trial spatio-spectral EEG features. The proposed protocol can be used for adapting the classifier or restoring alertness based on the cognitive state of the user during BCI applications.

Prediction of Reaction Time and Vigilance Variability From Spatio-Spectral Features of Resting-State EEG in a Long Sustained Attention Task.

Resting-state brain networks represent the intrinsic state of the brain during the majority of cognitive and sensorimotor tasks. However, no study has yet presented concise predictors of task-induced vigilance variability from spectro-spatial features of the resting-state electroencephalograms (EEG). In this study, ten healthy volunteers have participated in fixed-sequence, varying-duration sessions of sustained attention to response task (SART) for over 100 minutes. A novel and adaptive cumulative vigilance scoring (CVS) scheme is proposed based on tonic performance and response time. Multiple linear regression (MLR) using feature relevance analysis has shown that average CVS, average response time, and variabilities of these scores can be predicted (p < 0.05) from the resting-state band-power ratios of EEG signals. Cross-validated neural networks also captured different associations for narrow-band beta and wide-band gamma and differences between the high- and low-attention networks in temporal regions. The proposed framework and these first findings on stable and significant attention predictors from the power ratios of resting-state EEG can be useful in brain-computer interfacing and vigilance monitoring applications.

Pseudo-Marginal MCMC Sampling for Image Segmentation Using Nonparametric Shape Priors.

Segmenting images of low quality or with missing data is a challenging problem. In such scenarios, exploiting statistical prior information about the shapes to be segmented can improve the segmentation results significantly. Incorporating prior density of shapes into a Bayesian framework leads to the posterior density of segmenting shapes given the observed data. Most segmentation algorithms that exploit shape priors optimize a cost function based on the posterior density and find a point estimate (e.g., using maximum a posteriori estimation). However, especially when the prior shape density is multimodal leading to a multimodal posterior density, a point estimate does not provide a measure of the degree of confidence in that result, neither does it provide a picture of other probable solutions based on the observed data and the shape priors. With a statistical view, addressing these issues would involve the problem of characterizing the posterior distributions of the shapes of the objects to be segmented. An analytic computation of such posterior distributions is intractable; however, characterization is still possible through their samples. In this paper, we propose an efficient pseudo-marginal Markov chain Monte Carlo (MCMC) sampling approach to draw samples from posterior shape distributions for image segmentation. The computation time of the proposed approach is independent from the training set size. Therefore, it scales well for very large data sets. In addition to better characterization of the statistical structure of the problem, such an approach has the potential to address issues with getting stuck at local optima, suffered by existing shape-based segmentation methods. Our approach is able to characterize the posterior probability density in the space of shapes through its samples, and to return multiple solutions, potentially from different modes of a multimodal probability density, which would be encountered, e.g., in segmenting objects from multiple shape classes. We present promising results on a variety of synthetic and real data sets.

Prediction of Response Time and Vigilance Score in a Sustained Attention Task from Pre-trial Phase Synchrony using Deep Neural Networks.

A real-time assessment of sustained attention requires a continuous performance measure ideally obtained objectively and without disrupting the ongoing behavioral patterns. In this work, we investigate whether the phasic functional connectivity patterns from short- and long-range attention networks can predict the tonic performance in a long Sustained Attention to Response Task (SART). Pre-trial phase synchrony indices (PSIs) from individual experiment blocks are used as features for assessment of the proposed average cumulative vigilance score (CVS) and hit response time (HRT). Deep neural networks (DNNs) with the mean-squared-error (MSE) loss function outperformed the ones with mean-absolute-error (MAE) in 4-fold cross-validations. PSI features from the 16-20 Hz beta sub-band obtained the lowest RMSE of 0.043 and highest correlation of 0.806 for predicting the average CVS, and the alpha oscillation PSIs resulted in an RMSE of 51.91 ms and a correlation of 0.903 for predicting the mean HRT. The proposed system can be used for monitoring performance of users susceptible to hypo- or hyper-vigilance and the subsequent system adaptation without implemented eye trackers. To the best of our knowledge, functional connectivity features in general and phase locking values in particular have not been used for regression models of vigilance variations with neural networks.

Tracking-assisted Detection of Dendritic Spines in Time-Lapse Microscopic Images.

Detecting morphological changes of dendritic spines in time-lapse microscopy images and correlating them with functional properties such as memory and learning, are fundamental and challenging problems in neurobiology research. In this paper, we propose an algorithm for dendritic spine detection in time series. The proposed approach initially performs spine detection at each time point and improves the accuracy by exploiting the information obtained from tracking of individual spines over time. To detect dendritic spines in a time point image we employ an SVM classifier trained by pre-labeled SIFT feature descriptors in combination with a dot enhancement filter. Second, to track the growth or loss of spines, we apply a SIFT-based rigid registration method for the alignment of time-series images. This step takes into account both the structure and the movement of objects, combined with a robust dynamic scheme to link information about spines that disappear and reappear over time. Next, we improve spine detection by employing a probabilistic dynamic programming approach to search for an optimum solution to accurately detect missed spines. Finally, we determine the spine location more precisely by performing a watershed-geodesic active contour model. We quantitatively assess the performance of the proposed spine detection algorithm based on annotations performed by biologists and compare its performance with the results obtained by the noncommercial software NeuronIQ. Experiments show that our approach can accurately detect and quantify spines in 2-photon microscopy time-lapse data and is able to accurately identify spine elimination and formation.

Image Segmentation Using Disjunctive Normal Bayesian Shape and Appearance Models.

The use of appearance and shape priors in image segmentation is known to improve accuracy; however, existing techniques have several drawbacks. For instance, most active shape and appearance models require landmark points and assume unimodal shape and appearance distributions, and the level set representation does not support construction of local priors. In this paper, we present novel appearance and shape models for image segmentation based on a differentiable implicit parametric shape representation called a disjunctive normal shape model (DNSM). The DNSM is formed by the disjunction of polytopes, which themselves are formed by the conjunctions of half-spaces. The DNSM's parametric nature allows the use of powerful local prior statistics, and its implicit nature removes the need to use landmarks and easily handles topological changes. In a Bayesian inference framework, we model arbitrary shape and appearance distributions using nonparametric density estimations, at any local scale. The proposed local shape prior results in accurate segmentation even when very few training shapes are available, because the method generates a rich set of shape variations by locally combining training samples. We demonstrate the performance of the framework by applying it to both 2-D and 3-D data sets with emphasis on biomedical image segmentation applications.

Nonparametric Joint Shape and Feature Priors for Image Segmentation.

In many image segmentation problems involving limited and low-quality data, employing statistical prior information about the shapes of the objects to be segmented can significantly improve the segmentation result. However, defining probability densities in the space of shapes is an open and challenging problem, especially if the object to be segmented comes from a shape density involving multiple modes (classes). Existing techniques in the literature estimate the underlying shape distribution by extending Parzen density estimator to the space of shapes. In these methods, the evolving curve may converge to a shape from a wrong mode of the posterior density when the observed intensities provide very little information about the object boundaries. In such scenarios, employing both shape- and class-dependent discriminative feature priors can aid the segmentation process. Such features may involve, e.g., intensity-based, textural, or geometric information about the objects to be segmented. In this paper, we propose a segmentation algorithm that uses nonparametric joint shape and feature priors constructed by Parzen density estimation. We incorporate the learned joint shape and feature prior distribution into a maximum a posteriori estimation framework for segmentation. The resulting optimization problem is solved using active contours. We present experimental results on a variety of synthetic and real data sets from several fields involving multimodal shape densities. Experimental results demonstrate the potential of the proposed method.

Electroencephalographic identifiers of motor adaptation learning.

OBJECTIVE: Recent brain-computer interface (BCI) assisted stroke rehabilitation protocols tend to focus on sensorimotor activity of the brain. Relying on evidence claiming that a variety of brain rhythms beyond sensorimotor areas are related to the extent of motor deficits, we propose to identify neural correlates of motor learning beyond sensorimotor areas spatially and spectrally for further use in novel BCI-assisted neurorehabilitation settings. APPROACH: Electroencephalographic (EEG) data were recorded from healthy subjects participating in a physical force-field adaptation task involving reaching movements through a robotic handle. EEG activity recorded during rest prior to the experiment and during pre-trial movement preparation was used as features to predict motor adaptation learning performance across subjects. MAIN RESULTS: Subjects learned to perform straight movements under the force-field at different adaptation rates. Both resting-state and pre-trial EEG features were predictive of individual adaptation rates with relevance of a broad network of beta activity. Beyond sensorimotor regions, a parieto-occipital cortical component observed across subjects was involved strongly in predictions and a fronto-parietal cortical component showed significant decrease in pre-trial beta-powers for users with higher adaptation rates and increase in pre-trial beta-powers for users with lower adaptation rates. SIGNIFICANCE: Including sensorimotor areas, a large-scale network of beta activity is presented as predictive of motor learning. Strength of resting-state parieto-occipital beta activity or pre-trial fronto-parietal beta activity can be considered in BCI-assisted stroke rehabilitation protocols with neurofeedback training or volitional control of neural activity for brain-robot interfaces to induce plasticity.

Disjunctive Normal Parametric Level Set With Application to Image Segmentation.

Level set methods are widely used for image segmentation because of their convenient shape representation for numerical computations and capability to handle topological changes. However, in spite of the numerous works in the literature, the use of level set methods in image segmentation still has several drawbacks. These shortcomings include formation of irregularities of the signed distance function, sensitivity to initialization, lack of locality, and expensive computational cost, which increases dramatically as the number of objects to be simultaneously segmented grows. In this paper, we propose a novel parametric level set method called disjunctive normal level set (DNLS), and apply it to both two-phase (single object) and multiphase (multiobject) image segmentations. DNLS is a differentiable model formed by the union of polytopes, which themselves are created by intersections of half-spaces. We formulate the segmentation algorithm in a Bayesian framework and use a variational approach to minimize the energy with respect to the parameters of the model. The proposed DNLS can be considered as an open framework that allows the use of different appearance models and shape priors. Compared with the conventional level sets available in the literature, the proposed DNLS has the following major advantages: it requires significantly less computational time and memory, it naturally keeps the level set function regular during the evolution, it is more suitable for multiphase and local region-based image segmentations, and it is less sensitive to noise and initialization. The experimental results show the potential of the proposed method.

Recursive Bayesian Coding for BCIs.

Brain-Computer Interfaces (BCIs) seek to infer some task symbol, a task relevant instruction, from brain symbols, classifiable physiological states. For example, in a motor imagery robot control task a user would indicate their choice from a dictionary of task symbols (rotate arm left, grasp, etc.) by selecting from a smaller dictionary of brain symbols (imagined left or right hand movements). We examine how a BCI infers a task symbol using selections of brain symbols. We offer a recursive Bayesian decision framework which incorporates context prior distributions (e.g., language model priors in spelling applications), accounts for varying brain symbol accuracy and is robust to single brain symbol query errors. This framework is paired with Maximum Mutual Information (MMI) coding which maximizes a generalization of ITR. Both are applicable to any discrete task and brain phenomena (e.g., P300, SSVEP, MI). To demonstrate the efficacy of our approach we perform SSVEP "Shuffle" Speller experiments and compare our recursive coding scheme with traditional decision tree methods including Huffman coding. MMI coding leverages the asymmetry of the classifier's mistakes across a particular user's SSVEP responses; in doing so it offers a 33% increase in letter accuracy though it is 13% slower in our experiment.

Dendritic spine classification using shape and appearance features based on two-photon microscopy.

BACKGROUND: Neuronal morphology and function are highly coupled. In particular, dendritic spine morphology is strongly governed by the incoming neuronal activity. The first step towards understanding the structure-function relationships is to classify spine shapes into the main spine types suggested in the literature. Due to the lack of reliable automated analysis tools, classification is mostly performed manually, which is a time-intensive task and prone to subjectivity. NEW METHOD: We propose an automated method to classify dendritic spines using shape and appearance features based on challenging two-photon laser scanning microscopy (2PLSM) data. Disjunctive Normal Shape Models (DNSM) is a recently proposed parametric shape representation. We perform segmentation of spine images by applying DNSM and use the resulting representation as shape features. Furthermore, we use Histogram of oriented gradients (HOG) to extract appearance features. In this context, we propose a kernel density estimation (KDE) based framework for dendritic spine classification, which uses these shape and appearance features. RESULTS: Our shape and appearance features based approach combined with Neural Network (NN) correctly classifies 87.06% of spines on a dataset of 456 spines. COMPARISON WITH EXISTING METHODS: Our proposed method outperforms standard morphological feature based approaches. Our KDE based framework also enables neuroscientists to analyze the separability of spine shape classes in the likelihood ratio space, which leads to further insights about nature of the spine shape analysis problem. CONCLUSIONS: Results validate that performance of our proposed approach is comparable to a human expert. It also enable neuroscientists to study shape statistics in the likelihood ratio space.

Asynchronous decoding of finger movements from ECoG signals using long-range dependencies conditional random fields.

OBJECTIVE: In this work we propose the use of conditional random fields with long-range dependencies for the classification of finger movements from electrocorticographic recordings. APPROACH: The proposed method uses long-range dependencies taking into consideration time-lags between the brain activity and the execution of the motor task. In addition, the proposed method models the dynamics of the task executed by the subject and uses information about these dynamics as prior information during the classification stage. MAIN RESULTS: The results show that incorporating temporal information about the executed task as well as incorporating long-range dependencies between the brain signals and the labels effectively increases the system's classification performance compared to methods in the state of art. SIGNIFICANCE: The method proposed in this work makes use of probabilistic graphical models to incorporate temporal information in the classification of finger movements from electrocorticographic recordings. The proposed method highlights the importance of including prior information about the task that the subjects execute. As the results show, the combination of these two features effectively produce a significant improvement of the system's classification performance.

DISJUNCTIVE NORMAL LEVEL SET: AN EFFICIENT PARAMETRIC IMPLICIT METHOD.

Level set methods are widely used for image segmentation because of their capability to handle topological changes. In this paper, we propose a novel parametric level set method called Disjunctive Normal Level Set (DNLS), and apply it to both two phase (single object) and multiphase (multi-object) image segmentations. The DNLS is formed by union of polytopes which themselves are formed by intersections of half-spaces. The proposed level set framework has the following major advantages compared to other level set methods available in the literature. First, segmentation using DNLS converges much faster. Second, the DNLS level set function remains regular throughout its evolution. Third, the proposed multiphase version of the DNLS is less sensitive to initialization, and its computational cost and memory requirement remains almost constant as the number of objects to be simultaneously segmented grows. The experimental results show the potential of the proposed method.