Mental Workload Classification and Tasks Detection in Multitasking: Deep Learning Insights from EEG Study.

While the term task load (TL) refers to external task demands, the amount of work, or the number of tasks to be performed, mental workload (MWL) refers to the individual's effort, mental capacity, or cognitive resources utilized while performing a task. MWL in multitasking scenarios is often closely linked with the quantity of tasks a person is handling within a given timeframe. In this study, we challenge this hypothesis from the perspective of electroencephalography (EEG) using a deep learning approach. We conducted an EEG experiment with 50 participants performing NASA Multi-Attribute Task Battery II (MATB-II) under 4 different task load levels. We designed a convolutional neural network (CNN) to help with two distinct classification tasks. In one setting, the CNN was used to classify EEG segments based on their task load level. In another setting, the same CNN architecture was trained again to detect the presence of individual MATB-II subtasks. Results show that, while the model successfully learns to detect whether a particular subtask is active in a given segment (i.e., to differentiate between different subtasks-related EEG patterns), it struggles to differentiate between the two highest levels of task load (i.e., to distinguish MWL-related EEG patterns). We speculate that the challenge comes from two factors: first, the experiment was designed in a way that these two highest levels differed only in the quantity of work within a given timeframe; and second, the participants' effective adaptation to increased task demands, as evidenced by low error rates. Consequently, this indicates that under such conditions in multitasking, EEG may not reflect distinct enough patterns to differentiate higher levels of task load.

Enhancing functional abilities and cognitive integration of the lower limb prosthesis.

Lower limb amputation (LLA) destroys the sensory communication between the brain and the external world during standing and walking. Current prostheses do not restore sensory feedback to amputees, who, relying on very limited haptic information from the stump-socket interaction, are forced to deal with serious issues: the risk of falls, decreased mobility, prosthesis being perceived as an external object (low embodiment), and increased cognitive burden. Poor mobility is one of the causes of eventual device abandonment. Restoring sensory feedback from the missing leg of above-knee (transfemoral) amputees and integrating the sensory feedback into the sensorimotor loop would markedly improve the life of patients. In this study, we developed a leg neuroprosthesis, which provided real-time tactile and emulated proprioceptive feedback to three transfemoral amputees through nerve stimulation. The feedback was exploited in active tasks, which proved that our approach promoted improved mobility, fall prevention, and agility. We also showed increased embodiment of the lower limb prosthesis (LLP), through phantom leg displacement perception and questionnaires, and ease of the cognitive effort during a dual-task paradigm, through electroencephalographic recordings. Our results demonstrate that induced sensory feedback can be integrated at supraspinal levels to restore functional abilities of the missing leg. This work paves the way for further investigations about how the brain interprets different artificial feedback strategies and for the development of fully implantable sensory-enhanced leg neuroprostheses, which could drastically ameliorate life quality in people with disability.

Sensory feedback restoration in leg amputees improves walking speed, metabolic cost and phantom pain.

Conventional leg prostheses do not convey sensory information about motion or interaction with the ground to above-knee amputees, thereby reducing confidence and walking speed in the users that is associated with high mental and physical fatigue1-4. The lack of physiological feedback from the remaining extremity to the brain also contributes to the generation of phantom limb pain from the missing leg5,6. To determine whether neural sensory feedback restoration addresses these issues, we conducted a study with two transfemoral amputees, implanted with four intraneural stimulation electrodes7 in the remaining tibial nerve (ClinicalTrials.gov identifier NCT03350061). Participants were evaluated while using a neuroprosthetic device consisting of a prosthetic leg equipped with foot and knee sensors. These sensors drive neural stimulation, which elicits sensations of knee motion and the sole of the foot touching the ground. We found that walking speed and self-reported confidence increased while mental and physical fatigue decreased for both participants during neural sensory feedback compared to the no stimulation trials. Furthermore, participants exhibited reduced phantom limb pain with neural sensory feedback. The results from these proof-of-concept cases provide the rationale for larger population studies investigating the clinical utility of neuroprostheses that restore sensory feedback.

Multiple sparse volumetric priors for distributed EEG source reconstruction.

We revisit the multiple sparse priors (MSP) algorithm implemented in the statistical parametric mapping software (SPM) for distributed EEG source reconstruction (Friston et al., 2008). In the present implementation, multiple cortical patches are introduced as source priors based on a dipole source space restricted to a cortical surface mesh. In this note, we present a technique to construct volumetric cortical regions to introduce as source priors by restricting the dipole source space to a segmented gray matter layer and using a region growing approach. This extension allows to reconstruct brain structures besides the cortical surface and facilitates the use of more realistic volumetric head models including more layers, such as cerebrospinal fluid (CSF), compared to the standard 3-layered scalp-skull-brain head models. We illustrated the technique with ERP data and anatomical MR images in 12 subjects. Based on the segmented gray matter for each of the subjects, cortical regions were created and introduced as source priors for MSP-inversion assuming two types of head models. The standard 3-layered scalp-skull-brain head models and extended 4-layered head models including CSF. We compared these models with the current implementation by assessing the free energy corresponding with each of the reconstructions using Bayesian model selection for group studies. Strong evidence was found in favor of the volumetric MSP approach compared to the MSP approach based on cortical patches for both types of head models. Overall, the strongest evidence was found in favor of the volumetric MSP reconstructions based on the extended head models including CSF. These results were verified by comparing the reconstructed activity. The use of volumetric cortical regions as source priors is a useful complement to the present implementation as it allows to introduce more complex head models and volumetric source priors in future studies.

Bayesian model selection of template forward models for EEG source reconstruction.

Several EEG source reconstruction techniques have been proposed to identify the generating neuronal sources of electrical activity measured on the scalp. The solution of these techniques depends directly on the accuracy of the forward model that is inverted. Recently, a parametric empirical Bayesian (PEB) framework for distributed source reconstruction in EEG/MEG was introduced and implemented in the Statistical Parametric Mapping (SPM) software. The framework allows us to compare different forward modeling approaches, using real data, instead of using more traditional simulated data from an assumed true forward model. In the absence of a subject specific MR image, a 3-layered boundary element method (BEM) template head model is currently used including a scalp, skull and brain compartment. In this study, we introduced volumetric template head models based on the finite difference method (FDM). We constructed a FDM head model equivalent to the BEM model and an extended FDM model including CSF. These models were compared within the context of three different types of source priors related to the type of inversion used in the PEB framework: independent and identically distributed (IID) sources, equivalent to classical minimum norm approaches, coherence (COH) priors similar to methods such as LORETA, and multiple sparse priors (MSP). The resulting models were compared based on ERP data of 20 subjects using Bayesian model selection for group studies. The reconstructed activity was also compared with the findings of previous studies using functional magnetic resonance imaging. We found very strong evidence in favor of the extended FDM head model with CSF and assuming MSP. These results suggest that the use of realistic volumetric forward models can improve PEB EEG source reconstruction.

The dynamics of contour integration: A simultaneous EEG-fMRI study.

To study the dynamics of contour integration in the human brain, we simultaneously acquired EEG and fMRI data while participants were engaged in a passive viewing task. The stimuli were Gabor arrays with some Gabor elements positioned on the contour of an embedded shape, in three conditions: with local and global structure (perfect contour alignment), with global structure only (orthogonal orientations interrupting the alignment), or without contour. By applying JointICA to the EEG and fMRI responses of the subjects, new insights could be obtained that cannot be derived from unimodal recordings. In particular, only in the global structure condition, an ERP peak around 300ms was identified that involved a loop from LOC to the early visual areas. This component can be interpreted as being related to the verification of the consistency of the different local elements with the globally defined shape, which is necessary when perfect local-to-global alignment is absent. By modifying JointICA, a quantitative comparison of brain regions and the time-course of their interplay were obtained between different conditions. More generally, we provide additional support for the presence of feedback loops from higher areas to lower level sensory regions.

ICA extracts epileptic sources from fMRI in EEG-negative patients: a retrospective validation study.

Simultaneous EEG-fMRI has proven to be useful in localizing interictal epileptic activity. However, the applicability of traditional GLM-based analysis is limited as interictal spikes are often not seen on the EEG inside the scanner. Therefore, we aim at extracting epileptic activity purely from the fMRI time series using independent component analysis (ICA). To our knowledge, we show for the first time that ICA can find sources related to epileptic activity in patients where no interictal spikes were recorded in the EEG. The epileptic components were identified retrospectively based on the known localization of the ictal onset zone (IOZ). We demonstrate that the selected components truly correspond to epileptic activity, as sources extracted from patients resemble significantly better the IOZ than sources found in healthy controls. Furthermore, we show that the epileptic components in patients with and without spikes recorded inside the scanner resemble the IOZ in the same degree. We conclude that ICA of fMRI has the potential to extend the applicability of EEG-fMRI for presurgical evaluation in epilepsy.

Comparison of correlation analysis and JointICA for simultaneous EEG-fMRI recordings on contour integration task.

Multimodal approaches to brain imaging are getting increasingly popular among the neuroscience comunity. One such multimodal approach is the combination of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI). In this paper we demonstrate two EEG-fMRI integration methods for contour integration task. First, we derrive the contour-selectivity measures from event related potential (ERP) and fMRI data, and explore the correlation between the two. In this way, we connect the spatial information from fMRI with the temporal information from ERPs. Thereafter, the results from this approach are compare to JointICA integration approach [5], [6], which aims at extracting spatio-temporal independent components, which are the combination of ERP and fMRI activations.

The quest for single trial correlations in multimodal EEG-fMRI data.

In the past decade, technological advances have made it possible to reliably measure brain activity using simultaneous EEG-fMRI recordings inside an MR scanner. The main challenge then became to investigate the coupling between the EEG and fMRI signals in order to benefit from the simultaneously integrated temporal and spatial resolution. Although it is crucial to know when features in EEG and fMRI are expected to correlate with each other before the identification of common sources from multimodal data is possible, it is still a matter of debate. In this study, we address this question by analysing EEG and fMRI data separately from a face processing task. We show that we are able to reliably estimate single trial (ST) dynamics of face processing in EEG and fMRI data separately in four subjects. However, no correlation is found between the modalities. This implies that in this task modality-specific information is larger than the information that is shared by the modalities.

A new and fast approach towards sEMG decomposition.

The decomposition of high-density surface EMG (HD-sEMG) interference patterns into the contribution of motor units is still a challenging task. We introduce a new, fast solution to this problem. The method uses a data-driven approach for selecting a set of electrodes to enable discrimination of present motor unit action potentials (MUAPs). Then, using shapes detected on these channels, the hierarchical clustering algorithm as reported by Quian Quiroga et al. (Neural Comput 16:1661-1687, 2004) is extended for multichannel data in order to obtain the motor unit action potential (MUAP) signatures. After this first step, more motor unit firings are obtained using the extracted signatures by a novel demixing technique. In this demixing stage, we propose a time-efficient solution for the general convolutive system that models the motor unit firings on the HD-sEMG grid. We constrain this system by using the extracted signatures as prior knowledge and reconstruct the firing patterns in a computationally efficient way. The algorithm performance is successfully verified on simulated data containing up to 20 different MUAP signatures. Moreover, we tested the method on real low contraction recordings from the lateral vastus leg muscle by comparing the algorithm's output to the results obtained by manual analysis of the data from two independent trained operators. The proposed method showed to perform about equally successful as the operators.

Single trial ERP reading based on parallel factor analysis.

The extraction of task-related single trial ERP features has recently gained much interest, in particular in simultaneous EEG-fMRI applications. In this study, a specific decomposition known as parallel factor analysis (PARAFAC) was used, in order to retrieve the task-related activity from the raw signals. Using visual detection task data, acquired in normal circumstances and simultaneously with fMRI, differences between distinct task-related conditions can be captured in the trial signatures of specific PARAFAC components when applied to ERP data arranged in Channels × Time × Trials arrays, but the signatures did not correlate with the fMRI data. Despite the need for parameter tuning and careful preprocessing, the approach is shown to be successful, especially when prior knowledge about the expected ERPs is incorporated.

The "why" and "how" of JointICA: results from a visual detection task.

Since several years, neuroscience research started to focus on multimodal approaches. One such multimodal approach is the combination of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI). However, no standard integration procedure has been established so far. One promising data-driven approach consists of a joint decomposition of event-related potentials (ERPs) and fMRI maps derived from the response to a particular stimulus. Such an algorithm (joint independent component analysis or JointICA) has recently been proposed by Calhoun et al. (2006). This method provides sources with both a fine spatial and temporal resolution, and has shown to provide meaningful results. However, the algorithm's performance has not been fully characterized yet, and no procedure has been proposed to assess the quality of the decomposition. In this paper, we therefore try to answer why and how JointICA works. We show the performance of the algorithm on data obtained in a visual detection task, and compare the performance for EEG recorded simultaneously with fMRI data and for EEG recorded in a separate session (outside the scanner room). We perform several analyses in order to set the necessary conditions that lead to a sound decomposition, and to give additional insights for exploration in future studies. In that respect, we show how the algorithm behaves when different EEG electrodes are used and we test the robustness with respect to the number of subjects in the study. The performance of the algorithm in all the experiments is validated based on results from previous studies.

Independent component analysis as a preprocessing step for data compression of neonatal EEG.

We propose a novel approach for compressive sampling of the neonatal electro-encefalogram (EEG) data. The method assumes that the set of EEG data is generated by linearly mixing a fewer number of source signals. Another assumption is that the sources are nearly-sparse in Gabor dictionary. The presented method, instead of compressing original EEG channels, first performs a data-reduction, and then compresses the obtained sources. With this approach we showed that the gain in reconstruction speed is 33%-50%, whereas the compression rate is enhanced by 33%.

Automated way to obtain motor units' signatures and estimate their firing patterns during voluntary contractions using HD-sEMG.

A new, automated way to obtain signatures of active motor units (MUs) from high density surface EMG recordings during voluntary contractions is presented. It relies on clustering of repetitive shapes corresponding to different MU action potentials (MUAPs) present. The number of clusters and the mean shapes of the MUAPs as observed on the electrode grid, are estimated in a fast way without user interaction. The algorithm is tested on simulated signals mimicking a small muscle. Our results show that at least 8 MUAPs can be reliably reconstructed and their MU mean firing frequencies can be estimated.

Combining EMD with ICA for extracting independent sources from single channel and two-channel data.

Blind Source Separation (BSS) techniques are frequently needed in the processing of biomedical signals. This need comes from the fact that these signals are often composed of many different sources, which are mixed in the measured signal. However, we are usually only interested in examining one or a limited set of sources of interest separately. A variety of algorithms exist for separating multichannel mixtures into its independent sources (e.g. different Independent Component Analysis (ICA) techniques). These techniques only work if the number of channels is larger than, or equal to the number of sources present in the signal. On the other hand, only a few algorithms have been reported for the analysis of single channel sources, or other mixtures where the number of sources is higher than the number of channels. In this work we show a new technique which combines Empirical Mode Decomposition (EMD) and Independent Component Analysis (ICA). We will show that this technique is capable in separating independent sources when the number of these sources is higher than the number of channels available. We show the performance in single channel and two-channel biosignal processing.

Source separation from single-channel recordings by combining empirical-mode decomposition and independent component analysis.

In biomedical signal processing, it is often the case that many sources are mixed into the measured signal. The goal is usually to analyze one or several of them separately. In the case of multichannel measurements, several blind source separation techniques are available for decomposing the signal into its components [e.g., independent component analysis (ICA)]. However, only a few techniques have been reported for analyses of single-channel recordings. Examples are single-channel ICA (SCICA) and wavelet-ICA (WICA), which all have certain limitations. In this paper, we propose a new method for a single-channel signal decomposition. This method combines empirical-mode decomposition with ICA. We compare the separation performance of our algorithm with SCICA and WICA through simulations, and we show that our method outperforms the other two, especially for high noise-to-signal ratios. The performance of the new algorithm was also demonstrated in two real-life applications.

Decoupling between fundamental frequency and energy envelope of neonate cries.

BACKGROUND: The presence of decoupling, i.e. the absence of coupling between fundamental frequency variation and energy envelope during phonetic crying, and its extent, reflects the degree of maturation of the central nervous system. AIM: We hereby wanted to assess the existence and extent of decoupling in term neonates (neurodevelopmental relevance) and whether an association between decoupling and clinical pain expression could be unveiled (clinical relevance). STUDY DESIGN: To assess decoupling in healthy term neonates during procedural pain, newborns were videotaped and crying was recorded during venous blood sampling. Besides acoustic analysis, pain expression was quantified based on the Modified Behavioral Pain Scale (MBPS). SUBJECTS: 47 healthy term neonates underwent venous blood puncture at the 3rd day of life. OUTCOME MEASURES: Beside the MBPS score, the correlation coefficients were calculated between the fundamental frequency variation and energy envelope of the cries. RESULTS: Based on data collected in 47 healthy term neonates, correlation coefficients varied between 0.20 and 0.68. The degree of decoupling displayed extensive variability between the neonates and also in different cry bouts in a crying sequence within an individual neonate. A negative association was found between MBPS value and decoupling (r(2)=-0.12), the same as for the intra-subject variability although less extensive (r(2)=-0.02). CONCLUSION: Decoupling only relates weakly with the amount of distress in 3day old newborns, even though a great intra-subject variability is present. This study suggests that there is no evidence of extensive decoupling as the newborn still has to fully develop the control of larynx and abdominal muscles.