Using single-trial EEG to predict and analyze subsequent memory.

We show that it is possible to successfully predict subsequent memory performance based on single-trial EEG activity before and during item presentation in the study phase. Two-class classification was conducted to predict subsequently remembered vs. forgotten trials based on subjects' responses in the recognition phase. The overall accuracy across 18 subjects was 59.6% by combining pre- and during-stimulus information. The single-trial classification analysis provides a dimensionality reduction method to project the high-dimensional EEG data onto a discriminative space. These projections revealed novel findings in the pre- and during-stimulus periods related to levels of encoding. It was observed that the pre-stimulus information (specifically oscillatory activity between 25 and 35Hz) -300 to 0ms before stimulus presentation and during-stimulus alpha (7-12Hz) information between 1000 and 1400ms after stimulus onset distinguished between recollection and familiarity while the during-stimulus alpha information and temporal information between 400 and 800ms after stimulus onset mapped these two states to similar values.

Dynamic brightness induction causes flicker adaptation, but only along the edges: evidence against the neural filling-in of brightness.

Is brightness represented in a point-for-point neural map that is filled in from the response of small, contrast-sensitive edge detector cells? We tested for the presence of this filled-in map by adapting to illusory flicker caused by a dynamic brightness-induction stimulus. Thereafter flicker sensitivity was reduced when our test region was the same size as the induced region, but not for smaller, inset regions. This suggests induced brightness is represented by either small edge-selective cells with no filling-in stage, or by contrast-sensitive spatial filters at many different scales, but not by a population of filled-in neurons arranged in a point-for-point map.

Automated Pain Assessment in Children Using Electrodermal Activity and Video Data Fusion via Machine Learning.

OBJECTIVE: Pain assessment in children continues to challenge clinicians and researchers, as subjective experiences of pain require inference through observable behaviors, both involuntary and deliberate. The presented approach supplements the subjective self-report-based method by fusing electrodermal activity (EDA) recordings with video facial expressions to develop an objective pain assessment metric. Such an approach is specifically important for assessing pain in children who are not capable of providing accurate self-pain reports, requiring nonverbal pain assessment. We demonstrate the performance of our approach using data recorded from children in post-operative recovery following laparoscopic appendectomy. We examined separately and combined the usefulness of EDA and video facial expression data as predictors of children's self-reports of pain following surgery through recovery. Findings indicate that EDA and facial expression data independently provide above chance sensitivities and specificities, but their fusion for classifying clinically significant pain vs. clinically nonsignificant pain achieved substantial improvement, yielding 90.91% accuracy, with 100% sensitivity and 81.82% specificity. The multimodal measures capitalize upon different features of the complex pain response. Thus, this paper presents both evidence for the utility of a weighted maximum likelihood algorithm as a novel feature selection method for EDA and video facial expression data and an accurate and objective automated classification algorithm capable ofdiscriminating clinically significant pain from clinically nonsignificant pain in children.

Personalized Pain Detection in Facial Video with Uncertainty Estimation.

Pain is a personal, subjective experience, and the current gold standard to evaluate pain is the Visual Analog Scale (VAS), which is self-reported at the video level. One problem with the current automated pain detection systems is that the learned model doesn't generalize well to unseen subjects. In this work, we propose to improve pain detection in facial videos using individual models and uncertainty estimation. For a new test video, we jointly consider which individual models generalize well generally, and which individual models are more similar/accurate to this test video, in order to choose the optimal combination of individual models and get the best performance on new test videos. We show on the UNBC-McMaster Shoulder Pain Dataset that our method significantly improves the previous state-of-the-art performance.

Hybrid brain-computer interface with motor imagery and error-related brain activity.

OBJECTIVE: Brain-computer interface (BCI) systems read and interpret brain activity directly from the brain. They can provide a means of communication or locomotion for patients suffering from neurodegenerative diseases or stroke. However, non-stationarity of brain activity limits the reliable transfer of the algorithms that were trained during a calibration session to real-time BCI control. One source of non-stationarity is the user's brain response to the BCI output (feedback), for instance, whether the BCI feedback is perceived as an error by the user or not. By taking such sources of non-stationarity into account, the reliability of the BCI can be improved. APPROACH: In this work, we demonstrate a real-time implementation of a hybrid motor imagery BCI combining the information from the motor imagery signal and the error-related brain activity simultaneously so as to gain benefit from both sources. MAIN RESULTS: We show significantly improved performance in real-time BCI control across 12 participants, compared to a conventional motor imagery BCI. The significant improvement is in terms of classification accuracy, target hit rate, subjective perception of control and information-transfer rate. Moreover, our offline analyses of the recorded EEG data show that the error-related brain activity provides a more reliable source of information than the motor imagery signal. SIGNIFICANCE: This work shows, for the first time, that the error-related brain activity classifier compared to the motor imagery classifier is more consistent when trained on calibration data and tested during online control. This likely explains why the proposed hybrid BCI allows for a more reliable means of communication or rehabilitation for patients in need.

Temporally Adaptive Common Spatial Patterns with Deep Convolutional Neural Networks.

Brain-computer interface (BCI) systems are proposed as a means of communication for locked-in patients. One common BCI paradigm is motor imagery in which the user controls a BCI by imagining movements of different body parts. It is known that imagining different body parts results in event-related desynchronization (ERD) in various frequency bands. Existing methods such as common spatial patterns (CSP) and its refinement filterbank common spatial patterns (FB-CSP) aim at finding features that are informative for classification of the motor imagery class. Our proposed method is a temporally adaptive common spatial patterns implementation of the commonly used filter-bank common spatial patterns method using convolutional neural networks; hence it is called TA-CSPNN. With this method we aim to: (1) make the feature extraction and classification end-to-end, (2) base it on the way CSP/FBCSP extracts relevant features, and finally, (3) reduce the number of trainable parameters compared to existing deep learning methods to improve generalizability in noisy data such as EEG. More importantly, we show that this reduction in parameters does not affect performance and in fact the trained network generalizes better for data from some participants. We show our results on two datasets, one publicly available from BCI Competition IV, dataset 2a and another in-house motor imagery dataset.

Youth and Parent Appraisals of Participation in a Study of Spontaneous and Induced Pediatric Clinical Pain.

The current study examined youths' and their parents' perceptions concerning participation in an investigation of spontaneous and induced pain during recovery from laparoscopic appendectomy. Youth (age range 5-17 years) and their parents independently completed surveys about their study participation. On a 0 (very negative) -to-10 (very positive) scale, both parents 9.4(1.3) [mean(SD)] and youth 7.9(2.4) rated their experience as positive. Among youth, experience ratings did not differ by pain severity and survey responses did not differ by age. Most youth (83%) reported they would tell another youth to participate. Ethical issues regarding instigation of pain in youth for research purposes are examined.

Single-Trial EEG Analysis Predicts Memory Retrieval and Reveals Source-Dependent Differences.

We used pattern classifiers to extract features related to recognition memory retrieval from the temporal information in single-trial electroencephalography (EEG) data during attempted memory retrieval. Two-class classification was conducted on correctly remembered trials with accurate context (or source) judgments vs. correctly rejected trials. The average accuracy for datasets recorded in a single session was 61% while the average accuracy for datasets recorded in two separate sessions was 56%. To further understand the basis of the classifier's performance, two other pattern classifiers were trained on different pairs of behavioral conditions. The first of these was designed to use information related to remembering the item and the second to use information related to remembering the contextual information (or source) about the item. Mollison and Curran (2012) had earlier shown that subjects' familiarity judgments contributed to improved memory of spatial contextual information but not of extrinsic associated color information. These behavioral results were similarly reflected in the event-related potential (ERP) known as the FN400 (an early frontal effect relating to familiarity) which revealed differences between correct and incorrect context memories in the spatial but not color conditions. In our analyses we show that a classifier designed to distinguish between correct and incorrect context memories, more strongly involves early activity (400-500 ms) over the frontal channels for the location distinctions, than for the extrinsic color associations. In contrast, the classifier designed to classify memory for the item (without memory for the context), had more frontal channel involvement for the color associated experiments than for the spatial experiments. Taken together these results argue that location may be bound more tightly with the item than an extrinsic color association. The multivariate classification approach also showed that trial-by-trial variation in EEG corresponding to these ERP components were predictive of subjects' behavioral responses. Additionally, the multivariate classification approach enabled analysis of error conditions that did not have sufficient trials for standard ERP analyses. These results suggested that false alarms were primarily attributable to item memory (as opposed to memory of associated context), as commonly predicted, but with little previous corroborating EEG evidence.

Automated Pain Assessment using Electrodermal Activity Data and Machine Learning.

Objective pain assessment is required for appropriate pain management in the clinical setting. However, clinical gold standard pain assessment is based on subjective methods. Automated pain detection from physiological data may provide important objective information to better standardize pain assessment. Specifically, electrodermal activity (EDA) can identify features of stress and anxiety induced by varying pain levels. However, notable variability in EDA measurement exists and research to date has demonstrated sensitivity but lack of specificity in pain assessment. In this paper, we use timescale decomposition (TSD) to extract salient features from EDA signals to identify an accurate and automated EDA pain detection algorithm to sensitively and specifically distinguish pain from no-pain conditions.

Automated Pain Detection in Facial Videos of Children using Human-Assisted Transfer Learning.

Accurately determining pain levels in children is difficult, even for trained professionals and parents. Facial activity provides sensitive and specific information about pain, and computer vision algorithms have been developed to automatically detect Facial Action Units (AUs) defined by the Facial Action Coding System (FACS). Our prior work utilized information from computer vision, i.e., automatically detected facial AUs, to develop classifiers to distinguish between pain and no-pain conditions. However, application of pain/no-pain classifiers based on automated AU codings across different environmental domains results in diminished performance. In contrast, classifiers based on manually coded AUs demonstrate reduced environmentally-based variability in performance. In this paper, we train a machine learning model to recognize pain using AUs coded by a computer vision system embedded in a software package called iMotions. We also study the relationship between iMotions (automatically) and human (manually) coded AUs. We find that AUs coded automatically are different from those coded by a human trained in the FACS system, and that the human coder is less sensitive to environmental changes. To improve classification performance in the current work, we applied transfer learning by training another machine learning model to map automated AU codings to a subspace of manual AU codings to enable more robust pain recognition performance when only automatically coded AUs are available for the test data. With this transfer learning method, we improved the Area Under the ROC Curve (AUC) on independent data from new participants in our target domain from 0.67 to 0.72.

Explaining brightness illusions using spatial filtering and local response normalization.

We introduce two new low-level computational models of brightness perception that account for a wide range of brightness illusions, including many variations on White's Effect [Perception, 8, 1979, 413]. Our models extend Blakeslee and McCourt's ODOG model [Vision Research, 39, 1999, 4361], which combines multiscale oriented difference-of-Gaussian filters and response normalization. We extend the response normalization to be more neurally plausible by constraining normalization to nearby receptive fields (models 1 and 2) and spatial frequencies (model 2), and show that both of these changes increase the effectiveness of the models at predicting brightness illusions.

Preprocessing and meta-classification for brain-computer interfaces.

A brain-computer interface (BCI) is a system which allows direct translation of brain states into actions, bypassing the usual muscular pathways. A BCI system works by extracting user brain signals, applying machine learning algorithms to classify the user's brain state, and performing a computer-controlled action. Our goal is to improve brain state classification. Perhaps the most obvious way to improve classification performance is the selection of an advanced learning algorithm. However, it is now well known in the BCI community that careful selection of preprocessing steps is crucial to the success of any classification scheme. Furthermore, recent work indicates that combining the output of multiple classifiers (meta-classification) leads to improved classification rates relative to single classifiers (Dornhege et al., 2004). In this paper, we develop an automated approach which systematically analyzes the relative contributions of different preprocessing and meta-classification approaches. We apply this procedure to three data sets drawn from BCI Competition 2003 (Blankertz et al., 2004) and BCI Competition III (Blankertz et al., 2006), each of which exhibit very different characteristics. Our final classification results compare favorably with those from past BCI competitions. Additionally, we analyze the relative contributions of individual preprocessing and meta-classification choices and discuss which types of BCI data benefit most from specific algorithms.

A model of surround suppression through cortical feedback.

Surround suppression occurs when a visual stimulus outside a neuron's classical receptive field causes a reduction in firing rate. It has become clear that several mechanisms are working together to induce center-surround effects such as surround suppression. While several models exist that rely on lateral connections within V1 to explain surround suppression, few have been proposed that show how cortical feedback might play a role. In this work, we propose a theory in which reductions in excitatory feedback contribute to a neuron's suppressed firing rate. We also provide a computational model that incorporates this idea.

Homeostatic synaptic scaling in self-organizing maps.

Various forms of the self-organizing map (SOM) have been proposed as models of cortical development [Choe Y., Miikkulainen R., (2004). Contour integration and segmentation with self-organized lateral connections. Biological Cybernetics, 90, 75-88; Kohonen T., (2001). Self-organizing maps (3rd ed.). Springer; Sirosh J., Miikkulainen R., (1997). Topographic receptive fields and patterned lateral interaction in a self-organizing model of the primary visual cortex. Neural Computation, 9(3), 577-594]. Typically, these models use weight normalization to contain the weight growth associated with Hebbian learning. A more plausible mechanism for controlling the Hebbian process has recently emerged. Turrigiano and Nelson [Turrigiano G.G., Nelson S.B., (2004). Homeostatic plasticity in the developing nervous system. Nature Reviews Neuroscience, 5, 97-107] have shown that neurons in the cortex actively maintain an average firing rate by scaling their incoming weights. In this work, it is shown that this type of homeostatic synaptic scaling can replace the common, but unsupported, standard weight normalization. Organized maps still form and the output neurons are able to maintain an unsaturated firing rate, even in the face of large-scale cell proliferation or die-off. In addition, it is shown that in some cases synaptic scaling leads to networks that more accurately reflect the probability distribution of the input data.

On selecting features from splice junctions: an analysis using information theoretic and machine learning approaches.

The computational recognition of precise splice junctions is a challenge faced in the analysis of newly sequenced genomes. This is challenging due to the fact that the distribution of sequence patterns in these regions is not always distinct. Our objective is to understand the sequence signatures at the splice junctions, not simply to create an artificial recognition system. We use a combination of a neural network based calliper randomization approach and an information theoretic based feature selection approach for this purpose. This has been done in an effort to understand regions that harbor information content and to extract features relevant for the prediction of splice junctions. The analysis using the neural network based calliper randomization approach revealed regions important in the internal representation of the network model. The calliper approach captured both correlated as well as independently important features. The feature selection approach captures features that are independently informative. The two different methods can capture features with different properties. Comparative analysis of the results using both the methods help to infer about the kind of information present in the region.

Task-phase-specific dynamics of basal forebrain neuronal ensembles.

Cortically projecting basal forebrain neurons play a critical role in learning and attention, and their degeneration accompanies age-related impairments in cognition. Despite the impressive anatomical and cell-type complexity of this system, currently available data suggest that basal forebrain neurons lack complexity in their response fields, with activity primarily reflecting only macro-level brain states such as sleep and wake, onset of relevant stimuli and/or reward obtainment. The current study examined the spiking activity of basal forebrain neuron populations across multiple phases of a selective attention task, addressing, in particular, the issue of complexity in ensemble firing patterns across time. Clustering techniques applied to the full population revealed a large number of distinct categories of task-phase-specific activity patterns. Unique population firing-rate vectors defined each task phase and most categories of task-phase-specific firing had counterparts with opposing firing patterns. An analogous set of task-phase-specific firing patterns was also observed in a population of posterior parietal cortex neurons. Thus, consistent with the known anatomical complexity, basal forebrain population dynamics are capable of differentially modulating their cortical targets according to the unique sets of environmental stimuli, motor requirements, and cognitive processes associated with different task phases.

Effect of visual feedback on the occipital-parietal-motor network in Parkinson's disease with freezing of gait.

Freezing of gait (FOG) is an elusive phenomenon that debilitates a large number of Parkinson's disease (PD) patients regardless of stage of disease, medication status, or deep brain stimulation implantation. Sensory feedback cues, especially visual feedback cues, have been shown to alleviate FOG episodes or even prevent episodes from occurring. Here, we examine cortical information flow between occipital, parietal, and motor areas during the pre-movement stage of gait in a PD-with-FOG patient that had a strong positive behavioral response to visual cues, one PD-with-FOG patient without any behavioral response to visual cues, and age-matched healthy controls, before and after training with visual feedback. Results for this case study show differences in cortical information flow between the responding PD-with-FOG patient and the other two subject types, notably, an increased information flow in the beta range. Tentatively suggesting the formation of an alternative cortical sensory-motor pathway during training with visual feedback, these results are proposed as subject for further verification employing larger cohorts of patients.

Single-trial classification of gait and point movement preparation from human EEG.

Neuroimaging studies provide evidence of cortical involvement immediately before and during gait and during gait-related behaviors such as stepping in place or motor imagery of gait. Here we attempt to perform single-trial classification of gait intent from another movement plan (point intent) or from standing in place. Subjects walked naturally from a starting position to a designated ending position, pointed at a designated position from the starting position, or remained standing at the starting position. The 700 ms of recorded electroencephalography (EEG) before movement onset was used for single-trial classification of trials based on action type and direction (left walk, forward walk, right walk, left point, right point, and stand) as well as action type regardless of direction (stand, walk, point). Classification using regularized LDA was performed on a principal components analysis (PCA) reduced feature space composed of coefficients from levels 1 to 9 of a discrete wavelet decomposition using the Daubechies 4 wavelet. We achieved significant classification for all conditions, with errors as low as 17% when averaged across nine subjects. LDA and PCA highly weighted frequency ranges that included movement related potentials (MRPs), with smaller contributions from frequency ranges that included mu and beta idle motor rhythms. Additionally, error patterns suggested a spatial structure to the EEG signal. Future applications of the cortical gait intent signal may include an additional dimension of control for prosthetics, preemptive corrective feedback for gait disturbances, or human computer interfaces (HCI).

Spatial properties of flicker adaptation.

Prolonged viewing of a flickering region reduces sensitivity to a subsequently flickered test patch of identical extent, but the spatial properties of this adaptation are unknown. What happens to the sensitivity to a smaller flickered test patch completely contained in, but inset from, the adapted region? We show that sensitivity to the inset test patch is only slightly affected by adaptation of the larger region. This suggests that neurons that respond to the edges of the smaller test patch are not adapted by the larger flickering region. We then show that an annulus adapter designed specifically to adapt only those edges only slightly reduces sensitivity, demonstrating that neurons that do not adapt to the flickered edges are also involved in detecting flicker. This gives further evidence that flicker detection depends on at least two mechanisms - one sensitive to flickering edges and one sensitive to local flicker, and shows that these mechanisms can operate in isolation.

Brief presentations reveal the temporal dynamics of brightness induction and White's illusion.

We measured the timecourse of brightness processing by briefly presenting brightness illusions and then masking them. Brightness induction (brightness contrast) was visible when presented for only 58 ms, was stronger at short presentation times, and its visibility did not depend on spatial frequency. We also found that White's illusion was visible at 82 ms. Together, these results suggest that (1) brightness perception depends on the surrounding context, even at very short presentation times, (2) the initial brightness percept is generated very quickly, but additional exposure can modulate it, and (3) the temporal dynamics are not dependent on a slow filling-in process.