Modulation transfer functions for audiovisual speech.

Temporal synchrony between facial motion and acoustic modulations is a hallmark feature of audiovisual speech. The moving face and mouth during natural speech is known to be correlated with low-frequency acoustic envelope fluctuations (below 10 Hz), but the precise rates at which envelope information is synchronized with motion in different parts of the face are less clear. Here, we used regularized canonical correlation analysis (rCCA) to learn speech envelope filters whose outputs correlate with motion in different parts of the speakers face. We leveraged recent advances in video-based 3D facial landmark estimation allowing us to examine statistical envelope-face correlations across a large number of speakers (∼4000). Specifically, rCCA was used to learn modulation transfer functions (MTFs) for the speech envelope that significantly predict correlation with facial motion across different speakers. The AV analysis revealed bandpass speech envelope filters at distinct temporal scales. A first set of MTFs showed peaks around 3-4 Hz and were correlated with mouth movements. A second set of MTFs captured envelope fluctuations in the 1-2 Hz range correlated with more global face and head motion. These two distinctive timescales emerged only as a property of natural AV speech statistics across many speakers. A similar analysis of fewer speakers performing a controlled speech task highlighted only the well-known temporal modulations around 4 Hz correlated with orofacial motion. The different bandpass ranges of AV correlation align notably with the average rates at which syllables (3-4 Hz) and phrases (1-2 Hz) are produced in natural speech. Whereas periodicities at the syllable rate are evident in the envelope spectrum of the speech signal itself, slower 1-2 Hz regularities thus only become prominent when considering crossmodal signal statistics. This may indicate a motor origin of temporal regularities at the timescales of syllables and phrases in natural speech.

Real-Time Decoding of Attentional States Using Closed-Loop EEG Neurofeedback.

Sustained attention is a cognitive ability to maintain task focus over extended periods of time (Mackworth, 1948; Chun, Golomb, & Turk-Browne, 2011). In this study, scalp electroencephalography (EEG) signals were processed in real time using a 32 dry-electrode system during a sustained visual attention task. An attention training paradigm was implemented, as designed in DeBettencourt, Cohen, Lee, Norman, and Turk-Browne (2015) in which the composition of a sequence of blended images is updated based on the participant's decoded attentional level to a primed image category. It was hypothesized that a single neurofeedback training session would improve sustained attention abilities. Twenty-two participants were trained on a single neurofeedback session with behavioral pretraining and posttraining sessions within three consecutive days. Half of the participants functioned as controls in a double-blinded design and received sham neurofeedback. During the neurofeedback session, attentional states to primed categories were decoded in real time and used to provide a continuous feedback signal customized to each participant in a closed-loop approach. We report a mean classifier decoding error rate of 34.3% (chance = 50%). Within the neurofeedback group, there was a greater level of task-relevant attentional information decoded in the participant's brain before making a correct behavioral response than before an incorrect response. This effect was not visible in the control group (interaction p=7.23e-4), which strongly indicates that we were able to achieve a meaningful measure of subjective attentional state in real time and control participants' behavior during the neurofeedback session. We do not provide conclusive evidence whether the single neurofeedback session per se provided lasting effects in sustained attention abilities. We developed a portable EEG neurofeedback system capable of decoding attentional states and predicting behavioral choices in the attention task at hand. The neurofeedback code framework is Python based and open source, and it allows users to actively engage in the development of neurofeedback tools for scientific and translational use.

Sequence Classification Using Third-Order Moments.

Model-based classification of sequence data using a set of hidden Markov models is a well-known technique. The involved score function, which is often based on the class-conditional likelihood, can, however, be computationally demanding, especially for long data sequences. Inspired by recent theoretical advances in spectral learning of hidden Markov models, we propose a score function based on third-order moments. In particular, we propose to use the Kullback-Leibler divergence between theoretical and empirical third-order moments for classification of sequence data with discrete observations. The proposed method provides lower computational complexity at classification time than the usual likelihood-based methods. In order to demonstrate the properties of the proposed method, we perform classification of both simulated data and empirical data from a human activity recognition study.

Neural Markers of Responsiveness to the Environment in Human Sleep.

UNLABELLED: Sleep is characterized by a loss of behavioral responsiveness. However, recent research has shown that the sleeping brain is not completely disconnected from its environment. How neural activity constrains the ability to process sensory information while asleep is yet unclear. Here, we instructed human volunteers to classify words with lateralized hand responses while falling asleep. Using an electroencephalographic (EEG) marker of motor preparation, we show how responsiveness is modulated across sleep. These modulations are tracked using classic event-related potential analyses complemented by Lempel-Ziv complexity (LZc), a measure shown to track arousal in sleep and anesthesia. Neural activity related to the semantic content of stimuli was conserved in light non-rapid eye movement (NREM) sleep. However, these processes were suppressed in deep NREM sleep and, importantly, also in REM sleep, despite the recovery of wake-like neural activity in the latter. In NREM sleep, sensory activations were counterbalanced by evoked down states, which, when present, blocked further processing of external information. In addition, responsiveness markers correlated positively with baseline complexity, which could be related to modulation in sleep depth. In REM sleep, however, this relationship was reversed. We therefore propose that, in REM sleep, endogenously generated processes compete with the processing of external input. Sleep can thus be seen as a self-regulated process in which external information can be processed in lighter stages but suppressed in deeper stages. Last, our results suggest drastically different gating mechanisms in NREM and REM sleep. SIGNIFICANCE STATEMENT: Previous research has tempered the notion that sleepers are isolated from their environment. Here, we pushed this idea forward and examined, across all sleep stages, the brain's ability to flexibly process sensory information, up to the decision level. We extracted an EEG marker of motor preparation to determine the completion of the sensory processing chain and explored how it is constrained by baseline and evoked neural activity. In NREM sleep, slow waves elicited by stimuli appeared to block response preparation. We also used a novel analytic approach (Lempel-Ziv complexity) and showed that the ability to process external information correlates with neural complexity. A reversal of the correlation between complexity and motor indices in REM sleep suggests drastically different gating mechanisms across sleep stages.

Spatio-temporal reconstruction of brain dynamics from EEG with a Markov prior.

Electroencephalography (EEG) can capture brain dynamics in high temporal resolution. By projecting the scalp EEG signal back to its origin in the brain also high spatial resolution can be achieved. Source localized EEG therefore has potential to be a very powerful tool for understanding the functional dynamics of the brain. Solving the inverse problem of EEG is however highly ill-posed as there are many more potential locations of the EEG generators than EEG measurement points. Several well-known properties of brain dynamics can be exploited to alleviate this problem. More short ranging connections exist in the brain than long ranging, arguing for spatially focal sources. Additionally, recent work (Delorme et al., 2012) argues that EEG can be decomposed into components having sparse source distributions. On the temporal side both short and long term stationarity of brain activation are seen. We summarize these insights in an inverse solver, the so-called "Variational Garrote" (Kappen and Gómez, 2013). Using a Markov prior we can incorporate flexible degrees of temporal stationarity. Through spatial basis functions spatially smooth distributions are obtained. Sparsity of these are inherent to the Variational Garrote solver. We name our method the MarkoVG and demonstrate its ability to adapt to the temporal smoothness and spatial sparsity in simulated EEG data. Finally a benchmark EEG dataset is used to demonstrate MarkoVG's ability to recover non-stationary brain dynamics.

Data-driven forward model inference for EEG brain imaging.

Electroencephalography (EEG) is a flexible and accessible tool with excellent temporal resolution but with a spatial resolution hampered by volume conduction. Reconstruction of the cortical sources of measured EEG activity partly alleviates this problem and effectively turns EEG into a brain imaging device. The quality of the source reconstruction depends on the forward model which details head geometry and conductivities of different head compartments. These person-specific factors are complex to determine, requiring detailed knowledge of the subject's anatomy and physiology. In this proof-of-concept study, we show that, even when anatomical knowledge is unavailable, a suitable forward model can be estimated directly from the EEG. We propose a data-driven approach that provides a low-dimensional parametrization of head geometry and compartment conductivities, built using a corpus of forward models. Combined with only a recorded EEG signal, we are able to estimate both the brain sources and a person-specific forward model by optimizing this parametrization. We thus not only solve an inverse problem, but also optimize over its specification. Our work demonstrates that personalized EEG brain imaging is possible, even when the head geometry and conductivities are unknown.

The Center for Integrated Molecular Brain Imaging (Cimbi) database.

We here describe a multimodality neuroimaging containing data from healthy volunteers and patients, acquired within the Lundbeck Foundation Center for Integrated Molecular Brain Imaging (Cimbi) in Copenhagen, Denmark. The data is of particular relevance for neurobiological research questions related to the serotonergic transmitter system with its normative data on the serotonergic subtype receptors 5-HT1A, 5-HT1B, 5-HT2A, and 5-HT4 and the 5-HT transporter (5-HTT), but can easily serve other purposes. The Cimbi database and Cimbi biobank were formally established in 2008 with the purpose to store the wealth of Cimbi-acquired data in a highly structured and standardized manner in accordance with the regulations issued by the Danish Data Protection Agency as well as to provide a quality-controlled resource for future hypothesis-generating and hypothesis-driven studies. The Cimbi database currently comprises a total of 1100 PET and 1000 structural and functional MRI scans and it holds a multitude of additional data, such as genetic and biochemical data, and scores from 17 self-reported questionnaires and from 11 neuropsychological paper/computer tests. The database associated Cimbi biobank currently contains blood and in some instances saliva samples from about 500 healthy volunteers and 300 patients with e.g., major depression, dementia, substance abuse, obesity, and impulsive aggression. Data continue to be added to the Cimbi database and biobank.

Multiview Bayesian Correlated Component Analysis.

Correlated component analysis as proposed by Dmochowski, Sajda, Dias, and Parra (2012) is a tool for investigating brain process similarity in the responses to multiple views of a given stimulus. Correlated components are identified under the assumption that the involved spatial networks are identical. Here we propose a hierarchical probabilistic model that can infer the level of universality in such multiview data, from completely unrelated representations, corresponding to canonical correlation analysis, to identical representations as in correlated component analysis. This new model, which we denote Bayesian correlated component analysis, evaluates favorably against three relevant algorithms in simulated data. A well-established benchmark EEG data set is used to further validate the new model and infer the variability of spatial representations across multiple subjects.

Frontal alpha oscillations distinguish leaders from followers: multivariate decoding of mutually interacting brains.

Successful social interactions rely upon the abilities of two or more people to mutually exchange information in real-time, while simultaneously adapting to one another. The neural basis of social cognition has mostly been investigated in isolated individuals, and more recently using two-person paradigms to quantify the neuronal dynamics underlying social interaction. While several studies have shown the relevance of understanding complementary and mutually adaptive processes, the neural mechanisms underlying such coordinative behavioral patterns during joint action remain largely unknown. Here, we employed a synchronized finger-tapping task while measuring dual-EEG from pairs of human participants who either mutually adjusted to each other in an interactive task or followed a computer metronome. Neurophysiologically, the interactive condition was characterized by a stronger suppression of alpha and low-beta oscillations over motor and frontal areas in contrast to the non-interactive computer condition. A multivariate analysis of two-brain activity to classify interactive versus non-interactive trials revealed asymmetric patterns of the frontal alpha-suppression in each pair, during both task anticipation and execution, such that only one member showed the frontal component. Analysis of the behavioral data showed that this distinction coincided with the leader-follower relationship in 8/9 pairs, with the leaders characterized by the stronger frontal alpha-suppression. This suggests that leaders invest more resources in prospective planning and control. Hence our results show that the spontaneous emergence of leader-follower relationships in dyadic interactions can be predicted from EEG recordings of brain activity prior to and during interaction. Furthermore, this emphasizes the importance of investigating complementarity in joint action.

Non-parametric Bayesian graph models reveal community structure in resting state fMRI.

Modeling of resting state functional magnetic resonance imaging (rs-fMRI) data using network models is of increasing interest. It is often desirable to group nodes into clusters to interpret the communication patterns between nodes. In this study we consider three different nonparametric Bayesian models for node clustering in complex networks. In particular, we test their ability to predict unseen data and their ability to reproduce clustering across datasets. The three generative models considered are the Infinite Relational Model (IRM), Bayesian Community Detection (BCD), and the Infinite Diagonal Model (IDM). The models define probabilities of generating links within and between clusters and the difference between the models lies in the restrictions they impose upon the between-cluster link probabilities. IRM is the most flexible model with no restrictions on the probabilities of links between clusters. BCD restricts the between-cluster link probabilities to be strictly lower than within-cluster link probabilities to conform to the community structure typically seen in social networks. IDM only models a single between-cluster link probability, which can be interpreted as a background noise probability. These probabilistic models are compared against three other approaches for node clustering, namely Infomap, Louvain modularity, and hierarchical clustering. Using 3 different datasets comprising healthy volunteers' rs-fMRI we found that the BCD model was in general the most predictive and reproducible model. This suggests that rs-fMRI data exhibits community structure and furthermore points to the significance of modeling heterogeneous between-cluster link probabilities.

Image classification with symbolic hints using limited resources.

Typical machine learning classification benchmark problems often ignore the full input data structures present in real-world classification problems. Here we aim to represent additional information as "hints" for classification. We show that under a specific realistic conditional independence assumption, the hint information can be included by late fusion. In two experiments involving image classification with hints taking the form of text metadata, we demonstrate the feasibility and performance of the fusion scheme. We fuse the output of pre-trained image classifiers with the output of pre-trained text models. We show that calibration of the pre-trained models is crucial for the performance of the fused model. We compare the performance of the fusion scheme with a mid-level fusion scheme based on support vector machines and find that these two methods tend to perform quite similarly, albeit the late fusion scheme has only negligible computational costs.

Using sequences of life-events to predict human lives.

Here we represent human lives in a way that shares structural similarity to language, and we exploit this similarity to adapt natural language processing techniques to examine the evolution and predictability of human lives based on detailed event sequences. We do this by drawing on a comprehensive registry dataset, which is available for Denmark across several years, and that includes information about life-events related to health, education, occupation, income, address and working hours, recorded with day-to-day resolution. We create embeddings of life-events in a single vector space, showing that this embedding space is robust and highly structured. Our models allow us to predict diverse outcomes ranging from early mortality to personality nuances, outperforming state-of-the-art models by a wide margin. Using methods for interpreting deep learning models, we probe the algorithm to understand the factors that enable our predictions. Our framework allows researchers to discover potential mechanisms that impact life outcomes as well as the associated possibilities for personalized interventions.

Nonlinear denoising and analysis of neuroimages with kernel principal component analysis and pre-image estimation.

We investigate the use of kernel principal component analysis (PCA) and the inverse problem known as pre-image estimation in neuroimaging: i) We explore kernel PCA and pre-image estimation as a means for image denoising as part of the image preprocessing pipeline. Evaluation of the denoising procedure is performed within a data-driven split-half evaluation framework. ii) We introduce manifold navigation for exploration of a nonlinear data manifold, and illustrate how pre-image estimation can be used to generate brain maps in the continuum between experimentally defined brain states/classes. We base these illustrations on two fMRI BOLD data sets - one from a simple finger tapping experiment and the other from an experiment on object recognition in the ventral temporal lobe.

Noise-assisted variational quantum thermalization.

Preparing thermal states on a quantum computer can have a variety of applications, from simulating many-body quantum systems to training machine learning models. Variational circuits have been proposed for this task on near-term quantum computers, but several challenges remain, such as finding a scalable cost-function, avoiding the need of purification, and mitigating noise effects. We propose a new algorithm for thermal state preparation that tackles those three challenges by exploiting the noise of quantum circuits. We consider a variational architecture containing a depolarizing channel after each unitary layer, with the ability to directly control the level of noise. We derive a closed-form approximation for the free-energy of such circuit and use it as a cost function for our variational algorithm. By evaluating our method on a variety of Hamiltonians and system sizes, we find several systems for which the thermal state can be approximated with a high fidelity. However, we also show that the ability for our algorithm to learn the thermal state strongly depends on the temperature: while a high fidelity can be obtained for high and low temperatures, we identify a specific range for which the problem becomes more challenging. We hope that this first study on noise-assisted thermal state preparation will inspire future research on exploiting noise in variational algorithms.

Uncovering Cortical Units of Processing From Multi-Layered Connectomes.

Modern diffusion and functional magnetic resonance imaging (dMRI/fMRI) provide non-invasive high-resolution images from which multi-layered networks of whole-brain structural and functional connectivity can be derived. Unfortunately, the lack of observed correspondence between the connectivity profiles of the two modalities challenges the understanding of the relationship between the functional and structural connectome. Rather than focusing on correspondence at the level of connections we presently investigate correspondence in terms of modular organization according to shared canonical processing units. We use a stochastic block-model (SBM) as a data-driven approach for clustering high-resolution multi-layer whole-brain connectivity networks and use prediction to quantify the extent to which a given clustering accounts for the connectome within a modality. The employed SBM assumes a single underlying parcellation exists across modalities whilst permitting each modality to possess an independent connectivity structure between parcels thereby imposing concurrent functional and structural units but different structural and functional connectivity profiles. We contrast the joint processing units to their modality specific counterparts and find that even though data-driven structural and functional parcellations exhibit substantial differences, attributed to modality specific biases, the joint model is able to achieve a consensus representation that well accounts for both the functional and structural connectome providing improved representations of functional connectivity compared to using functional data alone. This implies that a representation persists in the consensus model that is shared by the individual modalities. We find additional support for this viewpoint when the anatomical correspondence between modalities is removed from the joint modeling. The resultant drop in predictive performance is in general substantial, confirming that the anatomical correspondence of processing units is indeed present between the two modalities. Our findings illustrate how multi-modal integration admits consensus representations well-characterizing each individual modality despite their biases and points to the importance of multi-layered connectomes as providing supplementary information regarding the brain's canonical processing units.

Visualization of nonlinear kernel models in neuroimaging by sensitivity maps.

There is significant current interest in decoding mental states from neuroimages. In this context kernel methods, e.g., support vector machines (SVM) are frequently adopted to learn statistical relations between patterns of brain activation and experimental conditions. In this paper we focus on visualization of such nonlinear kernel models. Specifically, we investigate the sensitivity map as a technique for generation of global summary maps of kernel classification models. We illustrate the performance of the sensitivity map on functional magnetic resonance (fMRI) data based on visual stimuli. We show that the performance of linear models is reduced for certain scan labelings/categorizations in this data set, while the nonlinear models provide more flexibility. We show that the sensitivity map can be used to visualize nonlinear versions of kernel logistic regression, the kernel Fisher discriminant, and the SVM, and conclude that the sensitivity map is a versatile and computationally efficient tool for visualization of nonlinear kernel models in neuroimaging.

Motion correction of single-voxel spectroscopy by independent component analysis applied to spectra from nonanesthetized pediatric subjects.

For single-voxel spectroscopy, the acquisition of the spectrum is typically repeated n times and then combined with a factor sqrt[n] in order to improve the signal-to-noise ratio. In practice, the acquisitions are not only affected by random noise but also by physiologic motion and subject movements. Since the influence of physiologic motion such as cardiac and respiratory motion on the data is limited, it can be compensated for without data loss. Individual acquisitions hampered by subject movements, on the other hand, need to be rejected if no correction or compensation is possible. If the individual acquisitions are stored, it is possible to identify and reject the motion-disturbed acquisitions before averaging. Several automatic algorithms were investigated using a dataset of spectra from nonanesthetized infants with a gestational age of 40 weeks. Median filtering removed most subject movement artifacts, but at the cost of increased sensitivity to random noise. Neither independent component analysis nor outlier identification with multiple comparisons has this problem. These two algorithms are novel in this context. The peak height values of the metabolites were increased compared to the mean of all acquisitions for both methods, although primarily for the ICA method.

Single-Trial Decoding of Scalp EEG under Natural Conditions.

There is significant current interest in decoding mental states from electroencephalography (EEG) recordings. EEG signals are subject-specific, are sensitive to disturbances, and have a low signal-to-noise ratio, which has been mitigated by the use of laboratory-grade EEG acquisition equipment under highly controlled conditions. In the present study, we investigate single-trial decoding of natural, complex stimuli based on scalp EEG acquired with a portable, 32 dry-electrode sensor system in a typical office setting. We probe generalizability by a leave-one-subject-out cross-validation approach. We demonstrate that support vector machine (SVM) classifiers trained on a relatively small set of denoised (averaged) pseudotrials perform on par with classifiers trained on a large set of noisy single-trial samples. We propose a novel method for computing sensitivity maps of EEG-based SVM classifiers for visualization of EEG signatures exploited by the SVM classifiers. Moreover, we apply an NPAIRS resampling framework for estimation of map uncertainty, and thus show that effect sizes of sensitivity maps for classifiers trained on small samples of denoised data and large samples of noisy data are similar. Finally, we demonstrate that the average pseudotrial classifier can successfully predict the class of single trials from withheld subjects, which allows for fast classifier training, parameter optimization, and unbiased performance evaluation in machine learning approaches for brain decoding.

Unmixing Oscillatory Brain Activity by EEG Source Localization and Empirical Mode Decomposition.

Neuronal activity is composed of synchronous and asynchronous oscillatory activity at different frequencies. The neuronal oscillations occur at time scales well matched to the temporal resolution of electroencephalography (EEG); however, to derive meaning from the electrical brain activity as measured from the scalp, it is useful to decompose the EEG signal in space and time. In this study, we elaborate on the investigations into source-based signal decomposition of EEG. Using source localization, the electrical brain signal is spatially unmixed and the neuronal dynamics from a region of interest are analyzed using empirical mode decomposition (EMD), a technique aimed at detecting periodic signals. We demonstrate, first in simulations, that the EMD is more accurate when applied to the spatially unmixed signal compared to the scalp-level signal. Furthermore, on EEG data recorded simultaneously with transcranial magnetic stimulation (TMS) over the hand area of the primary motor cortex, we observe a link between the peak to peak amplitude of the motor-evoked potential (MEP) and the phase of the decomposed localized electrical activity before TMS onset. The results thus encourage combination of source localization and EMD in the pursuit of further insight into the mechanisms of the brain with respect to the phase and frequency of the electrical oscillations and their cortical origin.

Shift-invariant multilinear decomposition of neuroimaging data.

We present an algorithm for multilinear decomposition that allows for arbitrary shifts along one modality. The method is applied to neural activity arranged in the three modalities space, time, and trial. Thus, the algorithm models neural activity as a linear superposition of components with a fixed time course that may vary across either trials or space in its overall intensity and latency. Its utility is demonstrated on simulated data as well as actual EEG, and fMRI data. We show how shift-invariant multilinear decompositions of multiway data can successfully cope with variable latencies in data derived from neural activity--a problem that has caused degenerate solutions especially in modeling neuroimaging data with instantaneous multilinear decompositions. Our algorithm is available for download at www.erpwavelab.org.