Early language dissociation in bilingual minds: magnetoencephalography evidence through a machine learning approach.

Does neural activity reveal how balanced bilinguals choose languages? Despite using diverse neuroimaging techniques, prior studies haven't provided a definitive solution to this problem. Nonetheless, studies involving direct brain stimulation in bilinguals have identified distinct brain regions associated with language production in different languages. In this magnetoencephalography study with 45 proficient Spanish-Basque bilinguals, we investigated language selection during covert picture naming and word reading tasks. Participants were prompted to name line drawings or read words if the color of the stimulus changed to green, in 10% of trials. The task was performed either in Spanish or Basque. Despite similar sensor-level evoked activity for both languages in both tasks, decoding analyses revealed language-specific classification ~100 ms post-stimulus onset. During picture naming, right occipital-temporal sensors predominantly contributed to language decoding, while left occipital-temporal sensors were crucial for decoding during word reading. Cross-task decoding analysis unveiled robust generalization effects from picture naming to word reading. Our methodology involved a fine-grained examination of neural responses using magnetoencephalography, offering insights into the dynamics of language processing in bilinguals. This study refines our understanding of the neural underpinnings of language selection and bridges the gap between non-invasive and invasive experimental evidence in bilingual language production.

Direct brain recordings suggest a causal subsequent-memory effect.

Endogenous variation in brain state and stimulus-specific evoked activity can both contribute to successful encoding. Previous studies, however, have not clearly distinguished among these components. We address this question by analysing intracranial EEG recorded from epilepsy patients as they studied and subsequently recalled lists of words. We first trained classifiers to predict recall of either single items or entire lists and found that both classifiers exhibited similar performance. We found that list-level classifier output-a biomarker of successful encoding-tracked item presentation and recall events, despite having no information about the trial structure. Across widespread brain regions, decreased low- and increased high-frequency activity (HFA) marked successful encoding of both items and lists. We found regional differences in the hippocampus and prefrontal cortex, where in the hippocampus HFA correlated more strongly with item recall, whereas, in the prefrontal cortex, HFA correlated more strongly with list performance. Despite subtle differences in item- and list-level features, the similarity in overall classification performance, spectral signatures of successful recall and fluctuations of spectral activity across the encoding period argue for a shared endogenous process that causally impacts the brain's ability to learn new information.

Functional alterations in cortical processing of speech in glioma-infiltrated cortex.

Recent developments in the biology of malignant gliomas have demonstrated that glioma cells interact with neurons through both paracrine signaling and electrochemical synapses. Glioma-neuron interactions consequently modulate the excitability of local neuronal circuits, and it is unclear the extent to which glioma-infiltrated cortex can meaningfully participate in neural computations. For example, gliomas may result in a local disorganization of activity that impedes the transient synchronization of neural oscillations. Alternatively, glioma-infiltrated cortex may retain the ability to engage in synchronized activity in a manner similar to normal-appearing cortex but exhibit other altered spatiotemporal patterns of activity with subsequent impact on cognitive processing. Here, we use subdural electrocorticography to sample both normal-appearing and glioma-infiltrated cortex during speech. We find that glioma-infiltrated cortex engages in synchronous activity during task performance in a manner similar to normal-appearing cortex but recruits a diffuse spatial network. On a temporal scale, we show that signals from glioma-infiltrated cortex have decreased entropy, which may affect its ability to encode information during nuanced tasks such as production of monosyllabic versus polysyllabic words. Furthermore, we show that temporal decoding strategies for distinguishing monosyllabic from polysyllabic words were feasible for signals arising from normal-appearing cortex but not from glioma-infiltrated cortex. These findings inform our understanding of cognitive processing in chronic disease states and have implications for neuromodulation and prosthetics in patients with malignant gliomas.

Stable readout of observed actions from format-dependent activity of monkey's anterior intraparietal neurons.

Humans accurately identify observed actions despite large dynamic changes in their retinal images and a variety of visual presentation formats. A large network of brain regions in primates participates in the processing of others' actions, with the anterior intraparietal area (AIP) playing a major role in routing information about observed manipulative actions (OMAs) to the other nodes of the network. This study investigated whether the AIP also contributes to invariant coding of OMAs across different visual formats. We recorded AIP neuronal activity from two macaques while they observed videos portraying seven manipulative actions (drag, drop, grasp, push, roll, rotate, squeeze) in four visual formats. Each format resulted from the combination of two actor's body postures (standing, sitting) and two viewpoints (lateral, frontal). Out of 297 recorded units, 38% were OMA-selective in at least one format. Robust population code for viewpoint and actor's body posture emerged shortly after stimulus presentation, followed by OMA selectivity. Although we found no fully invariant OMA-selective neuron, we discovered a population code that allowed us to classify action exemplars irrespective of the visual format. This code depends on a multiplicative mixing of signals about OMA identity and visual format, particularly evidenced by a set of units maintaining a relatively stable OMA selectivity across formats despite considerable rescaling of their firing rate depending on the visual specificities of each format. These findings suggest that the AIP integrates format-dependent information and the visual features of others' actions, leading to a stable readout of observed manipulative action identity.

The precedence effect in spatial hearing manifests in cortical neural population responses.

BACKGROUND: To localize sound sources accurately in a reverberant environment, human binaural hearing strongly favors analyzing the initial wave front of sounds. Behavioral studies of this "precedence effect" have so far largely been confined to human subjects, limiting the scope of complementary physiological approaches. Similarly, physiological studies have mostly looked at neural responses in the inferior colliculus, the main relay point between the inner ear and the auditory cortex, or used modeling of cochlear auditory transduction in an attempt to identify likely underlying mechanisms. Studies capable of providing a direct comparison of neural coding and behavioral measures of sound localization under the precedence effect are lacking. RESULTS: We adapted a "temporal weighting function" paradigm previously developed to quantify the precedence effect in human for use in laboratory rats. The animals learned to lateralize click trains in which each click in the train had a different interaural time difference. Computing the "perceptual weight" of each click in the train revealed a strong onset bias, very similar to that reported for humans. Follow-on electrocorticographic recording experiments revealed that onset weighting of interaural time differences is a robust feature of the cortical population response, but interestingly, it often fails to manifest at individual cortical recording sites. CONCLUSION: While previous studies suggested that the precedence effect may be caused by early processing mechanisms in the cochlea or inhibitory circuitry in the brainstem and midbrain, our results indicate that the precedence effect is not fully developed at the level of individual recording sites in the auditory cortex, but robust and consistent precedence effects are observable only in the auditory cortex at the level of cortical population responses. This indicates that the precedence effect emerges at later cortical processing stages and is a significantly "higher order" feature than has hitherto been assumed.

Zero-Shot Neural Decoding with Semi-Supervised Multi-View Embedding.

Zero-shot neural decoding aims to decode image categories, which were not previously trained, from functional magnetic resonance imaging (fMRI) activity evoked when a person views images. However, having insufficient training data due to the difficulty in collecting fMRI data causes poor generalization capability. Thus, models suffer from the projection domain shift problem when novel target categories are decoded. In this paper, we propose a zero-shot neural decoding approach with semi-supervised multi-view embedding. We introduce the semi-supervised approach that utilizes additional images related to the target categories without fMRI activity patterns. Furthermore, we project fMRI activity patterns into a multi-view embedding space, i.e., visual and semantic feature spaces of viewed images to effectively exploit the complementary information. We define several source and target groups whose image categories are very different and verify the zero-shot neural decoding performance. The experimental results demonstrate that the proposed approach rectifies the projection domain shift problem and outperforms existing methods.

Persistent neuronal firing in the medial temporal lobe supports performance and workload of visual working memory in humans.

The involvement of the medial temporal lobe (MTL) in working memory is controversially discussed. Recent findings suggest that persistent neural firing in the hippocampus during maintenance in verbal working memory is associated with workload. Here, we recorded single neuron firing in 13 epilepsy patients (7 male) while they performed a visual working memory task. The number of colored squares in the stimulus set determined the workload of the trial. Performance was almost perfect for low workload (1 and 2 squares) and dropped at high workload (4 and 6 squares), suggesting that high workload exceeded working memory capacity. We identified maintenance neurons in MTL neurons that showed persistent firing during the maintenance period. More maintenance neurons were found in the hippocampus for trials with correct compared to incorrect performance. Maintenance neurons increased and decreased firing in the hippocampus and increased firing in the entorhinal cortex for high compared to low workload. Population firing predicted workload particularly during the maintenance period. Prediction accuracy of workload based on single-trial activity during maintenance was strongest for neurons in the entorhinal cortex and hippocampus. The data suggest that persistent neural firing in the MTL reflects a domain-general process of maintenance supporting performance and workload of multiple items in working memory below and beyond working memory capacity. Persistent neural firing during maintenance in the entorhinal cortex may be associated with its preference to process visual-spatial arrays.

Subjective time is predicted by local and early visual processing.

Time is as pervasive as it is elusive to study, and how the brain keeps track of millisecond time is still unclear. Here we addressed the mechanisms underlying duration perception by looking for a neural signature of subjective time distortion induced by motion adaptation. We recorded electroencephalographic signals in human participants while they were asked to discriminate the duration of visual stimuli after different types of translational motion adaptation. Our results show that perceived duration can be predicted by the amplitude of the N200 event-related potential evoked by the adapted stimulus. Moreover, we show that the distortion of subjective time can be predicted by the activity in the Beta band frequency spectrum, at the offset of the adaptor and during the presentation of the subsequent adapted stimulus. Both effects were observed from posterior electrodes contralateral to the adapted stimulus. Overall, our findings suggest that local and low-level perceptual processes are involved in generating a subjective sense of time.

Decoding neural representational spaces using multivariate pattern analysis.

A major challenge for systems neuroscience is to break the neural code. Computational algorithms for encoding information into neural activity and extracting information from measured activity afford understanding of how percepts, memories, thought, and knowledge are represented in patterns of brain activity. The past decade and a half has seen significant advances in the development of methods for decoding human neural activity, such as multivariate pattern classification, representational similarity analysis, hyperalignment, and stimulus-model-based encoding and decoding. This article reviews these advances and integrates neural decoding methods into a common framework organized around the concept of high-dimensional representational spaces.

Reconstructing Synergy-Based Hand Grasp Kinematics from Electroencephalographic Signals.

Brain-machine interfaces (BMIs) have become increasingly popular in restoring the lost motor function in individuals with disabilities. Several research studies suggest that the CNS may employ synergies or movement primitives to reduce the complexity of control rather than controlling each DoF independently, and the synergies can be used as an optimal control mechanism by the CNS in simplifying and achieving complex movements. Our group has previously demonstrated neural decoding of synergy-based hand movements and used synergies effectively in driving hand exoskeletons. In this study, ten healthy right-handed participants were asked to perform six types of hand grasps representative of the activities of daily living while their neural activities were recorded using electroencephalography (EEG). From half of the participants, hand kinematic synergies were derived, and a neural decoder was developed, based on the correlation between hand synergies and corresponding cortical activity, using multivariate linear regression. Using the synergies and the neural decoder derived from the first half of the participants and only cortical activities from the remaining half of the participants, their hand kinematics were reconstructed with an average accuracy above 70%. Potential applications of synergy-based BMIs for controlling assistive devices in individuals with upper limb motor deficits, implications of the results in individuals with stroke and the limitations of the study were discussed.

The Rapid Emergence of Musical Pitch Structure in Human Cortex.

In tonal music, continuous acoustic waveforms are mapped onto discrete, hierarchically arranged, internal representations of pitch. To examine the neural dynamics underlying this transformation, we presented male and female human listeners with tones embedded within a Western tonal context while recording their cortical activity using magnetoencephalography. Machine learning classifiers were then trained to decode different tones from their underlying neural activation patterns at each peristimulus time sample, providing a dynamic measure of their dissimilarity in cortex. Comparing the time-varying dissimilarity between tones with the predictions of acoustic and perceptual models, we observed a temporal evolution in the brain's representational structure. Whereas initial dissimilarities mirrored their fundamental-frequency separation, dissimilarities beyond 200 ms reflected the perceptual status of each tone within the tonal hierarchy of Western music. These effects occurred regardless of stimulus regularities within the context or whether listeners were engaged in a task requiring explicit pitch analysis. Lastly, patterns of cortical activity that discriminated between tones became increasingly stable in time as the information coded by those patterns transitioned from low-to-high level properties. Current results reveal the dynamics with which the complex perceptual structure of Western tonal music emerges in cortex at the timescale of an individual tone.SIGNIFICANCE STATEMENT Little is understood about how the brain transforms an acoustic waveform into the complex perceptual structure of musical pitch. Applying neural decoding techniques to the cortical activity of human subjects engaged in music listening, we measured the dynamics of information processing in the brain on a moment-to-moment basis as subjects heard each tone. In the first 200 ms after onset, transient patterns of neural activity coded the fundamental frequency of tones. Subsequently, a period emerged during which more temporally stable activation patterns coded the perceptual status of each tone within the "tonal hierarchy" of Western music. Our results provide a crucial link between the complex perceptual structure of tonal music and the underlying neural dynamics from which it emerges.

A distributed dynamic brain network mediates linguistic tone representation and categorization.

Successful categorization requires listeners to represent the incoming sensory information, resolve the "blooming, buzzing confusion" inherent to noisy sensory signals, and leverage the accumulated evidence towards making a decision. Despite decades of intense debate, the neural systems underlying speech categorization remain unresolved. Here we assessed the neural representation and categorization of lexical tones by native Mandarin speakers (N = 31) across a range of acoustic and contextual variabilities (talkers, perceptual saliences, and stimulus-contexts) using functional magnetic imaging (fMRI) and an evidence accumulation model of decision-making. Univariate activation and multivariate pattern analyses reveal that the acoustic-variability-tolerant representations of tone category are observed within the middle portion of the left superior temporal gyrus (STG). Activation patterns in the frontal and parietal regions also contained category-relevant information that was differentially sensitive to various forms of variability. The robustness of neural representations of tone category in a distributed fronto-temporoparietal network is associated with trial-by-trial decision-making parameters. These findings support a hybrid model involving a representational core within the STG that operates dynamically within an extensive frontoparietal network to support the representation and categorization of linguistic pitch patterns.

Application of Empirical Mode Decomposition for Decoding Perception of Faces Using Magnetoencephalography.

Neural decoding is useful to explore the timing and source location in which the brain encodes information. Higher classification accuracy means that an analysis is more likely to succeed in extracting useful information from noises. In this paper, we present the application of a nonlinear, nonstationary signal decomposition technique-the empirical mode decomposition (EMD), on MEG data. We discuss the fundamental concepts and importance of nonlinear methods when it comes to analyzing brainwave signals and demonstrate the procedure on a set of open-source MEG facial recognition task dataset. The improved clarity of data allowed further decoding analysis to capture distinguishing features between conditions that were formerly over-looked in the existing literature, while raising interesting questions concerning hemispheric dominance to the encoding process of facial and identity information.

ARTFLOW: A Fast, Biologically Inspired Neural Network that Learns Optic Flow Templates for Self-Motion Estimation.

Most algorithms for steering, obstacle avoidance, and moving object detection rely on accurate self-motion estimation, a problem animals solve in real time as they navigate through diverse environments. One biological solution leverages optic flow, the changing pattern of motion experienced on the eye during self-motion. Here I present ARTFLOW, a biologically inspired neural network that learns patterns in optic flow to encode the observer's self-motion. The network combines the fuzzy ART unsupervised learning algorithm with a hierarchical architecture based on the primate visual system. This design affords fast, local feature learning across parallel modules in each network layer. Simulations show that the network is capable of learning stable patterns from optic flow simulating self-motion through environments of varying complexity with only one epoch of training. ARTFLOW trains substantially faster and yields self-motion estimates that are far more accurate than a comparable network that relies on Hebbian learning. I show how ARTFLOW serves as a generative model to predict the optic flow that corresponds to neural activations distributed across the network.

Neural Decoding on Imbalanced Calcium Imaging Data with a Network of Support Vector Machines.

We present a novel neural decoding system for calcium imaging data. Miniature calcium imaging is of great utility for examining population neural activity of animals. Our neural decoding system is developed using a carefully-designed support vector machine subsystem together with dataflow-based techniques for system design, which capture the high-level structure of the application and enable powerful system-level analysis and optimization. Also, we introduce a framework for handling imbalanced data. This addresses a problem of imbalanced datasets, which arises commonly in neural decoding applications, as well as in a wide variety of other applications in biomedical engineering and advanced robotics. We developed an ensemble learning based method to tackle this problem. The proposed framework systemically incorporates two heterogeneous model characteristics into a combined model. Through extensive experiments, we evaluate the proposed system using calcium imaging datasets in which neural activities of D1 medium spiny neurons in the dorsal striatum were recorded. The results show that the F 1 score of the proposed system is significantly better than those of previously developed neural decoding systems for calcium imaging.

Elucidating the Neural Representation and the Processing Dynamics of Face Ensembles.

Extensive behavioral work has documented the ability of the human visual system to extract summary representations from face ensembles (e.g., the average identity of a crowd of faces). Yet, the nature of such representations, their underlying neural mechanisms, and their temporal dynamics await elucidation. Here, we examine summary representations of facial identity in human adults (of both sexes) with the aid of pattern analyses, as applied to EEG data, along with behavioral testing. Our findings confirm the ability of the visual system to form such representations both explicitly and implicitly (i.e., with or without the use of specific instructions). We show that summary representations, rather than individual ensemble constituents, can be decoded from neural signals elicited by ensemble perception, we describe the properties of such representations by appeal to multidimensional face space constructs, and we visualize their content through neural-based image reconstruction. Further, we show that the temporal profile of ensemble processing diverges systematically from that of single faces consistent with a slower, more gradual accumulation of perceptual information. Thus, our findings reveal the representational basis of ensemble processing, its fine-grained visual content, and its neural dynamics.SIGNIFICANCE STATEMENT Humans encounter groups of faces, or ensembles, in a variety of environments. Previous behavioral research has investigated how humans process face ensembles as well as the types of summary representations that can be derived from them, such as average emotion, gender, and identity. However, the neural mechanisms mediating these processes are unclear. Here, we demonstrate that ensemble representations, with different facial identity summaries, can be decoded and even visualized from neural data through multivariate analyses. These results provide, to our knowledge, the first detailed investigation into the status and the visual content of neural ensemble representations of faces. Further, the current findings shed light on the temporal dynamics of face ensembles and its relationship with single-face processing.

Natural music evokes correlated EEG responses reflecting temporal structure and beat.

The brain activity of multiple subjects has been shown to synchronize during salient moments of natural stimuli, suggesting that correlation of neural responses indexes a brain state operationally termed 'engagement'. While past electroencephalography (EEG) studies have considered both auditory and visual stimuli, the extent to which these results generalize to music-a temporally structured stimulus for which the brain has evolved specialized circuitry-is less understood. Here we investigated neural correlation during natural music listening by recording EEG responses from N=48 adult listeners as they heard real-world musical works, some of which were temporally disrupted through shuffling of short-term segments (measures), reversal, or randomization of phase spectra. We measured correlation between multiple neural responses (inter-subject correlation) and between neural responses and stimulus envelope fluctuations (stimulus-response correlation) in the time and frequency domains. Stimuli retaining basic musical features, such as rhythm and melody, elicited significantly higher behavioral ratings and neural correlation than did phase-scrambled controls. However, while unedited songs were self-reported as most pleasant, time-domain correlations were highest during measure-shuffled versions. Frequency-domain measures of correlation (coherence) peaked at frequencies related to the musical beat, although the magnitudes of these spectral peaks did not explain the observed temporal correlations. Our findings show that natural music evokes significant inter-subject and stimulus-response correlations, and suggest that the neural correlates of musical 'engagement' may be distinct from those of enjoyment.

The future of upper extremity rehabilitation robotics: research and practice.

The loss of upper limb motor function can have a devastating effect on people's lives. To restore upper limb control and functionality, researchers and clinicians have developed interfaces to interact directly with the human body's motor system. In this invited review, we aim to provide details on the peripheral nerve interfaces and brain-machine interfaces that have been developed in the past 30 years for upper extremity control, and we highlight the challenges that still remain to transition the technology into the clinical market. The findings show that peripheral nerve interfaces and brain-machine interfaces have many similar characteristics that enable them to be concurrently developed. Decoding neural information from both interfaces may lead to novel physiological models that may one day fully restore upper limb motor function for a growing patient population.

Trial-by-trial surprise-decoding model for visual and auditory binary oddball tasks.

Having to survive in a continuously changing environment has driven the human brain to actively predict the future state of its surroundings. Oddball tasks are specific types of experiments in which this nature of the human brain is studied. Detailed mathematical models have been constructed to explain the brain's perception in these tasks. These models consider a subject as an ideal observer who abstracts a hypothesis from the previous stimuli, and estimates its hyper-parameters - in order to make the next prediction. The corresponding prediction error is assumed to manifest the subjective surprise of the brain. While the approach of earlier works to this problem has been to suggest an encoding model, we investigated the reverse model: if the stimuli's surprise is assumed as the cause of the observer's surprise, it must be possible to decode the surprise of each stimulus, for every single subject, given only their neural responses, i.e. to tell how unexpected a specific stimulus has been for them. Employing machine learning tools, we developed a surprise decoding model for binary oddball tasks. We constructed our model using the ideal observer proposed by Meyniel et al. in 2016, and applied it to three datasets, one with visual, one with auditory, and one with both visual and auditory stimuli. We demonstrated that our decoding model performs very well for both of the sensory modalities with or without the presence of the subject's motor response.

Neural signatures underlying deliberation in human foraging decisions.

Humans have a remarkable capacity to mentally project themselves far ahead in time. This ability, which entails the mental simulation of events, is thought to be fundamental to deliberative decision making, as it allows us to search through and evaluate possible choices. Many decisions that humans make are foraging decisions, in which one must decide whether an available offer is worth taking, when compared to unknown future possibilities (i.e., the background). Using a translational decision-making paradigm designed to reveal decision preferences in rats, we found that humans engaged in deliberation when making foraging decisions. A key feature of this task is that preferences (and thus, value) are revealed as a function of serial choices. Like rats, humans also took longer to respond when faced with difficult decisions near their preference boundary, which was associated with prefrontal and hippocampal activation, exemplifying cross-species parallels in deliberation. Furthermore, we found that voxels within the visual cortices encoded neural representations of the available possibilities specifically following regret-inducing experiences, in which the subject had previously rejected a good offer only to encounter a low-valued offer on the subsequent trial.