Music can be reconstructed from human auditory cortex activity using nonlinear decoding models.

Music is core to human experience, yet the precise neural dynamics underlying music perception remain unknown. We analyzed a unique intracranial electroencephalography (iEEG) dataset of 29 patients who listened to a Pink Floyd song and applied a stimulus reconstruction approach previously used in the speech domain. We successfully reconstructed a recognizable song from direct neural recordings and quantified the impact of different factors on decoding accuracy. Combining encoding and decoding analyses, we found a right-hemisphere dominance for music perception with a primary role of the superior temporal gyrus (STG), evidenced a new STG subregion tuned to musical rhythm, and defined an anterior-posterior STG organization exhibiting sustained and onset responses to musical elements. Our findings show the feasibility of applying predictive modeling on short datasets acquired in single patients, paving the way for adding musical elements to brain-computer interface (BCI) applications.

Spatio-temporal evolution of human neural activity during visually cued hand movements.

Making hand movements in response to visual cues is common in daily life. It has been well known that this process activates multiple areas in the brain, but how these neural activations progress across space and time remains largely unknown. Taking advantage of intracranial electroencephalographic (iEEG) recordings using depth and subdural electrodes from 36 human subjects using the same task, we applied single-trial and cross-trial analyses to high-frequency iEEG activity. The results show that the neural activation was widely distributed across the human brain both within and on the surface of the brain, and focused specifically on certain areas in the parietal, frontal, and occipital lobes, where parietal lobes present significant left lateralization on the activation. We also demonstrate temporal differences across these brain regions. Finally, we evaluated the degree to which the timing of activity within these regions was related to sensory or motor function. The findings of this study promote the understanding of task-related neural processing of the human brain, and may provide important insights for translational applications.

Unexpected sound omissions are signaled in human posterior superior temporal gyrus: an intracranial study.

Context modulates sensory neural activations enhancing perceptual and behavioral performance and reducing prediction errors. However, the mechanism of when and where these high-level expectations act on sensory processing is unclear. Here, we isolate the effect of expectation absent of any auditory evoked activity by assessing the response to omitted expected sounds. Electrocorticographic signals were recorded directly from subdural electrode grids placed over the superior temporal gyrus (STG). Subjects listened to a predictable sequence of syllables, with some infrequently omitted. We found high-frequency band activity (HFA, 70-170 Hz) in response to omissions, which overlapped with a posterior subset of auditory-active electrodes in STG. Heard syllables could be distinguishable reliably from STG, but not the identity of the omitted stimulus. Both omission- and target-detection responses were also observed in the prefrontal cortex. We propose that the posterior STG is central for implementing predictions in the auditory environment. HFA omission responses in this region appear to index mismatch-signaling or salience detection processes.

Modulation in cortical excitability disrupts information transfer in perceptual-level stimulus processing.

Despite significant interest in the neural underpinnings of behavioral variability, little light has been shed on the cortical mechanism underlying the failure to respond to perceptual-level stimuli. We hypothesized that cortical activity resulting from perceptual-level stimuli is sensitive to the moment-to-moment fluctuations in cortical excitability, and thus may not suffice to produce a behavioral response. We tested this hypothesis using electrocorticographic recordings to follow the propagation of cortical activity in six human subjects that responded to perceptual-level auditory stimuli. Here we show that for presentations that did not result in a behavioral response, the likelihood of cortical activity decreased from auditory cortex to motor cortex, and was related to reduced local cortical excitability. Cortical excitability was quantified using instantaneous voltage during a short window prior to cortical activity onset. Therefore, when humans are presented with an auditory stimulus close to perceptual-level threshold, moment-by-moment fluctuations in cortical excitability determine whether cortical responses to sensory stimulation successfully connect auditory input to a resultant behavioral response.

Within-subject reaction time variability: Role of cortical networks and underlying neurophysiological mechanisms.

Variations in reaction time are a ubiquitous characteristic of human behavior. Extensively documented, they have been successfully modeled using parameters of the subject or the task, but the neural basis of behavioral reaction time that varies within the same subject and the same task has been minimally studied. In this paper, we investigate behavioral reaction time variance using 28 datasets of direct cortical recordings in humans who engaged in four different types of simple sensory-motor reaction time tasks. Using a previously described technique that can identify the onset of population-level cortical activity and a novel functional connectivity algorithm described herein, we show that the cumulative latency difference of population-level neural activity across the task-related cortical network can explain up to 41% of the trial-by-trial variance in reaction time. Furthermore, we show that reaction time variance may primarily be due to the latencies in specific brain regions and demonstrate that behavioral latency variance is accumulated across the whole task-related cortical network. Our results suggest that population-level neural activity monotonically increases prior to movement execution, and that trial-by-trial changes in that increase are, in part, accounted for by inhibitory activity indexed by low-frequency oscillations. This pre-movement neural activity explains 19% of the measured variance in neural latencies in our data. Thus, our study provides a mechanistic explanation for a sizable fraction of behavioral reaction time when the subject's task is the same from trial to trial.

Facephenes and rainbows: Causal evidence for functional and anatomical specificity of face and color processing in the human brain.

Neuroscientists have long debated whether some regions of the human brain are exclusively engaged in a single specific mental process. Consistent with this view, fMRI has revealed cortical regions that respond selectively to certain stimulus classes such as faces. However, results from multivoxel pattern analyses (MVPA) challenge this view by demonstrating that category-selective regions often contain information about "nonpreferred" stimulus dimensions. But is this nonpreferred information causally relevant to behavior? Here we report a rare opportunity to test this question in a neurosurgical patient implanted for clinical reasons with strips of electrodes along his fusiform gyri. Broadband gamma electrocorticographic responses in multiple adjacent electrodes showed strong selectivity for faces in a region corresponding to the fusiform face area (FFA), and preferential responses to color in a nearby site, replicating earlier reports. To test the causal role of these regions in the perception of nonpreferred dimensions, we then electrically stimulated individual sites while the patient viewed various objects. When stimulated in the FFA, the patient reported seeing an illusory face (or "facephene"), independent of the object viewed. Similarly, stimulation of color-preferring sites produced illusory "rainbows." Crucially, the patient reported no change in the object viewed, apart from the facephenes and rainbows apparently superimposed on them. The functional and anatomical specificity of these effects indicate that some cortical regions are exclusively causally engaged in a single specific mental process, and prompt caution about the widespread assumption that any information scientists can decode from the brain is causally relevant to behavior.

Spatiotemporal dynamics of word retrieval in speech production revealed by cortical high-frequency band activity.

Word retrieval is core to language production and relies on complementary processes: the rapid activation of lexical and conceptual representations and word selection, which chooses the correct word among semantically related competitors. Lexical and conceptual activation is measured by semantic priming. In contrast, word selection is indexed by semantic interference and is hampered in semantically homogeneous (HOM) contexts. We examined the spatiotemporal dynamics of these complementary processes in a picture naming task with blocks of semantically heterogeneous (HET) or HOM stimuli. We used electrocorticography data obtained from frontal and temporal cortices, permitting detailed spatiotemporal analysis of word retrieval processes. A semantic interference effect was observed with naming latencies longer in HOM versus HET blocks. Cortical response strength as indexed by high-frequency band (HFB) activity (70-150 Hz) amplitude revealed effects linked to lexical-semantic activation and word selection observed in widespread regions of the cortical mantle. Depending on the subsecond timing and cortical region, HFB indexed semantic interference (i.e., more activity in HOM than HET blocks) or semantic priming effects (i.e., more activity in HET than HOM blocks). These effects overlapped in time and space in the left posterior inferior temporal gyrus and the left prefrontal cortex. The data do not support a modular view of word retrieval in speech production but rather support substantial overlap of lexical-semantic activation and word selection mechanisms in the brain.

Neural correlate of the construction of sentence meaning.

The neural processes that underlie your ability to read and understand this sentence are unknown. Sentence comprehension occurs very rapidly, and can only be understood at a mechanistic level by discovering the precise sequence of underlying computational and neural events. However, we have no continuous and online neural measure of sentence processing with high spatial and temporal resolution. Here we report just such a measure: intracranial recordings from the surface of the human brain show that neural activity, indexed by γ-power, increases monotonically over the course of a sentence as people read it. This steady increase in activity is absent when people read and remember nonword-lists, despite the higher cognitive demand entailed, ruling out accounts in terms of generic attention, working memory, and cognitive load. Response increases are lower for sentence structure without meaning ("Jabberwocky" sentences) and word meaning without sentence structure (word-lists), showing that this effect is not explained by responses to syntax or word meaning alone. Instead, the full effect is found only for sentences, implicating compositional processes of sentence understanding, a striking and unique feature of human language not shared with animal communication systems. This work opens up new avenues for investigating the sequence of neural events that underlie the construction of linguistic meaning.

Optimal referencing for stereo-electroencephalographic (SEEG) recordings.

Stereo-electroencephalography (SEEG) is an intracranial recording technique in which depth electrodes are inserted in the brain as part of presurgical assessments for invasive brain surgery. SEEG recordings can tap into neural signals across the entire brain and thereby sample both cortical and subcortical sites. However, even though signal referencing is important for proper assessment of SEEG signals, no previous study has comprehensively evaluated the optimal referencing method for SEEG. In our study, we recorded SEEG data from 15 human subjects during a motor task, referencing them against the average of two white matter contacts (monopolar reference). We then subjected these signals to 5 different re-referencing approaches: common average reference (CAR), gray-white matter reference (GWR), electrode shaft reference (ESR), bipolar reference, and Laplacian reference. The results from three different signal quality metrics suggest the use of the Laplacian re-reference for study of local population-level activity and low-frequency oscillatory activity.

Contralesional Brain-Computer Interface Control of a Powered Exoskeleton for Motor Recovery in Chronic Stroke Survivors.

BACKGROUND AND PURPOSE: There are few effective therapies to achieve functional recovery from motor-related disabilities affecting the upper limb after stroke. This feasibility study tested whether a powered exoskeleton driven by a brain-computer interface (BCI), using neural activity from the unaffected cortical hemisphere, could affect motor recovery in chronic hemiparetic stroke survivors. This novel system was designed and configured for a home-based setting to test the feasibility of BCI-driven neurorehabilitation in outpatient environments. METHODS: Ten chronic hemiparetic stroke survivors with moderate-to-severe upper-limb motor impairment (mean Action Research Arm Test=13.4) used a powered exoskeleton that opened and closed the affected hand using spectral power from electroencephalographic signals from the unaffected hemisphere associated with imagined hand movements of the paretic limb. Patients used the system at home for 12 weeks. Motor function was evaluated before, during, and after the treatment. RESULTS: Across patients, our BCI-driven approach resulted in a statistically significant average increase of 6.2 points in the Action Research Arm Test. This behavioral improvement significantly correlated with improvements in BCI control. Secondary outcomes of grasp strength, Motricity Index, and the Canadian Occupational Performance Measure also significantly improved. CONCLUSIONS: The findings demonstrate the therapeutic potential of a BCI-driven neurorehabilitation approach using the unaffected hemisphere in this uncontrolled sample of chronic stroke survivors. They also demonstrate that BCI-driven neurorehabilitation can be effectively delivered in the home environment, thus increasing the probability of future clinical translation. CLINICAL TRIAL REGISTRATION: URL: http://www.clinicaltrials.gov. Unique identifier: NCT02552368.

Instantaneous voltage as an alternative to power- and phase-based interpretation of oscillatory brain activity.

For decades, oscillatory brain activity has been characterized primarily by measurements of power and phase. While many studies have linked those measurements to cortical excitability, their relationship to each other and to the physiological underpinnings of excitability is unclear. The recently proposed Function-through-Biased-Oscillations (FBO) hypothesis (Schalk, 2015) addressed these issues by suggesting that the voltage potential at the cortical surface directly reflects the excitability of cortical populations, that this voltage is rhythmically driven away from a low resting potential (associated with depolarized cortical populations) towards positivity (associated with hyperpolarized cortical populations). This view explains how oscillatory power and phase together influence the instantaneous voltage potential that directly regulates cortical excitability. This implies that the alternative measurement of instantaneous voltage of oscillatory activity should better predict cortical excitability compared to either of the more traditional measurements of power or phase. Using electrocorticographic (ECoG) data from 28 human subjects, the results of our study confirm this prediction: compared to oscillatory power and phase, the instantaneous voltage explained 20% and 31% more of the variance in broadband gamma, respectively, and power and phase together did not produce better predictions than the instantaneous voltage. These results synthesize the previously separate power- and phase-based interpretations and associate oscillatory activity directly with a physiological interpretation of cortical excitability. This alternative view has implications for the interpretation of studies of oscillatory activity and for current theories of cortical information transmission.

Spontaneous Decoding of the Timing and Content of Human Object Perception from Cortical Surface Recordings Reveals Complementary Information in the Event-Related Potential and Broadband Spectral Change.

The link between object perception and neural activity in visual cortical areas is a problem of fundamental importance in neuroscience. Here we show that electrical potentials from the ventral temporal cortical surface in humans contain sufficient information for spontaneous and near-instantaneous identification of a subject's perceptual state. Electrocorticographic (ECoG) arrays were placed on the subtemporal cortical surface of seven epilepsy patients. Grayscale images of faces and houses were displayed rapidly in random sequence. We developed a template projection approach to decode the continuous ECoG data stream spontaneously, predicting the occurrence, timing and type of visual stimulus. In this setting, we evaluated the independent and joint use of two well-studied features of brain signals, broadband changes in the frequency power spectrum of the potential and deflections in the raw potential trace (event-related potential; ERP). Our ability to predict both the timing of stimulus onset and the type of image was best when we used a combination of both the broadband response and ERP, suggesting that they capture different and complementary aspects of the subject's perceptual state. Specifically, we were able to predict the timing and type of 96% of all stimuli, with less than 5% false positive rate and a ~20ms error in timing.

Proceedings of the Eighth International Workshop on Advances in Electrocorticography.

Excerpted proceedings of the Eighth International Workshop on Advances in Electrocorticography (ECoG), which convened October 15-16, 2015 in Chicago, IL, are presented. The workshop series has become the foremost gathering to present current basic and clinical research in subdural brain signal recording and analysis.

Proceedings of the Seventh International Workshop on Advances in Electrocorticography.

The Seventh International Workshop on Advances in Electrocorticography (ECoG) convened in Washington, DC, on November 13-14, 2014. Electrocorticography-based research continues to proliferate widely across basic science and clinical disciplines. The 2014 workshop highlighted advances in neurolinguistics, brain-computer interface, functional mapping, and seizure termination facilitated by advances in the recording and analysis of the ECoG signal. The following proceedings document summarizes the content of this successful multidisciplinary gathering.

Spatial and temporal relationships of electrocorticographic alpha and gamma activity during auditory processing.

Neuroimaging approaches have implicated multiple brain sites in musical perception, including the posterior part of the superior temporal gyrus and adjacent perisylvian areas. However, the detailed spatial and temporal relationship of neural signals that support auditory processing is largely unknown. In this study, we applied a novel inter-subject analysis approach to electrophysiological signals recorded from the surface of the brain (electrocorticography (ECoG)) in ten human subjects. This approach allowed us to reliably identify those ECoG features that were related to the processing of a complex auditory stimulus (i.e., continuous piece of music) and to investigate their spatial, temporal, and causal relationships. Our results identified stimulus-related modulations in the alpha (8-12 Hz) and high gamma (70-110 Hz) bands at neuroanatomical locations implicated in auditory processing. Specifically, we identified stimulus-related ECoG modulations in the alpha band in areas adjacent to primary auditory cortex, which are known to receive afferent auditory projections from the thalamus (80 of a total of 15,107 tested sites). In contrast, we identified stimulus-related ECoG modulations in the high gamma band not only in areas close to primary auditory cortex but also in other perisylvian areas known to be involved in higher-order auditory processing, and in superior premotor cortex (412/15,107 sites). Across all implicated areas, modulations in the high gamma band preceded those in the alpha band by 280 ms, and activity in the high gamma band causally predicted alpha activity, but not vice versa (Granger causality, p<1e(-8)). Additionally, detailed analyses using Granger causality identified causal relationships of high gamma activity between distinct locations in early auditory pathways within superior temporal gyrus (STG) and posterior STG, between posterior STG and inferior frontal cortex, and between STG and premotor cortex. Evidence suggests that these relationships reflect direct cortico-cortical connections rather than common driving input from subcortical structures such as the thalamus. In summary, our inter-subject analyses defined the spatial and temporal relationships between music-related brain activity in the alpha and high gamma bands. They provide experimental evidence supporting current theories about the putative mechanisms of alpha and gamma activity, i.e., reflections of thalamo-cortical interactions and local cortical neural activity, respectively, and the results are also in agreement with existing functional models of auditory processing.

Proceedings of the Fifth International Workshop on Advances in Electrocorticography.

The Fifth International Workshop on Advances in Electrocorticography convened in San Diego, CA, on November 7-8, 2013. Advancements in methodology, implementation, and commercialization across both research and in the interval year since the last workshop were the focus of the gathering. Electrocorticography (ECoG) is now firmly established as a preferred signal source for advanced research in functional, cognitive, and neuroprosthetic domains. Published output in ECoG fields has increased tenfold in the past decade. These proceedings attempt to summarize the state of the art.

Feasibility of Stereo EEG Based Brain Computer Interfacing in An Adult and Pediatric Cohort.

Introduction: Stereoelectroencephalography (sEEG) is a mesoscale intracranial monitoring method which records from the brain volumetrically with depth electrodes. Implementation of sEEG in BCI has not been well-described across a diverse patient cohort. Methods: Across eighteen subjects, channels with high frequency broadband (HFB, 65-115Hz) power increases during hand, tongue, or foot movements during a motor screening task were provided real-time feedback based on these HFB power changes to control a cursor on a screen. Results: Seventeen subjects established successful control of the overt motor BCI, but only nine were able to control imagery BCI with ≥ 80% accuracy. In successful imagery BCI, HFB power in the two target conditions separated into distinct subpopulations, which appear to engage unique subnetworks of the motor cortex compared to cued movement or imagery alone. Conclusion: sEEG-based motor BCI utilizing overt movement and kinesthetic imagery is robust across patient ages and cortical regions with substantial differences in learning proficiency between real or imagined movement.

Anatomical registration of intracranial electrodes. Robust model-based localization and deformable smooth brain-shift compensation methods.

BACKGROUND: Intracranial electrodes are typically localized from post-implantation CT artifacts. Automatic algorithms localizing low signal-to-noise ratio artifacts and high-density electrode arrays are missing. Additionally, implantation of grids/strips introduces brain deformations, resulting in registration errors when fusing post-implantation CT and pre-implantation MR images. Brain-shift compensation methods project electrode coordinates to cortex, but either fail to produce smooth solutions or do not account for brain deformations. NEW METHODS: We first introduce GridFit, a model-based fitting approach that simultaneously localizes all electrodes' CT artifacts in grids, strips, or depth arrays. Second, we present CEPA, a brain-shift compensation algorithm combining orthogonal-based projections, spring-mesh models, and spatial regularization constraints. RESULTS: We tested GridFit on ∼6000 simulated scenarios. The localization of CT artifacts showed robust performance under difficult scenarios, such as noise, overlaps, and high-density implants (<1 mm errors). Validation with data from 20 challenging patients showed 99% accurate localization of the electrodes (3160/3192). We tested CEPA brain-shift compensation with data from 15 patients. Projections accounted for simple mechanical deformation principles with < 0.4 mm errors. The inter-electrode distances smoothly changed across neighbor electrodes, while changes in inter-electrode distances linearly increased with projection distance. COMPARISON WITH EXISTING METHODS: GridFit succeeded in difficult scenarios that challenged available methods and outperformed visual localization by preserving the inter-electrode distance. CEPA registration errors were smaller than those obtained for well-established alternatives. Additionally, modeling resting-state high-frequency activity in five patients further supported CEPA. CONCLUSION: GridFit and CEPA are versatile tools for registering intracranial electrode coordinates, providing highly accurate results even in the most challenging implantation scenarios. The methods are implemented in the iElectrodes open-source toolbox.

A low-frequency oscillatory neural signal in humans encodes a developing decision variable.

We often make decisions based on sensory evidence that is accumulated over a period of time. How the evidence for such decisions is represented in the brain and how such a neural representation is used to guide a subsequent action are questions of considerable interest to decision sciences. The neural correlates of developing perceptual decisions have been thoroughly investigated in the oculomotor system of macaques who communicated their decisions using an eye movement. It has been found that the evidence informing a decision to make an eye movement is in part accumulated within the same oculomotor circuits that signal the upcoming eye movement. Recent evidence suggests that the somatomotor system may exhibit an analogous property for choices made using a hand movement. To investigate this possibility, we engaged humans in a decision task in which they integrated discrete quanta of sensory information over a period of time and signaled their decision using a hand movement or an eye movement. The discrete form of the sensory evidence allowed us to infer the decision variable on which subjects base their decision on each trial and to assess the neural processes related to each quantum of the incoming decision evidence. We found that a low-frequency electrophysiological signal recorded over centroparietal regions strongly encodes the decision variable inferred in this task, and that it does so specifically for hand movement choices. The signal ramps up with a rate that is proportional to the decision variable, remains graded by the decision variable throughout the delay period, reaches a common peak shortly before a hand movement, and falls off shortly after the hand movement. Furthermore, the signal encodes the polarity of each evidence quantum, with a short latency, and retains the response level over time. Thus, this neural signal shows properties of evidence accumulation. These findings suggest that the decision-related effects observed in the oculomotor system of the monkey during eye movement choices may share the same basic properties with the decision-related effects in the somatomotor system of humans during hand movement choices.

Proceedings of the Fourth International Workshop on Advances in Electrocorticography.

The Fourth International Workshop on Advances in Electrocorticography (ECoG) convened in New Orleans, LA, on October 11-12, 2012. The proceedings of the workshop serves as an accurate record of the most contemporary clinical and experimental work on brain surface recording and represents the insights of a unique multidisciplinary ensemble of expert clinicians and scientists. Presentations covered a broad range of topics, including innovations in passive functional mapping, increased understanding of pathologic high-frequency oscillations, evolving sensor technologies, a human trial of ECoG-driven brain-machine interface, as well as fresh insights into brain electrical stimulation.