Online speech synthesis using a chronically implanted brain-computer interface in an individual with ALS.

Brain-computer interfaces (BCIs) that reconstruct and synthesize speech using brain activity recorded with intracranial electrodes may pave the way toward novel communication interfaces for people who have lost their ability to speak, or who are at high risk of losing this ability, due to neurological disorders. Here, we report online synthesis of intelligible words using a chronically implanted brain-computer interface (BCI) in a man with impaired articulation due to ALS, participating in a clinical trial (ClinicalTrials.gov, NCT03567213) exploring different strategies for BCI communication. The 3-stage approach reported here relies on recurrent neural networks to identify, decode and synthesize speech from electrocorticographic (ECoG) signals acquired across motor, premotor and somatosensory cortices. We demonstrate a reliable BCI that synthesizes commands freely chosen and spoken by the participant from a vocabulary of 6 keywords previously used for decoding commands to control a communication board. Evaluation of the intelligibility of the synthesized speech indicates that 80% of the words can be correctly recognized by human listeners. Our results show that a speech-impaired individual with ALS can use a chronically implanted BCI to reliably produce synthesized words while preserving the participant's voice profile, and provide further evidence for the stability of ECoG for speech-based BCIs.

A click-based electrocorticographic brain-computer interface enables long-term high-performance switch-scan spelling.

Background: Brain-computer interfaces (BCIs) can restore communication in movement- and/or speech-impaired individuals by enabling neural control of computer typing applications. Single command "click" decoders provide a basic yet highly functional capability. Methods: We sought to test the performance and long-term stability of click-decoding using a chronically implanted high density electrocorticographic (ECoG) BCI with coverage of the sensorimotor cortex in a human clinical trial participant (ClinicalTrials.gov, NCT03567213) with amyotrophic lateral sclerosis (ALS). We trained the participant's click decoder using a small amount of training data (< 44 minutes across four days) collected up to 21 days prior to BCI use, and then tested it over a period of 90 days without any retraining or updating. Results: Using this click decoder to navigate a switch-scanning spelling interface, the study participant was able to maintain a median spelling rate of 10.2 characters per min. Though a transient reduction in signal power modulation interrupted testing with this fixed model, a new click decoder achieved comparable performance despite being trained with even less data (< 15 min, within one day). Conclusion: These results demonstrate that a click decoder can be trained with a small ECoG dataset while retaining robust performance for extended periods, providing functional text-based communication to BCI users.

Stable Decoding from a Speech BCI Enables Control for an Individual with ALS without Recalibration for 3 Months.

Brain-computer interfaces (BCIs) can be used to control assistive devices by patients with neurological disorders like amyotrophic lateral sclerosis (ALS) that limit speech and movement. For assistive control, it is desirable for BCI systems to be accurate and reliable, preferably with minimal setup time. In this study, a participant with severe dysarthria due to ALS operates computer applications with six intuitive speech commands via a chronic electrocorticographic (ECoG) implant over the ventral sensorimotor cortex. Speech commands are accurately detected and decoded (median accuracy: 90.59%) throughout a 3-month study period without model retraining or recalibration. Use of the BCI does not require exogenous timing cues, enabling the participant to issue self-paced commands at will. These results demonstrate that a chronically implanted ECoG-based speech BCI can reliably control assistive devices over long time periods with only initial model training and calibration, supporting the feasibility of unassisted home use.

Online speech synthesis using a chronically implanted brain-computer interface in an individual with ALS.

Recent studies have shown that speech can be reconstructed and synthesized using only brain activity recorded with intracranial electrodes, but until now this has only been done using retrospective analyses of recordings from able-bodied patients temporarily implanted with electrodes for epilepsy surgery. Here, we report online synthesis of intelligible words using a chronically implanted brain-computer interface (BCI) in a clinical trial participant (ClinicalTrials.gov, NCT03567213) with dysarthria due to amyotrophic lateral sclerosis (ALS). We demonstrate a reliable BCI that synthesizes commands freely chosen and spoken by the user from a vocabulary of 6 keywords originally designed to allow intuitive selection of items on a communication board. Our results show for the first time that a speech-impaired individual with ALS can use a chronically implanted BCI to reliably produce synthesized words that are intelligible to human listeners while preserving the participants voice profile.

Lexicality-Modulated Influence of Auditory Cortex on Subthalamic Nucleus During Motor Planning for Speech.

Speech requires successful information transfer within cortical-basal ganglia loop circuits to produce the desired acoustic output. For this reason, up to 90% of Parkinson's disease patients experience impairments of speech articulation. Deep brain stimulation (DBS) is highly effective in controlling the symptoms of Parkinson's disease, sometimes alongside speech improvement, but subthalamic nucleus (STN) DBS can also lead to decreases in semantic and phonological fluency. This paradox demands better understanding of the interactions between the cortical speech network and the STN, which can be investigated with intracranial EEG recordings collected during DBS implantation surgery. We analyzed the propagation of high-gamma activity between STN, superior temporal gyrus (STG), and ventral sensorimotor cortices during reading aloud via event-related causality, a method that estimates strengths and directionalities of neural activity propagation. We employed a newly developed bivariate smoothing model based on a two-dimensional moving average, which is optimal for reducing random noise while retaining a sharp step response, to ensure precise embedding of statistical significance in the time-frequency space. Sustained and reciprocal neural interactions between STN and ventral sensorimotor cortex were observed. Moreover, high-gamma activity propagated from the STG to the STN prior to speech onset. The strength of this influence was affected by the lexical status of the utterance, with increased activity propagation during word versus pseudoword reading. These unique data suggest a potential role for the STN in the feedforward control of speech.

Towards optimizing single pulse electrical stimulation: High current intensity, short pulse width stimulation most effectively elicits evoked potentials.

BACKGROUND: While single pulse electrical stimulation (SPES) is increasingly used to study effective connectivity, the effects of varying stimulation parameters on the resulting cortico-cortical evoked potentials (CCEPs) have not been systematically explored. OBJECTIVE: We sought to understand the interacting effects of stimulation pulse width, current intensity, and charge on CCEPs through an extensive testing of this parameter space and analysis of several response metrics. METHODS: We conducted SPES in 11 patients undergoing intracranial EEG monitoring using five combinations of current intensity (1.5, 2.0, 3.0, 5.0, and 7.5 mA) and pulse width at each of three charges (0.750, 1.125, and 1.500 µC/phase) to study how CCEP amplitude, distribution, latency, morphology, and stimulus artifact amplitude vary with each parameter. RESULTS: Stimulations with a greater charge or a greater current intensity and shorter pulse width at a given charge generally resulted in greater CCEP amplitudes and spatial distributions, shorter latencies, and increased waveform correlation. These effects interacted such that stimulations with the lowest charge and highest current intensities resulted in greater response amplitudes and spatial distributions than stimulations with the highest charge and lowest current intensities. Stimulus artifact amplitude increased with charge, but this could be mitigated by using shorter pulse widths. CONCLUSIONS: Our results indicate that individual combinations of current intensity and pulse width, in addition to charge, are important determinants of CCEP magnitude, morphology, and spatial extent. Together, these findings suggest that high current intensity, short pulse width stimulations are optimal SPES settings for eliciting strong and consistent responses while minimizing charge.

A supervised data-driven spatial filter denoising method for speech artifacts in intracranial electrophysiological recordings.

Neurosurgical procedures that enable direct brain recordings in awake patients offer unique opportunities to explore the neurophysiology of human speech. The scarcity of these opportunities and the altruism of participating patients compel us to apply the highest rigor to signal analysis. Intracranial electroencephalography (iEEG) signals recorded during overt speech can contain a speech artifact that tracks the fundamental frequency (F0) of the participant's voice, involving the same high-gamma frequencies that are modulated during speech production and perception. To address this artifact, we developed a spatial-filtering approach to identify and remove acoustic-induced contaminations of the recorded signal. We found that traditional reference schemes jeopardized signal quality, whereas our data-driven method denoised the recordings while preserving underlying neural activity.

Cross-task specificity and within-task invariance of cognitive control processes.

Cognitive control involves flexibly combining multiple sensory inputs with task-dependent goals during decision making. Several tasks involving conflicting sensory inputs and motor outputs have been proposed to examine cognitive control, including the Stroop, Flanker, and multi-source interference task. Because these tasks have been studied independently, it remains unclear whether the neural signatures of cognitive control reflect abstract control mechanisms or specific combinations of sensory and behavioral aspects of each task. To address these questions, we record invasive neurophysiological signals from 16 patients with pharmacologically intractable epilepsy and compare neural responses within and between tasks. Neural signals differ between incongruent and congruent conditions, showing strong modulation by conflicting task demands. These neural signals are mostly specific to each task, generalizing within a task but not across tasks. These results highlight the complex interplay between sensory inputs, motor outputs, and task demands underlying cognitive control processes.

The dynamics of cortical interactions in visual recognition of object category: living versus nonliving.

Noninvasive brain imaging studies have shown that higher visual processing of objects occurs in neural populations that are separable along broad semantic categories, particularly living versus nonliving objects. However, because of their limited temporal resolution, these studies have not been able to determine whether broad semantic categories are also reflected in the dynamics of neural interactions within cortical networks. We investigated the time course of neural propagation among cortical areas activated during object naming in 12 patients implanted with subdural electrode grids prior to epilepsy surgery, with a special focus on the visual recognition phase of the task. Analysis of event-related causality revealed significantly stronger neural propagation among sites within ventral temporal lobe (VTL) at early latencies, around 250 ms, for living objects compared to nonliving objects. Differences in other features, including familiarity, visual complexity, and age of acquisition, did not significantly change the patterns of neural propagation. Our findings suggest that the visual processing of living objects relies on stronger causal interactions among sites within VTL, perhaps reflecting greater integration of visual feature processing. In turn, this may help explain the fragility of naming living objects in neurological diseases affecting VTL.

Cortico-cortical evoked potentials in response to varying stimulation intensity improves seizure localization.

OBJECTIVE: As single pulse electrical stimulation (SPES) is increasingly utilized to help localize the seizure onset zone (SOZ), it is important to understand how stimulation intensity can affect the ability to use cortico-cortical evoked potentials (CCEPs) to delineate epileptogenic regions. METHODS: We studied 15 drug-resistant epilepsy patients undergoing intracranial EEG monitoring and SPES with titrations of stimulation intensity. The N1 amplitude and distribution of CCEPs elicited in the SOZ and non-seizure onset zone (nSOZ) were quantified at each intensity. The separability of the SOZ and nSOZ using N1 amplitudes was compared between models using responses to titrations, responses to one maximal intensity, or both. RESULTS: At 2 mA and above, the increase in N1 amplitude with current intensity was greater for responses within the SOZ, and SOZ response distribution was maximized by 4-6 mA. Models incorporating titrations achieved better separability of SOZ and nSOZ compared to those using one maximal intensity. CONCLUSIONS: We demonstrated that differences in CCEP amplitude over a range of current intensities can improve discriminability of SOZ regions. SIGNIFICANCE: This study provides insight into the underlying excitability of the SOZ and how differences in current-dependent amplitudes of CCEPs may be used to help localize epileptogenic sites.

Source-sink connectivity: a novel interictal EEG marker for seizure localization.

Over 15 million epilepsy patients worldwide have drug-resistant epilepsy. Successful surgery is a standard of care treatment but can only be achieved through complete resection or disconnection of the epileptogenic zone, the brain region(s) where seizures originate. Surgical success rates vary between 20% and 80%, because no clinically validated biological markers of the epileptogenic zone exist. Localizing the epileptogenic zone is a costly and time-consuming process, which often requires days to weeks of intracranial EEG (iEEG) monitoring. Clinicians visually inspect iEEG data to identify abnormal activity on individual channels occurring immediately before seizures or spikes that occur interictally (i.e. between seizures). In the end, the clinical standard mainly relies on a small proportion of the iEEG data captured to assist in epileptogenic zone localization (minutes of seizure data versus days of recordings), missing opportunities to leverage these largely ignored interictal data to better diagnose and treat patients. IEEG offers a unique opportunity to observe epileptic cortical network dynamics but waiting for seizures increases patient risks associated with invasive monitoring. In this study, we aimed to leverage interictal iEEG data by developing a new network-based interictal iEEG marker of the epileptogenic zone. We hypothesized that when a patient is not clinically seizing, it is because the epileptogenic zone is inhibited by other regions. We developed an algorithm that identifies two groups of nodes from the interictal iEEG network: those that are continuously inhibiting a set of neighbouring nodes ('sources') and the inhibited nodes themselves ('sinks'). Specifically, patient-specific dynamical network models were estimated from minutes of iEEG and their connectivity properties revealed top sources and sinks in the network, with each node being quantified by source-sink metrics. We validated the algorithm in a retrospective analysis of 65 patients. The source-sink metrics identified epileptogenic regions with 73% accuracy and clinicians agreed with the algorithm in 93% of seizure-free patients. The algorithm was further validated by using the metrics of the annotated epileptogenic zone to predict surgical outcomes. The source-sink metrics predicted outcomes with an accuracy of 79% compared to an accuracy of 43% for clinicians' predictions (surgical success rate of this dataset). In failed outcomes, we identified brain regions with high metrics that were untreated. When compared with high frequency oscillations, the most commonly proposed interictal iEEG feature for epileptogenic zone localization, source-sink metrics outperformed in predictive power (by a factor of 1.2), suggesting they may be an interictal iEEG fingerprint of the epileptogenic zone.

Shared Control of Bimanual Robotic Limbs With a Brain-Machine Interface for Self-Feeding.

Advances in intelligent robotic systems and brain-machine interfaces (BMI) have helped restore functionality and independence to individuals living with sensorimotor deficits; however, tasks requiring bimanual coordination and fine manipulation continue to remain unsolved given the technical complexity of controlling multiple degrees of freedom (DOF) across multiple limbs in a coordinated way through a user input. To address this challenge, we implemented a collaborative shared control strategy to manipulate and coordinate two Modular Prosthetic Limbs (MPL) for performing a bimanual self-feeding task. A human participant with microelectrode arrays in sensorimotor brain regions provided commands to both MPLs to perform the self-feeding task, which included bimanual cutting. Motor commands were decoded from bilateral neural signals to control up to two DOFs on each MPL at a time. The shared control strategy enabled the participant to map his four-DOF control inputs, two per hand, to as many as 12 DOFs for specifying robot end effector position and orientation. Using neurally-driven shared control, the participant successfully and simultaneously controlled movements of both robotic limbs to cut and eat food in a complex bimanual self-feeding task. This demonstration of bimanual robotic system control via a BMI in collaboration with intelligent robot behavior has major implications for restoring complex movement behaviors for those living with sensorimotor deficits.

Characteristics and stability of sensorimotor activity driven by isolated-muscle group activation in a human with tetraplegia.

Understanding the cortical representations of movements and their stability can shed light on improved brain-machine interface (BMI) approaches to decode these representations without frequent recalibration. Here, we characterize the spatial organization (somatotopy) and stability of the bilateral sensorimotor map of forearm muscles in an incomplete-high spinal-cord injury study participant implanted bilaterally in the primary motor and sensory cortices with Utah microelectrode arrays (MEAs). We built representation maps by recording bilateral multiunit activity (MUA) and surface electromyography (EMG) as the participant executed voluntary contractions of the extensor carpi radialis (ECR), and attempted motions in the flexor carpi radialis (FCR), which was paralytic. To assess stability, we repeatedly mapped and compared left- and right-wrist-extensor-related activity throughout several sessions, comparing somatotopy of active electrodes, as well as neural signals both at the within-electrode (multiunit) and cross-electrode (network) levels. Wrist motions showed significant activation in motor and sensory cortical electrodes. Within electrodes, firing strength stability diminished as the time increased between consecutive measurements (hours within a session, or days across sessions), with higher stability observed in sensory cortex than in motor, and in the contralateral hemisphere than in the ipsilateral. However, we observed no differences at network level, and no evidence of decoding instabilities for wrist EMG, either across timespans of hours or days, or across recording area. While map stability differs between brain area and hemisphere at multiunit/electrode level, these differences are nullified at ensemble level.

Stimulating native seizures with neural resonance: a new approach to localize the seizure onset zone.

Successful outcomes in epilepsy surgery rely on the accurate localization of the seizure onset zone. Localizing the seizure onset zone is often a costly and time-consuming process wherein a patient undergoes intracranial EEG monitoring, and a team of clinicians wait for seizures to occur. Clinicians then analyse the intracranial EEG before each seizure onset to identify the seizure onset zone and localization accuracy increases when more seizures are captured. In this study, we develop a new approach to guide clinicians to actively elicit seizures with electrical stimulation. We propose that a brain region belongs to the seizure onset zone if a periodic stimulation at a particular frequency produces large amplitude oscillations in the intracranial EEG network that propagate seizure activity. Such responses occur when there is 'resonance' in the intracranial EEG network, and the resonant frequency can be detected by observing a sharp peak in the magnitude versus frequency response curve, called a Bode plot. To test our hypothesis, we analysed single-pulse electrical stimulation response data in 32 epilepsy patients undergoing intracranial EEG monitoring. For each patient and each stimulated brain region, we constructed a Bode plot by estimating a transfer function model from the intracranial EEG 'impulse' or single-pulse electrical stimulation response. The Bode plots were then analysed for evidence of resonance. First, we showed that when Bode plot features were used as a marker of the seizure onset zone, it distinguished successful from failed surgical outcomes with an area under the curve of 0.83, an accuracy that surpassed current methods of analysis with cortico-cortical evoked potential amplitude and cortico-cortical spectral responses. Then, we retrospectively showed that three out of five native seizures accidentally triggered in four patients during routine periodic stimulation at a given frequency corresponded to a resonant peak in the Bode plot. Last, we prospectively stimulated peak resonant frequencies gleaned from the Bode plots to elicit seizures in six patients, and this resulted in an induction of three seizures and three auras in these patients. These findings suggest neural resonance as a new biomarker of the seizure onset zone that can guide clinicians in eliciting native seizures to more quickly and accurately localize the seizure onset zone.

Significance of event related causality (ERC) in eloquent neural networks.

Neural activity emerges and propagates swiftly between brain areas. Investigation of these transient large-scale flows requires sophisticated statistical models. We present a method for assessing the statistical confidence of event-related neural propagation. Furthermore, we propose a criterion for statistical model selection, based on both goodness of fit and width of confidence intervals. We show that event-related causality (ERC) with two-dimensional (2D) moving average, is an efficient estimator of task-related neural propagation and that it can be used to determine how different cognitive task demands affect the strength and directionality of neural propagation across human cortical networks. Using electrodes surgically implanted on the surface of the brain for clinical testing prior to epilepsy surgery, we recorded electrocorticographic (ECoG) signals as subjects performed three naming tasks: naming of ambiguous and unambiguous visual objects, and as a contrast, naming to auditory description. ERC revealed robust and statistically significant patterns of high gamma activity propagation, consistent with models of visually and auditorily cued word production. Interestingly, ambiguous visual stimuli elicited more robust propagation from visual to auditory cortices relative to unambiguous stimuli, whereas naming to auditory description elicited propagation in the opposite direction, consistent with recruitment of modalities other than those of the stimulus during object recognition and naming. The new method introduced here is uniquely suitable to both research and clinical applications and can be used to estimate the statistical significance of neural propagation for both cognitive neuroscientific studies and functional brain mapping prior to resective surgery for epilepsy and brain tumors.

Ethical commitments, principles, and practices guiding intracranial neuroscientific research in humans.

Leveraging firsthand experience, BRAIN-funded investigators conducting intracranial human neuroscience research propose two fundamental ethical commitments: (1) maintaining the integrity of clinical care and (2) ensuring voluntariness. Principles, practices, and uncertainties related to these commitments are offered for future investigation.

Brain-Computer Interface: Applications to Speech Decoding and Synthesis to Augment Communication.

Damage or degeneration of motor pathways necessary for speech and other movements, as in brainstem strokes or amyotrophic lateral sclerosis (ALS), can interfere with efficient communication without affecting brain structures responsible for language or cognition. In the worst-case scenario, this can result in the locked in syndrome (LIS), a condition in which individuals cannot initiate communication and can only express themselves by answering yes/no questions with eye blinks or other rudimentary movements. Existing augmentative and alternative communication (AAC) devices that rely on eye tracking can improve the quality of life for people with this condition, but brain-computer interfaces (BCIs) are also increasingly being investigated as AAC devices, particularly when eye tracking is too slow or unreliable. Moreover, with recent and ongoing advances in machine learning and neural recording technologies, BCIs may offer the only means to go beyond cursor control and text generation on a computer, to allow real-time synthesis of speech, which would arguably offer the most efficient and expressive channel for communication. The potential for BCI speech synthesis has only recently been realized because of seminal studies of the neuroanatomical and neurophysiological underpinnings of speech production using intracranial electrocorticographic (ECoG) recordings in patients undergoing epilepsy surgery. These studies have shown that cortical areas responsible for vocalization and articulation are distributed over a large area of ventral sensorimotor cortex, and that it is possible to decode speech and reconstruct its acoustics from ECoG if these areas are recorded with sufficiently dense and comprehensive electrode arrays. In this article, we review these advances, including the latest neural decoding strategies that range from deep learning models to the direct concatenation of speech units. We also discuss state-of-the-art vocoders that are integral in constructing natural-sounding audio waveforms for speech BCIs. Finally, this review outlines some of the challenges ahead in directly synthesizing speech for patients with LIS.

Intracortical Somatosensory Stimulation to Elicit Fingertip Sensations in an Individual With Spinal Cord Injury.

BACKGROUND AND OBJECTIVES: The restoration of touch to fingers and fingertips is critical to achieving dexterous neuroprosthetic control for individuals with sensorimotor dysfunction. However, localized fingertip sensations have not been evoked via intracortical microstimulation (ICMS). METHODS: Using a novel intraoperative mapping approach, we implanted electrode arrays in the finger areas of left and right somatosensory cortex and delivered ICMS over a 2-year period in a human participant with spinal cord injury. RESULTS: Stimulation evoked tactile sensations in 8 fingers, including fingertips, spanning both hands. Evoked percepts followed expected somatotopic arrangements. The subject was able to reliably identify up to 7 finger-specific sites spanning both hands in a finger discrimination task. The size of the evoked percepts was on average 33% larger than a finger pad, as assessed via manual markings of a hand image. The size of the evoked percepts increased modestly with increased stimulation intensity, growing 21% as pulse amplitude increased from 20 to 80 µA. Detection thresholds were estimated on a subset of electrodes, with estimates of 9.2 to 35 µA observed, roughly consistent with prior studies. DISCUSSION: These results suggest that ICMS can enable the delivery of consistent and localized fingertip sensations during object manipulation by neuroprostheses for individuals with somatosensory deficits. CLINICALTRIALSGOV IDENTIFIER: NCT03161067.