Online speech synthesis using a chronically implanted brain-computer interface in an individual with ALS.

Brain-computer interfaces (BCIs) that reconstruct and synthesize speech using brain activity recorded with intracranial electrodes may pave the way toward novel communication interfaces for people who have lost their ability to speak, or who are at high risk of losing this ability, due to neurological disorders. Here, we report online synthesis of intelligible words using a chronically implanted brain-computer interface (BCI) in a man with impaired articulation due to ALS, participating in a clinical trial (ClinicalTrials.gov, NCT03567213) exploring different strategies for BCI communication. The 3-stage approach reported here relies on recurrent neural networks to identify, decode and synthesize speech from electrocorticographic (ECoG) signals acquired across motor, premotor and somatosensory cortices. We demonstrate a reliable BCI that synthesizes commands freely chosen and spoken by the participant from a vocabulary of 6 keywords previously used for decoding commands to control a communication board. Evaluation of the intelligibility of the synthesized speech indicates that 80% of the words can be correctly recognized by human listeners. Our results show that a speech-impaired individual with ALS can use a chronically implanted BCI to reliably produce synthesized words while preserving the participant's voice profile, and provide further evidence for the stability of ECoG for speech-based BCIs.

Stable Decoding from a Speech BCI Enables Control for an Individual with ALS without Recalibration for 3 Months.

Brain-computer interfaces (BCIs) can be used to control assistive devices by patients with neurological disorders like amyotrophic lateral sclerosis (ALS) that limit speech and movement. For assistive control, it is desirable for BCI systems to be accurate and reliable, preferably with minimal setup time. In this study, a participant with severe dysarthria due to ALS operates computer applications with six intuitive speech commands via a chronic electrocorticographic (ECoG) implant over the ventral sensorimotor cortex. Speech commands are accurately detected and decoded (median accuracy: 90.59%) throughout a 3-month study period without model retraining or recalibration. Use of the BCI does not require exogenous timing cues, enabling the participant to issue self-paced commands at will. These results demonstrate that a chronically implanted ECoG-based speech BCI can reliably control assistive devices over long time periods with only initial model training and calibration, supporting the feasibility of unassisted home use.

Online speech synthesis using a chronically implanted brain-computer interface in an individual with ALS.

Recent studies have shown that speech can be reconstructed and synthesized using only brain activity recorded with intracranial electrodes, but until now this has only been done using retrospective analyses of recordings from able-bodied patients temporarily implanted with electrodes for epilepsy surgery. Here, we report online synthesis of intelligible words using a chronically implanted brain-computer interface (BCI) in a clinical trial participant (ClinicalTrials.gov, NCT03567213) with dysarthria due to amyotrophic lateral sclerosis (ALS). We demonstrate a reliable BCI that synthesizes commands freely chosen and spoken by the user from a vocabulary of 6 keywords originally designed to allow intuitive selection of items on a communication board. Our results show for the first time that a speech-impaired individual with ALS can use a chronically implanted BCI to reliably produce synthesized words that are intelligible to human listeners while preserving the participants voice profile.

Changes in human brain dynamics during behavioral priming and repetition suppression.

Behavioral responses to a perceptual stimulus are typically faster with repeated exposure to the stimulus (behavioral priming). This implicit learning mechanism is critical for survival but impaired in a variety of neurological disorders, including Alzheimer's disease. Many studies of the neural bases for behavioral priming have encountered an interesting paradox: in spite of faster behavioral responses, repeated stimuli usually elicit weaker neural responses (repetition suppression). Several neurophysiological models have been proposed to resolve this paradox, but noninvasive techniques for human studies have had insufficient spatial-temporal precision for testing their predictions. Here, we used the unparalleled precision of electrocorticography (ECoG) to analyze the timing and magnitude of task-related changes in neural activation and propagation while patients named novel vs repeated visual objects. Stimulus repetition was associated with faster verbal responses and decreased neural activation (repetition suppression) in ventral occipito-temporal cortex (VOTC) and left prefrontal cortex (LPFC). Interestingly, we also observed increased neural activation (repetition enhancement) in LPFC and other recording sites. Moreover, with analysis of high gamma propagation we observed increased top-down propagation from LPFC into VOTC, preceding repetition suppression. The latter results indicate that repetition suppression and behavioral priming are associated with strengthening of top-down network influences on perceptual processing, consistent with predictive coding models of repetition suppression, and they support a central role for changes in large-scale cortical dynamics in achieving more efficient and rapid behavioral responses.

Keyword Spotting Using Human Electrocorticographic Recordings.

Neural keyword spotting could form the basis of a speech brain-computer-interface for menu-navigation if it can be done with low latency and high specificity comparable to the "wake-word" functionality of modern voice-activated AI assistant technologies. This study investigated neural keyword spotting using motor representations of speech via invasively-recorded electrocorticographic signals as a proof-of-concept. Neural matched filters were created from monosyllabic consonant-vowel utterances: one keyword utterance, and 11 similar non-keyword utterances. These filters were used in an analog to the acoustic keyword spotting problem, applied for the first time to neural data. The filter templates were cross-correlated with the neural signal, capturing temporal dynamics of neural activation across cortical sites. Neural vocal activity detection (VAD) was used to identify utterance times and a discriminative classifier was used to determine if these utterances were the keyword or non-keyword speech. Model performance appeared to be highly related to electrode placement and spatial density. Vowel height (/a/ vs /i/) was poorly discriminated in recordings from sensorimotor cortex, but was highly discriminable using neural features from superior temporal gyrus during self-monitoring. The best performing neural keyword detection (5 keyword detections with two false-positives across 60 utterances) and neural VAD (100% sensitivity, ~1 false detection per 10 utterances) came from high-density (2 mm electrode diameter and 5 mm pitch) recordings from ventral sensorimotor cortex, suggesting the spatial fidelity and extent of high-density ECoG arrays may be sufficient for the purpose of speech brain-computer-interfaces.

The Potential for a Speech Brain-Computer Interface Using Chronic Electrocorticography.

A brain-computer interface (BCI) is a technology that uses neural features to restore or augment the capabilities of its user. A BCI for speech would enable communication in real time via neural correlates of attempted or imagined speech. Such a technology would potentially restore communication and improve quality of life for locked-in patients and other patients with severe communication disorders. There have been many recent developments in neural decoders, neural feature extraction, and brain recording modalities facilitating BCI for the control of prosthetics and in automatic speech recognition (ASR). Indeed, ASR and related fields have developed significantly over the past years, and many lend many insights into the requirements, goals, and strategies for speech BCI. Neural speech decoding is a comparatively new field but has shown much promise with recent studies demonstrating semantic, auditory, and articulatory decoding using electrocorticography (ECoG) and other neural recording modalities. Because the neural representations for speech and language are widely distributed over cortical regions spanning the frontal, parietal, and temporal lobes, the mesoscopic scale of population activity captured by ECoG surface electrode arrays may have distinct advantages for speech BCI, in contrast to the advantages of microelectrode arrays for upper-limb BCI. Nevertheless, there remain many challenges for the translation of speech BCIs to clinical populations. This review discusses and outlines the current state-of-the-art for speech BCI and explores what a speech BCI using chronic ECoG might entail.

Cortical Responses to Input From Distant Areas are Modulated by Local Spontaneous Alpha/Beta Oscillations.

Any given area in human cortex may receive input from multiple, functionally heterogeneous areas, potentially representing different processing threads. Alpha (8-13 Hz) and beta oscillations (13-20 Hz) have been hypothesized by other investigators to gate local cortical processing, but their influence on cortical responses to input from other cortical areas is unknown. To study this, we measured the effect of local oscillatory power and phase on cortical responses elicited by single-pulse electrical stimulation (SPES) at distant cortical sites, in awake human subjects implanted with intracranial electrodes for epilepsy surgery. In 4 out of 5 subjects, the amplitudes of corticocortical evoked potentials (CCEPs) elicited by distant SPES were reproducibly modulated by the power, but not the phase, of local oscillations in alpha and beta frequencies. Specifically, CCEP amplitudes were higher when average oscillatory power just before distant SPES (-110 to -10 ms) was high. This effect was observed in only a subset (0-33%) of sites with CCEPs and, like the CCEPs themselves, varied with stimulation at different distant sites. Our results suggest that although alpha and beta oscillations may gate local processing, they may also enhance the responsiveness of cortex to input from distant cortical sites.

BCI2000Web and WebFM: Browser-Based Tools for Brain Computer Interfaces and Functional Brain Mapping.

BCI2000 has been a popular platform for development of real-time brain computer interfaces (BCIs). Since BCI2000's initial release, web browsers have evolved considerably, enabling rapid development of internet-enabled applications and interactive visualizations. Linking the amplifier abstraction and signal processing native to BCI2000 with the host of technologies and ease of development afforded by modern web browsers could enable a new generation of browser-based BCIs and visualizations. We developed a server and filter module called BCI2000Web providing an HTTP connection capable of escalation into an RFC6455 WebSocket, which enables direct communication between a browser and a BCI2000 distribution in real-time, facilitating a number of novel applications. We also present a JavaScript module, bci2k.js, that allows web developers to create paradigms and visualizations using this interface in an easy-to-use and intuitive manner. To illustrate the utility of BCI2000Web, we demonstrate a browser-based implementation of a real-time electrocorticographic (ECoG) functional mapping suite called WebFM. We also explore how the unique characteristics of our browser-based framework make BCI2000Web an attractive tool for future BCI applications. BCI2000Web leverages the advances of BCI2000 to provide real-time browser-based interactions with human neurophysiological recordings, allowing for web-based BCIs and other applications, including real-time functional brain mapping. Both BCI2000 and WebFM are provided under open source licenses. Enabling a powerful BCI suite to communicate with today's most technologically progressive software empowers a new cohort of developers to engage with BCI technology, and could serve as a platform for internet-enabled BCIs.

Feature Selection Methods for Zero-Shot Learning of Neural Activity.

Dimensionality poses a serious challenge when making predictions from human neuroimaging data. Across imaging modalities, large pools of potential neural features (e.g., responses from particular voxels, electrodes, and temporal windows) have to be related to typically limited sets of stimuli and samples. In recent years, zero-shot prediction models have been introduced for mapping between neural signals and semantic attributes, which allows for classification of stimulus classes not explicitly included in the training set. While choices about feature selection can have a substantial impact when closed-set accuracy, open-set robustness, and runtime are competing design objectives, no systematic study of feature selection for these models has been reported. Instead, a relatively straightforward feature stability approach has been adopted and successfully applied across models and imaging modalities. To characterize the tradeoffs in feature selection for zero-shot learning, we compared correlation-based stability to several other feature selection techniques on comparable data sets from two distinct imaging modalities: functional Magnetic Resonance Imaging and Electrocorticography. While most of the feature selection methods resulted in similar zero-shot prediction accuracies and spatial/spectral patterns of selected features, there was one exception; A novel feature/attribute correlation approach was able to achieve those accuracies with far fewer features, suggesting the potential for simpler prediction models that yield high zero-shot classification accuracy.

Presurgical language localization with visual naming associated ECoG high- gamma modulation in pediatric drug-resistant epilepsy.

OBJECTIVE: This prospective study compared presurgical language localization with visual naming-associated high-γ modulation (HGM) and conventional electrical cortical stimulation (ECS) in children with intracranial electrodes. METHODS: Patients with drug-resistant epilepsy who were undergoing intracranial monitoring were included if able to name pictures. Electrocorticography (ECoG) signals were recorded during picture naming (overt and covert) and quiet baseline. For each electrode the likelihood of high-γ (70-116 Hz) power modulation during naming task relative to the baseline was estimated. Electrodes with significant HGM were plotted on a three-dimensional (3D) cortical surface model. Sensitivity, specificity, and accuracy were calculated compared to clinical ECS. RESULTS: Seventeen patients with mean age of 11.3 years (range 4-19) were included. In patients with left hemisphere electrodes (n = 10), HGM during overt naming showed high specificity (0.81, 95% confidence interval [CI] 0.78-0.85), and accuracy (0.71, 95% CI 0.66-0.75, p < 0.001), but modest sensitivity (0.47) when ECS interference with naming (aphasia or paraphasic errors) and/or oral motor function was regarded as the gold standard. Similar results were reproduced by comparing covert naming-associated HGM with ECS naming sites. With right hemisphere electrodes (n = 7), no ECS-naming deficits were seen without interference with oral-motor function. HGM mapping showed a high specificity (0.81, 95% CI 0.78-0.84), and accuracy (0.76, 95% CI 0.71-0.81, p = 0.006), but modest sensitivity (0.44) compared to ECS interference with oral-motor function. Naming-associated ECoG HGM was consistently observed over Broca's area (left posterior inferior-frontal gyrus), bilateral oral/facial motor cortex, and sometimes over the temporal pole. SIGNIFICANCE: This study supports the use of ECoG HGM mapping in children in whom adverse events preclude ECS, or as a screening method to prioritize electrodes for ECS testing.

Semantic attributes are encoded in human electrocorticographic signals during visual object recognition.

Non-invasive neuroimaging studies have shown that semantic category and attribute information are encoded in neural population activity. Electrocorticography (ECoG) offers several advantages over non-invasive approaches, but the degree to which semantic attribute information is encoded in ECoG responses is not known. We recorded ECoG while patients named objects from 12 semantic categories and then trained high-dimensional encoding models to map semantic attributes to spectral-temporal features of the task-related neural responses. Using these semantic attribute encoding models, untrained objects were decoded with accuracies comparable to whole-brain functional Magnetic Resonance Imaging (fMRI), and we observed that high-gamma activity (70-110Hz) at basal occipitotemporal electrodes was associated with specific semantic dimensions (manmade-animate, canonically large-small, and places-tools). Individual patient results were in close agreement with reports from other imaging modalities on the time course and functional organization of semantic processing along the ventral visual pathway during object recognition. The semantic attribute encoding model approach is critical for decoding objects absent from a training set, as well as for studying complex semantic encodings without artificially restricting stimuli to a small number of semantic categories.

Cortical subnetwork dynamics during human language tasks.

Language tasks require the coordinated activation of multiple subnetworks-groups of related cortical interactions involved in specific components of task processing. Although electrocorticography (ECoG) has sufficient temporal and spatial resolution to capture the dynamics of event-related interactions between cortical sites, it is difficult to decompose these complex spatiotemporal patterns into functionally discrete subnetworks without explicit knowledge of each subnetwork's timing. We hypothesized that subnetworks corresponding to distinct components of task-related processing could be identified as groups of interactions with co-varying strengths. In this study, five subjects implanted with ECoG grids over language areas performed word repetition and picture naming. We estimated the interaction strength between each pair of electrodes during each task using a time-varying dynamic Bayesian network (tvDBN) model constructed from the power of high gamma (70-110Hz) activity, a surrogate for population firing rates. We then reduced the dimensionality of this model using principal component analysis (PCA) to identify groups of interactions with co-varying strengths, which we term functional network components (FNCs). This data-driven technique estimates both the weight of each interaction's contribution to a particular subnetwork, and the temporal profile of each subnetwork's activation during the task. We found FNCs with temporal and anatomical features consistent with articulatory preparation in both tasks, and with auditory and visual processing in the word repetition and picture naming tasks, respectively. These FNCs were highly consistent between subjects with similar electrode placement, and were robust enough to be characterized in single trials. Furthermore, the interaction patterns uncovered by FNC analysis correlated well with recent literature suggesting important functional-anatomical distinctions between processing external and self-produced speech. Our results demonstrate that subnetwork decomposition of event-related cortical interactions is a powerful paradigm for interpreting the rich dynamics of large-scale, distributed cortical networks during human cognitive tasks.

Neuroprosthetic limb control with electrocorticography: approaches and challenges.

Advanced upper limb prosthetics, such as the Johns Hopkins Applied Physics Lab Modular Prosthetic Limb (MPL), are now available for research and preliminary clinical applications. Research attention has shifted to developing means of controlling these prostheses. Penetrating microelectrode arrays are often used in animal and human models to decode action potentials for cortical control. These arrays may suffer signal loss over the long-term and therefore should not be the only implant type investigated for chronic BMI use. Electrocorticographic (ECoG) signals from electrodes on the cortical surface may provide more stable long-term recordings. Several studies have demonstrated ECoG's potential for decoding cortical activity. As a result, clinical studies are investigating ECoG encoding of limb movement, as well as its use for interfacing with and controlling advanced prosthetic arms. This overview presents the technical state of the art in the use of ECoG in controlling prostheses. Technical limitations of the current approach and future directions are also presented.