Decoding the brain: From neural representations to mechanistic models.

A central principle in neuroscience is that neurons within the brain act in concert to produce perception, cognition, and adaptive behavior. Neurons are organized into specialized brain areas, dedicated to different functions to varying extents, and their function relies on distributed circuits to continuously encode relevant environmental and body-state features, enabling other areas to decode (interpret) these representations for computing meaningful decisions and executing precise movements. Thus, the distributed brain can be thought of as a series of computations that act to encode and decode information. In this perspective, we detail important concepts of neural encoding and decoding and highlight the mathematical tools used to measure them, including deep learning methods. We provide case studies where decoding concepts enable foundational and translational science in motor, visual, and language processing.

Parallel and distributed encoding of speech across human auditory cortex.

Speech perception is thought to rely on a cortical feedforward serial transformation of acoustic into linguistic representations. Using intracranial recordings across the entire human auditory cortex, electrocortical stimulation, and surgical ablation, we show that cortical processing across areas is not consistent with a serial hierarchical organization. Instead, response latency and receptive field analyses demonstrate parallel and distinct information processing in the primary and nonprimary auditory cortices. This functional dissociation was also observed where stimulation of the primary auditory cortex evokes auditory hallucination but does not distort or interfere with speech perception. Opposite effects were observed during stimulation of nonprimary cortex in superior temporal gyrus. Ablation of the primary auditory cortex does not affect speech perception. These results establish a distributed functional organization of parallel information processing throughout the human auditory cortex and demonstrate an essential independent role for nonprimary auditory cortex in speech processing.

The Control of Vocal Pitch in Human Laryngeal Motor Cortex.

In speech, the highly flexible modulation of vocal pitch creates intonation patterns that speakers use to convey linguistic meaning. This human ability is unique among primates. Here, we used high-density cortical recordings directly from the human brain to determine the encoding of vocal pitch during natural speech. We found neural populations in bilateral dorsal laryngeal motor cortex (dLMC) that selectively encoded produced pitch but not non-laryngeal articulatory movements. This neural population controlled short pitch accents to express prosodic emphasis on a word in a sentence. Other larynx cortical representations controlling voicing and longer pitch phrase contours were found at separate sites. dLMC sites also encoded vocal pitch during a non-speech singing task. Finally, direct focal stimulation of dLMC evoked laryngeal movements and involuntary vocalization, confirming its causal role in feedforward control. Together, these results reveal the neural basis for the voluntary control of vocal pitch in human speech. VIDEO ABSTRACT.

An Amygdala-Hippocampus Subnetwork that Encodes Variation in Human Mood.

Human brain networks that encode variation in mood on naturalistic timescales remain largely unexplored. Here we combine multi-site, semi-chronic, intracranial electroencephalography recordings from the human limbic system with machine learning methods to discover a brain subnetwork that correlates with variation in individual subjects' self-reported mood over days. First we defined the subnetworks that influence intrinsic brain dynamics by identifying regions that showed coordinated changes in spectral coherence. The most common subnetwork, found in 13 of 21 subjects, was characterized by ß-frequency coherence (13-30 Hz) between the amygdala and hippocampus. Increased variability of this subnetwork correlated with worsening mood across these 13 subjects. Moreover, these subjects had significantly higher trait anxiety than the 8 of 21 for whom this amygdala-hippocampus subnetwork was absent. These results demonstrate an approach for extracting network-behavior relationships from complex datasets, and they reveal a conserved subnetwork associated with a psychological trait that significantly influences intrinsic brain dynamics and encodes fluctuations in mood.

Large-scale neurophysiology and single-cell profiling in human neuroscience.

Advances in large-scale single-unit human neurophysiology, single-cell RNA sequencing, spatial transcriptomics and long-term ex vivo tissue culture of surgically resected human brain tissue have provided an unprecedented opportunity to study human neuroscience. In this Perspective, we describe the development of these paradigms, including Neuropixels and recent brain-cell atlas efforts, and discuss how their convergence will further investigations into the cellular underpinnings of network-level activity in the human brain. Specifically, we introduce a workflow in which functionally mapped samples of human brain tissue resected during awake brain surgery can be cultured ex vivo for multi-modal cellular and functional profiling. We then explore how advances in human neuroscience will affect clinical practice, and conclude by discussing societal and ethical implications to consider. Potential findings from the field of human neuroscience will be vast, ranging from insights into human neurodiversity and evolution to providing cell-type-specific access to study and manipulate diseased circuits in pathology. This Perspective aims to provide a unifying framework for the field of human neuroscience as we welcome an exciting era for understanding the functional cytoarchitecture of the human brain.

Large-scale single-neuron speech sound encoding across the depth of human cortex.

Understanding the neural basis of speech perception requires that we study the human brain both at the scale of the fundamental computational unit of neurons and in their organization across the depth of cortex. Here we used high-density Neuropixels arrays1-3 to record from 685 neurons across cortical layers at nine sites in a high-level auditory region that is critical for speech, the superior temporal gyrus4,5, while participants listened to spoken sentences. Single neurons encoded a wide range of speech sound cues, including features of consonants and vowels, relative vocal pitch, onsets, amplitude envelope and sequence statistics. Neurons at each cross-laminar recording exhibited dominant tuning to a primary speech feature while also containing a substantial proportion of neurons that encoded other features contributing to heterogeneous selectivity. Spatially, neurons at similar cortical depths tended to encode similar speech features. Activity across all cortical layers was predictive of high-frequency field potentials (electrocorticography), providing a neuronal origin for macroelectrode recordings from the cortical surface. Together, these results establish single-neuron tuning across the cortical laminae as an important dimension of speech encoding in human superior temporal gyrus.

The speech neuroprosthesis.

Loss of speech after paralysis is devastating, but circumventing motor-pathway injury by directly decoding speech from intact cortical activity has the potential to restore natural communication and self-expression. Recent discoveries have defined how key features of speech production are facilitated by the coordinated activity of vocal-tract articulatory and motor-planning cortical representations. In this Review, we highlight such progress and how it has led to successful speech decoding, first in individuals implanted with intracranial electrodes for clinical epilepsy monitoring and subsequently in individuals with paralysis as part of early feasibility clinical trials to restore speech. We discuss high-spatiotemporal-resolution neural interfaces and the adaptation of state-of-the-art speech computational algorithms that have driven rapid and substantial progress in decoding neural activity into text, audible speech, and facial movements. Although restoring natural speech is a long-term goal, speech neuroprostheses already have performance levels that surpass communication rates offered by current assistive-communication technology. Given this accelerated rate of progress in the field, we propose key evaluation metrics for speed and accuracy, among others, to help standardize across studies. We finish by highlighting several directions to more fully explore the multidimensional feature space of speech and language, which will continue to accelerate progress towards a clinically viable speech neuroprosthesis.

A high-performance neuroprosthesis for speech decoding and avatar control.

Speech neuroprostheses have the potential to restore communication to people living with paralysis, but naturalistic speed and expressivity are elusive1. Here we use high-density surface recordings of the speech cortex in a clinical-trial participant with severe limb and vocal paralysis to achieve high-performance real-time decoding across three complementary speech-related output modalities: text, speech audio and facial-avatar animation. We trained and evaluated deep-learning models using neural data collected as the participant attempted to silently speak sentences. For text, we demonstrate accurate and rapid large-vocabulary decoding with a median rate of 78 words per minute and median word error rate of 25%. For speech audio, we demonstrate intelligible and rapid speech synthesis and personalization to the participant's pre-injury voice. For facial-avatar animation, we demonstrate the control of virtual orofacial movements for speech and non-speech communicative gestures. The decoders reached high performance with less than two weeks of training. Our findings introduce a multimodal speech-neuroprosthetic approach that has substantial promise to restore full, embodied communication to people living with severe paralysis.

Glioblastoma remodelling of human neural circuits decreases survival.

Gliomas synaptically integrate into neural circuits1,2. Previous research has demonstrated bidirectional interactions between neurons and glioma cells, with neuronal activity driving glioma growth1-4 and gliomas increasing neuronal excitability2,5-8. Here we sought to determine how glioma-induced neuronal changes influence neural circuits underlying cognition and whether these interactions influence patient survival. Using intracranial brain recordings during lexical retrieval language tasks in awake humans together with site-specific tumour tissue biopsies and cell biology experiments, we find that gliomas remodel functional neural circuitry such that task-relevant neural responses activate tumour-infiltrated cortex well beyond the cortical regions that are normally recruited in the healthy brain. Site-directed biopsies from regions within the tumour that exhibit high functional connectivity between the tumour and the rest of the brain are enriched for a glioblastoma subpopulation that exhibits a distinct synaptogenic and neuronotrophic phenotype. Tumour cells from functionally connected regions secrete the synaptogenic factor thrombospondin-1, which contributes to the differential neuron-glioma interactions observed in functionally connected tumour regions compared with tumour regions with less functional connectivity. Pharmacological inhibition of thrombospondin-1 using the FDA-approved drug gabapentin decreases glioblastoma proliferation. The degree of functional connectivity between glioblastoma and the normal brain negatively affects both patient survival and performance in language tasks. These data demonstrate that high-grade gliomas functionally remodel neural circuits in the human brain, which both promotes tumour progression and impairs cognition.

Anatomical and behavioural correlates of auditory perception in developmental dyslexia.

Developmental dyslexia is typically associated with difficulties in basic auditory processing and in manipulating speech sounds. However, the neuroanatomical correlates of auditory difficulties in developmental dyslexia (DD) and their contribution to individual clinical phenotypes are still unknown. Recent intracranial electrocorticography findings associated processing of sound amplitude rises and speech sounds with posterior and middle superior temporal gyrus (STG), respectively. We hypothesize that regional STG anatomy will relate to specific auditory abilities in DD, and that auditory processing abilities will relate to behavioral difficulties with speech and reading. One hundred and ten children (78 DD, 32 typically developing, age 7-15 years) completed amplitude rise time and speech in noise discrimination tasks. They also underwent a battery of cognitive tests. Anatomical MRI scans were used to identify regions in which local cortical gyrification complexity correlated with auditory behavior. Behaviorally, amplitude rise time but not speech in noise performance was impaired in DD. Neurally, amplitude rise time and speech in noise performance correlated with gyrification in posterior and middle STG, respectively. Furthermore, amplitude rise time significantly contributed to reading impairments in DD, while speech in noise only explained variance in phonological awareness. Finally, amplitude rise time and speech in noise performance were not correlated, and each task was correlated with distinct neuropsychological measures, emphasizing their unique contributions to DD. Overall, we provide a direct link between the neurodevelopment of the left STG and individual variability in auditory processing abilities in neurotypical and dyslexic populations.

Speech synthesis from neural decoding of spoken sentences.

Technology that translates neural activity into speech would be transformative for people who are unable to communicate as a result of neurological impairments. Decoding speech from neural activity is challenging because speaking requires very precise and rapid multi-dimensional control of vocal tract articulators. Here we designed a neural decoder that explicitly leverages kinematic and sound representations encoded in human cortical activity to synthesize audible speech. Recurrent neural networks first decoded directly recorded cortical activity into representations of articulatory movement, and then transformed these representations into speech acoustics. In closed vocabulary tests, listeners could readily identify and transcribe speech synthesized from cortical activity. Intermediate articulatory dynamics enhanced performance even with limited data. Decoded articulatory representations were highly conserved across speakers, enabling a component of the decoder to be transferrable across participants. Furthermore, the decoder could synthesize speech when a participant silently mimed sentences. These findings advance the clinical viability of using speech neuroprosthetic technology to restore spoken communication.

Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria.

BACKGROUND: Technology to restore the ability to communicate in paralyzed persons who cannot speak has the potential to improve autonomy and quality of life. An approach that decodes words and sentences directly from the cerebral cortical activity of such patients may represent an advancement over existing methods for assisted communication. METHODS: We implanted a subdural, high-density, multielectrode array over the area of the sensorimotor cortex that controls speech in a person with anarthria (the loss of the ability to articulate speech) and spastic quadriparesis caused by a brain-stem stroke. Over the course of 48 sessions, we recorded 22 hours of cortical activity while the participant attempted to say individual words from a vocabulary set of 50 words. We used deep-learning algorithms to create computational models for the detection and classification of words from patterns in the recorded cortical activity. We applied these computational models, as well as a natural-language model that yielded next-word probabilities given the preceding words in a sequence, to decode full sentences as the participant attempted to say them. RESULTS: We decoded sentences from the participant's cortical activity in real time at a median rate of 15.2 words per minute, with a median word error rate of 25.6%. In post hoc analyses, we detected 98% of the attempts by the participant to produce individual words, and we classified words with 47.1% accuracy using cortical signals that were stable throughout the 81-week study period. CONCLUSIONS: In a person with anarthria and spastic quadriparesis caused by a brain-stem stroke, words and sentences were decoded directly from cortical activity during attempted speech with the use of deep-learning models and a natural-language model. (Funded by Facebook and others; ClinicalTrials.gov number, NCT03698149.).

Quantitative Assessment of Preanalytic Variables on Clinical Evaluation of PI3/AKT/mTOR Signaling Activity in Diffuse Glioma.

Biomarker-driven therapeutic clinical trials require the implementation of standardized, evidence-based practices for sample collection. In diffuse glioma, phosphatidylinositol 3 (PI3)-kinase/AKT/mTOR (PI3/AKT/mTOR) signaling is an attractive therapeutic target for which window-of-opportunity clinical trials could facilitate the identification of promising new agents. Yet, the relevant preanalytic variables and optimal tumor sampling methods necessary to measure pathway activity are unknown. To address this, we used a murine model for isocitrate dehydrogenase (IDH)-wildtype glioblastoma (GBM) and human tumor tissue, including IDH-wildtype GBM and IDH-mutant diffuse glioma. First, we determined the impact of delayed time-to-formalin fixation, or cold ischemia time (CIT), on the quantitative assessment of cellular expression of 6 phosphoproteins that are readouts of PI3K/AK/mTOR activity (phosphorylated-proline-rich Akt substrate of 40 kDa (p-PRAS40, T246), -mechanistic target of rapamycin (p-mTOR; S2448); -AKT (p-AKT, S473); -ribosomal protein S6 (p-RPS6, S240/244 and S235/236), and -eukaryotic initiation factor 4E-binding protein 1 (p-4EBP1, T37/46). With CITs ≥ 2 hours, typical of routine clinical handling, all had reduced or altered expression with p-RPS6 (S240/244) exhibiting relatively greater stability. A similar pattern was observed using patient tumor samples from the operating room with p-4EBP1 more sensitive to delayed fixation than p-RPS6 (S240/244). Many clinical trials utilize unstained slides for biomarker evaluation. Thus, we evaluated the impact of slide storage conditions on the detection of p-RPS6 (S240/244), p-4EBP1, and p-AKT. After 5 months, storage at -80°C was required to preserve the expression of p-4EBP1 and p-AKT, whereas p-RPS6 (240/244) expression was not stable regardless of storage temperature. Biomarker heterogeneity impacts optimal tumor sampling. Quantification of p-RPS6 (240/244) expression in multiple regionally distinct human tumor samples from 8 patients revealed significant intratumoral heterogeneity. Thus, the accurate assessment of PI3K/AKT/mTOR signaling in diffuse glioma must overcome intratumoral heterogeneity and multiple preanalytic factors, including time-to-formalin fixation, slide storage conditions, and phosphoprotein of interest.

Development and validation of a nonverbal consensus-based semantic memory paradigm in patients with epilepsy.

OBJECTIVE: Brain areas implicated in semantic memory can be damaged in patients with epilepsy (PWE). However, it is challenging to delineate semantic processing deficits from acoustic, linguistic, and other verbal aspects in current neuropsychological assessments. We developed a new Visual-based Semantic Association Task (ViSAT) to evaluate nonverbal semantic processing in PWE. METHOD: The ViSAT was adapted from similar predecessors (Pyramids & Palm Trees test, PPT; Camels & Cactus Test, CCT) comprised of 100 unique trials using real-life color pictures that avoid demographic, cultural, and other potential confounds. We obtained performance data from 23 PWE participants and 24 control participants (Control), along with crowdsourced normative data from 54 Amazon Mechanical Turk (Mturk) workers. RESULTS: ViSAT reached a consensus >90% in 91.3% of trials compared to 83.6% in PPT and 82.9% in CCT. A deep learning model demonstrated that visual features of the stimulus images (color, shape; i.e., non-semantic) did not influence top answer choices (p = 0.577). The PWE group had lower accuracy than the Control group (p = 0.019). PWE had longer response times than the Control group in general and this was augmented for the semantic processing (trial answer) stage (both p < 0.001). CONCLUSIONS: This study demonstrated performance impairments in PWE that may reflect dysfunction of nonverbal semantic memory circuits, such as seizure onset zones overlapping with key semantic regions (e.g., anterior temporal lobe). The ViSAT paradigm avoids confounds, is repeatable/longitudinal, captures behavioral data, and is open-source, thus we propose it as a strong alternative for clinical and research assessment of nonverbal semantic memory.

Human hippocampal neurogenesis drops sharply in children to undetectable levels in adults.

New neurons continue to be generated in the subgranular zone of the dentate gyrus of the adult mammalian hippocampus. This process has been linked to learning and memory, stress and exercise, and is thought to be altered in neurological disease. In humans, some studies have suggested that hundreds of new neurons are added to the adult dentate gyrus every day, whereas other studies find many fewer putative new neurons. Despite these discrepancies, it is generally believed that the adult human hippocampus continues to generate new neurons. Here we show that a defined population of progenitor cells does not coalesce in the subgranular zone during human fetal or postnatal development. We also find that the number of proliferating progenitors and young neurons in the dentate gyrus declines sharply during the first year of life and only a few isolated young neurons are observed by 7 and 13 years of age. In adult patients with epilepsy and healthy adults (18-77 years; n = 17 post-mortem samples from controls; n = 12 surgical resection samples from patients with epilepsy), young neurons were not detected in the dentate gyrus. In the monkey (Macaca mulatta) hippocampus, proliferation of neurons in the subgranular zone was found in early postnatal life, but this diminished during juvenile development as neurogenesis decreased. We conclude that recruitment of young neurons to the primate hippocampus decreases rapidly during the first years of life, and that neurogenesis in the dentate gyrus does not continue, or is extremely rare, in adult humans. The early decline in hippocampal neurogenesis raises questions about how the function of the dentate gyrus differs between humans and other species in which adult hippocampal neurogenesis is preserved.

Grasp-specific high-frequency broadband mirror neuron activity during reach-and-grasp movements in humans.

Broadly congruent mirror neurons, responding to any grasp movement, and strictly congruent mirror neurons, responding only to specific grasp movements, have been reported in single-cell studies with primates. Delineating grasp properties in humans is essential to understand the human mirror neuron system with implications for behavior and social cognition. We analyzed electrocorticography data from a natural reach-and-grasp movement observation and delayed imitation task with 3 different natural grasp types of everyday objects. We focused on the classification of grasp types from high-frequency broadband mirror activation patterns found in classic mirror system areas, including sensorimotor, supplementary motor, inferior frontal, and parietal cortices. Classification of grasp types was successful during movement observation and execution intervals but not during movement retention. Our grasp type classification from combined and single mirror electrodes provides evidence for grasp-congruent activity in the human mirror neuron system potentially arising from strictly congruent mirror neurons.

Learning nonnative speech sounds changes local encoding in the adult human cortex.

Adults can learn to identify nonnative speech sounds with training, albeit with substantial variability in learning behavior. Increases in behavioral accuracy are associated with increased separability for sound representations in cortical speech areas. However, it remains unclear whether individual auditory neural populations all show the same types of changes with learning, or whether there are heterogeneous encoding patterns. Here, we used high-resolution direct neural recordings to examine local population response patterns, while native English listeners learned to recognize unfamiliar vocal pitch patterns in Mandarin Chinese tones. We found a distributed set of neural populations in bilateral superior temporal gyrus and ventrolateral frontal cortex, where the encoding of Mandarin tones changed throughout training as a function of trial-by-trial accuracy ("learning effect"), including both increases and decreases in the separability of tones. These populations were distinct from populations that showed changes as a function of exposure to the stimuli regardless of trial-by-trial accuracy. These learning effects were driven in part by more variable neural responses to repeated presentations of acoustically identical stimuli. Finally, learning effects could be predicted from speech-evoked activity even before training, suggesting that intrinsic properties of these populations make them amenable to behavior-related changes. Together, these results demonstrate that nonnative speech sound learning involves a wide array of changes in neural representations across a distributed set of brain regions.

Functional alterations in cortical processing of speech in glioma-infiltrated cortex.

Recent developments in the biology of malignant gliomas have demonstrated that glioma cells interact with neurons through both paracrine signaling and electrochemical synapses. Glioma-neuron interactions consequently modulate the excitability of local neuronal circuits, and it is unclear the extent to which glioma-infiltrated cortex can meaningfully participate in neural computations. For example, gliomas may result in a local disorganization of activity that impedes the transient synchronization of neural oscillations. Alternatively, glioma-infiltrated cortex may retain the ability to engage in synchronized activity in a manner similar to normal-appearing cortex but exhibit other altered spatiotemporal patterns of activity with subsequent impact on cognitive processing. Here, we use subdural electrocorticography to sample both normal-appearing and glioma-infiltrated cortex during speech. We find that glioma-infiltrated cortex engages in synchronous activity during task performance in a manner similar to normal-appearing cortex but recruits a diffuse spatial network. On a temporal scale, we show that signals from glioma-infiltrated cortex have decreased entropy, which may affect its ability to encode information during nuanced tasks such as production of monosyllabic versus polysyllabic words. Furthermore, we show that temporal decoding strategies for distinguishing monosyllabic from polysyllabic words were feasible for signals arising from normal-appearing cortex but not from glioma-infiltrated cortex. These findings inform our understanding of cognitive processing in chronic disease states and have implications for neuromodulation and prosthetics in patients with malignant gliomas.

A Neurosurgical Functional Dissection of the Middle Precentral Gyrus during Speech Production.

Classical models have traditionally focused on the left posterior inferior frontal gyrus (Broca's area) as a key region for motor planning of speech production. However, converging evidence suggests that it is not critical for either speech motor planning or execution. Alternative cortical areas supporting high-level speech motor planning have yet to be defined. In this review, we focus on the precentral gyrus, whose role in speech production is often thought to be limited to lower-level articulatory muscle control. In particular, we highlight neurosurgical investigations that have shed light on a cortical region anatomically located near the midpoint of the precentral gyrus, hence called the middle precentral gyrus (midPrCG). The midPrCG is functionally located between dorsal hand and ventral orofacial cortical representations and exhibits unique sensorimotor and multisensory functions relevant for speech processing. This includes motor control of the larynx, auditory processing, as well as a role in reading and writing. Furthermore, direct electrical stimulation of midPrCG can evoke complex movements, such as vocalization, and selective injury can cause deficits in verbal fluency, such as pure apraxia of speech. Based on these findings, we propose that midPrCG is essential to phonological-motoric aspects of speech production, especially syllabic-level speech sequencing, a role traditionally ascribed to Broca's area. The midPrCG is a cortical brain area that should be included in contemporary models of speech production with a unique role in speech motor planning and execution.