Decoding the brain: From neural representations to mechanistic models.

A central principle in neuroscience is that neurons within the brain act in concert to produce perception, cognition, and adaptive behavior. Neurons are organized into specialized brain areas, dedicated to different functions to varying extents, and their function relies on distributed circuits to continuously encode relevant environmental and body-state features, enabling other areas to decode (interpret) these representations for computing meaningful decisions and executing precise movements. Thus, the distributed brain can be thought of as a series of computations that act to encode and decode information. In this perspective, we detail important concepts of neural encoding and decoding and highlight the mathematical tools used to measure them, including deep learning methods. We provide case studies where decoding concepts enable foundational and translational science in motor, visual, and language processing.

Restoring the Sense of Touch Using a Sensorimotor Demultiplexing Neural Interface.

Paralyzed muscles can be reanimated following spinal cord injury (SCI) using a brain-computer interface (BCI) to enhance motor function alone. Importantly, the sense of touch is a key component of motor function. Here, we demonstrate that a human participant with a clinically complete SCI can use a BCI to simultaneously reanimate both motor function and the sense of touch, leveraging residual touch signaling from his own hand. In the primary motor cortex (M1), residual subperceptual hand touch signals are simultaneously demultiplexed from ongoing efferent motor intention, enabling intracortically controlled closed-loop sensory feedback. Using the closed-loop demultiplexing BCI almost fully restored the ability to detect object touch and significantly improved several sensorimotor functions. Afferent grip-intensity levels are also decoded from M1, enabling grip reanimation regulated by touch signaling. These results demonstrate that subperceptual neural signals can be decoded from the cortex and transformed into conscious perception, significantly augmenting function.

Dynamic Ligand Discrimination in the Notch Signaling Pathway.

The Notch signaling pathway comprises multiple ligands that are used in distinct biological contexts. In principle, different ligands could activate distinct target programs in signal-receiving cells, but it is unclear how such ligand discrimination could occur. Here, we show that cells use dynamics to discriminate signaling by the ligands Dll1 and Dll4 through the Notch1 receptor. Quantitative single-cell imaging revealed that Dll1 activates Notch1 in discrete, frequency-modulated pulses that specifically upregulate the Notch target gene Hes1. By contrast, Dll4 activates Notch1 in a sustained, amplitude-modulated manner that predominantly upregulates Hey1 and HeyL. Ectopic expression of Dll1 or Dll4 in chick neural crest produced opposite effects on myogenic differentiation, showing that ligand discrimination can occur in vivo. Finally, analysis of chimeric ligands suggests that ligand-receptor clustering underlies dynamic encoding of ligand identity. The ability of the pathway to utilize ligands as distinct communication channels has implications for diverse Notch-dependent processes.

The Code for Facial Identity in the Primate Brain.

Primates recognize complex objects such as faces with remarkable speed and reliability. Here, we reveal the brain's code for facial identity. Experiments in macaques demonstrate an extraordinarily simple transformation between faces and responses of cells in face patches. By formatting faces as points in a high-dimensional linear space, we discovered that each face cell's firing rate is proportional to the projection of an incoming face stimulus onto a single axis in this space, allowing a face cell ensemble to encode the location of any face in the space. Using this code, we could precisely decode faces from neural population responses and predict neural firing rates to faces. Furthermore, this code disavows the long-standing assumption that face cells encode specific facial identities, confirmed by engineering faces with drastically different appearance that elicited identical responses in single face cells. Our work suggests that other objects could be encoded by analogous metric coordinate systems. PAPERCLIP.

Decoding Mammalian Ribosome-mRNA States by Translational GTPase Complexes.

In eukaryotes, accurate protein synthesis relies on a family of translational GTPases that pair with specific decoding factors to decipher the mRNA code on ribosomes. We present structures of the mammalian ribosome engaged with decoding factor⋅GTPase complexes representing intermediates of translation elongation (aminoacyl-tRNA⋅eEF1A), termination (eRF1⋅eRF3), and ribosome rescue (Pelota⋅Hbs1l). Comparative analyses reveal that each decoding factor exploits the plasticity of the ribosomal decoding center to differentially remodel ribosomal proteins and rRNA. This leads to varying degrees of large-scale ribosome movements and implies distinct mechanisms for communicating information from the decoding center to each GTPase. Additional structural snapshots of the translation termination pathway reveal the conformational changes that choreograph the accommodation of decoding factors into the peptidyl transferase center. Our results provide a structural framework for how different states of the mammalian ribosome are selectively recognized by the appropriate decoding factor⋅GTPase complex to ensure translational fidelity.

Lack of evidence for ribosomal frameshifting in ATP7B mRNA decoding.

The research article describing the discovery of ribosomal frameshifting in the bacterial CopA gene also reported the occurrence of frameshifting in the expression of the human ortholog ATP7B based on assays using dual luciferase reporters. An examination of the publicly available ribosome profiling data and the phylogenetic analysis of the proposed frameshifting site cast doubt on the validity of this claim and prompted us to reexamine the evidence. We observed similar apparent frameshifting efficiencies as the original authors using the same type of vector that synthesizes both luciferases as a single polyprotein. However, we noticed anomalously low absolute luciferase activities from the N-terminal reporter that suggests interference of reporter activity or levels by the ATP7B test cassette. When we tested the same proposed ATP7B frameshifting cassette in a more recently developed reporter system in which the reporters are released without being included in a polyprotein, no frameshifting was detected above background levels.

Peeling the Onion of Brain Representations.

The brain's function is to enable adaptive behavior in the world. To this end, the brain processes information about the world. The concept of representation links the information processed by the brain back to the world and enables us to understand what the brain does at a functional level. The appeal of making the connection between brain activity and what it represents has been irresistible to neuroscience, despite the fact that representational interpretations pose several challenges: We must define which aspects of brain activity matter, how the code works, and how it supports computations that contribute to adaptive behavior. It has been suggested that we might drop representational language altogether and seek to understand the brain, more simply, as a dynamical system. In this review, we argue that the concept of representation provides a useful link between dynamics and computational function and ask which aspects of brain activity should be analyzed to achieve a representational understanding. We peel the onion of brain representations in search of the layers (the aspects of brain activity) that matter to computation. The article provides an introduction to the motivation and mathematics of representational models, a critical discussion of their assumptions and limitations, and a preview of future directions in this area.

How auditory neurons count temporal intervals and decode information.

The numerical sense of animals includes identifying the numerosity of a sequence of events that occur with specific intervals, e.g., notes in a call or bar of music. Across nervous systems, the temporal patterning of spikes can code these events, but how this information is decoded (counted) remains elusive. In the anuran auditory system, temporal information of this type is decoded in the midbrain, where "interval-counting" neurons spike only after at least a threshold number of sound pulses have occurred with specific timing. We show that this decoding process, i.e., interval counting, arises from integrating phasic, onset-type and offset inhibition with excitation that augments across successive intervals, possibly due to a progressive decrease in "shunting" effects of inhibition. Because these physiological properties are ubiquitous within and across central nervous systems, interval counting may be a general mechanism for decoding diverse information coded/encoded in temporal patterns of spikes, including "bursts," and estimating elapsed time.

High-level cognition is supported by information-rich but compressible brain activity patterns.

To efficiently yet reliably represent and process information, our brains need to produce information-rich signals that differentiate between moments or cognitive states, while also being robust to noise or corruption. For many, though not all, natural systems, these two properties are often inversely related: More information-rich signals are less robust, and vice versa. Here, we examined how these properties change with ongoing cognitive demands. To this end, we applied dimensionality reduction algorithms and pattern classifiers to functional neuroimaging data collected as participants listened to a story, temporally scrambled versions of the story, or underwent a resting state scanning session. We considered two primary aspects of the neural data recorded in these different experimental conditions. First, we treated the maximum achievable decoding accuracy across participants as an indicator of the "informativeness" of the recorded patterns. Second, we treated the number of features (components) required to achieve a threshold decoding accuracy as a proxy for the "compressibility" of the neural patterns (where fewer components indicate greater compression). Overall, we found that the peak decoding accuracy (achievable without restricting the numbers of features) was highest in the intact (unscrambled) story listening condition. However, the number of features required to achieve comparable classification accuracy was also lowest in the intact story listening condition. Taken together, our work suggests that our brain networks flexibly reconfigure according to ongoing task demands and that the activity patterns associated with higher-order cognition and high engagement are both more informative and more compressible than the activity patterns associated with lower-order tasks and lower engagement.

Brain decoding of spontaneous thought: Predictive modeling of self-relevance and valence using personal narratives.

The contents and dynamics of spontaneous thought are important factors for personality traits and mental health. However, assessing spontaneous thoughts is challenging due to their unconstrained nature, and directing participants' attention to report their thoughts may fundamentally alter them. Here, we aimed to decode two key content dimensions of spontaneous thought-self-relevance and valence-directly from brain activity. To train functional MRI-based predictive models, we used individually generated personal stories as stimuli in a story-reading task to mimic narrative-like spontaneous thoughts (n = 49). We then tested these models on multiple test datasets (total n = 199). The default mode, ventral attention, and frontoparietal networks played key roles in the predictions, with the anterior insula and midcingulate cortex contributing to self-relevance prediction and the left temporoparietal junction and dorsomedial prefrontal cortex contributing to valence prediction. Overall, this study presents brain models of internal thoughts and emotions, highlighting the potential for the brain decoding of spontaneous thought.

tRNAVal allows four-way decoding with unmodified uridine at the wobble position in Lactobacillus casei.

Modifications at the wobble position (position 34) of tRNA facilitate interactions that enable or stabilize non-Watson-Crick basepairs. In bacterial tRNA, 5-hydroxyuridine (ho5U) derivatives xo5U [x: methyl (mo5U), carboxymethyl (cmo5U), and methoxycarbonylmethyl (mcmo5U)] present at the wobble positions of tRNAs are responsible for recognition of NYN codon families. These modifications of U34 allow basepairing not only with A and G but also with U and in some cases C. mo5U was originally found in Gram-positive bacteria, and cmo5U and mcmo5U were found in Gram-negative bacteria. tRNAs of Mycoplasma species, mitochondria, and chloroplasts adopt four-way decoding in which unmodified U34 recognizes codons ending in A, G, C, and U. Lactobacillus casei, Gram-positive bacteria and lactic acid bacteria, lacks the modification enzyme genes for xo5U biosynthesis. Nevertheless, L. casei has only one type of tRNAVal with the anticodon UAC [tRNAVal(UAC)]. However, the genome of L. casei encodes an undetermined tRNA (tRNAUnd) gene, and the sequence corresponding to the anticodon region is GAC. Here, we confirm that U34 in L. casei tRNAVal is unmodified and that there is no tRNAUnd expression in the cells. In addition, in vitro transcribed tRNAUnd was not aminoacylated by L. casei valyl-tRNA synthetase suggesting that tRNAUnd is not able to accept valine, even if expressed in cells. Correspondingly, native tRNAVal(UAC) with unmodified U34 bound to all four valine codons in the ribosome A site. This suggests that L. casei tRNAVal decodes all valine codons by four-way decoding, similarly to tRNAs from Mycoplasma species, mitochondria, and chloroplasts.

Propagating spatiotemporal activity patterns across macaque motor cortex carry kinematic information.

Propagating spatiotemporal neural patterns are widely evident across sensory, motor, and association cortical areas. However, it remains unclear whether any characteristics of neural propagation carry information about specific behavioral details. Here, we provide the first evidence for a link between the direction of cortical propagation and specific behavioral features of an upcoming movement on a trial-by-trial basis. We recorded local field potentials (LFPs) from multielectrode arrays implanted in the primary motor cortex of two rhesus macaque monkeys while they performed a 2D reach task. Propagating patterns were extracted from the information-rich high-gamma band (200 to 400 Hz) envelopes in the LFP amplitude. We found that the exact direction of propagating patterns varied systematically according to initial movement direction, enabling kinematic predictions. Furthermore, characteristics of these propagation patterns provided additional predictive capability beyond the LFP amplitude themselves, which suggests the value of including mesoscopic spatiotemporal characteristics in refining brain-machine interfaces.

Calibrating Bayesian Decoders of Neural Spiking Activity.

Accurately decoding external variables from observations of neural activity is a major challenge in systems neuroscience. Bayesian decoders, which provide probabilistic estimates, are some of the most widely used. Here we show how, in many common settings, the probabilistic predictions made by traditional Bayesian decoders are overconfident. That is, the estimates for the decoded stimulus or movement variables are more certain than they should be. We then show how Bayesian decoding with latent variables, taking account of low-dimensional shared variability in the observations, can improve calibration, although additional correction for overconfidence is still needed. Using data from males, we examine (1) decoding the direction of grating stimuli from spike recordings in the primary visual cortex in monkeys, (2) decoding movement direction from recordings in the primary motor cortex in monkeys, (3) decoding natural images from multiregion recordings in mice, and (4) decoding position from hippocampal recordings in rats. For each setting, we characterize the overconfidence, and we describe a possible method to correct miscalibration post hoc. Properly calibrated Bayesian decoders may alter theoretical results on probabilistic population coding and lead to brain-machine interfaces that more accurately reflect confidence levels when identifying external variables.

Trying Harder: How Cognitive Effort Sculpts Neural Representations during Working Memory.

While the exertion of mental effort improves performance on cognitive tasks, the neural mechanisms by which motivational factors impact cognition remain unknown. Here, we used fMRI to test how changes in cognitive effort, induced by changes in task difficulty, impact neural representations of working memory (WM). Participants (both sexes) were precued whether WM difficulty would be hard or easy. We hypothesized that hard trials demanded more effort as a later decision required finer mnemonic precision. Behaviorally, pupil size was larger and response times were slower on hard compared with easy trials suggesting our manipulation of effort succeeded. Neurally, we observed robust persistent activity during delay periods in the prefrontal cortex (PFC), especially during hard trials. Yet, details of the memoranda could not be decoded from patterns in prefrontal activity. In the patterns of activity in the visual cortex, however, we found strong decoding of memorized targets, where accuracy was higher on hard trials. To potentially link these across-region effects, we hypothesized that effort, carried by persistent activity in the PFC, impacts the quality of WM representations encoded in the visual cortex. Indeed, we found that the amplitude of delay period activity in the frontal cortex predicted decoded accuracy in the visual cortex on a trial-wise basis. These results indicate that effort-related feedback signals sculpt population activity in the visual cortex, improving mnemonic fidelity.

Integrated Perceptual Decisions Rely on Parallel Evidence Accumulation.

The ability to make accurate and timely decisions, such as judging when it is safe to cross the road, is the foundation of adaptive behavior. While the computational and neural processes supporting simple decisions on isolated stimuli have been well characterized, decision-making in the real world often requires integration of discrete sensory events over time and space. Most previous experimental work on perceptual decision-making has focused on tasks that involve only a single, task-relevant source of sensory input. It remains unclear, therefore, how such integrative decisions are regulated computationally. Here we used psychophysics, electroencephalography, and computational modeling to understand how the human brain combines visual motion signals across space in the service of a single, integrated decision. To that purpose, we presented two random-dot kinematograms in the left and the right visual hemifields. Coherent motion signals were shown briefly and concurrently in each location, and healthy adult human participants of both sexes reported the average of the two motion signals. We directly tested competing predictions arising from influential serial and parallel accounts of visual processing. Using a biologically plausible model of motion filtering, we found evidence in favor of parallel integration as the fundamental computational mechanism regulating integrated perceptual decisions.

Pupil-Linked Arousal Modulates Precision of Stimulus Representation in Cortex.

Neural responses are naturally variable from one moment to the next, even when the stimulus is held constant. What factors might underlie this variability in neural population activity? We hypothesized that spontaneous fluctuations in cortical stimulus representations are created by changes in arousal state. We tested the hypothesis using a combination of fMRI, probabilistic decoding methods, and pupillometry. Human participants (20 female, 12 male) were presented with gratings of random orientation. Shortly after viewing the grating, participants reported its orientation and gave their level of confidence in this judgment. Using a probabilistic fMRI decoding technique, we quantified the precision of the stimulus representation in the visual cortex on a trial-by-trial basis. Pupil size was recorded and analyzed to index the observer's arousal state. We found that the precision of the cortical stimulus representation, reported confidence, and variability in the behavioral orientation judgments varied from trial to trial. Interestingly, these trial-by-trial changes in cortical and behavioral precision and confidence were linked to pupil size and its temporal rate of change. Specifically, when the cortical stimulus representation was more precise, the pupil dilated more strongly prior to stimulus onset and remained larger during stimulus presentation. Similarly, stronger pupil dilation during stimulus presentation was associated with higher levels of subjective confidence, a secondary measure of sensory precision, as well as improved behavioral performance. Taken together, our findings support the hypothesis that spontaneous fluctuations in arousal state modulate the fidelity of the stimulus representation in the human visual cortex, with clear consequences for behavior.

Distinct Neuron Types Contribute to Hybrid Auditory Spatial Coding.

Neural decoding is a tool for understanding how activities from a population of neurons inside the brain relate to the outside world and for engineering applications such as brain-machine interfaces. However, neural decoding studies mainly focused on different decoding algorithms rather than different neuron types which could use different coding strategies. In this study, we used two-photon calcium imaging to assess three auditory spatial decoders (space map, opponent channel, and population pattern) in excitatory and inhibitory neurons in the dorsal inferior colliculus of male and female mice. Our findings revealed a clustering of excitatory neurons that prefer similar interaural level difference (ILD), the primary spatial cues in mice, while inhibitory neurons showed random local ILD organization. We found that inhibitory neurons displayed lower decoding variability under the opponent channel decoder, while excitatory neurons achieved higher decoding accuracy under the space map and population pattern decoders. Further analysis revealed that the inhibitory neurons' preference for ILD off the midline and the excitatory neurons' heterogeneous ILD tuning account for their decoding differences. Additionally, we discovered a sharper ILD tuning in the inhibitory neurons. Our computational model, linking this to increased presynaptic inhibitory inputs, was corroborated using monaural and binaural stimuli. Overall, this study provides experimental and computational insight into how excitatory and inhibitory neurons uniquely contribute to the coding of sound locations.

Dissociable contributions of the hippocampus and orbitofrontal cortex to representing task space in a social context.

The hippocampus (HC) and the orbitofrontal cortex (OFC) jointly encode a map-like representation of a task space to guide behavior. It remains unclear how the OFC and HC interact in encoding this map-like representation, though previous studies indicated that both regions have different functions. We acquired the functional magnetic resonance imaging data under a social navigation task in which participants interacted with characters in a two-dimensional "social space." We calculate the social relationships between the participants and characters and used a drift-diffusion model to capture the inner process of social interaction. Then we used multivoxel pattern analysis to explore the brain-behavior relationship. We found that (i) both the HC and the OFC showed higher activations during the selective trial than the narrative trial; (ii) the neural pattern of the right HC was associated with evidence accumulation during social interaction, and the pattern of the right lateral OFC was associated with the social relationship; (iii) the neural pattern of the HC can decode the participants choices, while the neural pattern of the OFC can decode the task information about trials. The study provided evidence for distinct roles of the HC and the OFC in encoding different information when representing social space.

Investigating the different mechanisms in related neural activities: a focus on auditory perception and imagery.

Neuroimaging studies have shown that the neural representation of imagery is closely related to the perception modality; however, the undeniable different experiences between perception and imagery indicate that there are obvious neural mechanism differences between them, which cannot be explained by the simple theory that imagery is a form of weak perception. Considering the importance of functional integration of brain regions in neural activities, we conducted correlation analysis of neural activity in brain regions jointly activated by auditory imagery and perception, and then brain functional connectivity (FC) networks were obtained with a consistent structure. However, the connection values between the areas in the superior temporal gyrus and the right precentral cortex were significantly higher in auditory perception than in the imagery modality. In addition, the modality decoding based on FC patterns showed that the FC network of auditory imagery and perception can be significantly distinguishable. Subsequently, voxel-level FC analysis further verified the distribution regions of voxels with significant connectivity differences between the 2 modalities. This study complemented the correlation and difference between auditory imagery and perception in terms of brain information interaction, and it provided a new perspective for investigating the neural mechanisms of different modal information representations.

Early language dissociation in bilingual minds: magnetoencephalography evidence through a machine learning approach.

Does neural activity reveal how balanced bilinguals choose languages? Despite using diverse neuroimaging techniques, prior studies haven't provided a definitive solution to this problem. Nonetheless, studies involving direct brain stimulation in bilinguals have identified distinct brain regions associated with language production in different languages. In this magnetoencephalography study with 45 proficient Spanish-Basque bilinguals, we investigated language selection during covert picture naming and word reading tasks. Participants were prompted to name line drawings or read words if the color of the stimulus changed to green, in 10% of trials. The task was performed either in Spanish or Basque. Despite similar sensor-level evoked activity for both languages in both tasks, decoding analyses revealed language-specific classification ~100 ms post-stimulus onset. During picture naming, right occipital-temporal sensors predominantly contributed to language decoding, while left occipital-temporal sensors were crucial for decoding during word reading. Cross-task decoding analysis unveiled robust generalization effects from picture naming to word reading. Our methodology involved a fine-grained examination of neural responses using magnetoencephalography, offering insights into the dynamics of language processing in bilinguals. This study refines our understanding of the neural underpinnings of language selection and bridges the gap between non-invasive and invasive experimental evidence in bilingual language production.