Crossmodal hierarchical predictive coding for audiovisual sequences in the human brain.

Predictive coding theory suggests the brain anticipates sensory information using prior knowledge. While this theory has been extensively researched within individual sensory modalities, evidence for predictive processing across sensory modalities is limited. Here, we examine how crossmodal knowledge is represented and learned in the brain, by identifying the hierarchical networks underlying crossmodal predictions when information of one sensory modality leads to a prediction in another modality. We record electroencephalogram (EEG) during a crossmodal audiovisual local-global oddball paradigm, in which the predictability of transitions between tones and images are manipulated at both the stimulus and sequence levels. To dissect the complex predictive signals in our EEG data, we employed a model-fitting approach to untangle neural interactions across modalities and hierarchies. The model-fitting result demonstrates that audiovisual integration occurs at both the levels of individual stimulus interactions and multi-stimulus sequences. Furthermore, we identify the spatio-spectro-temporal signatures of prediction-error signals across hierarchies and modalities, and reveal that auditory and visual prediction errors are rapidly redirected to the central-parietal electrodes during learning through alpha-band interactions. Our study suggests a crossmodal predictive coding mechanism where unimodal predictions are processed by distributed brain networks to form crossmodal knowledge.

Erroneous predictive coding across brain hierarchies in a non-human primate model of autism spectrum disorder.

In autism spectrum disorder (ASD), atypical sensory experiences are often associated with irregularities in predictive coding, which proposes that the brain creates hierarchical sensory models via a bidirectional process of predictions and prediction errors. However, it remains unclear how these irregularities manifest across different functional hierarchies in the brain. To address this, we study a marmoset model of ASD induced by valproic acid (VPA) treatment. We record high-density electrocorticography (ECoG) during an auditory task with two layers of temporal control, and applied a quantitative model to quantify the integrity of predictive coding across two distinct hierarchies. Our results demonstrate a persistent pattern of sensory hypersensitivity and unstable predictions across two brain hierarchies in VPA-treated animals, and reveal the associated spatio-spectro-temporal neural signatures. Despite the regular occurrence of imprecise predictions in VPA-treated animals, we observe diverse configurations of underestimation or overestimation of sensory regularities within the hierarchies. Our results demonstrate the coexistence of the two primary Bayesian accounts of ASD: overly-precise sensory observations and weak prior beliefs, and offer a potential multi-layered biomarker for ASD, which could enhance our understanding of its diverse symptoms.

Assessing novelty, feasibility and value of creative ideas with an unsupervised approach using GPT-4.

Creativity is defined by three key factors: novelty, feasibility and value. While many creativity tests focus primarily on novelty, they often neglect feasibility and value, thereby limiting their reflection of real-world creativity. In this study, we employ GPT-4, a large language model, to assess these three dimensions in a Japanese-language Alternative Uses Test (AUT). Using a crowdsourced evaluation method, we acquire ground truth data for 30 question items and test various GPT prompt designs. Our findings show that asking for multiple responses in a single prompt, using an 'explain first, rate later' design, is both cost-effective and accurate (r = .62, .59 and .33 for novelty, feasibility and value, respectively). Moreover, our method offers comparable accuracy to existing methods in assessing novelty, without the need for training data. We also evaluate additional models such as GPT-4 Turbo, GPT-4 Omni and Claude 3.5 Sonnet. Comparable performance across these models demonstrates the universal applicability of our prompt design. Our results contribute a straightforward platform for instant AUT evaluation and provide valuable ground truth data for future methodological research.

Multivariate sharp-wave ripples in schizophrenia during awake state.

AIMS: Schizophrenia (SZ) is a brain disorder characterized by psychotic symptoms and cognitive dysfunction. Recently, irregularities in sharp-wave ripples (SPW-Rs) have been reported in SZ. As SPW-Rs play a critical role in memory, their irregularities can cause psychotic symptoms and cognitive dysfunction in patients with SZ. In this study, we investigated the SPW-Rs in human SZ. METHODS: We measured whole-brain activity using magnetoencephalography (MEG) in patients with SZ (n = 20) and sex- and age-matched healthy participants (n = 20) during open-eye rest. We identified SPW-Rs and analyzed their occurrence and time-frequency traits. Furthermore, we developed a novel multivariate analysis method, termed "ripple-gedMEG" to extract the global features of SPW-Rs. We also examined the association between SPW-Rs and brain state transitions. The outcomes of these analyses were modeled to predict the positive and negative syndrome scale (PANSS) scores of SZ. RESULTS: We found that SPW-Rs in the SZ (1) occurred more frequently, (2) the delay of the coupling phase (3) appeared in different brain areas, (4) consisted of a less organized spatiotemporal pattern, and (5) were less involved in brain state transitions. Finally, some of the neural features associated with the SPW-Rs were found to be PANSS-positive, a pathological indicator of SZ. These results suggest that widespread but disorganized SPW-Rs underlies the symptoms of SZ. CONCLUSION: We identified irregularities in SPW-Rs in SZ and confirmed that their alternations were strongly associated with SZ neuropathology. These results suggest a new direction for human SZ research.

Dissecting Mismatch Negativity: Early and Late Subcomponents for Detecting Deviants in Local and Global Sequence Regularities.

Mismatch negativity (MMN) is commonly recognized as a neural signal of prediction error evoked by deviants from the expected patterns of sensory input. Studies show that MMN diminishes when sequence patterns become more predictable over a longer timescale. This implies that MMN is composed of multiple subcomponents, each responding to different levels of temporal regularities. To probe the hypothesized subcomponents in MMN, we record human electroencephalography during an auditory local-global oddball paradigm where the tone-to-tone transition probability (local regularity) and the overall sequence probability (global regularity) are manipulated to control temporal predictabilities at two hierarchical levels. We find that the size of MMN is correlated with both probabilities and the spatiotemporal structure of MMN can be decomposed into two distinct subcomponents. Both subcomponents appear as negative waveforms, with one peaking early in the central-frontal area and the other late in a more frontal area. With a quantitative predictive coding model, we map the early and late subcomponents to the prediction errors that are tied to local and global regularities, respectively. Our study highlights the hierarchical complexity of MMN and offers an experimental and analytical platform for developing a multitiered neural marker applicable in clinical settings.

Short-term neuronal and synaptic plasticity act in synergy for deviance detection in spiking networks.

Sensory areas of cortex respond more strongly to infrequent stimuli when these violate previously established regularities, a phenomenon known as deviance detection (DD). Previous modeling work has mainly attempted to explain DD on the basis of synaptic plasticity. However, a large fraction of cortical neurons also exhibit firing rate adaptation, an underexplored potential mechanism. Here, we investigate DD in a spiking neuronal network model with two types of short-term plasticity, fast synaptic short-term depression (STD) and slower threshold adaptation (TA). We probe the model with an oddball stimulation paradigm and assess DD by evaluating the network responses. We find that TA is sufficient to elicit DD. It achieves this by habituating neurons near the stimulation site that respond earliest to the frequently presented standard stimulus (local fatigue), which diminishes the response and promotes the recovery (global fatigue) of the wider network. Further, we find a synergy effect between STD and TA, where they interact with each other to achieve greater DD than the sum of their individual effects. We show that this synergy is caused by the local fatigue added by STD, which inhibits the global response to the frequently presented stimulus, allowing greater recovery of TA-mediated global fatigue and making the network more responsive to the deviant stimulus. Finally, we show that the magnitude of DD strongly depends on the timescale of stimulation. We conclude that highly predictable information can be encoded in strong local fatigue, which allows greater global recovery and subsequent heightened sensitivity for DD.

Efficient, continual, and generalized learning in the brain - neural mechanism of Mental Schema 2.0.

There has been tremendous progress in artificial neural networks (ANNs) over the past decade; however, the gap between ANNs and the biological brain as a learning device remains large. With the goal of closing this gap, this paper reviews learning mechanisms in the brain by focusing on three important issues in ANN research: efficiency, continuity, and generalization. We first discuss the method by which the brain utilizes a variety of self-organizing mechanisms to maximize learning efficiency, with a focus on the role of spontaneous activity of the brain in shaping synaptic connections to facilitate spatiotemporal learning and numerical processing. Then, we examined the neuronal mechanisms that enable lifelong continual learning, with a focus on memory replay during sleep and its implementation in brain-inspired ANNs. Finally, we explored the method by which the brain generalizes learned knowledge in new situations, particularly from the mathematical generalization perspective of topology. Besides a systematic comparison in learning mechanisms between the brain and ANNs, we propose "Mental Schema 2.0," a new computational property underlying the brain's unique learning ability that can be implemented in ANNs.

Altered coordination between frontal delta and parietal alpha networks underlies anhedonia and depressive rumination in major depressive disorder.

BACKGROUND: A hyperactive default mode network (DMN) has been observed in people with major depressive disorder (MDD), and weak DMN suppression has been linked to depressive symptoms. However, whether dysregulation of the DMN contributes to blunted positive emotional experience in people with MDD is unclear. METHODS: We recorded 128-channel electroencephalograms (EEGs) from 24 participants with MDD and 31 healthy controls in a resting state (RS) and an emotion-induction state (ES), in which participants engaged with emotionally positive pictures. We combined Granger causality analysis and data-driven decomposition to extract latent brain networks shared among states and groups, and we further evaluated their interactions across individuals. RESULTS: We extracted 2 subnetworks. Subnetwork 1 represented a delta (δ)-band (1~4 Hz) frontal network that was activated more in the ES than the RS (i.e., task-positive). Subnetwork 2 represented an alpha (α)-band (8~13 Hz) parietal network that was suppressed more in the ES than the RS (i.e., task-negative). These subnetworks were anticorrelated in both the healthy control and MDD groups, but with different sensitivities: for participants with MDD to achieve the same level of task-positive (subnetwork 1) activation as healthy controls, more suppression of task-negative (subnetwork 2) activation was necessary. Furthermore, the anticorrelation strength in participants with MDD correlated with the severity of 2 core MDD symptoms: anhedonia and rumination. LIMITATIONS: The sample size was small. CONCLUSION: Our findings revealed altered coordination between 2 functional networks in MDD and suggest that weak suppression of the task-negative α-band parietal network contributes to blunted positive emotional responses in adults with depression. The subnetworks identified here could be used for diagnosis or targeted for treatment in the future.

A quantitative model reveals a frequency ordering of prediction and prediction-error signals in the human brain.

The human brain is proposed to harbor a hierarchical predictive coding neuronal network underlying perception, cognition, and action. In support of this theory, feedforward signals for prediction error have been reported. However, the identification of feedback prediction signals has been elusive due to their causal entanglement with prediction-error signals. Here, we use a quantitative model to decompose these signals in electroencephalography during an auditory task, and identify their spatio-spectral-temporal signatures across two functional hierarchies. Two prediction signals are identified in the period prior to the sensory input: a low-level signal representing the tone-to-tone transition in the high beta frequency band, and a high-level signal for the multi-tone sequence structure in the low beta band. Subsequently, prediction-error signals dependent on the prior predictions are found in the gamma band. Our findings reveal a frequency ordering of prediction signals and their hierarchical interactions with prediction-error signals supporting predictive coding theory.

Dynamic Reorganization of Motor Networks During Recovery from Partial Spinal Cord Injury in Monkeys.

After spinal cord injury (SCI), the motor-related cortical areas can be a potential substrate for functional recovery in addition to the spinal cord. However, a dynamic description of how motor cortical circuits reorganize after SCI is lacking. Here, we captured the comprehensive dynamics of motor networks across SCI in a nonhuman primate model. Using electrocorticography over the sensorimotor areas in monkeys, we collected broadband neuronal signals during a reaching-and-grasping task at different stages of recovery of dexterous finger movements after a partial SCI at the cervical levels. We identified two distinct network dynamics: grasping-related intrahemispheric interactions from the contralesional premotor cortex (PM) to the contralesional primary motor cortex (M1) in the high-γ band (>70 Hz), and motor-preparation-related interhemispheric interactions from the contralesional to ipsilesional PM in the α and low-ß bands (10-15 Hz). The strengths of these networks correlated to the time course of behavioral recovery. The grasping-related network showed enhanced activation immediately after the injury, but gradually returned to normal while the strength of the motor-preparation-related network gradually increased. Our findings suggest a cortical compensatory mechanism after SCI, where two interdependent motor networks redirect activity from the contralesional hemisphere to the other hemisphere to facilitate functional recovery.

Large-Scale Cortical Networks for Hierarchical Prediction and Prediction Error in the Primate Brain.

According to predictive-coding theory, cortical areas continuously generate and update predictions of sensory inputs at different hierarchical levels and emit prediction errors when the predicted and actual inputs differ. However, predictions and prediction errors are simultaneous and interdependent processes, making it difficult to disentangle their constituent neural network organization. Here, we test the theory by using high-density electrocorticography (ECoG) in monkeys during an auditory "local-global" paradigm in which the temporal regularities of the stimuli were controlled at two hierarchical levels. We decomposed the broadband data and identified lower- and higher-level prediction-error signals in early auditory cortex and anterior temporal cortex, respectively, and a prediction-update signal sent from prefrontal cortex back to temporal cortex. The prediction-error and prediction-update signals were transmitted via γ (>40 Hz) and α/ß (<30 Hz) oscillations, respectively. Our findings provide strong support for hierarchical predictive coding and outline how it is dynamically implemented using distinct cortical areas and frequencies.

Cortical network architecture for context processing in primate brain.

Context is information linked to a situation that can guide behavior. In the brain, context is encoded by sensory processing and can later be retrieved from memory. How context is communicated within the cortical network in sensory and mnemonic forms is unknown due to the lack of methods for high-resolution, brain-wide neuronal recording and analysis. Here, we report the comprehensive architecture of a cortical network for context processing. Using hemisphere-wide, high-density electrocorticography, we measured large-scale neuronal activity from monkeys observing videos of agents interacting in situations with different contexts. We extracted five context-related network structures including a bottom-up network during encoding and, seconds later, cue-dependent retrieval of the same network with the opposite top-down connectivity. These findings show that context is represented in the cortical network as distributed communication structures with dynamic information flows. This study provides a general methodology for recording and analyzing cortical network neuronal communication during cognition.

Studying brain functions with mesoscopic measurements: Advances in electrocorticography for non-human primates.

Our brain is organized in a modular structure. Information in different modalities is processed within distinct cortical areas. However, individual cortical areas cannot enable complex cognitive functions without interacting with other cortical areas. Electrocorticography (ECoG) has recently become an important tool for studying global network activity across cortical areas in animal models. With stable recordings of electrical field potentials from multiple cortical areas, ECoG provides an opportunity to systematically study large-scale cortical activity at a mesoscopic spatiotemporal resolution under various experimental conditions. Recent developments in thin, flexible ECoG electrodes permit recording field potentials from not only gyral but intrasulcal cortical surfaces. Our review here focuses on the recent advances of ECoG applications to non-human primates.

An artificial network model for estimating the network structure underlying partially observed neuronal signals.

Many previous studies have proposed methods for quantifying neuronal interactions. However, these methods evaluated the interactions between recorded signals in an isolated network. In this study, we present a novel approach for estimating interactions between observed neuronal signals by theorizing that those signals are observed from only a part of the network that also includes unobserved structures. We propose a variant of the recurrent network model that consists of both observable and unobservable units. The observable units represent recorded neuronal activity, and the unobservable units are introduced to represent activity from unobserved structures in the network. The network structures are characterized by connective weights, i.e., the interaction intensities between individual units, which are estimated from recorded signals. We applied this model to multi-channel brain signals recorded from monkeys, and obtained robust network structures with physiological relevance. Furthermore, the network exhibited common features that portrayed cortical dynamics as inversely correlated interactions between excitatory and inhibitory populations of neurons, which are consistent with the previous view of cortical local circuits. Our results suggest that the novel concept of incorporating an unobserved structure into network estimations has theoretical advantages and could provide insights into brain dynamics beyond what can be directly observed.

Large-scale information flow in conscious and unconscious states: an ECoG study in monkeys.

Consciousness is an emergent property of the complex brain network. In order to understand how consciousness is constructed, neural interactions within this network must be elucidated. Previous studies have shown that specific neural interactions between the thalamus and frontoparietal cortices; frontal and parietal cortices; and parietal and temporal cortices are correlated with levels of consciousness. However, due to technical limitations, the network underlying consciousness has not been investigated in terms of large-scale interactions with high temporal and spectral resolution. In this study, we recorded neural activity with dense electrocorticogram (ECoG) arrays and used the spectral Granger causality to generate a more comprehensive network that relates to consciousness in monkeys. We found that neural interactions were significantly different between conscious and unconscious states in all combinations of cortical region pairs. Furthermore, the difference in neural interactions between conscious and unconscious states could be represented in 4 frequency-specific large-scale networks with unique interaction patterns: 2 networks were related to consciousness and showed peaks in alpha and beta bands, while the other 2 networks were related to unconsciousness and showed peaks in theta and gamma bands. Moreover, networks in the unconscious state were shared amongst 3 different unconscious conditions, which were induced either by ketamine and medetomidine, propofol, or sleep. Our results provide a novel picture that the difference between conscious and unconscious states is characterized by a switch in frequency-specific modes of large-scale communications across the entire cortex, rather than the cessation of interactions between specific cortical regions.

Higher order partial least squares (HOPLS): a generalized multilinear regression method.

A new generalized multilinear regression model, termed the higher order partial least squares (HOPLS), is introduced with the aim to predict a tensor (multiway array) Y from a tensor X through projecting the data onto the latent space and performing regression on the corresponding latent variables. HOPLS differs substantially from other regression models in that it explains the data by a sum of orthogonal Tucker tensors, while the number of orthogonal loadings serves as a parameter to control model complexity and prevent overfitting. The low-dimensional latent space is optimized sequentially via a deflation operation, yielding the best joint subspace approximation for both X and Y. Instead of decomposing X and Y individually, higher order singular value decomposition on a newly defined generalized cross-covariance tensor is employed to optimize the orthogonal loadings. A systematic comparison on both synthetic data and real-world decoding of 3D movement trajectories from electrocorticogram signals demonstrate the advantages of HOPLS over the existing methods in terms of better predictive ability, suitability to handle small sample sizes, and robustness to noise.

Spontaneous synchronization of arm motion between Japanese macaques.

Humans show spontaneous synchronization of movements during social interactions; this coordination has been shown to facilitate smooth communication. Although human studies exploring spontaneous synchronization are increasing in number, little is known about this phenomenon in other species. In this study, we examined spontaneous behavioural synchronization between monkeys in a laboratory setting. Synchronization was quantified by changes in button-pressing behaviour while pairs of monkeys were facing one another. Synchronization between the monkeys was duly observed and it was participant-partner dependent. Further tests confirmed that the speed of button pressing changed to harmonic or sub-harmonic levels in relation to the partner's speed. In addition, the visual information from the partner induced a higher degree of synchronization than auditory information. This study establishes advanced tasks for testing social coordination in monkeys, and illustrates ways in which monkeys coordinate their actions to establish synchronization.

Decoding continuous three-dimensional hand trajectories from epidural electrocorticographic signals in Japanese macaques.

Brain­machine interface (BMI) technology captures brain signals to enable control of prosthetic or communication devices with the goal of assisting patients who have limited or no ability to perform voluntary movements. Decoding of inherent information in brain signals to interpret the user's intention is one of main approaches for developing BMI technology. Subdural electrocorticography (sECoG)-based decoding provides good accuracy, but surgical complications are one of the major concerns for this approach to be applied in BMIs. In contrast, epidural electrocorticography (eECoG) is less invasive, thus it is theoretically more suitable for long-term implementation, although it is unclear whether eECoG signals carry sufficient information for decoding natural movements. We successfully decoded continuous three-dimensional hand trajectories from eECoG signals in Japanese macaques. A steady quantity of information of continuous hand movements could be acquired from the decoding system for at least several months, and a decoding model could be used for ∼10 days without significant degradation in accuracy or recalibration. The correlation coefficients between observed and predicted trajectories were lower than those for sECoG-based decoding experiments we previously reported, owing to a greater degree of chewing artifacts in eECoG-based decoding than is found in sECoG-based decoding. As one of the safest invasive recording methods available, eECoG provides an acceptable level of performance. With the ease of replacement and upgrades, eECoG systems could become the first-choice interface for real-life BMI applications.

On the decoding of intracranial data using sparse orthonormalized partial least squares.

It has recently been shown that robust decoding of motor output from electrocorticogram signals in monkeys over prolonged periods of time has become feasible (Chao et al 2010 Front. Neuroeng. 3 1-10). In order to achieve these results, multivariate partial least-squares (PLS) regression was used. PLS uses a set of latent variables, referred to as components, to model the relationship between the input and the output data and is known to handle high-dimensional and possibly strongly correlated inputs and outputs well. We developed a new decoding method called sparse orthonormalized partial least squares (SOPLS) which was tested on a subset of the data used in Chao et al (2010) (freely obtainable from neurotycho.org (Nagasaka et al 2011 PLoS ONE 6 e22561)). We show that SOPLS reaches the same decoding performance as PLS using just two sparse components which can each be interpreted as encoding particular combinations of motor parameters. Furthermore, the sparse solution afforded by the SOPLS model allowed us to show the functional involvement of beta and gamma band responses in premotor and motor cortex for predicting the first component. Based on the literature, we conjecture that this first component is involved in the encoding of movement direction. Hence, the sparse and compact representation afforded by the SOPLS model facilitates interpretation of which spectral, spatial and temporal components are involved in successful decoding. These advantages make the proposed decoding method an important new tool in neuroprosthetics.

Long-term asynchronous decoding of arm motion using electrocorticographic signals in monkeys.

Brain-machine interfaces (BMIs) employ the electrical activity generated by cortical neurons directly for controlling external devices and have been conceived as a means for restoring human cognitive or sensory-motor functions. The dominant approach in BMI research has been to decode motor variables based on single-unit activity (SUA). Unfortunately, this approach suffers from poor long-term stability and daily recalibration is normally required to maintain reliable performance. A possible alternative is BMIs based on electrocorticograms (ECoGs), which measure population activity and may provide more durable and stable recording. However, the level of long-term stability that ECoG-based decoding can offer remains unclear. Here we propose a novel ECoG-based decoding paradigm and show that we have successfully decoded hand positions and arm joint angles during an asynchronous food-reaching task in monkeys when explicit cues prompting the onset of movement were not required. Performance using our ECoG-based decoder was comparable to existing SUA-based systems while evincing far superior stability and durability. In addition, the same decoder could be used for months without any drift in accuracy or recalibration. These results were achieved by incorporating the spatio-spectro-temporal integration of activity across multiple cortical areas to compensate for the lower fidelity of ECoG signals. These results show the feasibility of high-performance, chronic and versatile ECoG-based neuroprosthetic devices for real-life applications. This new method provides a stable platform for investigating cortical correlates for understanding motor control, sensory perception, and high-level cognitive processes.