A Reinforcement Learning Neural Network for Robotic Manipulator Control.

We propose a neural network model for reinforcement learning to control a robotic manipulator with unknown parameters and dead zones. The model is composed of three networks. The state of the robotic manipulator is predicted by the state network of the model, the action policy is learned by the action network, and the performance index of the action policy is estimated by a critic network. The three networks work together to optimize the performance index based on the reinforcement learning control scheme. The convergence of the learning methods is analyzed. Application of the proposed model on a simulated two-link robotic manipulator demonstrates the effectiveness and the stability of the model.

Selforganization of modular activity of grid cells.

A unique topographical representation of space is found in the concerted activity of grid cells in the rodent medial entorhinal cortex. Many among the principal cells in this region exhibit a hexagonal firing pattern, in which each cell expresses its own set of place fields (spatial phases) at the vertices of a triangular grid, the spacing and orientation of which are typically shared with neighboring cells. Grid spacing, in particular, has been found to increase along the dorso-ventral axis of the entorhinal cortex but in discrete steps, that is, with a modular structure. In this study, we show that such a modular activity may result from the self-organization of interacting units, which individually would not show discrete but rather continuously varying grid spacing. Within our "adaptation" network model, the effect of a continuously varying time constant, which determines grid spacing in the isolated cell model, is modulated by recurrent collateral connections, which tend to produce a few subnetworks, akin to magnetic domains, each with its own grid spacing. In agreement with experimental evidence, the modular structure is tightly defined by grid spacing, but also involves grid orientation and distortion, due to interactions across modules. Thus, our study sheds light onto a possible mechanism, other than simply assuming separate networks a priori, underlying the formation of modular grid representations.

Continuous attractor network model for conjunctive position-by-velocity tuning of grid cells.

The spatial responses of many of the cells recorded in layer II of rodent medial entorhinal cortex (MEC) show a triangular grid pattern, which appears to provide an accurate population code for animal spatial position. In layer III, V and VI of the rat MEC, grid cells are also selective to head-direction and are modulated by the speed of the animal. Several putative mechanisms of grid-like maps were proposed, including attractor network dynamics, interactions with theta oscillations or single-unit mechanisms such as firing rate adaptation. In this paper, we present a new attractor network model that accounts for the conjunctive position-by-velocity selectivity of grid cells. Our network model is able to perform robust path integration even when the recurrent connections are subject to random perturbations.

A prefrontal-thalamic circuit encodes social information for social recognition.

Social recognition encompasses encoding social information and distinguishing unfamiliar from familiar individuals to form social relationships. Although the medial prefrontal cortex (mPFC) is known to play a role in social behavior, how identity information is processed and by which route it is communicated in the brain remains unclear. Here we report that a ventral midline thalamic area, nucleus reuniens (Re) that has reciprocal connections with the mPFC, is critical for social recognition in male mice. In vivo single-unit recordings and decoding analysis reveal that neural populations in both mPFC and Re represent different social stimuli, however, mPFC coding capacity is stronger. We demonstrate that chemogenetic inhibitions of Re impair the mPFC-Re neural synchronization and the mPFC social coding. Projection pathway-specific inhibitions by optogenetics reveal that the reciprocal connectivity between the mPFC and the Re is necessary for social recognition. These results reveal an mPFC-thalamic circuit for social information processing.

Reviving Bistable Perception in Patients With Depression by Decreasing the Overestimation of Prior Precision.

Slower perceptual alternations, a notable perceptual effect observed in psychiatric disorders, can be alleviated by antidepressant therapies that affect serotonin levels in the brain. While these phenomena have been well documented, the underlying neurocognitive mechanisms remain to be elucidated. Our study bridges this gap by employing a computational cognitive approach within a Bayesian predictive coding framework to explore these mechanisms in depression. We fitted a prediction error (PE) model to behavioral data from a binocular rivalry task, uncovering that significantly higher initial prior precision and lower PE led to a slower switch rate in patients with depression. Furthermore, serotonin-targeting antidepressant treatments significantly decreased the prior precision and increased PE, both of which were predictive of improvements in the perceptual alternation rate of depression patients. These findings indicated that the substantially slower perception switch rate in patients with depression was caused by the greater reliance on top-down priors and that serotonin treatment's efficacy was in its recalibration of these priors and enhancement of PE. Our study not only elucidates the cognitive underpinnings of depression, but also suggests computational modeling as a potent tool for integrating cognitive science with clinical psychology, advancing our understanding and treatment of cognitive impairments in depression.

A model for the differentiation between grid and conjunctive units in medial entorhinal cortex.

The multiple layers of medial entorhinal cortex (mEC) contain cells that differ in selectivity, connectivity, and cellular properties. Grid cells in layer II and in the deeper layers express triangular grid patterns in the environment. The firing rate of the conjunctive cells found in layer III and below, on the other hand, show grid-by-head direction tuning. In this study, we model the differentiation between grid and conjunctive cells in a network with self-organized connections. Arranged into distinct "layers", the model grid units and conjunctive units develop, with a similar time course, grid fields resulting from firing rate adaptation and competitive learning. Grid alignment in both layers is delayed with respect to the formation of triangular grids. A common grid orientation among conjunctive units is produced, in the model, by head-direction modulated collateral interactions, while the grids of grid units inherit the same orientation through connections from conjunctive units. Grid units as well as conjunctive units share a similar spacing but show a random distribution of spatial phases. Grid units however carry more spatial information than conjunctive units, thus providing better inputs for the hippocampus to form spatial memories.

Grid maps for spaceflight, anyone? They are for free!

We show that, given extensive exploration of a three-dimensional volume, grid units can form with the approximate periodicity of a face-centered cubic crystal, as the spontaneous product of a self-organizing process at the single unit level, driven solely by firing rate adaptation.

Synchronization and oscillation behaviors of excitatory and inhibitory populations with spike-timing-dependent plasticity.

The effect of synaptic plasticity on the synchronization mechanism of the cerebral cortex has been a hot research topic over the past two decades. There are a great deal of literatures on excitatory pyramidal neurons, but the mechanism of interaction between the inhibitory interneurons is still under exploration. In this study, we consider a complex network consisting of excitatory (E) pyramidal neurons and inhibitory (I) interneurons interacting with chemical synapses through spike-timing-dependent plasticity (STDP). To study the effects of eSTDP and iSTDP on synchronization and oscillation behaviors emerged in an excitatory-inhibitory balanced network, we analyzed three different cases, a small-world network of purely excitatory neurons with eSTDP, a small-world network of purely inhibitory neurons with iSTDP and a small-world network with excitatory-inhibitory balanced neurons. By varying the number of inhibitory interneurons, and that of connected edges in a small-world network, and the coupling strength, these networks exhibit different synchronization and oscillation behaviors. We found that the eSTDP facilitates synchronization effectively, while iSTDP has no significant impact on it. In addition, eSTDP and iSTDP restrict the balance of the excitatory-inhibitory balanced neuronal network together and play a fundamental role in maintaining network stability and synchronization. They also can be used to guide the treatment and further research of neurodegenerative diseases.

The effects of first-dose methylphenidate on the neural signatures of visual selective attention in children with attention-deficit/hyperactivity disorder.

Although methylphenidate (MPH) has been shown to significantly improve selective attention in children with attention-deficit/hyperactivity disorder (ADHD), the neural mechanism of this effect remains unclear. We investigated the effects of first-dose MPH on the neural signatures of visual selective attention in children with ADHD. We measured the impact of first-dose MPH on electrophysiological indexes from eighteen children with ADHD (8.9-15.2 years; 15 boys) while they performed a visual search task. MPH was administered in a double-blind placebo-controlled crossover design. MPH led to decreases in behavioral error rates and reaction times. For the electrophysiological indexes, MPH significantly increased the target-elicited N2pc amplitude and posterior P3 amplitude during the selective attention process. The trial-based correlation analysis revealed that the enhanced N2pc (more negative) and P3 (more positive) promoted the behavioral response speed for children with ADHD. The lower individual P3 amplitude was associated with higher severity of inattention symptoms. The severer inattention symptoms were related to weaker MPH effect on N2pc amplitude. These findings suggest that N2pc and P3 are closely related to the mechanism of MPH in the ADHD treatment.

Abnormal Reactivity of Brain Oscillations to Visual Search Target in Children With Attention-Deficit/Hyperactivity Disorder.

BACKGROUND: Previous studies have shown that impaired goal-directed alpha lateralization and functional disconnection within attention networks during the cue period are significant features of attention-deficit/hyperactivity disorder (ADHD). This study aimed to explore the role of brain oscillations in the visual search process, focusing on target-induced posterior alpha lateralization, midfrontal theta synchronization, and their functional connection in children with ADHD. METHODS: Electroencephalograms were recorded from typically developing (TD) children (n = 72) and children with ADHD (n = 96) while they performed a visual search task. RESULTS: Both the TD and ADHD groups showed significant midfrontal theta event-related synchronization (ERS) and posterior alpha lateralization. Compared with TD children, children with ADHD showed significantly lower theta ERS and higher target-induced alpha lateralization. TD children showed a positive trial-based correlation between theta ERS and alpha lateralization and a negative correlation between theta ERS and reaction time variability. However, all these correlations were absent in children with ADHD. CONCLUSIONS: Abnormal brain oscillations in children with ADHD indicate insufficient executive control function and the compensation of attention networks for attention deficits in visual selective attention. Cross-frequency disconnection reflects the common deficiency of executive control in the gating of target information. Our findings provide novel evidence for interpreting the features of brain oscillations during stimulus-driven selective attention in children with ADHD.

Grid alignment in entorhinal cortex.

The spatial responses of many of the cells recorded in all layers of rodent medial entorhinal cortex (mEC) show mutually aligned grid patterns. Recent experimental findings have shown that grids can often be better described as elliptical rather than purely circular and that, beyond the mutual alignment of their grid axes, ellipses tend to also orient their long axis along preferred directions. Are grid alignment and ellipse orientation aspects of the same phenomenon? Does the grid alignment result from single-unit mechanisms or does it require network interactions? We address these issues by refining a single-unit adaptation model of grid formation, to describe specifically the spontaneous emergence of conjunctive grid-by-head-direction cells in layers III, V, and VI of mEC. We find that tight alignment can be produced by recurrent collateral interactions, but this requires head-direction (HD) modulation. Through a competitive learning process driven by spatial inputs, grid fields then form already aligned, and with randomly distributed spatial phases. In addition, we find that the self-organization process is influenced by any anisotropy in the behavior of the simulated rat. The common grid alignment often orients along preferred running directions (RDs), as induced in a square environment. When speed anisotropy is present in exploration behavior, the shape of individual grids is distorted toward an ellipsoid arrangement. Speed anisotropy orients the long ellipse axis along the fast direction. Speed anisotropy on its own also tends to align grids, even without collaterals, but the alignment is seen to be loose. Finally, the alignment of spatial grid fields in multiple environments shows that the network expresses the same set of grid fields across environments, modulo a coherent rotation and translation. Thus, an efficient metric encoding of space may emerge through spontaneous pattern formation at the single-unit level, but it is coherent, hence context-invariant, if aided by collateral interactions.

Entorhinal-hippocampal interactions lead to globally coherent representations of space.

The firing maps of grid cells in the entorhinal cortex are thought to provide an efficient metric system capable of supporting spatial inference in all environments. However, whether spatial representations of grid cells are determined by local environment cues or are organized into globally coherent patterns remains undetermined. We propose a navigation model containing a path integration system in the entorhinal cortex and a cognitive map system in the hippocampus. In the path integration system, grid cell network and head direction (HD) cell network integrate movement and visual information, and form attractor states to represent the positions and head directions of the animal. In the cognitive map system, a topological map is constructed capturing the attractor states of the path integration system as nodes and the transitions between attractor states as links. On loop closure, when the animal revisits a familiar place, the topological map is calibrated to minimize odometry errors. The change of the topological map is mapped back to the path integration system, to correct the states of the grid cells and the HD cells. The proposed model was tested on iRat, a rat-like miniature robot, in a realistic maze. Experimental results showed that, after familiarization of the environment, both grid cells and HD cells develop globally coherent firing maps by map calibration and activity correction. These results demonstrate that the hippocampus and the entorhinal cortex work together to form globally coherent metric representations of the environment. The underlying mechanisms of the hippocampal-entorhinal circuit in capturing the structure of the environment from sequences of experience are critical for understanding episodic memory.

A theory of geometry representations for spatial navigation.

The geometric information of space, such as environment boundaries, is represented heterogeneously across brain regions. The computational mechanisms of encoding the spatial layout of environments remain to be determined. Here, we postulate a conjunctive encoding theory to illustrate the construct of cognitive maps from geometric perception. The theory naturally describes a spectrum of cell types including experimentally observed boundary vector cells, border cells, "annulus" and "bulls-eye" cells as special examples. In a similar way, inspired by the integration of egocentric and allocentric information as found in the postrhinal cortex, the theory also predicts a new cell type, named geometry cell. Geometry cells encode the geometric layout of the local space relative to the environment center, independent of the animal's positions and headings within the local space. The predicted geometry cell provides pure allocentric high-level representations of local scenes to support the quick formation of cognitive map representations capturing the spatial layout of complex environments. The theory sheds new light on the neural mechanisms of spatial cognition and brain-inspired autonomous intelligent systems.

Learning Cognitive Map Representations for Navigation by Sensory-Motor Integration.

How to transform a mixed flow of sensory and motor information into memory state of self-location and to build map representations of the environment are central questions in the navigation research. Studies in neuroscience have shown that place cells in the hippocampus of the rodent brains form dynamic cognitive representations of locations in the environment. We propose a neural-network model called sensory-motor integration network model (SeMINet) to learn cognitive map representations by integrating sensory and motor information while an agent is exploring a virtual environment. This biologically inspired model consists of a deep neural network representing visual features of the environment, a recurrent network of place units encoding spatial information by sensorimotor integration, and a secondary network to decode the locations of the agent from spatial representations. The recurrent connections between the place units sustain an activity bump in the network without the need of sensory inputs, and the asymmetry in the connections propagates the activity bump in the network, forming a dynamic memory state which matches the motion of the agent. A competitive learning process establishes the association between the sensory representations and the memory state of the place units, and is able to correct the cumulative path-integration errors. The simulation results demonstrate that the network forms neural codes that convey location information of the agent independent of its head direction. The decoding network reliably predicts the location even when the movement is subject to noise. The proposed SeMINet thus provides a brain-inspired neural-network model for cognitive map updated by both self-motion cues and visual cues.

A brain-inspired compact cognitive mapping system.

In many simultaneous localization and mapping (SLAM) systems, the map of the environment grows over time as the robot explores the environment. The ever-growing map prevents long-term mapping, especially in large-scale environments. In this paper, we develop a compact cognitive mapping approach inspired by neurobiological experiments. Mimicking the firing activities of neighborhood cells, neighborhood fields determined by movement information, i.e. translation and rotation, are modeled to describe one of the distinct segments of the explored environment. The vertices with low neighborhood field activities are avoided to be added into the cognitive map. The optimization of the cognitive map is formulated as a robust non-linear least squares problem constrained by the transitions between vertices, and is numerically solved efficiently. According to the cognitive decision-making of place familiarity, loop closure edges are clustered depending on time intervals, and then batch global optimization of the cognitive map is performed to satisfy the combined constraint of the whole cluster. After the loop closure process, scene integration is performed, in which revisited vertices are removed subsequently to further reduce the size of the cognitive map. The compact cognitive mapping approach is tested on a monocular visual SLAM system in a naturalistic maze for a biomimetic animated robot. Our results demonstrate that the proposed method largely restricts the growth of the size of the cognitive map over time, and meanwhile, the compact cognitive map correctly represents the overall layout of the environment. The compact cognitive mapping method is well suitable for the representation of large-scale environments to achieve long-term robot navigation.

Probabilistic learning vector quantization on manifold of symmetric positive definite matrices.

In this paper, we develop a new classification method for manifold-valued data in the framework of probabilistic learning vector quantization. In many classification scenarios, the data can be naturally represented by symmetric positive definite matrices, which are inherently points that live on a curved Riemannian manifold. Due to the non-Euclidean geometry of Riemannian manifolds, traditional Euclidean machine learning algorithms yield poor results on such data. In this paper, we generalize the probabilistic learning vector quantization algorithm for data points living on the manifold of symmetric positive definite matrices equipped with Riemannian natural metric (affine-invariant metric). By exploiting the induced Riemannian distance, we derive the probabilistic learning Riemannian space quantization algorithm, obtaining the learning rule through Riemannian gradient descent. Empirical investigations on synthetic data, image data , and motor imagery electroencephalogram (EEG) data demonstrate the superior performance of the proposed method.

Characterization of exploratory patterns and hippocampal-prefrontal network oscillations during the emergence of free exploration.

During free exploration, the emergence of patterned and sequential behavioral responses to an unknown environment reflects exploration traits and adaptation. However, the behavioral dynamics and neural substrates underlying the exploratory behavior remain poorly understood. We developed computational tools to quantify the exploratory behavior and performed in vivo electrophysiological recordings in a large arena in which mice made sequential excursions into unknown territory. Occupancy entropy was calculated to characterize the cumulative and moment-to-moment behavioral dynamics in explored and unexplored territories. Local field potential analysis revealed that the theta activity in the dorsal hippocampus (dHPC) was highly correlated with the occupancy entropy. Individual dHPC and prefrontal cortex (PFC) oscillatory activities could classify various aspects of free exploration. Initiation of exploration was accompanied by a coordinated decrease and increase in theta activity in PFC and dHPC, respectively. Our results indicate that dHPC and PFC work synergistically in shaping free exploration by modulating exploratory traits during emergence and visits to an unknown environment.

Video data for the cognitive mapping process of NeuroBayesSLAM system.

Simultaneous localization and mapping (SLAM), which addresses the problem of constructing a spatial map of an unknown environment while simultaneously determining the mobile robot's position relative to this map, is regarded as one of the key technologies in mobile robot navigation. This data article presents four raw video files, demonstrating the mapping and localization processes of NeuroBayesSLAM, a neurobiologically inspired SLAM system, on two publicly available datasets, namely the St Lucia suburb dataset and the iRat Australia dataset. The cognitive mapping process was recorded by a free screen recorder software on ubuntu Linux system. Neural activities of the head-direction cells and the grid cells, the local view templates of visual scenes, and experience map were included. These data envision the possibility of transferring the multisensory integration mechanism found in the spatial memory circuits of the mammalian brain to develop intelligent cognitive mapping systems for indoor and large outdoor environments as in the research article "NeuroBayesSLAM: Neurobiologically Inspired Bayesian Integration of Multisensory Information for Robot Navigation" Zeng et al., 2020.

NeuroBayesSLAM: Neurobiologically inspired Bayesian integration of multisensory information for robot navigation.

Spatial navigation depends on the combination of multiple sensory cues from idiothetic and allothetic sources. The computational mechanisms of mammalian brains in integrating different sensory modalities under uncertainty for navigation is enlightening for robot navigation. We propose a Bayesian attractor network model to integrate visual and vestibular inputs inspired by the spatial memory systems of mammalian brains. In the model, the pose of the robot is encoded separately by two sub-networks, namely head direction network for angle representation and grid cell network for position representation, using similar neural codes of head direction cells and grid cells observed in mammalian brains. The neural codes in each of the sub-networks are updated in a Bayesian manner by a population of integrator cells for vestibular cue integration, as well as a population of calibration cells for visual cue calibration. The conflict between vestibular cue and visual cue is resolved by the competitive dynamics between the two populations. The model, implemented on a monocular visual simultaneous localization and mapping (SLAM) system, termed NeuroBayesSLAM, successfully builds semi-metric topological maps and self-localizes in outdoor and indoor environments of difference characteristics, achieving comparable performance as previous neurobiologically inspired navigation systems but with much less computation complexity. The proposed multisensory integration method constitutes a concise yet robust and biologically plausible method for robot navigation in large environments. The model provides a viable Bayesian mechanism for multisensory integration that may pertain to other neural subsystems beyond spatial cognition.

The DIAMOND Model: Deep Recurrent Neural Networks for Self-Organizing Robot Control.

The proposed architecture applies the principle of predictive coding and deep learning in a brain-inspired approach to robotic sensorimotor control. It is composed of many layers each of which is a recurrent network. The component networks can be spontaneously active due to the homeokinetic learning rule, a principle that has been studied previously for the purpose of self-organized generation of behavior. We present robotic simulations that illustrate the function of the network and show evidence that deeper networks enable more complex exploratory behavior.