Linking global top-down views to first-person views in the brain.

Humans and other animals have a remarkable capacity to translate their position from one spatial frame of reference to another. The ability to seamlessly move between top-down and first-person views is important for navigation, memory formation, and other cognitive tasks. Evidence suggests that the medial temporal lobe and other cortical regions contribute to this function. To understand how a neural system might carry out these computations, we used variational autoencoders (VAEs) to reconstruct the first-person view from the top-down view of a robot simulation, and vice versa. Many latent variables in the VAEs had similar responses to those seen in neuron recordings, including location-specific activity, head direction tuning, and encoding of distance to local objects. Place-specific responses were prominent when reconstructing a first-person view from a top-down view, but head direction-specific responses were prominent when reconstructing a top-down view from a first-person view. In both cases, the model could recover from perturbations without retraining, but rather through remapping. These results could advance our understanding of how brain regions support viewpoint linkages and transformations.

Cortical Motion Perception Emerges from Dimensionality Reduction with Evolved Spike-Timing-Dependent Plasticity Rules.

The nervous system is under tight energy constraints and must represent information efficiently. This is particularly relevant in the dorsal part of the medial superior temporal area (MSTd) in primates where neurons encode complex motion patterns to support a variety of behaviors. A sparse decomposition model based on a dimensionality reduction principle known as non-negative matrix factorization (NMF) was previously shown to account for a wide range of monkey MSTd visual response properties. This model resulted in sparse, parts-based representations that could be regarded as basis flow fields, a linear superposition of which accurately reconstructed the input stimuli. This model provided evidence that the seemingly complex response properties of MSTd may be a by-product of MSTd neurons performing dimensionality reduction on their input. However, an open question is how a neural circuit could carry out this function. In the current study, we propose a spiking neural network (SNN) model of MSTd based on evolved spike-timing-dependent plasticity and homeostatic synaptic scaling (STDP-H) learning rules. We demonstrate that the SNN model learns compressed and efficient representations of the input patterns similar to the patterns that emerge from NMF, resulting in MSTd-like receptive fields observed in monkeys. This SNN model suggests that STDP-H observed in the nervous system may be performing a similar function as NMF with sparsity constraints, which provides a test bed for mechanistic theories of how MSTd may efficiently encode complex patterns of visual motion to support robust self-motion perception.SIGNIFICANCE STATEMENT The brain may use dimensionality reduction and sparse coding to efficiently represent stimuli under metabolic constraints. Neurons in monkey area MSTd respond to complex optic flow patterns resulting from self-motion. We developed a spiking neural network model that showed MSTd-like response properties can emerge from evolving spike-timing-dependent plasticity with STDP-H parameters of the connections between then middle temporal area and MSTd. Simulated MSTd neurons formed a sparse, reduced population code capable of encoding perceptual variables important for self-motion perception. This model demonstrates that complex neuronal responses observed in MSTd may emerge from efficient coding and suggests that neurobiological plasticity, like STDP-H, may contribute to reducing the dimensions of input stimuli and allowing spiking neurons to learn sparse representations.

Neural correlates of sparse coding and dimensionality reduction.

Supported by recent computational studies, there is increasing evidence that a wide range of neuronal responses can be understood as an emergent property of nonnegative sparse coding (NSC), an efficient population coding scheme based on dimensionality reduction and sparsity constraints. We review evidence that NSC might be employed by sensory areas to efficiently encode external stimulus spaces, by some associative areas to conjunctively represent multiple behaviorally relevant variables, and possibly by the basal ganglia to coordinate movement. In addition, NSC might provide a useful theoretical framework under which to understand the often complex and nonintuitive response properties of neurons in other brain areas. Although NSC might not apply to all brain areas (for example, motor or executive function areas) the success of NSC-based models, especially in sensory areas, warrants further investigation for neural correlates in other regions.

Neurobiologically Inspired Self-Monitoring Systems.

In this article, we explore neurobiological principles that could be deployed in systems requiring self-preservation, adaptive control, and contextual awareness. We start with low-level control for sensor processing and motor reflexes. We then discuss how critical it is at an intermediate level to maintain homeostasis and predict system set points. We end with a discussion at a high-level, or cognitive level, where planning and prediction can further monitor the system and optimize performance. We emphasize the information flow between these levels both from a systems neuroscience and an engineering point of view. Throughout the paper, we describe the brain systems that carry out these functions and provide examples from artificial intelligence, machine learning, and robotics that include these features. Our goal is to show how biological organisms performing self-monitoring can inspire the design of autonomous and embedded systems.

A neural model of schemas and memory encoding.

The ability to rapidly assimilate new information is essential for survival in a dynamic environment. This requires experiences to be encoded alongside the contextual schemas in which they occur. Tse et al. (Science 316(5821):76-82, 2007) showed that new information matching a preexisting schema is learned rapidly. To better understand the neurobiological mechanisms for creating and maintaining schemas, we constructed a biologically plausible neural network to learn context in a spatial memory task. Our model suggests that this occurs through two processing streams of indexing and representation, in which the medial prefrontal cortex and hippocampus work together to index cortical activity. Additionally, our study shows how neuromodulation contributes to rapid encoding within consistent schemas. The level of abstraction of our model further provides a basis for creating context-dependent memories while preventing catastrophic forgetting in artificial neural networks.

Merging information in the entorhinal cortex: what can we learn from robotics experiments and modeling?

Place recognition is a complex process involving idiothetic and allothetic information. In mammals, evidence suggests that visual information stemming from the temporal and parietal cortical areas ('what' and 'where' information) is merged at the level of the entorhinal cortex (EC) to build a compact code of a place. Local views extracted from specific feature points can provide information important for view cells (in primates) and place cells (in rodents) even when the environment changes dramatically. Robotics experiments using conjunctive cells merging 'what' and 'where' information related to different local views show their important role for obtaining place cells with strong generalization capabilities. This convergence of information may also explain the formation of grid cells in the medial EC if we suppose that: (1) path integration information is computed outside the EC, (2) this information is compressed at the level of the EC owing to projection (which follows a modulo principle) of cortical activities associated with discretized vector fields representing angles and/or path integration, and (3) conjunctive cells merge the projections of different modalities to build grid cell activities. Applying modulo projection to visual information allows an interesting compression of information and could explain more recent results on grid cells related to visual exploration. In conclusion, the EC could be dedicated to the build-up of a robust yet compact code of cortical activity whereas the hippocampus proper recognizes these complex codes and learns to predict the transition from one state to another.

3D Visual Response Properties of MSTd Emerge from an Efficient, Sparse Population Code.

UNLABELLED: Neurons in the dorsal subregion of the medial superior temporal (MSTd) area of the macaque respond to large, complex patterns of retinal flow, implying a role in the analysis of self-motion. Some neurons are selective for the expanding radial motion that occurs as an observer moves through the environment ("heading"), and computational models can account for this finding. However, ample evidence suggests that MSTd neurons exhibit a continuum of visual response selectivity to large-field motion stimuli. Furthermore, the underlying computational principles by which these response properties are derived remain poorly understood. Here we describe a computational model of macaque MSTd based on the hypothesis that neurons in MSTd efficiently encode the continuum of large-field retinal flow patterns on the basis of inputs received from neurons in MT with receptive fields that resemble basis vectors recovered with non-negative matrix factorization. These assumptions are sufficient to quantitatively simulate neurophysiological response properties of MSTd cells, such as 3D translation and rotation selectivity, suggesting that these properties might simply be a byproduct of MSTd neurons performing dimensionality reduction on their inputs. At the population level, model MSTd accurately predicts eye velocity and heading using a sparse distributed code, consistent with the idea that biological MSTd might be well equipped to efficiently encode various self-motion variables. The present work aims to add some structure to the often contradictory findings about macaque MSTd, and offers a biologically plausible account of a wide range of visual response properties ranging from single-unit selectivity to population statistics. SIGNIFICANCE STATEMENT: Using a dimensionality reduction technique known as non-negative matrix factorization, we found that a variety of medial superior temporal (MSTd) neural response properties could be derived from MT-like input features. The responses that emerge from this technique, such as 3D translation and rotation selectivity, spiral tuning, and heading selectivity, can account for a number of empirical results. These findings (1) provide a further step toward a scientific understanding of the often nonintuitive response properties of MSTd neurons; (2) suggest that response properties, such as complex motion tuning and heading selectivity, might simply be a byproduct of MSTd neurons performing dimensionality reduction on their inputs; and (3) imply that motion perception in the cortex is consistent with ideas from the efficient-coding and free-energy principles.

Learning to ignore: a modeling study of a decremental cholinergic pathway and its influence on attention and learning.

Learning to ignore irrelevant stimuli is essential to achieving efficient and fluid attention, and serves as the complement to increasing attention to relevant stimuli. The different cholinergic (ACh) subsystems within the basal forebrain regulate attention in distinct but complementary ways. ACh projections from the substantia innominata/nucleus basalis region (SI/nBM) to the neocortex are necessary to increase attention to relevant stimuli and have been well studied. Lesser known are ACh projections from the medial septum/vertical limb of the diagonal band (MS/VDB) to the hippocampus and the cingulate that are necessary to reduce attention to irrelevant stimuli. We developed a neural simulation to provide insight into how ACh can decrement attention using this distinct pathway from the MS/VDB. We tested the model in behavioral paradigms that require decremental attention. The model exhibits behavioral effects such as associative learning, latent inhibition, and persisting behavior. Lesioning the MS/VDB disrupts latent inhibition, and drastically increases perseverative behavior. Taken together, the model demonstrates that the ACh decremental pathway is necessary for appropriate learning and attention under dynamic circumstances and suggests a canonical neural architecture for decrementing attention.

Mechanisms underlying the basal forebrain enhancement of top-down and bottom-up attention.

Both attentional signals from frontal cortex and neuromodulatory signals from basal forebrain (BF) have been shown to influence information processing in the primary visual cortex (V1). These two systems exert complementary effects on their targets, including increasing firing rates and decreasing interneuronal correlations. Interestingly, experimental research suggests that the cholinergic system is important for increasing V1's sensitivity to both sensory and attentional information. To see how the BF and top-down attention act together to modulate sensory input, we developed a spiking neural network model of V1 and thalamus that incorporated cholinergic neuromodulation and top-down attention. In our model, activation of the BF had a broad effect that decreases the efficacy of top-down projections and increased the reliance of bottom-up sensory input. In contrast, we demonstrated how local release of acetylcholine in the visual cortex, which was triggered through top-down gluatmatergic projections, could enhance top-down attention with high spatial specificity. Our model matched experimental data showing that the BF and top-down attention decrease interneuronal correlations and increase between-trial reliability. We found that decreases in correlations were primarily between excitatory-inhibitory pairs rather than excitatory-excitatory pairs and suggest that excitatory-inhibitory decorrelation is necessary for maintaining low levels of excitatory-excitatory correlations. Increased inhibitory drive via release of acetylcholine in V1 may then act as a buffer, absorbing increases in excitatory-excitatory correlations that occur with attention and BF stimulation. These findings will lead to a better understanding of the mechanisms underyling the BF's interactions with attention signals and influences on correlations.

A sparse quantized hopfield network for online-continual memory.

An important difference between brains and deep neural networks is the way they learn. Nervous systems learn online where a stream of noisy data points are presented in a non-independent, identically distributed way. Further, synaptic plasticity in the brain depends only on information local to synapses. Deep networks, on the other hand, typically use non-local learning algorithms and are trained in an offline, non-noisy, independent, identically distributed setting. Understanding how neural networks learn under the same constraints as the brain is an open problem for neuroscience and neuromorphic computing. A standard approach to this problem has yet to be established. In this paper, we propose that discrete graphical models that learn via an online maximum a posteriori learning algorithm could provide such an approach. We implement this kind of model in a neural network called the Sparse Quantized Hopfield Network. We show our model outperforms state-of-the-art neural networks on associative memory tasks, outperforms these networks in online, continual settings, learns efficiently with noisy inputs, and is better than baselines on an episodic memory task.

Importance of Path Planning Variability: A Simulation Study.

Individuals vary in the way they navigate through space. Some take novel shortcuts, while others rely on known routes to find their way around. We wondered how and why there is so much variation in the population. To address this, we first compared the trajectories of 368 human subjects navigating a virtual maze with simulated trajectories. The simulated trajectories were generated by strategy-based path planning algorithms from robotics. Based on the similarities between human trajectories and different strategy-based simulated trajectories, we found that there is a variation in the type of strategy individuals apply to navigate space, as well as variation within individuals on a trial-by-trial basis. Moreover, we observed variation within a trial when subjects occasionally switched the navigation strategies halfway through a trajectory. In these cases, subjects started with a route strategy, in which they followed a familiar path, and then switched to a survey strategy, in which they took shortcuts by considering the layout of the environment. Then we simulated a second set of trajectories using five different but comparable artificial maps. These trajectories produced the similar pattern of strategy variation within and between trials. Furthermore, we varied the relative cost, that is, the assumed mental effort or required timesteps to choose a learned route over alternative paths. When the learned route was relatively costly, the simulated agents tended to take shortcuts. Conversely, when the learned route was less costly, the simulated agents showed preference toward a route strategy. We suggest that cost or assumed mental effort may be the reason why in previous studies, subjects used survey knowledge when instructed to take the shortest path. We suggest that this variation we observe in humans may be beneficial for robotic swarms or collections of autonomous agents during information gathering.

Achieving efficient interpretability of reinforcement learning via policy distillation and selective input gradient regularization.

Although deep Reinforcement Learning (RL) has proven successful in a wide range of tasks, one challenge it faces is interpretability when applied to real-world problems. Saliency maps are frequently used to provide interpretability for deep neural networks. However, in the RL domain, existing saliency map approaches are either computationally expensive and thus cannot satisfy the real-time requirement of real-world scenarios or cannot produce interpretable saliency maps for RL policies. In this work, we propose an approach of Distillation with selective Input Gradient Regularization (DIGR) which uses policy distillation and input gradient regularization to produce new policies that achieve both high interpretability and computation efficiency in generating saliency maps. Our approach is also found to improve the robustness of RL policies to multiple adversarial attacks. We conduct experiments on three tasks, MiniGrid (Fetch Object), Atari (Breakout) and CARLA Autonomous Driving, to demonstrate the importance and effectiveness of our approach.

Robust Resting-State Dynamics in a Large-Scale Spiking Neural Network Model of Area CA3 in the Mouse Hippocampus.

Hippocampal area CA3 performs the critical auto-associative function underlying pattern completion in episodic memory. Without external inputs, the electrical activity of this neural circuit reflects the spontaneous spiking interplay among glutamatergic pyramidal neurons and GABAergic interneurons. However, the network mechanisms underlying these resting-state firing patterns are poorly understood. Leveraging the Hippocampome.org knowledge base, we developed a data-driven, large-scale spiking neural network (SNN) model of mouse CA3 with 8 neuron types, 90,000 neurons, 51 neuron-type specific connections, and 250,000,000 synapses. We instantiated the SNN in the CARLsim4 multi-GPU simulation environment using the Izhikevich and Tsodyks-Markram formalisms for neuronal and synaptic dynamics, respectively. We analyzed the resultant population activity upon transient activation. The SNN settled into stable oscillations with a biologically plausible grand-average firing frequency, which was robust relative to a wide range of transient activation. The diverse firing patterns of individual neuron types were consistent with existing knowledge of cell type-specific activity in vivo. Altered network structures that lacked neuron- or connection-type specificity were neither stable nor robust, highlighting the importance of neuron type circuitry. Additionally, external inputs reflecting dentate mossy fibers shifted the observed rhythms to the gamma band. We freely released the CARLsim4-Hippocampome framework on GitHub to test hippocampal hypotheses. Our SNN may be useful to investigate the circuit mechanisms underlying the computational functions of CA3. Moreover, our approach can be scaled to the whole hippocampal formation, which may contribute to elucidating how the unique neuronal architecture of this system subserves its crucial cognitive roles.

Evaluation of the Toyota Human Support Robot (HSR) for Social Interaction and Learning.

Tele-operated social robots (telerobots) offer an innovative means of allowing children who are medically restricted to their homes (MRH) to return to their local schools and physical communities. Most commercially available telerobots have three foundational features that facilitate child-robot interaction: remote mobility, synchronous two-way vision capabilities, and synchronous two-way audio capabilities. We conducted a comparative analysis between the Toyota Human Support Robot (HSR) and commercially available telerobots, focusing on these foundational features. Children who used these robots and these features on a daily basis to attend school were asked to pilot the HSR in a simulated classroom for learning activities. As the HSR has three additional features that are not available on commercial telerobots: (1) pan-tilt camera, (2) mapping and autonomous navigation, and (3) robot arm and gripper for children to "reach" into remote environments, participants were also asked to evaluate the use of these features for learning experiences. To expand on earlier work on the use of telerobots by remote children, this study provides novel empirical findings on (1) the capabilities of the Toyota HSR for robot-mediated learning similar to commercially available telerobots and (2) the efficacy of novel HSR features (i.e., pan-tilt camera, autonomous navigation, robot arm/hand hardware) for future learning experiences. We found that among our participants, autonomous navigation and arm/gripper hardware were rated as highly valuable for social and learning activities.

Design Principles for Neurorobotics.

In their book "How the Body Shapes the Way We Think: A New View of Intelligence," Pfeifer and Bongard put forth an embodied approach to cognition. Because of this position, many of their robot examples demonstrated "intelligent" behavior despite limited neural processing. It is our belief that neurorobots should attempt to follow many of these principles. In this article, we discuss a number of principles to consider when designing neurorobots and experiments using robots to test brain theories. These principles are strongly inspired by Pfeifer and Bongard, but build on their design principles by grounding them in neuroscience and by adding principles based on neuroscience research. Our design principles fall into three categories. First, organisms must react quickly and appropriately to events. Second, organisms must have the ability to learn and remember over their lifetimes. Third, organisms must weigh options that are crucial for survival. We believe that by following these design principles a robot's behavior will be more naturalistic and more successful.

Deep Reinforcement Learning With Modulated Hebbian Plus Q-Network Architecture.

In this article, we consider a subclass of partially observable Markov decision process (POMDP) problems which we termed confounding POMDPs. In these types of POMDPs, temporal difference (TD)-based reinforcement learning (RL) algorithms struggle, as TD error cannot be easily derived from observations. We solve these types of problems using a new bio-inspired neural architecture that combines a modulated Hebbian network (MOHN) with deep Q-network (DQN), which we call modulated Hebbian plus Q-network architecture (MOHQA). The key idea is to use a Hebbian network with rarely correlated bio-inspired neural traces to bridge temporal delays between actions and rewards when confounding observations and sparse rewards result in inaccurate TD errors. In MOHQA, DQN learns low-level features and control, while the MOHN contributes to high-level decisions by associating rewards with past states and actions. Thus, the proposed architecture combines two modules with significantly different learning algorithms, a Hebbian associative network and a classical DQN pipeline, exploiting the advantages of both. Simulations on a set of POMDPs and on the Malmo environment show that the proposed algorithm improved DQN's results and even outperformed control tests with advantage-actor critic (A2C), quantile regression DQN with long short-term memory (QRDQN + LSTM), Monte Carlo policy gradient (REINFORCE), and aggregated memory for reinforcement learning (AMRL) algorithms on most difficult POMDPs with confounding stimuli and sparse rewards.

Sparse Representations for Object- and Ego-Motion Estimations in Dynamic Scenes.

Disentangling the sources of visual motion in a dynamic scene during self-movement or ego motion is important for autonomous navigation and tracking. In the dynamic image segments of a video frame containing independently moving objects, optic flow relative to the next frame is the sum of the motion fields generated due to camera and object motion. The traditional ego-motion estimation methods assume the scene to be static, and the recent deep learning-based methods do not separate pixel velocities into object- and ego-motion components. We propose a learning-based approach to predict both ego-motion parameters and object-motion field (OMF) from image sequences using a convolutional autoencoder while being robust to variations due to the unconstrained scene depth. This is achieved by: 1) training with continuous ego-motion constraints that allow solving for ego-motion parameters independently of depth and 2) learning a sparsely activated overcomplete ego-motion field (EMF) basis set, which eliminates the irrelevant components in both static and dynamic segments for the task of ego-motion estimation. In order to learn the EMF basis set, we propose a new differentiable sparsity penalty function that approximates the number of nonzero activations in the bottleneck layer of the autoencoder and enforces sparsity more effectively than L1- and L2-norm-based penalties. Unlike the existing direct ego-motion estimation methods, the predicted global EMF can be used to extract OMF directly by comparing it against the optic flow. Compared with the state-of-the-art baselines, the proposed model performs favorably on pixelwise object- and ego-motion estimation tasks when evaluated on real and synthetic data sets of dynamic scenes.

Modeling uncertainty-seeking behavior mediated by cholinergic influence on dopamine.

Recent findings suggest that acetylcholine mediates uncertainty-seeking behaviors through its projection to dopamine neurons - another neuromodulatory system known for its major role in reinforcement learning and decision-making. In this paper, we propose a leaky-integrate-and-fire model of this mechanism. It implements a softmax-like selection with an uncertainty bonus by a cholinergic drive to dopaminergic neurons, which in turn influence synaptic currents of downstream neurons. The model is able to reproduce experimental data in two decision-making tasks. It also predicts that: (i) in the absence of cholinergic input, dopaminergic activity would not correlate with uncertainty, and that (ii) the adaptive advantage brought by the implemented uncertainty-seeking mechanism is most useful when sources of reward are not highly uncertain. Moreover, this modeling work allows us to propose novel experiments which might shed new light on the role of acetylcholine in both random and directed exploration. Overall, this study contributes to a more comprehensive understanding of the role of the cholinergic system and, in particular, its involvement in decision-making.

Neuromodulated attention and goal-driven perception in uncertain domains.

In uncertain domains, the goals are often unknown and need to be predicted by the organism or system. In this paper, contrastive Excitation Backprop (c-EB) was used in two goal-driven perception tasks - one with pairs of noisy MNIST digits and the other with a robot in an action-based attention scenario. The first task included attending to even, odd, low, and high digits, whereas the second task included action goals, such as "eat", "work-on-computer", "read", and "say-hi" that led to attention to objects associated with those actions. The system needed to increase attention to target items and decrease attention to distractor items and background noise. Because the valid goal was unknown, an online learning model based on the cholinergic and noradrenergic neuromodulatory systems was used to predict a noisy goal (expected uncertainty) and re-adapt when the goal changed (unexpected uncertainty). This neurobiologically plausible model demonstrates how neuromodulatory systems can predict goals in uncertain domains and how attentional mechanisms can enhance the perception for that goal.