Learning with sparse reward in a gap junction network inspired by the insect mushroom body.

Animals can learn in real-life scenarios where rewards are often only available when a goal is achieved. This 'distal' or 'sparse' reward problem remains a challenge for conventional reinforcement learning algorithms. Here we investigate an algorithm for learning in such scenarios, inspired by the possibility that axo-axonal gap junction connections, observed in neural circuits with parallel fibres such as the insect mushroom body, could form a resistive network. In such a network, an active node represents the task state, connections between nodes represent state transitions and their connection to actions, and current flow to a target state can guide decision making. Building on evidence that gap junction weights are adaptive, we propose that experience of a task can modulate the connections to form a graph encoding the task structure. We demonstrate that the approach can be used for efficient reinforcement learning under sparse rewards, and discuss whether it is plausible as an account of the insect mushroom body.

Beyond prediction error: 25 years of modeling the associations formed in the insect mushroom body.

The insect mushroom body has gained increasing attention as a system in which the computational basis of neural learning circuits can be unraveled. We now understand in detail the key locations in this circuit where synaptic associations are formed between sensory patterns and values leading to actions. However, the actual learning rule (or rules) implemented by neural activity and leading to synaptic change is still an open question. Here, I survey the diversity of answers that have been offered in computational models of this system over the past decades, including the recurring assumption-in line with top-down theories of associative learning-that the core function is to reduce prediction error. However, I will argue, a more bottom-up approach may ultimately reveal a richer algorithmic capacity in this still enigmatic brain neuropil.

Emergent spatial goals in an integrative model of the insect central complex.

The insect central complex appears to encode and process spatial information through vector manipulation. Here, we draw on recent insights into circuit structure to fuse previous models of sensory-guided navigation, path integration and vector memory. Specifically, we propose that the allocentric encoding of location provided by path integration creates a spatially stable anchor for converging sensory signals that is relevant in multiple behavioural contexts. The allocentric reference frame given by path integration transforms a goal direction into a goal location and we demonstrate through modelling that it can enhance approach of a sensory target in noisy, cluttered environments or with temporally sparse stimuli. We further show the same circuit can improve performance in the more complex navigational task of route following. The model suggests specific functional roles for circuit elements of the central complex that helps explain their high preservation across insect species.

A model of cue integration as vector summation in the insect brain.

Ball-rolling dung beetles are known to integrate multiple cues in order to facilitate their straight-line orientation behaviour. Recent work has suggested that orientation cues are integrated according to a vector sum, that is, compass cues are represented by vectors and summed to give a combined orientation estimate. Further, cue weight (vector magnitude) appears to be set according to cue reliability. This is consistent with the popular Bayesian view of cue integration: cues are integrated to reduce or minimize an agent's uncertainty about the external world. Integration of orientation cues is believed to occur at the input to the insect central complex. Here, we demonstrate that a model of the head direction circuit of the central complex, including plasticity in input synapses, can act as a substrate for cue integration as vector summation. Further, we show that cue influence is not necessarily driven by cue reliability. Finally, we present a dung beetle behavioural experiment which, in combination with simulation, strongly suggests that these beetles do not weight cues according to reliability. We suggest an alternative strategy whereby cues are weighted according to relative contrast, which can also explain previous results.

Impact of central complex lesions on innate and learnt visual navigation in ants.

Wood ants are excellent navigators, using a combination of innate and learnt navigational strategies to travel between their nest and feeding sites. Visual navigation in ants has been studied extensively, however, we have little direct evidence for the underlying neural mechanisms. Here, we perform lateralized mechanical lesions in the central complex (CX) of wood ants, a midline structure known to allow an insect to keep track of the direction of sensory cues relative to its own orientation and to control movement. We lesioned two groups of ants and observed their behaviour in an arena with a large visual landmark present. The first group of ants were naïve and when intact such ants show a clear innate attraction to the conspicuous landmark. The second group of ants were trained to aim to a food location to the side of the landmark. The general heading of naïve ants towards a visual cue was not altered by the lesions, but the heading of ants trained to a landmark adjacent food position was affected. Thus, CX lesions had a specific impact on learnt visual guidance. We also observed that lateralised lesions altered the fine details of turning with lesioned ants spending less time turning to the side ipsilateral of the lesion. The results confirm the role of the CX in turn control and highlight its important role in the implementation of learnt behaviours that rely on information from other brain regions.

A Neural Model for Insect Steering Applied to Olfaction and Path Integration.

Many animal behaviors require orientation and steering with respect to the environment. For insects, a key brain area involved in spatial orientation and navigation is the central complex. Activity in this neural circuit has been shown to track the insect's current heading relative to its environment and has also been proposed to be the substrate of path integration. However, it remains unclear how the output of the central complex is integrated into motor commands. Central complex output neurons project to the lateral accessory lobes (LAL), from which descending neurons project to thoracic motor centers. Here, we present a computational model of a simple neural network that has been described anatomically and physiologically in the LALs of male silkworm moths, in the context of odor-mediated steering. We present and analyze two versions of this network, one rate based and one based on spiking neurons. The modeled network consists of an inhibitory local interneuron and a bistable descending neuron (flip-flop) that both receive input in the LAL. The flip-flop neuron projects onto neck motor neurons to induce steering. We show that this simple computational model not only replicates the basic parameters of male silkworm moth behavior in a simulated odor plume but can also take input from a computational model of path integration in the central complex and use it to steer back to a point of origin. Furthermore, we find that increasing the level of detail within the model improves the realism of the model's behavior, leading to the emergence of looping behavior as an orientation strategy. Our results suggest that descending neurons originating in the LALs, such as flip-flop neurons, are sufficient to mediate multiple steering behaviors. This study is therefore a first step to close the gap between orientation circuits in the central complex and downstream motor centers.

A unified mechanism for innate and learned visual landmark guidance in the insect central complex.

Insects can navigate efficiently in both novel and familiar environments, and this requires flexiblity in how they are guided by sensory cues. A prominent landmark, for example, can elicit strong innate behaviours (attraction or menotaxis) but can also be used, after learning, as a specific directional cue as part of a navigation memory. However, the mechanisms that allow both pathways to co-exist, interact or override each other are largely unknown. Here we propose a model for the behavioural integration of innate and learned guidance based on the neuroanatomy of the central complex (CX), adapted to control landmark guided behaviours. We consider a reward signal provided either by an innate attraction to landmarks or a long-term visual memory in the mushroom bodies (MB) that modulates the formation of a local vector memory in the CX. Using an operant strategy for a simulated agent exploring a simple world containing a single visual cue, we show how the generated short-term memory can support both innate and learned steering behaviour. In addition, we show how this architecture is consistent with the observed effects of unilateral MB lesions in ants that cause a reversion to innate behaviour. We suggest the formation of a directional memory in the CX can be interpreted as transforming rewarding (positive or negative) sensory signals into a mapping of the environment that describes the geometrical attractiveness (or repulsion). We discuss how this scheme might represent an ideal way to combine multisensory information gathered during the exploration of an environment and support optimal cue integration.

Object manipulation without hands.

Our current understanding of manipulation is based on primate hands, resulting in a detailed but narrow perspective of ways to handle objects. Although most other animals lack hands, they are still capable of flexible manipulation of diverse objects, including food and nest materials, and depend on dexterity in object handling to survive and reproduce. Birds, for instance, use their bills and feet to forage and build nests, while insects carry food and construct nests with their mandibles and legs. Bird bills and insect mandibles are much simpler than a primate hand, resembling simple robotic grippers. A better understanding of manipulation in these and other species would provide a broader comparative perspective on the origins of dexterity. Here we contrast data from primates, birds and insects, describing how they sense and grasp objects, and the neural architectures that control manipulation. Finally, we outline techniques for collecting comparable manipulation data from animals with diverse morphologies and describe the practical applications of studying manipulation in a wide range of species, including providing inspiration for novel designs of robotic manipulators.

A motion compensation treadmill for untethered wood ants (Formica rufa): evidence for transfer of orientation memories from free-walking training.

The natural scale of insect navigation during foraging makes it challenging to study under controlled conditions. Virtual reality and trackball setups have offered experimental control over visual environments while studying tethered insects, but potential limitations and confounds introduced by tethering motivates the development of alternative untethered solutions. In this paper, we validate the use of a motion compensator (or 'treadmill') to study visually driven behaviour of freely moving wood ants (Formica rufa). We show how this setup allows naturalistic walking behaviour and preserves foraging motivation over long time frames. Furthermore, we show that ants are able to transfer associative and navigational memories from classical maze and arena contexts to our treadmill. Thus, we demonstrate the possibility to study navigational behaviour over ecologically relevant durations (and virtual distances) in precisely controlled environments, bridging the gap between natural and highly controlled laboratory experiments.

Route-following ants respond to alterations of the view sequence.

Ants can navigate by comparing the currently perceived view with memorised views along a familiar foraging route. Models regarding route-following suggest that the views are stored and recalled independently of the sequence in which they occur. Hence, the ant only needs to evaluate the instantaneous familiarity of the current view to obtain a heading direction. This study investigates whether ant homing behaviour is influenced by alterations in the sequence of views experienced along a familiar route, using the frequency of stop-and-scan behaviour as an indicator of the ant's navigational uncertainty. Ants were trained to forage between their nest and a feeder which they exited through a short channel before proceeding along the homeward route. In tests, ants were collected before entering the nest and released again in the channel, which was placed either in its original location or halfway along the route. Ants exiting the familiar channel in the middle of the route would thus experience familiar views in a novel sequence. Results show that ants exiting the channel scan significantly more when they find themselves in the middle of the route, compared with when emerging at the expected location near the feeder. This behaviour suggests that previously encountered views influence the recognition of current views, even when these views are highly familiar, revealing a sequence component to route memory. How information about view sequences could be implemented in the insect brain, as well as potential alternative explanations to our results, are discussed.

Modelling the mechanics of exploration in larval Drosophila.

The Drosophila larva executes a stereotypical exploratory routine that appears to consist of stochastic alternation between straight peristaltic crawling and reorientation events through lateral bending. We present a model of larval mechanics for axial and transverse motion over a planar substrate, and use it to develop a simple, reflexive neuromuscular model from physical principles. The mechanical model represents the midline of the larva as a set of point masses which interact with each other via damped translational and torsional springs, and with the environment via sliding friction forces. The neuromuscular model consists of: 1. segmentally localised reflexes that amplify axial compression in order to counteract frictive energy losses, and 2. long-range mutual inhibition between reflexes in distant segments, enabling overall motion of the model larva relative to its substrate. In the absence of damping and driving, the mechanical model produces axial travelling waves, lateral oscillations, and unpredictable, chaotic deformations. The neuromuscular model counteracts friction to recover these motion patterns, giving rise to forward and backward peristalsis in addition to turning. Our model produces spontaneous exploration, even though the nervous system has no intrinsic pattern generating or decision making ability, and neither senses nor drives bending motions. Ultimately, our model suggests a novel view of larval exploration as a deterministic superdiffusion process which is mechanistically grounded in the chaotic mechanics of the body. We discuss how this may provide new interpretations for existing observations at the level of tissue-scale activity patterns and neural circuitry, and provide some experimental predictions that would test the extent to which the mechanisms we present translate to the real larva.

From skylight input to behavioural output: A computational model of the insect polarised light compass.

Many insects navigate by integrating the distances and directions travelled on an outward path, allowing direct return to the starting point. Fundamental to the reliability of this process is the use of a neural compass based on external celestial cues. Here we examine how such compass information could be reliably computed by the insect brain, given realistic constraints on the sky polarisation pattern and the insect eye sensor array. By processing the degree of polarisation in different directions for different parts of the sky, our model can directly estimate the solar azimuth and also infer the confidence of the estimate. We introduce a method to correct for tilting of the sensor array, as might be caused by travel over uneven terrain. We also show that the confidence can be used to approximate the change in sun position over time, allowing the compass to remain fixed with respect to 'true north' during long excursions. We demonstrate that the compass is robust to disturbances and can be effectively used as input to an existing neural model of insect path integration. We discuss the plausibility of our model to be mapped to known neural circuits, and to be implemented for robot navigation.

The internal maps of insects.

Insect navigation is strikingly geometric. Many species use path integration to maintain an accurate estimate of their distance and direction (a vector) to their nest and can store the vector information for multiple salient locations in the world, such as food sources, in a common coordinate system. Insects can also use remembered views of the terrain around salient locations or along travelled routes to guide return, which is a fundamentally geometric process. Recent modelling of these abilities shows convergence on a small set of algorithms and assumptions that appear sufficient to account for a wide range of behavioural data. Notably, this 'base model' does not include any significant topological knowledge: the insect does not need to recover the information (implicit in their vector memory) about the relationships between salient places; nor to maintain any connectedness or ordering information between view memories; nor to form any associations between views and vectors. However, there remains some experimental evidence not fully explained by this base model that may point towards the existence of a more complex or integrated mental map in insects.

Using an Insect Mushroom Body Circuit to Encode Route Memory in Complex Natural Environments.

Ants, like many other animals, use visual memory to follow extended routes through complex environments, but it is unknown how their small brains implement this capability. The mushroom body neuropils have been identified as a crucial memory circuit in the insect brain, but their function has mostly been explored for simple olfactory association tasks. We show that a spiking neural model of this circuit originally developed to describe fruitfly (Drosophila melanogaster) olfactory association, can also account for the ability of desert ants (Cataglyphis velox) to rapidly learn visual routes through complex natural environments. We further demonstrate that abstracting the key computational principles of this circuit, which include one-shot learning of sparse codes, enables the theoretical storage capacity of the ant mushroom body to be estimated at hundreds of independent images.

A Model of Drosophila Larva Chemotaxis.

Detailed observations of larval Drosophila chemotaxis have characterised the relationship between the odour gradient and the runs, head casts and turns made by the animal. We use a computational model to test whether hypothesised sensorimotor control mechanisms are sufficient to account for larval behaviour. The model combines three mechanisms based on simple transformations of the recent history of odour intensity at the head location. The first is an increased probability of terminating runs in response to gradually decreasing concentration, the second an increased probability of terminating head casts in response to rapidly increasing concentration, and the third a biasing of run directions up concentration gradients through modulation of small head casts. We show that this model can be tuned to produce behavioural statistics comparable to those reported for the larva, and that this tuning results in similar chemotaxis performance to the larva. We demonstrate that each mechanism can enable odour approach but the combination of mechanisms is most effective, and investigate how these low-level control mechanisms relate to behavioural measures such as the preference indices used to investigate larval learning behaviour in group assays.

Optimal cue integration in ants.

In situations with redundant or competing sensory information, humans have been shown to perform cue integration, weighting different cues according to their certainty in a quantifiably optimal manner. Ants have been shown to merge the directional information available from their path integration (PI) and visual memory, but as yet it is not clear that they do so in a way that reflects the relative certainty of the cues. In this study, we manipulate the variance of the PI home vector by allowing ants (Cataglyphis velox) to run different distances and testing their directional choice when the PI vector direction is put in competition with visual memory. Ants show progressively stronger weighting of their PI direction as PI length increases. The weighting is quantitatively predicted by modelling the expected directional variance of home vectors of different lengths and assuming optimal cue integration. However, a subsequent experiment suggests ants may not actually compute an internal estimate of the PI certainty, but are using the PI home vector length as a proxy.

How variation in head pitch could affect image matching algorithms for ant navigation.

Desert ants are a model system for animal navigation, using visual memory to follow long routes across both sparse and cluttered environments. Most accounts of this behaviour assume retinotopic image matching, e.g. recovering heading direction by finding a minimum in the image difference function as the viewpoint rotates. But most models neglect the potential image distortion that could result from unstable head motion. We report that for ants running across a short section of natural substrate, the head pitch varies substantially: by over 20 degrees with no load; and 60 degrees when carrying a large food item. There is no evidence of head stabilisation. Using a realistic simulation of the ant's visual world, we demonstrate that this range of head pitch significantly degrades image matching. The effect of pitch variation can be ameliorated by a memory bank of densely sampled along a route so that an image sufficiently similar in pitch and location is available for comparison. However, with large pitch disturbance, inappropriate memories sampled at distant locations are often recalled and navigation along a route can be adversely affected. Ignoring images obtained at extreme pitches, or averaging images over several pitches, does not significantly improve performance.

Dynamic antennal positioning allows honeybee followers to decode the dance.

The honeybee waggle dance has been widely studied as a communication system, yet we know little about how nestmates assimilate the information needed to navigate toward the signaled resource. They are required to detect the dancer's orientation relative to gravity and duration of the waggle phase and translate this into a flight vector with a direction relative to the sun1 and distance from the hive.2,3 Moreover, they appear capable of doing so from varied, dynamically changing positions around the dancer. Using high-speed, high-resolution video, we have uncovered a previously unremarked correlation between antennal position and the relative body axes of dancer and follower bees. Combined with new information about antennal inputs4,5 and spatial encoding in the insect central complex,6,7 we show how a neural circuit first proposed to underlie path integration could be adapted to decoding the dance and acquiring the signaled information as a flight vector that can be followed to the resource. This provides the first plausible account of how the bee brain could support the interpretation of its dance language.