The Dynamics of Social Interaction Among Evolved Model Agents.

We offer three advances to the perceptual crossing simulation studies, which are aimed at challenging methodological individualism in the analysis of social cognition. First, we evolve and systematically test agents in rigorous conditions, identifying a set of 26 "robust circuits" with consistently high and generalizing performance. Next, we transform the sensor from discrete to continuous, facilitating a bifurcation analysis of the dynamics that shows that nonequilibrium dynamics are key to the mutual maintenance of interaction. Finally, we examine agents' performance with partners whose neural controllers are different from their own and with decoy objects of fixed frequency and amplitude. Nonclonal performance varies and is not predicted by genotypic distance. Frequency-amplitude values that fool the focal agent do not include the agent's own values. Altogether, our findings accentuate the importance of dynamical and nonclonal analyses for simulated sociality, emphasize the role of dialogue between artificial and human studies, and highlight the contributions of simulation studies to understanding social interactions.

Reinforcement Learning for Central Pattern Generation in Dynamical Recurrent Neural Networks.

Lifetime learning, or the change (or acquisition) of behaviors during a lifetime, based on experience, is a hallmark of living organisms. Multiple mechanisms may be involved, but biological neural circuits have repeatedly demonstrated a vital role in the learning process. These neural circuits are recurrent, dynamic, and non-linear and models of neural circuits employed in neuroscience and neuroethology tend to involve, accordingly, continuous-time, non-linear, and recurrently interconnected components. Currently, the main approach for finding configurations of dynamical recurrent neural networks that demonstrate behaviors of interest is using stochastic search techniques, such as evolutionary algorithms. In an evolutionary algorithm, these dynamic recurrent neural networks are evolved to perform the behavior over multiple generations, through selection, inheritance, and mutation, across a population of solutions. Although, these systems can be evolved to exhibit lifetime learning behavior, there are no explicit rules built into these dynamic recurrent neural networks that facilitate learning during their lifetime (e.g., reward signals). In this work, we examine a biologically plausible lifetime learning mechanism for dynamical recurrent neural networks. We focus on a recently proposed reinforcement learning mechanism inspired by neuromodulatory reward signals and ongoing fluctuations in synaptic strengths. Specifically, we extend one of the best-studied and most-commonly used dynamic recurrent neural networks to incorporate the reinforcement learning mechanism. First, we demonstrate that this extended dynamical system (model and learning mechanism) can autonomously learn to perform a central pattern generation task. Second, we compare the robustness and efficiency of the reinforcement learning rules in relation to two baseline models, a random walk and a hill-climbing walk through parameter space. Third, we systematically study the effect of the different meta-parameters of the learning mechanism on the behavioral learning performance. Finally, we report on preliminary results exploring the generality and scalability of this learning mechanism for dynamical neural networks as well as directions for future work.

A Neuromechanical Model of Multiple Network Rhythmic Pattern Generators for Forward Locomotion in C. elegans.

Multiple mechanisms contribute to the generation, propagation, and coordination of the rhythmic patterns necessary for locomotion in Caenorhabditis elegans. Current experiments have focused on two possibilities: pacemaker neurons and stretch-receptor feedback. Here, we focus on whether it is possible that a chain of multiple network rhythmic pattern generators in the ventral nerve cord also contribute to locomotion. We use a simulation model to search for parameters of the anatomically constrained ventral nerve cord circuit that, when embodied and situated, can drive forward locomotion on agar, in the absence of pacemaker neurons or stretch-receptor feedback. Systematic exploration of the space of possible solutions reveals that there are multiple configurations that result in locomotion that is consistent with certain aspects of the kinematics of worm locomotion on agar. Analysis of the best solutions reveals that gap junctions between different classes of motorneurons in the ventral nerve cord can play key roles in coordinating the multiple rhythmic pattern generators.

Persistent thermal input controls steering behavior in Caenorhabditis elegans.

Motile organisms actively detect environmental signals and migrate to a preferable environment. Especially, small animals convert subtle spatial difference in sensory input into orientation behavioral output for directly steering toward a destination, but the neural mechanisms underlying steering behavior remain elusive. Here, we analyze a C. elegans thermotactic behavior in which a small number of neurons are shown to mediate steering toward a destination temperature. We construct a neuroanatomical model and use an evolutionary algorithm to find configurations of the model that reproduce empirical thermotactic behavior. We find that, in all the evolved models, steering curvature are modulated by temporally persistent thermal signals sensed beyond the time scale of sinusoidal locomotion of C. elegans. Persistent rise in temperature decreases steering curvature resulting in straight movement of model worms, whereas fall in temperature increases curvature resulting in crooked movement. This relation between temperature change and steering curvature reproduces the empirical thermotactic migration up thermal gradients and steering bias toward higher temperature. Further, spectrum decomposition of neural activities in model worms show that thermal signals are transmitted from a sensory neuron to motor neurons on the longer time scale than sinusoidal locomotion of C. elegans. Our results suggest that employments of temporally persistent sensory signals enable small animals to steer toward a destination in natural environment with variable, noisy, and subtle cues.

Sources of predictive information in dynamical neural networks.

Behavior involves the ongoing interaction between an organism and its environment. One of the prevailing theories of adaptive behavior is that organisms are constantly making predictions about their future environmental stimuli. However, how they acquire that predictive information is still poorly understood. Two complementary mechanisms have been proposed: predictions are generated from an agent's internal model of the world or predictions are extracted directly from the environmental stimulus. In this work, we demonstrate that predictive information, measured using bivariate mutual information, cannot distinguish between these two kinds of systems. Furthermore, we show that predictive information cannot distinguish between organisms that are adapted to their environments and random dynamical systems exposed to the same environment. To understand the role of predictive information in adaptive behavior, we need to be able to identify where it is generated. To do this, we decompose information transfer across the different components of the organism-environment system and track the flow of information in the system over time. To validate the proposed framework, we examined it on a set of computational models of idealized agent-environment systems. Analysis of the systems revealed three key insights. First, predictive information, when sourced from the environment, can be reflected in any agent irrespective of its ability to perform a task. Second, predictive information, when sourced from the nervous system, requires special dynamics acquired during the process of adapting to the environment. Third, the magnitude of predictive information in a system can be different for the same task if the environmental structure changes.

Embodied Dyadic Interaction Increases Complexity of Neural Dynamics: A Minimal Agent-Based Simulation Model.

The concept of social interaction is at the core of embodied and enactive approaches to social cognitive processes, yet scientifically it remains poorly understood. Traditionally, cognitive science had relegated all behavior to being the end result of internal neural activity. However, the role of feedback from the interactions between agent and their environment has become increasingly important to understanding behavior. We focus on the role that social interaction plays in the behavioral and neural activity of the individuals taking part in it. Is social interaction merely a source of complex inputs to the individual, or can social interaction increase the individuals' own complexity? Here we provide a proof of concept of the latter possibility by artificially evolving pairs of simulated mobile robots to increase their neural complexity, which consistently gave rise to strategies that take advantage of their capacity for interaction. We found that during social interaction, the neural controllers exhibited dynamics of higher-dimensionality than were possible in social isolation. Moreover, by testing evolved strategies against unresponsive ghost partners, we demonstrated that under some conditions this effect was dependent on mutually responsive co-regulation, rather than on the mere presence of another agent's behavior as such. Our findings provide an illustration of how social interaction can augment the internal degrees of freedom of individuals who are actively engaged in participation.

Potential role of a ventral nerve cord central pattern generator in forward and backward locomotion in Caenorhabditis elegans.

C. elegans locomotes in an undulatory fashion, generating thrust by propagating dorsoventral bends along its body. Although central pattern generators (CPGs) are typically involved in animal locomotion, their presence in C. elegans has been questioned, mainly because there has been no evident circuit that supports intrinsic network oscillations. With a fully reconstructed connectome, the question of whether it is possible to have a CPG in the ventral nerve cord (VNC) of C. elegans can be answered through computational models. We modeled a repeating neural unit based on segmentation analysis of the connectome. We then used an evolutionary algorithm to determine the unknown physiological parameters of each neuron so as to match the features of the neural traces of the worm during forward and backward locomotion. We performed 1,000 evolutionary runs and consistently found configurations of the neural circuit that produced oscillations matching the main characteristic observed in experimental recordings. In addition to providing an existence proof for the possibility of a CPG in the VNC, we suggest a series of testable hypotheses about its operation. More generally, we show the feasibility and fruitfulness of a methodology to study behavior based on a connectome, in the absence of complete neurophysiological details.

From head to tail: a neuromechanical model of forward locomotion in Caenorhabditis elegans.

With 302 neurons and a near-complete reconstruction of the neural and muscle anatomy at the cellular level, Caenorhabditis elegans is an ideal candidate organism to study the neuromechanical basis of behaviour. Yet despite the breadth of knowledge about the neurobiology, anatomy and physics of C. elegans, there are still a number of unanswered questions about one of its most basic and fundamental behaviours: forward locomotion. How the rhythmic pattern is generated and propagated along the body is not yet well understood. We report on the development and analysis of a model of forward locomotion that integrates the neuroanatomy, neurophysiology and body mechanics of the worm. Our model is motivated by experimental analysis of the structure of the ventral cord circuitry and the effect of local body curvature on nearby motoneurons. We developed a neuroanatomically grounded model of the head motoneuron circuit and the ventral nerve cord circuit. We integrated the neural model with an existing biomechanical model of the worm's body, with updated musculature and stretch receptors. Unknown parameters were evolved using an evolutionary algorithm to match the speed of the worm on agar. We performed 100 evolutionary runs and consistently found electrophysiological configurations that reproduced realistic control of forward movement. The ensemble of successful solutions reproduced key experimental observations that they were not designed to fit, including the wavelength and frequency of the propagating wave. Analysis of the ensemble revealed that head motoneurons SMD and RMD are sufficient to drive dorsoventral undulations in the head and neck and that short-range posteriorly directed proprioceptive feedback is sufficient to propagate the wave along the rest of the body.This article is part of a discussion meeting issue 'Connectome to behaviour: modelling C. elegans at cellular resolution'.

The whole worm: brain-body-environment models of C. elegans.

Brain, body and environment are in continuous dynamical interaction, and it is becoming increasingly clear that an animal's behavior must be understood as a product not only of its nervous system, but also of the ongoing feedback of this neural activity through the biomechanics of its body and the ecology of its environment. Modeling has an essential integrative role to play in such an understanding. But successful whole-animal modeling requires an animal for which detailed behavioral, biomechanical and neural information is available and a modeling methodology which can gracefully cope with the constantly changing balance of known and unknown biological constraints. Here we review recent progress on both optogenetic techniques for imaging and manipulating neural activity and neuromechanical modeling in the nematode worm Caenorhabditis elegans. This work demonstrates both the feasibility and challenges of whole-animal modeling.

A stochastic neuronal model predicts random search behaviors at multiple spatial scales in C. elegans.

Random search is a behavioral strategy used by organisms from bacteria to humans to locate food that is randomly distributed and undetectable at a distance. We investigated this behavior in the nematode Caenorhabditis elegans, an organism with a small, well-described nervous system. Here we formulate a mathematical model of random search abstracted from the C. elegans connectome and fit to a large-scale kinematic analysis of C. elegans behavior at submicron resolution. The model predicts behavioral effects of neuronal ablations and genetic perturbations, as well as unexpected aspects of wild type behavior. The predictive success of the model indicates that random search in C. elegans can be understood in terms of a neuronal flip-flop circuit involving reciprocal inhibition between two populations of stochastic neurons. Our findings establish a unified theoretical framework for understanding C. elegans locomotion and a testable neuronal model of random search that can be applied to other organisms.

Information Flow through a Model of the C. elegans Klinotaxis Circuit.

Understanding how information about external stimuli is transformed into behavior is one of the central goals of neuroscience. Here we characterize the information flow through a complete sensorimotor circuit: from stimulus, to sensory neurons, to interneurons, to motor neurons, to muscles, to motion. Specifically, we apply a recently developed framework for quantifying information flow to a previously published ensemble of models of salt klinotaxis in the nematode worm Caenorhabditis elegans. Despite large variations in the neural parameters of individual circuits, we found that the overall information flow architecture circuit is remarkably consistent across the ensemble. This suggests structural connectivity is not necessarily predictive of effective connectivity. It also suggests information flow analysis captures general principles of operation for the klinotaxis circuit. In addition, information flow analysis reveals several key principles underlying how the models operate: (1) Interneuron class AIY is responsible for integrating information about positive and negative changes in concentration, and exhibits a strong left/right information asymmetry. (2) Gap junctions play a crucial role in the transfer of information responsible for the information symmetry observed in interneuron class AIZ. (3) Neck motor neuron class SMB implements an information gating mechanism that underlies the circuit's state-dependent response. (4) The neck carries more information about small changes in concentration than about large ones, and more information about positive changes in concentration than about negative ones. Thus, not all directions of movement are equally informative for the worm. Each of these findings corresponds to hypotheses that could potentially be tested in the worm. Knowing the results of these experiments would greatly refine our understanding of the neural circuit underlying klinotaxis.

Connecting a connectome to behavior: an ensemble of neuroanatomical models of C. elegans klinotaxis.

Increased efforts in the assembly and analysis of connectome data are providing new insights into the principles underlying the connectivity of neural circuits. However, despite these considerable advances in connectomics, neuroanatomical data must be integrated with neurophysiological and behavioral data in order to obtain a complete picture of neural function. Due to its nearly complete wiring diagram and large behavioral repertoire, the nematode worm Caenorhaditis elegans is an ideal organism in which to explore in detail this link between neural connectivity and behavior. In this paper, we develop a neuroanatomically-grounded model of salt klinotaxis, a form of chemotaxis in which changes in orientation are directed towards the source through gradual continual adjustments. We identify a minimal klinotaxis circuit by systematically searching the C. elegans connectome for pathways linking chemosensory neurons to neck motor neurons, and prune the resulting network based on both experimental considerations and several simplifying assumptions. We then use an evolutionary algorithm to find possible values for the unknown electrophsyiological parameters in the network such that the behavioral performance of the entire model is optimized to match that of the animal. Multiple runs of the evolutionary algorithm produce an ensemble of such models. We analyze in some detail the mechanisms by which one of the best evolved circuits operates and characterize the similarities and differences between this mechanism and other solutions in the ensemble. Finally, we propose a series of experiments to determine which of these alternatives the worm may be using.

Evolution and analysis of minimal neural circuits for klinotaxis in Caenorhabditis elegans.

Chemotaxis during sinusoidal locomotion in nematodes captures in simplified form the general problem of how dynamical interactions between the nervous system, body, and environment are exploited in the generation of adaptive behavior. We used an evolutionary algorithm to generate neural networks that exhibit klinotaxis, a common form of chemotaxis in which the direction of locomotion in a chemical gradient closely follows the line of steepest ascent. Sensory inputs and motor outputs of the model networks were constrained to match the inputs and outputs of the Caenorhabditis elegans klinotaxis network. We found that a minimalistic neural network, comprised of an ON-OFF pair of chemosensory neurons and a pair of neck muscle motor neurons, is sufficient to generate realistic klinotaxis behavior. Importantly, emergent properties of model networks reproduced two key experimental observations that they were not designed to fit, suggesting that the model may be operating according to principles similar to those of the biological network. A dynamical systems analysis of 77 evolved networks revealed a novel neural mechanism for spatial orientation behavior. This mechanism provides a testable hypothesis that is likely to accelerate the discovery and analysis of the biological circuitry for chemotaxis in C. elegans.