Evolutionary algorithms as an alternative to backpropagation for supervised training of Biophysical Neural Networks and Neural ODEs.

Training networks consisting of biophysically accurate neuron models could allow for new insights into how brain circuits can organize and solve tasks. We begin by analyzing the extent to which the central algorithm for neural network learning -- stochastic gradient descent through backpropagation (BP) -- can be used to train such networks. We find that properties of biophysically based neural network models needed for accurate modelling such as stiffness, high nonlinearity and long evaluation timeframes relative to spike times makes BP unstable and divergent in a variety of cases. To address these instabilities and inspired by recent work, we investigate the use of "gradient-estimating" evolutionary algorithms (EAs) for training biophysically based neural networks. We find that EAs have several advantages making them desirable over direct BP, including being forward-pass only, robust to noisy and rigid losses, allowing for discrete loss formulations, and potentially facilitating a more global exploration of parameters. We apply our method to train a recurrent network of Morris-Lecar neuron models on a stimulus integration and working memory task, and show how it can succeed in cases where direct BP is inapplicable. To expand on the viability of EAs in general, we apply them to a general neural ODE problem and a stiff neural ODE benchmark and find again that EAs can out-perform direct BP here, especially for the over-parameterized regime. Our findings suggest that biophysical neurons could provide useful benchmarks for testing the limits of BP-adjacent methods, and demonstrate the viability of EAs for training networks with complex components.

AL-SAR: Active Learning for Skeleton-Based Action Recognition.

Action recognition from temporal multivariate sequences of features, such as identifying human actions, is typically approached by supervised training as it requires many ground truth annotations to reach high recognition accuracy. Unsupervised methods for the organization of sequences into clusters have been introduced, however, such methods continue to require annotations to associate clusters with actions. The challenges in annotation necessitate an effective classification methodology that minimizes the required number of labels. Active learning (AL) approaches have been proposed to address these challenges and were able to establish robust results on image classification. Such approaches are not directly applicable to sequences, since for sequences, the variations are in both spatial and temporal domains. In this brief, we introduce a novel method for AL for sequences, called "AL-SAR", which combines unsupervised training with sparsely supervised annotation. In particular, AL-SAR employs a multi-head mechanism for robust uncertainty evaluation of the latent space learned by an encoder-decoder framework. It aims to iteratively select a sparse set of samples, which annotation contributes the most to the disentanglement of the latent space. We evaluate our system on common benchmark datasets with multiple sequences and actions, such as NW-UCLA, NTU RGB + D 60, and UWA3D. Our results indicate that AL-SAR coupled with encoder-decoder network outperforms other AL methods coupled with the same network structure.

Statistical perspective on functional and causal neural connectomics: The Time-Aware PC algorithm.

The representation of the flow of information between neurons in the brain based on their activity is termed the causal functional connectome. Such representation incorporates the dynamic nature of neuronal activity and causal interactions between them. In contrast to connectome, the causal functional connectome is not directly observed and needs to be inferred from neural time series. A popular statistical framework for inferring causal connectivity from observations is the directed probabilistic graphical modeling. Its common formulation is not suitable for neural time series since it was developed for variables with independent and identically distributed static samples. In this work, we propose to model and estimate the causal functional connectivity from neural time series using a novel approach that adapts directed probabilistic graphical modeling to the time series scenario. In particular, we develop the Time-Aware PC (TPC) algorithm for estimating the causal functional connectivity, which adapts the PC algorithm-a state-of-the-art method for statistical causal inference. We show that the model outcome of TPC has the properties of reflecting causality of neural interactions such as being non-parametric, exhibits the directed Markov property in a time-series setting, and is predictive of the consequence of counterfactual interventions on the time series. We demonstrate the utility of the methodology to obtain the causal functional connectome for several datasets including simulations, benchmark datasets, and recent multi-array electro-physiological recordings from the mouse visual cortex.

Statistical Perspective on Functional and Causal Neural Connectomics: A Comparative Study.

Representation of brain network interactions is fundamental to the translation of neural structure to brain function. As such, methodologies for mapping neural interactions into structural models, i.e., inference of functional connectome from neural recordings, are key for the study of brain networks. While multiple approaches have been proposed for functional connectomics based on statistical associations between neural activity, association does not necessarily incorporate causation. Additional approaches have been proposed to incorporate aspects of causality to turn functional connectomes into causal functional connectomes, however, these methodologies typically focus on specific aspects of causality. This warrants a systematic statistical framework for causal functional connectomics that defines the foundations of common aspects of causality. Such a framework can assist in contrasting existing approaches and to guide development of further causal methodologies. In this work, we develop such a statistical guide. In particular, we consolidate the notions of associations and representations of neural interaction, i.e., types of neural connectomics, and then describe causal modeling in the statistics literature. We particularly focus on the introduction of directed Markov graphical models as a framework through which we define the Directed Markov Property-an essential criterion for examining the causality of proposed functional connectomes. We demonstrate how based on these notions, a comparative study of several existing approaches for finding causal functional connectivity from neural activity can be conducted. We proceed by providing an outlook ahead regarding the additional properties that future approaches could include to thoroughly address causality.

Knowledge distillation circumvents nonlinearity for optical convolutional neural networks.

In recent years, convolutional neural networks (CNNs) have enabled ubiquitous image processing applications. As such, CNNs require fast forward propagation runtime to process high-resolution visual streams in real time. This is still a challenging task even with state-of-the-art graphics and tensor processing units. The bottleneck in computational efficiency primarily occurs in the convolutional layers. Performing convolutions in the Fourier domain is a promising way to accelerate forward propagation since it transforms convolutions into elementwise multiplications, which are considerably faster to compute for large kernels. Furthermore, such computation could be implemented using an optical 4f system with orders of magnitude faster operation. However, a major challenge in using this spectral approach, as well as in an optical implementation of CNNs, is the inclusion of a nonlinearity between each convolutional layer, without which CNN performance drops dramatically. Here, we propose a spectral CNN linear counterpart (SCLC) network architecture and its optical implementation. We propose a hybrid platform with an optical front end to perform a large number of linear operations, followed by an electronic back end. The key contribution is to develop a knowledge distillation (KD) approach to circumvent the need for nonlinear layers between the convolutional layers and successfully train such networks. While the KD approach is known in machine learning as an effective process for network pruning, we adapt the approach to transfer the knowledge from a nonlinear network (teacher) to a linear counterpart (student), where we can exploit the inherent parallelism of light. We show that the KD approach can achieve performance that easily surpasses the standard linear version of a CNN and could approach the performance of the nonlinear network. Our simulations show that the possibility of increasing the resolution of the input image allows our proposed 4f optical linear network to perform more efficiently than a nonlinear network with the same accuracy on two fundamental image processing tasks: (i) object classification and (ii) semantic segmentation.

Clustering and Recognition of Spatiotemporal Features Through Interpretable Embedding of Sequence to Sequence Recurrent Neural Networks.

Encoder-decoder recurrent neural network models (RNN Seq2Seq) have achieved success in ubiquitous areas of computation and applications. They were shown to be effective in modeling data with both temporal and spatial dependencies for translation or prediction tasks. In this study, we propose an embedding approach to visualize and interpret the representation of data by these models. Furthermore, we show that the embedding is an effective method for unsupervised learning and can be utilized to estimate the optimality of model training. In particular, we demonstrate that embedding space projections of the decoder states of RNN Seq2Seq model trained on sequences prediction are organized in clusters capturing similarities and differences in the dynamics of these sequences. Such performance corresponds to an unsupervised clustering of any spatio-temporal features and can be employed for time-dependent problems such as temporal segmentation, clustering of dynamic activity, self-supervised classification, action recognition, failure prediction, etc. We test and demonstrate the application of the embedding methodology to time-sequences of 3D human body poses. We show that the methodology provides a high-quality unsupervised categorization of movements. The source code with examples is available in a Github repository.

Neural Interactome: Interactive Simulation of a Neuronal System.

Connectivity and biophysical processes determine the functionality of neuronal networks. We, therefore, developed a real-time framework, called Neural Interactome,, to simultaneously visualize and interact with the structure and dynamics of such networks. Neural Interactome is a cross-platform framework, which combines graph visualization with the simulation of neural dynamics, or experimentally recorded multi neural time series, to allow application of stimuli to neurons to examine network responses. In addition, Neural Interactome supports structural changes, such as disconnection of neurons from the network (ablation feature). Neural dynamics can be explored on a single neuron level (using a zoom feature), back in time (using a review feature), and recorded (using presets feature). The development of the Neural Interactome was guided by generic concepts to be applicable to neuronal networks with different neural connectivity and dynamics. We implement the framework using a model of the nervous system of Caenorhabditis elegans (C. elegans) nematode, a model organism with resolved connectome and neural dynamics. We show that Neural Interactome assists in studying neural response patterns associated with locomotion and other stimuli. In particular, we demonstrate how stimulation and ablation help in identifying neurons that shape particular dynamics. We examine scenarios that were experimentally studied, such as touch response circuit, and explore new scenarios that did not undergo elaborate experimental studies.

Functional connectomics from neural dynamics: probabilistic graphical models for neuronal network of Caenorhabditis elegans.

We propose an approach to represent neuronal network dynamics as a probabilistic graphical model (PGM). To construct the PGM, we collect time series of neuronal responses produced by the neuronal network and use singular value decomposition to obtain a low-dimensional projection of the time-series data. We then extract dominant patterns from the projections to get pairwise dependency information and create a graphical model for the full network. The outcome model is a functional connectome that captures how stimuli propagate through the network and thus represents causal dependencies between neurons and stimuli. We apply our methodology to a model of the Caenorhabditis elegans somatic nervous system to validate and show an example of our approach. The structure and dynamics of the C. elegans nervous system are well studied and a model that generates neuronal responses is available. The resulting PGM enables us to obtain and verify underlying neuronal pathways for known behavioural scenarios and detect possible pathways for novel scenarios.This article is part of a discussion meeting issue 'Connectome to behaviour: modelling C. elegans at cellular resolution'.

Classification of Fixed Point Network Dynamics from Multiple Node Timeseries Data.

Fixed point networks are dynamic networks encoding stimuli via distinct output patterns. Although, such networks are common in neural systems, their structures are typically unknown or poorly characterized. It is thereby valuable to use a supervised approach for resolving how a network encodes inputs of interest and the superposition of those inputs from sampled multiple node time series. In this paper, we show that accomplishing such a task involves finding a low-dimensional state space from supervised noisy recordings. We demonstrate that while standard methods for dimension reduction are unable to provide optimal separation of fixed points and transient trajectories approaching them, the combination of dimension reduction with selection (clustering) and optimization can successfully provide such functionality. Specifically, we propose two methods: Exclusive Threshold Reduction (ETR) and Optimal Exclusive Threshold Reduction (OETR) for finding a basis for the classification state space. We show that the classification space-constructed through the combination of dimension reduction and optimal separation-can directly facilitate recognition of stimuli, and classify complex inputs (mixtures) into similarity classes. We test our methodology on a benchmark data-set recorded from the olfactory system. We also use the benchmark to compare our results with the state-of-the-art. The comparison shows that our methods are capable to construct classification spaces and perform recognition at a significantly better rate than previously proposed approaches.

Symmetries Constrain Dynamics in a Family of Balanced Neural Networks.

We examine a family of random firing-rate neural networks in which we enforce the neurobiological constraint of Dale's Law-each neuron makes either excitatory or inhibitory connections onto its post-synaptic targets. We find that this constrained system may be described as a perturbation from a system with nontrivial symmetries. We analyze the symmetric system using the tools of equivariant bifurcation theory and demonstrate that the symmetry-implied structures remain evident in the perturbed system. In comparison, spectral characteristics of the network coupling matrix are relatively uninformative about the behavior of the constrained system.

Multistability and Long-Timescale Transients Encoded by Network Structure in a Model of C. elegans Connectome Dynamics.

The neural dynamics of the nematode Caenorhabditis elegans are experimentally low-dimensional and may be understood as long-timescale transitions between multiple low-dimensional attractors. Previous modeling work has found that dynamic models of the worm's full neuronal network are capable of generating reasonable dynamic responses to certain inputs, even when all neurons are treated as identical save for their connectivity. This study investigates such a model of C. elegans neuronal dynamics, finding that a wide variety of multistable responses are generated in response to varied inputs. Specifically, we generate bifurcation diagrams for all possible single-neuron inputs, showing the existence of fixed points and limit cycles for different input regimes. The nature of the dynamical response is seen to vary according to the type of neuron receiving input; for example, input into sensory neurons is more likely to drive a bifurcation in the system than input into motor neurons. As a specific example we consider compound input into the neuron pairs PLM and ASK, discovering bistability of a limit cycle and a fixed point. The transient timescales in approaching each of these states are much longer than any intrinsic timescales of the system. This suggests consistency of our model with the characterization of dynamics in neural systems as long-timescale transitions between discrete, low-dimensional attractors corresponding to behavioral states.

Neural Integration Underlying a Time-Compensated Sun Compass in the Migratory Monarch Butterfly.

Migrating eastern North American monarch butterflies use a time-compensated sun compass to adjust their flight to the southwest direction. Although the antennal genetic circadian clock and the azimuth of the sun are instrumental for proper function of the compass, it is unclear how these signals are represented on a neuronal level and how they are integrated to produce flight control. To address these questions, we constructed a receptive field model of the compound eye that encodes the solar azimuth. We then derived a neural circuit model that integrates azimuthal and circadian signals to correct flight direction. The model demonstrates an integration mechanism, which produces robust trajectories reaching the southwest regardless of the time of day and includes a configuration for remigration. Comparison of model simulations with flight trajectories of butterflies in a flight simulator shows analogous behaviors and affirms the prediction that midday is the optimal time for migratory flight.

Low-dimensional functionality of complex network dynamics: neurosensory integration in the Caenorhabditis Elegans connectome.

We develop a biophysical model of neurosensory integration in the model organism Caenorhabditis elegans. Building on experimental findings on the neuron conductances and their resolved connectome, we posit the first full dynamic model of the neural voltage excitations that allows for a characterization of network structures which link input stimuli to neural proxies of behavioral responses. Full connectome simulations of neural responses to prescribed inputs show that robust, low-dimensional bifurcation structures drive neural voltage activity modes. Comparison of these modes with experimental studies allows us to link these network structures to behavioral responses. Thus the underlying bifurcation structures discovered, i.e., induced Hopf bifurcations, are critical in explaining behavioral responses such as swimming and crawling.

Data-driven inference of network connectivity for modeling the dynamics of neural codes in the insect antennal lobe.

The antennal lobe (AL), olfactory processing center in insects, is able to process stimuli into distinct neural activity patterns, called olfactory neural codes. To model their dynamics we perform multichannel recordings from the projection neurons in the AL driven by different odorants. We then derive a dynamic neuronal network from the electrophysiological data. The network consists of lateral-inhibitory neurons and excitatory neurons (modeled as firing-rate units), and is capable of producing unique olfactory neural codes for the tested odorants. To construct the network, we (1) design a projection, an odor space, for the neural recording from the AL, which discriminates between distinct odorants trajectories (2) characterize scent recognition, i.e., decision-making based on olfactory signals and (3) infer the wiring of the neural circuit, the connectome of the AL. We show that the constructed model is consistent with biological observations, such as contrast enhancement and robustness to noise. The study suggests a data-driven approach to answer a key biological question in identifying how lateral inhibitory neurons can be wired to excitatory neurons to permit robust activity patterns.

Sensory biology. Flower discrimination by pollinators in a dynamic chemical environment.

Pollinators use their sense of smell to locate flowers from long distances, but little is known about how they are able to discriminate their target odor from a mélange of other natural and anthropogenic odors. Here, we measured the plume from Datura wrightii flowers, a nectar resource for Manduca sexta moths, and show that the scent was dynamic and rapidly embedded among background odors. The moth's ability to track the odor was dependent on the background and odor frequency. By influencing the balance of excitation and inhibition in the antennal lobe, background odors altered the neuronal representation of the target odor and the ability of the moth to track the plume. These results show that the mix of odors present in the environment influences the pollinator's olfactory ability.

High-Energy Passive Mode-Locking of Fiber Lasers.

Mode-locking refers to the generation of ultrashort optical pulses in laser systems. A comprehensive study of achieving high-energy pulses in a ring cavity fiber laser that is passively mode-locked by a series of waveplates and a polarizer is presented in this paper. Specifically, it is shown that the multipulsing instability can be circumvented in favor of bifurcating to higher-energy single pulses by appropriately adjusting the group velocity dispersion in the fiber and the waveplate/polarizer settings in the saturable absorber. The findings may be used as practical guidelines for designing high-power lasers since the theoretical model relates directly to the experimental settings.

Neural dynamics, bifurcations, and firing rates in a quadratic integrate-and-fire model with a recovery variable. I: Deterministic behavior.

We study the dynamics of a quadratic integrate-and-fire model of a single-compartment neuron with a slow recovery variable, as input current and parameters describing timescales, recovery variable, and postspike reset change. Analysis of a codimension 2 bifurcation reveals that the domain of attraction of a stable hyperpolarized rest state interacts subtly with reset parameters, which reposition the system state after spiking. We obtain explicit approximations of instantaneous firing rates for fixed values of the recovery variable, and use the averaging theorem to obtain asymptotic firing rates as a function of current and reset parameters. Along with the different phase-plane geometries, these computations provide explicit tools for the interpretation of different spiking patterns and guide parameter selection in modeling different cortical cell types.

Parabolic resonance: a route to hamiltonian spatiotemporal chaos.

We show that initial data near an unperturbed stable plane wave can evolve into a regime of spatiotemporal chaos in the slightly forced conservative periodic one-dimensional nonlinear Schrödinger equation. Statistical measures are employed to demonstrate that this spatiotemporal chaos is intermittent: there are windows in time for which the solution gains spatial coherence. The parameters and initial profiles that lead to such intermittency are predicted by utilizing a novel geometrical description of the integrable unforced equation.

Three types of chaos in the forced nonlinear Schrödinger equation.

Three different types of chaotic behavior and instabilities (homoclinic chaos, hyperbolic resonance, and parabolic resonance) in Hamiltonian perturbations of the nonlinear Schrödinger (NLS) equation are described. The analysis is performed on a truncated model using a novel framework in which a hierarchy of bifurcations is constructed. It is demonstrated numerically that the forced NLS equation exhibits analogous types of chaotic phenomena. Thus, by adjusting the forcing frequency, the behavior near the plane wave solution may be set to any one of the three different types of chaos for any periodic box length.