Quantifying and Maximizing the Information Flux in Recurrent Neural Networks.

Free-running recurrent neural networks (RNNs), especially probabilistic models, generate an ongoing information flux that can be quantified with the mutual information I[x→(t),x→(t+1)] between subsequent system states x→. Although previous studies have shown that I depends on the statistics of the network's connection weights, it is unclear how to maximize I systematically and how to quantify the flux in large systems where computing the mutual information becomes intractable. Here, we address these questions using Boltzmann machines as model systems. We find that in networks with moderately strong connections, the mutual information I is approximately a monotonic transformation of the root-mean-square averaged Pearson correlations between neuron pairs, a quantity that can be efficiently computed even in large systems. Furthermore, evolutionary maximization of I[x→(t),x→(t+1)] reveals a general design principle for the weight matrices enabling the systematic construction of systems with a high spontaneous information flux. Finally, we simultaneously maximize information flux and the mean period length of cyclic attractors in the state-space of these dynamical networks. Our results are potentially useful for the construction of RNNs that serve as short-time memories or pattern generators.

Synchronization in STDP-driven memristive neural networks with time-varying topology.

Synchronization is a widespread phenomenon in the brain. Despite numerous studies, the specific parameter configurations of the synaptic network structure and learning rules needed to achieve robust and enduring synchronization in neurons driven by spike-timing-dependent plasticity (STDP) and temporal networks subject to homeostatic structural plasticity (HSP) rules remain unclear. Here, we bridge this gap by determining the configurations required to achieve high and stable degrees of complete synchronization (CS) and phase synchronization (PS) in time-varying small-world and random neural networks driven by STDP and HSP. In particular, we found that decreasing P (which enhances the strengthening effect of STDP on the average synaptic weight) and increasing F (which speeds up the swapping rate of synapses between neurons) always lead to higher and more stable degrees of CS and PS in small-world and random networks, provided that the network parameters such as the synaptic time delay [Formula: see text], the average degree [Formula: see text], and the rewiring probability [Formula: see text] have some appropriate values. When [Formula: see text], [Formula: see text], and [Formula: see text] are not fixed at these appropriate values, the degree and stability of CS and PS may increase or decrease when F increases, depending on the network topology. It is also found that the time delay [Formula: see text] can induce intermittent CS and PS whose occurrence is independent F. Our results could have applications in designing neuromorphic circuits for optimal information processing and transmission via synchronization phenomena.

The stochastic Fitzhugh-Nagumo neuron model in the excitable regime embeds a leaky integrate-and-fire model.

In this paper, we provide a complete mathematical construction for a stochastic leaky-integrate-and-fire model (LIF) mimicking the interspike interval (ISI) statistics of a stochastic FitzHugh-Nagumo neuron model (FHN) in the excitable regime, where the unique fixed point is stable. Under specific types of noises, we prove that there exists a global random attractor for the stochastic FHN system. The linearization method is then applied to estimate the firing time and to derive the associated radial equation representing a LIF equation. This result confirms the previous prediction in Ditlevsen and Greenwood (J Math Biol 67(2):239-259, 2013) for the Morris-Lecar neuron model in the bistability regime consisting of a stable fixed point and a stable limit cycle.

Weak-noise-induced transitions with inhibition and modulation of neural oscillations.

We analyze the effect of weak-noise-induced transitions on the dynamics of the FitzHugh-Nagumo neuron model in a bistable state consisting of a stable fixed point and a stable unforced limit cycle. Bifurcation and slow-fast analysis give conditions on the parameter space for the establishment of this bi-stability. In the parametric zone of bi-stability, weak-noise amplitudes may strongly inhibit the neuron's spiking activity. Surprisingly, increasing the noise strength leads to a minimum in the spiking activity, after which the activity starts to increase monotonically with an increase in noise strength. We investigate this inhibition and modulation of neural oscillations by weak-noise amplitudes by looking at the variation of the mean number of spikes per unit time with the noise intensity. We show that this phenomenon always occurs when the initial conditions lie in the basin of attraction of the stable limit cycle. For initial conditions in the basin of attraction of the stable fixed point, the phenomenon, however, disappears, unless the timescale separation parameter of the model is bounded within some interval. We provide a theoretical explanation of this phenomenon in terms of the stochastic sensitivity functions of the attractors and their minimum Mahalanobis distances from the separatrix isolating the basins of attraction.

Dynamics of neural fields with exponential temporal kernel.

We consider the standard neural field equation with an exponential temporal kernel. We analyze the time-independent (static) and time-dependent (dynamic) bifurcations of the equilibrium solution and the emerging spatiotemporal wave patterns. We show that an exponential temporal kernel does not allow static bifurcations such as saddle-node, pitchfork, and in particular, static Turing bifurcations. However, the exponential temporal kernel possesses the important property that it takes into account the finite memory of past activities of neurons, which Green's function does not. Through a dynamic bifurcation analysis, we give explicit bifurcation conditions. Hopf bifurcations lead to temporally non-constant, but spatially constant solutions, but Turing-Hopf bifurcations generate spatially and temporally non-constant solutions, in particular, traveling waves. Bifurcation parameters are the coefficient of the exponential temporal kernel, the transmission speed of neural signals, the time delay rate of synapses, and the ratio of excitatory to inhibitory synaptic weights.

Combined effects of spike-timing-dependent plasticity and homeostatic structural plasticity on coherence resonance.

Efficient processing and transfer of information in neurons have been linked to noise-induced resonance phenomena such as coherence resonance (CR), and adaptive rules in neural networks have been mostly linked to two prevalent mechanisms: spike-timing-dependent plasticity (STDP) and homeostatic structural plasticity (HSP). Thus this paper investigates CR in small-world and random adaptive networks of Hodgkin-Huxley neurons driven by STDP and HSP. Our numerical study indicates that the degree of CR strongly depends, and in different ways, on the adjusting rate parameter P, which controls STDP, on the characteristic rewiring frequency parameter F, which controls HSP, and on the parameters of the network topology. In particular, we found two robust behaviors. (i) Decreasing P (which enhances the weakening effect of STDP on synaptic weights) and decreasing F (which slows down the swapping rate of synapses between neurons) always leads to higher degrees of CR in small-world and random networks, provided that the synaptic time delay parameter τ_{c} has some appropriate values. (ii) Increasing the synaptic time delay τ_{c} induces multiple CR (MCR)-the occurrence of multiple peaks in the degree of coherence as τ_{c} changes-in small-world and random networks, with MCR becoming more pronounced at smaller values of P and F. Our results imply that STDP and HSP can jointly play an essential role in enhancing the time precision of firing necessary for optimal information processing and transfer in neural systems and could thus have applications in designing networks of noisy artificial neural circuits engineered to use CR to optimize information processing and transfer.

Self-induced-stochastic-resonance breathing chimeras.

The study by Semenova et al. [Phys. Rev. Lett. 117, 014102 (2016)0031-900710.1103/PhysRevLett.117.014102] discovered a type of chimera state known as coherence-resonance chimera (CRC), which combines the effects of coherence resonance (CR) and the spatial property of classical chimeras. In this Letter, we present yet another form of chimera, which we refer to as self-induced-stochastic-resonance breathing chimera (SISR-BC), which differs fundamentally from the CRC in that it combines the mechanism and effects of self-induced stochastic resonance (SISR, previously shown by DeVille et al. [Phys. Rev. E 72, 031105 (2005)1539-375510.1103/PhysRevE.72.031105] to be intrinsically different from CR), the symmetry breaking in the rotational coupling between the slow and fast subsystems of the coupled oscillators, and the property of breathing chimera-a form of chimera state characterized by nonstationary periodic dynamics of coherent-incoherent patterns with a periodically oscillating global order parameter. Unlike other types of chimeras, including CRC, SISR-BC demonstrates remarkable resilience to a relatively wide range of stochastic perturbations and persists even when the purely excitable system is significantly distant from the Hopf bifurcation threshold-thanks to the mechanism of SISR-and globally attracts random distributions of initial conditions. Considering its potential impact on information processing in neuronal networks, SISR-BC could have special significance and applications.

Control of noise-induced coherent oscillations in three-neuron motifs.

The phenomenon of self-induced stochastic resonance (SISR) requires a nontrivial scaling limit between the deterministic and the stochastic timescales of an excitable system, leading to the emergence of coherent oscillations which are absent without noise. In this paper, we numerically investigate SISR and its control in single neurons and three-neuron motifs made up of the Morris-Lecar model. In single neurons, we compare the effects of electrical and chemical autapses on the degree of coherence of the oscillations due to SISR. In the motifs, we compare the effects of altering the synaptic time-delayed couplings and the topologies on the degree of SISR. Finally, we provide two enhancement strategies for a particularly poor degree of SISR in motifs with chemical synapses: (1) we show that a poor SISR can be significantly enhanced by attaching an electrical or an excitatory chemical autapse on one of the neurons, and (2) we show that by multiplexing the motif with a poor SISR to another motif (with a high SISR in isolation), the degree of SISR in the former motif can be significantly enhanced. We show that the efficiency of these enhancement strategies depends on the topology of the motifs and the nature of synaptic time-delayed couplings mediating the multiplexing connections.

Diversity-induced decoherence.

We analyze the effect of small-amplitude noise and heterogeneity in a network of coupled excitable oscillators with strong timescale separation. Using mean-field analysis, we uncover the mechanism of a nontrivial effect-diversity-induced decoherence (DIDC)-in which heterogeneity modulates the mechanism of self-induced stochastic resonance to inhibit the coherence of oscillations. We argue that DIDC may offer one possible mechanism via which, in excitable neural systems, generic heterogeneity and background noise can synergistically prevent unwanted resonances that may be related to hyperkinetic movement disorders.

Optimal Self-Induced Stochastic Resonance in Multiplex Neural Networks: Electrical vs. Chemical Synapses.

Electrical and chemical synapses shape the dynamics of neural networks, and their functional roles in information processing have been a longstanding question in neurobiology. In this paper, we investigate the role of synapses on the optimization of the phenomenon of self-induced stochastic resonance in a delayed multiplex neural network by using analytical and numerical methods. We consider a two-layer multiplex network in which, at the intra-layer level, neurons are coupled either by electrical synapses or by inhibitory chemical synapses. For each isolated layer, computations indicate that weaker electrical and chemical synaptic couplings are better optimizers of self-induced stochastic resonance. In addition, regardless of the synaptic strengths, shorter electrical synaptic delays are found to be better optimizers of the phenomenon than shorter chemical synaptic delays, while longer chemical synaptic delays are better optimizers than longer electrical synaptic delays; in both cases, the poorer optimizers are, in fact, worst. It is found that electrical, inhibitory, or excitatory chemical multiplexing of the two layers having only electrical synapses at the intra-layer levels can each optimize the phenomenon. Additionally, only excitatory chemical multiplexing of the two layers having only inhibitory chemical synapses at the intra-layer levels can optimize the phenomenon. These results may guide experiments aimed at establishing or confirming to the mechanism of self-induced stochastic resonance in networks of artificial neural circuits as well as in real biological neural networks.

Control of coherence resonance by self-induced stochastic resonance in a multiplex neural network.

We consider a two-layer multiplex network of diffusively coupled FitzHugh-Nagumo (FHN) neurons in the excitable regime. We show that the phenomenon of coherence resonance (CR) in one layer can not only be controlled by the network topology, the intra- and interlayer time-delayed couplings, but also by another phenomenon, namely, self-induced stochastic resonance (SISR) in the other layer. Numerical computations show that when the layers are isolated, each of these noise-induced phenomena is weakened (strengthened) by a sparser (denser) ring network topology, stronger (weaker) intralayer coupling forces, and longer (shorter) intralayer time delays. However, CR shows a much higher sensitivity than SISR to changes in these control parameters. It is also shown, in contrast to SISR in a single isolated FHN neuron, that the maximum noise amplitude at which SISR occurs in the network of coupled FHN neurons is controllable, especially in the regime of strong coupling forces and long time delays. In order to use SISR in the first layer of the multiplex network to control CR in the second layer, we first choose the control parameters of the second layer in isolation such that in one case CR is poor and in another case, nonexistent. It is then shown that a pronounced SISR can not only significantly improve a poor CR, but can also induce a pronounced CR, which was nonexistent in the isolated second layer. In contrast to strong intralayer coupling forces, strong interlayer coupling forces are found to enhance CR, while long interlayer time delays, just as long intralayer time delays, deteriorate CR. Most importantly, we find that in a strong interlayer coupling regime, SISR in the first layer performs better than CR in enhancing CR in the second layer. But in a weak interlayer coupling regime, CR in the first layer performs better than SISR in enhancing CR in the second layer. Our results could find novel applications in noisy neural network dynamics and engineering.