Mesoscale effects of trader learning behaviors in financial markets: A multi-agent reinforcement learning study.

Recent advances in the field of machine learning have yielded novel research perspectives in behavioural economics and financial markets microstructure studies. In this paper we study the impact of individual trader leaning characteristics on markets using a stock market simulator designed with a multi-agent architecture. Each agent, representing an autonomous investor, trades stocks through reinforcement learning, using a centralized double-auction limit order book. This approach allows us to study the impact of individual trader traits on the whole stock market at the mesoscale in a bottom-up approach. We chose to test three trader trait aspects: agent learning rate increases, herding behaviour and random trading. As hypothesized, we find that larger learning rates significantly increase the number of crashes. We also find that herding behaviour undermines market stability, while random trading tends to preserve it.

Breakdown of category-specific word representations in a brain-constrained neurocomputational model of semantic dementia.

The neurobiological nature of semantic knowledge, i.e., the encoding and storage of conceptual information in the human brain, remains a poorly understood and hotly debated subject. Clinical data on semantic deficits and neuroimaging evidence from healthy individuals have suggested multiple cortical regions to be involved in the processing of meaning. These include semantic hubs (most notably, anterior temporal lobe, ATL) that take part in semantic processing in general as well as sensorimotor areas that process specific aspects/categories according to their modality. Biologically inspired neurocomputational models can help elucidate the exact roles of these regions in the functioning of the semantic system and, importantly, in its breakdown in neurological deficits. We used a neuroanatomically constrained computational model of frontotemporal cortices implicated in word acquisition and processing, and adapted it to simulate and explain the effects of semantic dementia (SD) on word processing abilities. SD is a devastating, yet insufficiently understood progressive neurodegenerative disease, characterised by semantic knowledge deterioration that is hypothesised to be specifically related to neural damage in the ATL. The behaviour of our brain-based model is in full accordance with clinical data-namely, word comprehension performance decreases as SD lesions in ATL progress, whereas word repetition abilities remain less affected. Furthermore, our model makes predictions about lesion- and category-specific effects of SD: our simulation results indicate that word processing should be more impaired for object- than for action-related words, and that degradation of white matter should produce more severe consequences than the same proportion of grey matter decay. In sum, the present results provide a neuromechanistic explanatory account of cortical-level language impairments observed during the onset and progress of semantic dementia.

A universal description of stochastic oscillators.

Many systems in physics, chemistry, and biology exhibit oscillations with a pronounced random component. Such stochastic oscillations can emerge via different mechanisms, for example, linear dynamics of a stable focus with fluctuations, limit-cycle systems perturbed by noise, or excitable systems in which random inputs lead to a train of pulses. Despite their diverse origins, the phenomenology of random oscillations can be strikingly similar. Here, we introduce a nonlinear transformation of stochastic oscillators to a complex-valued function [Formula: see text](x) that greatly simplifies and unifies the mathematical description of the oscillator's spontaneous activity, its response to an external time-dependent perturbation, and the correlation statistics of different oscillators that are weakly coupled. The function [Formula: see text] (x) is the eigenfunction of the Kolmogorov backward operator with the least negative (but nonvanishing) eigenvalue λ1 = µ1 + iω1. The resulting power spectrum of the complex-valued function is exactly given by a Lorentz spectrum with peak frequency ω1 and half-width µ1; its susceptibility with respect to a weak external forcing is given by a simple one-pole filter, centered around ω1; and the cross-spectrum between two coupled oscillators can be easily expressed by a combination of the spontaneous power spectra of the uncoupled systems and their susceptibilities. Our approach makes qualitatively different stochastic oscillators comparable, provides simple characteristics for the coherence of the random oscillation, and gives a framework for the description of weakly coupled oscillators.

Force tuning explains changes in phasic dopamine signaling during stimulus-reward learning.

According to a popular hypothesis, phasic dopamine (DA) activity encodes a reward prediction error (RPE) necessary for reinforcement learning. However, recent work showed that DA neurons are necessary for performance rather than learning. One limitation of previous work on phasic DA signaling and RPE is the limited behavioral measures. Here, we measured subtle force exertion while recording and manipulating DA activity in the ventral tegmental area (VTA) during stimulus-reward learning. We found two major populations of DA neurons that increased firing before forward and backward force exertion. Force tuning is the same regardless of learning, reward predictability, or outcome valence. Changes in the pattern of force exertion can explain results traditionally used to support the RPE hypothesis, such as modulation by reward magnitude, probability, and unpredicted reward delivery or omission. Thus VTA DA neurons are not used to signal RPE but to regulate force exertion during motivated behavior.

Homeostatic Reinforcement Theory Accounts for Sodium Appetitive State- and Taste-Dependent Dopamine Responding.

Seeking and consuming nutrients is essential to survival and the maintenance of life. Dynamic and volatile environments require that animals learn complex behavioral strategies to obtain the necessary nutritive substances. While this has been classically viewed in terms of homeostatic regulation, recent theoretical work proposed that such strategies result from reinforcement learning processes. This theory proposed that phasic dopamine (DA) signals play a key role in signaling potentially need-fulfilling outcomes. To examine links between homeostatic and reinforcement learning processes, we focus on sodium appetite as sodium depletion triggers state- and taste-dependent changes in behavior and DA signaling evoked by sodium-related stimuli. We find that both the behavior and the dynamics of DA signaling underlying sodium appetite can be accounted for by a homeostatically regulated reinforcement learning framework (HRRL). We first optimized HRRL-based agents to sodium-seeking behavior measured in rodents. Agents successfully reproduced the state and the taste dependence of behavioral responding for sodium as well as for lithium and potassium salts. We then showed that these same agents account for the regulation of DA signals evoked by sodium tastants in a taste- and state-dependent manner. Our models quantitatively describe how DA signals evoked by sodium decrease with satiety and increase with deprivation. Lastly, our HRRL agents assigned equal preference for sodium versus the lithium containing salts, accounting for similar behavioral and neurophysiological observations in rodents. We propose that animals use orosensory signals as predictors of the internal impact of the consumed good and our results pose clear targets for future experiments. In sum, this work suggests that appetite-driven behavior may be driven by reinforcement learning mechanisms that are dynamically tuned by homeostatic need.

Spikebench: An open benchmark for spike train time-series classification.

Modern well-performing approaches to neural decoding are based on machine learning models such as decision tree ensembles and deep neural networks. The wide range of algorithms that can be utilized to learn from neural spike trains, which are essentially time-series data, results in the need for diverse and challenging benchmarks for neural decoding, similar to the ones in the fields of computer vision and natural language processing. In this work, we propose a spike train classification benchmark, based on open-access neural activity datasets and consisting of several learning tasks such as stimulus type classification, animal's behavioral state prediction, and neuron type identification. We demonstrate that an approach based on hand-crafted time-series feature engineering establishes a strong baseline performing on par with state-of-the-art deep learning-based models for neural decoding. We release the code allowing to reproduce the reported results.

Multistability and evolution of chimera states in a network of type II Morris-Lecar neurons with asymmetrical nonlocal inhibitory connections.

Formation of synchronous activity patterns is an essential property of neuronal networks that has been of central interest to synchronization theory. Chimera states, where both synchronous and asynchronous activities of neurons co-exist in a single network, are particularly poignant examples of such patterns, whose dynamics and multistability may underlie brain function, such as cognitive tasks. However, dynamical mechanisms of coherent state formation in spiking neuronal networks as well as ways to control these states remain unclear. In this paper, we take a step in this direction by considering the evolution of chimera states in a network of class II excitable Morris-Lecar neurons with asymmetrical nonlocal inhibitory connections. Using the adaptive coherence measure, we are able to partition the network parameter space into regions of various collective behaviors (antiphase synchronous clusters, traveling waves, different types of chimera states as well as a spiking death regime) and have shown multistability between the various regimes. We track the evolution of the chimera states as a function of changed key network parameters and found transitions between various types of chimera states. We further find that the network can demonstrate long transients leading to quasi-persistence of activity patterns in the border regions hinting at near-criticality behaviors.

Recurring Cholinergic Inputs Induce Local Hippocampal Plasticity through Feedforward Disinhibition.

The CA1 pyramidal neurons are embedded in an intricate local circuitry that contains a variety of interneurons. The roles these interneurons play in the regulation of the excitatory synaptic plasticity remains largely understudied. Recent experiments showed that recurring cholinergic activation of α7 nACh receptors expressed in oriens-lacunosum-moleculare (OLMα2) interneurons can directly induce LTP in Schaffer collateral (SC)-CA1 synapses. Here, we pair in vitro studies with biophysically based modeling to uncover the underlying mechanisms. According to our model, α7 nAChR activation increases OLM GABAergic activity. This results in the inhibition of the fast-spiking interneurons that provide feedforward inhibition onto CA1 pyramidal neurons. This disinhibition, paired with tightly timed SC stimulation, can induce potentiation at the excitatory synapses of CA1 pyramidal neurons. Our work details the role of cholinergic modulation in disinhibition-induced hippocampal plasticity. It relates the timing of cholinergic pairing found experimentally in previous studies with the timing between disinhibition and hippocampal stimulation necessary to induce potentiation and suggests the dynamics of the involved interneurons play a crucial role in determining this timing.Significance StatementWe use a combination of experiments and mechanistic modeling to uncover the key role for cholinergic neuromodulation of feedforward disinhibitory circuits in regulating hippocampal plasticity. We found that cholinergic activation of α7 nAChR on α7 nACh receptors expressed in oriens-lacunosum-moleculare interneurons, when tightly paired with stimulation of the Schaffer collaterals, can cancel feedforward inhibition onto CA1 pyramidal cells, enabling the potentiation of the SC-CA1 synapse. Our work details how cholinergic action on GABAergic interneurons can tightly regulate the excitability and plasticity of the hippocampal network, unraveling the intricate interplay of the hierarchal inhibitory circuitry and cholinergic neuromodulation as a mechanism for hippocampal plasticity.

A framework for macroscopic phase-resetting curves for generalised spiking neural networks.

Brain rhythms emerge from synchronization among interconnected spiking neurons. Key properties of such rhythms can be gleaned from the phase-resetting curve (PRC). Inferring the PRC and developing a systematic phase reduction theory for large-scale brain rhythms remains an outstanding challenge. Here we present a theoretical framework and methodology to compute the PRC of generic spiking networks with emergent collective oscillations. We adopt a renewal approach where neurons are described by the time since their last action potential, a description that can reproduce the dynamical feature of many cell types. For a sufficiently large number of neurons, the network dynamics are well captured by a continuity equation known as the refractory density equation. We develop an adjoint method for this equation giving a semi-analytical expression of the infinitesimal PRC. We confirm the validity of our framework for specific examples of neural networks. Our theoretical framework can link key biological properties at the individual neuron scale and the macroscopic oscillatory network properties. Beyond spiking networks, the approach is applicable to a broad class of systems that can be described by renewal processes.

Phase response approaches to neural activity models with distributed delay.

In weakly coupled neural oscillator networks describing brain dynamics, the coupling delay is often distributed. We present a theoretical framework to calculate the phase response curve of distributed-delay induced limit cycles with infinite-dimensional phase space. Extending previous works, in which non-delayed or discrete-delay systems were investigated, we develop analytical results for phase response curves of oscillatory systems with distributed delay using Gaussian and log-normal delay distributions. We determine the scalar product and normalization condition for the linearized adjoint of the system required for the calculation of the phase response curve. As a paradigmatic example, we apply our technique to the Wilson-Cowan oscillator model of excitatory and inhibitory neuronal populations under the two delay distributions. We calculate and compare the phase response curves for the Gaussian and log-normal delay distributions. The phase response curves obtained from our adjoint calculations match those compiled by the direct perturbation method, thereby proving that the theory of weakly coupled oscillators can be applied successfully for distributed-delay-induced limit cycles. We further use the obtained phase response curves to derive phase interaction functions and determine the possible phase locked states of multiple inter-coupled populations to illuminate different synchronization scenarios. In numerical simulations, we show that the coupling delay distribution can impact the stability of the synchronization between inter-coupled gamma-oscillatory networks.

Cholinergic modulation of hierarchical inhibitory control over cortical resting state dynamics: Local circuit modeling of schizophrenia-related hypofrontality.

Nicotinic acetylcholine receptors (nAChRs) modulate the cholinergic drive to a hierarchy of inhibitory neurons in the superficial layers of the PFC, critical to cognitive processes. It has been shown that genetic deletions of the various types of nAChRs impact the properties of ultra-slow transitions between high and low PFC activity states in mice during quiet wakefulness. The impact characteristics depend on specific interneuron populations expressing the manipulated receptor subtype. In addition, recent data indicate that a genetic mutation of the α5 nAChR subunit, located on vasoactive intestinal polypeptide (VIP) inhibitory neurons, the rs16969968 single nucleotide polymorphism (α5 SNP), plays a key role in the hypofrontality observed in schizophrenia patients carrying the SNP. Data also indicate that chronic nicotine application to α5 SNP mice relieves the hypofrontality. We developed a computational model to show that the activity patterns recorded in the genetically modified mice can be explained by changes in the dynamics of the local PFC circuit. Notably, our model shows that these altered PFC circuit dynamics are due to changes in the stability structure of the activity states. We identify how this stability structure is differentially modulated by cholinergic inputs to the parvalbumin (PV), somatostatin (SOM) or the VIP inhibitory populations. Our model uncovers that a change in amplitude, but not duration of the high activity states can account for the lowered pyramidal (PYR) population firing rates recorded in α5 SNP mice. We demonstrate how nicotine-induced desensitization and upregulation of the ß2 nAChRs located on SOM interneurons, as opposed to the activation of α5 nAChRs located on VIP interneurons, is sufficient to explain the nicotine-induced activity normalization in α5 SNP mice. The model further implies that subsequent nicotine withdrawal may exacerbate the hypofrontality over and beyond one caused by the SNP mutation.

Additively Combining Utilities and Beliefs: Research Gaps and Algorithmic Developments.

Value-based decision making in complex environments, such as those with uncertain and volatile mapping of reward probabilities onto options, may engender computational strategies that are not necessarily optimal in terms of normative frameworks but may ensure effective learning and behavioral flexibility in conditions of limited neural computational resources. In this article, we review a suboptimal strategy - additively combining reward magnitude and reward probability attributes of options for value-based decision making. In addition, we present computational intricacies of a recently developed model (named MIX model) representing an algorithmic implementation of the additive strategy in sequential decision-making with two options. We also discuss its opportunities; and conceptual, inferential, and generalization issues. Furthermore, we suggest future studies that will reveal the potential and serve the further development of the MIX model as a general model of value-based choice making.

Activity Stabilization in a Population Model of Working Memory by Sinusoidal and Noisy Inputs.

According to mechanistic theories of working memory (WM), information is retained as stimulus-dependent persistent spiking activity of cortical neural networks. Yet, how this activity is related to changes in the oscillatory profile observed during WM tasks remains a largely open issue. We explore joint effects of input gamma-band oscillations and noise on the dynamics of several firing rate models of WM. The considered models have a metastable active regime, i.e., they demonstrate long-lasting transient post-stimulus firing rate elevation. We start from a single excitatory-inhibitory circuit and demonstrate that either gamma-band or noise input could stabilize the active regime, thus supporting WM retention. We then consider a system of two circuits with excitatory intercoupling. We find that fast coupling allows for better stabilization by common noise compared to independent noise and stronger amplification of this effect by in-phase gamma inputs compared to anti-phase inputs. Finally, we consider a multi-circuit system comprised of two clusters, each containing a group of circuits receiving a common noise input and a group of circuits receiving independent noise. Each cluster is associated with its own local gamma generator, so all its circuits receive gamma-band input in the same phase. We find that gamma-band input differentially stabilizes the activity of the "common-noise" groups compared to the "independent-noise" groups. If the inter-cluster connections are fast, this effect is more pronounced when the gamma-band input is delivered to the clusters in the same phase rather than in the anti-phase. Assuming that the common noise comes from a large-scale distributed WM representation, our results demonstrate that local gamma oscillations can stabilize the activity of the corresponding parts of this representation, with stronger effect for fast long-range connections and synchronized gamma oscillations.

Efficient and robust coding in heterogeneous recurrent networks.

Cortical networks show a large heterogeneity of neuronal properties. However, traditional coding models have focused on homogeneous populations of excitatory and inhibitory neurons. Here, we analytically derive a class of recurrent networks of spiking neurons that close to optimally track a continuously varying input online, based on two assumptions: 1) every spike is decoded linearly and 2) the network aims to reduce the mean-squared error between the input and the estimate. From this we derive a class of predictive coding networks, that unifies encoding and decoding and in which we can investigate the difference between homogeneous networks and heterogeneous networks, in which each neurons represents different features and has different spike-generating properties. We find that in this framework, 'type 1' and 'type 2' neurons arise naturally and networks consisting of a heterogeneous population of different neuron types are both more efficient and more robust against correlated noise. We make two experimental predictions: 1) we predict that integrators show strong correlations with other integrators and resonators are correlated with resonators, whereas the correlations are much weaker between neurons with different coding properties and 2) that 'type 2' neurons are more coherent with the overall network activity than 'type 1' neurons.

Distinct Temporal Structure of Nicotinic ACh Receptor Activation Determines Responses of VTA Neurons to Endogenous ACh and Nicotine.

The addictive component of tobacco, nicotine, acts via nicotinic acetylcholine receptors (nAChRs). The ß2 subunit-containing nAChRs (ß2-nAChRs) play a crucial role in the rewarding properties of nicotine and are particularly densely expressed in the mesolimbic dopamine (DA) system. Specifically, nAChRs directly and indirectly affect DA neurons in the ventral tegmental area (VTA). The understanding of ACh and nicotinic regulation of DA neuron activity is incomplete. By computational modeling, we provide mechanisms for several apparently contradictory experimental results. First, systemic knockout of ß2-containing nAChRs drastically reduces DA neurons bursting, although the major glutamatergic (Glu) afferents that have been shown to evoke this bursting stay intact. Second, the most intuitive way to rescue this bursting-by re-expressing the nAChRs on VTA DA neurons-fails. Third, nAChR re-expression on VTA GABA neurons rescues bursting in DA neurons and increases their firing rate under the influence of ACh input, whereas nicotinic application results in the opposite changes in firing. Our model shows that, first, without ACh receptors, Glu excitation of VTA DA and GABA neurons remains balanced and GABA inhibition cancels the direct excitation. Second, re-expression of ACh receptors on DA neurons provides an input that impedes membrane repolarization and is ineffective in restoring firing of DA neurons. Third, the distinct responses to ACh and nicotine occur because of distinct temporal patterns of these inputs: pulsatile versus continuous. Altogether, this study highlights how ß2-nAChRs influence coactivation of the VTA DA and GABA neurons required for motivation and saliency signals carried by DA neuron activity.

Role of synaptic nonlinearity in persistent firing rate shifts caused by external periodic forcing.

Information storage and processing in the brain largely relies on the neural population coding principle. In this framework, information is reflected in the population firing rate that reflects asynchronous irregular spiking of its constituent neurons. Periodic modulations of neural activity can lead to neural activity oscillations. Data indicate that such oscillations are ubiquitous in brain activity and are modulated, in frequency and amplitude, in a functionally meaningful manner. The relationship between oscillations and the population rate code remains an open issue. While ample works show how changes in the mean firing rate may alter neural oscillations, the reverse connection is unclear. One notable possibility is that oscillatory activity impinging on a neural population modulates its mean firing rate, thereby impacting information processing. We suggest that such modulation requires nonlinearities and propose nonlinear excitatory coupling via slow N-methyl-D-aspartate (NMDA) receptors as the prevalent mechanism. The aim of our paper is to theoretically explore to what extent the NMDA-related mechanism could account for oscillation-induced mean firing rate changes. We consider a mean-field model of a neural circuit containing an excitatory and an inhibitory population with linear transfer functions. Along with NMDA excitation, the model included fast recurrent excitatory and inhibitory connectivity. To explicitly study the effects of impinging oscillation on the rate dynamics, we subjected the circuit to a sinusoidal input signal imitating an input from distant brain regions or from a larger network into which the circuit is embedded. Using time-scale separation and time-averaging techniques, we developed a geometric method to determine the oscillation-induced mean firing rate shifts and validated it by numeric simulations of the model. Our results indicate that a large-amplitude stable firing rate shift requires nonlinear NMDA synapses on both the excitatory and the inhibitory populations. Our results delineate specific neural synaptic properties that enable neural oscillations to act as flexible modulators of the population rate code.

Transition from quantum chaos to localization in spin chains.

Recent years have seen an increasing interest in quantum chaos and related aspects of spatially extended systems, such as spin chains. However, the results are strongly system dependent: generic approaches suggest the presence of many-body localization, while analytical calculations for certain system classes, here referred to as the "self-dual case," prove adherence to universal (chaotic) spectral behavior. We address these issues studying the level statistics in the vicinity of the latter case, thereby revealing transitions to many-body localization as well as the appearance of several nonstandard random-matrix universality classes.

Hippocampal Interneuronal α7 nAChRs Modulate Theta Oscillations in Freely Moving Mice.

Muscarinic acetylcholine receptors (mAChRs) are critically involved in hippocampal theta generation, but much less is known about the role of nicotinic AChRs (nAChRs). Here we provide evidence that α7 nAChRs expressed on interneurons, particularly those in oriens lacunosum moleculare (OLM), also regulate hippocampal theta generation. Local hippocampal infusion of a selective α7 nAChR antagonist significantly reduces hippocampal theta power and impairs Y-maze spontaneous alternation performance in freely moving mice. By knocking out receptors in different neuronal subpopulations, we find that α7 nAChRs expressed in OLM interneurons regulate theta generation. Our in vitro slice studies indicate that α7 nAChR activation increases OLM neuron activity that, in turn, enhances pyramidal cell excitatory postsynaptic currents (EPSCs). Our study also suggests that mAChR activation promotes transient theta generation, while α7 nAChR activation facilitates future theta generation by similar stimulations, revealing a complex mechanism whereby cholinergic signaling modulates different aspects of hippocampal theta oscillations through different receptor subtypes.

Progressive alignment of inhibitory and excitatory delay may drive a rapid developmental switch in cortical network dynamics.

Nervous system maturation occurs on multiple levels-synaptic, circuit, and network-at divergent timescales. For example, many synaptic properties mature gradually, whereas emergent network dynamics can change abruptly. Here we combine experimental and theoretical approaches to investigate a sudden transition in spontaneous and sensory evoked thalamocortical activity necessary for the development of vision. Inspired by in vivo measurements of timescales and amplitudes of synaptic currents, we extend the Wilson and Cowan model to take into account the relative onset timing and amplitudes of inhibitory and excitatory neural population responses. We study this system as these parameters are varied within amplitudes and timescales consistent with developmental observations to identify the bifurcations of the dynamics that might explain the network behaviors in vivo. Our findings indicate that the inhibitory timing is a critical determinant of thalamocortical activity maturation; a gradual decay of the ratio of inhibitory to excitatory onset time drives the system through a bifurcation that leads to a sudden switch of the network spontaneous activity from high-amplitude oscillations to a nonoscillatory active state. This switch also drives a change from a threshold bursting to linear response to transient stimuli, also consistent with in vivo observation. Thus we show that inhibitory timing is likely critical to the development of network dynamics and may underlie rapid changes in activity without similarly rapid changes in the underlying synaptic and cellular parameters.NEW & NOTEWORTHY Relying on a generalization of the Wilson-Cowan model, which allows a solid analytic foundation for the understanding of the link between maturation of inhibition and network dynamics, we propose a potential explanation for the role of developing excitatory/inhibitory synaptic delays in mediating a sudden switch in thalamocortical visual activity preceding vision onset.

Neurocomputational theories of homeostatic control.

Homeostasis is a problem for all living agents. It entails predictively regulating internal states within the bounds compatible with survival in order to maximise fitness. This can be achieved physiologically, through complex hierarchies of autonomic regulation, but it must also be achieved via behavioural control, both reactive and proactive. Here we briefly review some of the major theories of homeostatic control and their historical cognates, addressing how they tackle the optimisation of both physiological and behavioural homeostasis. We start with optimal control approaches, setting up key concepts, exploring their strengths and limitations. We then concentrate on contemporary neurocomputational approaches to homeostatic control. We primarily focus on a branch of reinforcement learning known as homeostatic reinforcement learning (HRL). A central premise of HRL is that reward optimisation is directly coupled to homeostatic control. A central construct in this framework is the drive function which maps from homeostatic state to motivational drive, where reductions in drive are operationally defined as reward values. We explain HRL's main advantages, empirical applications, and conceptual insights. Notably, we show how simple constraints on the drive function can yield a normative account of predictive control, as well as account for phenomena such as satiety, risk aversion, and interactions between competing homeostatic needs. We illustrate how HRL agents can learn to avoid hazardous states without any need to experience them, and how HRL can be applied in clinical domains. Finally, we outline several challenges to HRL, and how survival constraints and active inference models could circumvent these problems.