Differential Dopamine Receptor-Dependent Sensitivity Improves the Switch Between Hard and Soft Selection in a Model of the Basal Ganglia.

The problem of selecting one action from a set of different possible actions, simply referred to as the problem of action selection, is a ubiquitous challenge in the animal world. For vertebrates, the basal ganglia (BG) are widely thought to implement the core computation to solve this problem, as its anatomy and physiology are well suited to this end. However, the BG still display physiological features whose role in achieving efficient action selection remains unclear. In particular, it is known that the two types of dopaminergic receptors (D1 and D2) present in the BG give rise to mechanistically different responses. The overall effect will be a difference in sensitivity to dopamine, which may have ramifications for action selection. However, which receptor type leads to a stronger response is unclear due to the complexity of the intracellular mechanisms involved. In this study, we use an existing, high-level computational model of the BG, which assumes that dopamine contributes to action selection by enabling a switch between different selection regimes, to predict which of D1 or D2 has the greater sensitivity. Thus, we ask, Assuming dopamine enables a switch between action selection regimes in the BG, what functional sensitivity values would result in improved action selection computation? To do this, we quantitatively assessed the model's capacity to perform action selection as we parametrically manipulated the sensitivity weights of D1 and D2. We show that differential (rather than equal) D1 and D2 sensitivity to dopaminergic input improves the switch between selection regimes during the action selection computation in our model. Specifically, greater D2 sensitivity compared to D1 led to these improvements.

Roles for globus pallidus externa revealed in a computational model of action selection in the basal ganglia.

The basal ganglia are considered vital to action selection - a hypothesis supported by several biologically plausible computational models. Of the several subnuclei of the basal ganglia, the globus pallidus externa (GPe) has been thought of largely as a relay nucleus, and its intrinsic connectivity has not been incorporated in significant detail, in any model thus far. Here, we incorporate newly revealed subgroups of neurons within the GPe into an existing computational model of the basal ganglia, and investigate their role in action selection. Three main results ensued. First, using previously used metrics for selection, the new extended connectivity improved the action selection performance of the model. Second, low frequency theta oscillations were observed in the subpopulation of the GPe (the TA or 'arkypallidal' neurons) which project exclusively to the striatum. These oscillations were suppressed by increased dopamine activity - revealing a possible link with symptoms of Parkinson's disease. Third, a new phenomenon was observed in which the usual monotonic relationship between input to the basal ganglia and its output within an action 'channel' was, under some circumstances, reversed. Thus, at high levels of input, further increase of this input to the channel could cause an increase of the corresponding output rather than the more usually observed decrease. Moreover, this phenomenon was associated with the prevention of multiple channel selection, thereby assisting in optimal action selection. Examination of the mechanistic origin of our results showed the so-called 'prototypical' GPe neurons to be the principal subpopulation influencing action selection. They control the striatum via the arkypallidal neurons and are also able to regulate the output nuclei directly. Taken together, our results highlight the role of the GPe as a major control hub of the basal ganglia, and provide a mechanistic account for its control function.

A probabilistic, distributed, recursive mechanism for decision-making in the brain.

Decision formation recruits many brain regions, but the procedure they jointly execute is unknown. Here we characterize its essential composition, using as a framework a novel recursive Bayesian algorithm that makes decisions based on spike-trains with the statistics of those in sensory cortex (MT). Using it to simulate the random-dot-motion task, we demonstrate it quantitatively replicates the choice behaviour of monkeys, whilst predicting losses of otherwise usable information from MT. Its architecture maps to the recurrent cortico-basal-ganglia-thalamo-cortical loops, whose components are all implicated in decision-making. We show that the dynamics of its mapped computations match those of neural activity in the sensorimotor cortex and striatum during decisions, and forecast those of basal ganglia output and thalamus. This also predicts which aspects of neural dynamics are and are not part of inference. Our single-equation algorithm is probabilistic, distributed, recursive, and parallel. Its success at capturing anatomy, behaviour, and electrophysiology suggests that the mechanism implemented by the brain has these same characteristics.

Integrating Brain and Biomechanical Models-A New Paradigm for Understanding Neuro-muscular Control.

To date, realistic models of how the central nervous system governs behavior have been restricted in scope to the brain, brainstem or spinal column, as if these existed as disembodied organs. Further, the model is often exercised in relation to an in vivo physiological experiment with input comprising an impulse, a periodic signal or constant activation, and output as a pattern of neural activity in one or more neural populations. Any link to behavior is inferred only indirectly via these activity patterns. We argue that to discover the principles of operation of neural systems, it is necessary to express their behavior in terms of physical movements of a realistic motor system, and to supply inputs that mimic sensory experience. To do this with confidence, we must connect our brain models to neuro-muscular models and provide relevant visual and proprioceptive feedback signals, thereby closing the loop of the simulation. This paper describes an effort to develop just such an integrated brain and biomechanical system using a number of pre-existing models. It describes a model of the saccadic oculomotor system incorporating a neuromuscular model of the eye and its six extraocular muscles. The position of the eye determines how illumination of a retinotopic input population projects information about the location of a saccade target into the system. A pre-existing saccadic burst generator model was incorporated into the system, which generated motoneuron activity patterns suitable for driving the biomechanical eye. The model was demonstrated to make accurate saccades to a target luminance under a set of environmental constraints. Challenges encountered in the development of this model showed the importance of this integrated modeling approach. Thus, we exposed shortcomings in individual model components which were only apparent when these were supplied with the more plausible inputs available in a closed loop design. Consequently we were able to suggest missing functionality which the system would require to reproduce more realistic behavior. The construction of such closed-loop animal models constitutes a new paradigm of computational neurobehavior and promises a more thoroughgoing approach to our understanding of the brain's function as a controller for movement and behavior.

Decision-making and action selection in insects: inspiration from vertebrate-based theories.

Effective decision-making, one of the most crucial functions of the brain, entails the analysis of sensory information and the selection of appropriate behavior in response to stimuli. Here, we consider the current state of knowledge on the mechanisms of decision-making and action selection in the insect brain, with emphasis on the olfactory processing system. Theoretical and computational models of decision-making emphasize the importance of using inhibitory connections to couple evidence-accumulating pathways; this coupling allows for effective discrimination between competing alternatives and thus enables a decision maker to reach a stable unitary decision. Theory also shows that the coupling of pathways can be implemented using a variety of different mechanisms and vastly improves the performance of decision-making systems. The vertebrate basal ganglia appear to resolve stable action selection by being a point of convergence for multiple excitatory and inhibitory inputs such that only one possible response is selected and all other alternatives are suppressed. Similar principles appear to operate within the insect brain. The insect lateral protocerebrum (LP) serves as a point of convergence for multiple excitatory and inhibitory channels of olfactory information to effect stable decision and action selection, at least for olfactory information. The LP is a rather understudied region of the insect brain, yet this premotor region may be key to effective resolution of action section. We argue that it may be beneficial to use models developed to explore the operation of the vertebrate brain as inspiration when considering action selection in the invertebrate domain. Such an approach may facilitate the proposal of new hypotheses and furthermore frame experimental studies for how decision-making and action selection might be achieved in insects.

Probabilistic Decision Making with Spikes: From ISI Distributions to Behaviour via Information Gain.

Computational theories of decision making in the brain usually assume that sensory 'evidence' is accumulated supporting a number of hypotheses, and that the first accumulator to reach threshold triggers a decision in favour of its associated hypothesis. However, the evidence is often assumed to occur as a continuous process whose origins are somewhat abstract, with no direct link to the neural signals - action potentials or 'spikes' - that must ultimately form the substrate for decision making in the brain. Here we introduce a new variant of the well-known multi-hypothesis sequential probability ratio test (MSPRT) for decision making whose evidence observations consist of the basic unit of neural signalling - the inter-spike interval (ISI) - and which is based on a new form of the likelihood function. We dub this mechanism s-MSPRT and show its precise form for a range of realistic ISI distributions with positive support. In this way we show that, at the level of spikes, the refractory period may actually facilitate shorter decision times, and that the mechanism is robust against poor choice of the hypothesized data distribution. We show that s-MSPRT performance is related to the Kullback-Leibler divergence (KLD) or information gain between ISI distributions, through which we are able to link neural signalling to psychophysical observation at the behavioural level. Thus, we find the mean information needed for a decision is constant, thereby offering an account of Hick's law (relating decision time to the number of choices). Further, the mean decision time of s-MSPRT shows a power law dependence on the KLD offering an account of Piéron's law (relating reaction time to stimulus intensity). These results show the foundations for a research programme in which spike train analysis can be made the basis for predictions about behavior in multi-alternative choice tasks.

A new framework for cortico-striatal plasticity: behavioural theory meets in vitro data at the reinforcement-action interface.

Operant learning requires that reinforcement signals interact with action representations at a suitable neural interface. Much evidence suggests that this occurs when phasic dopamine, acting as a reinforcement prediction error, gates plasticity at cortico-striatal synapses, and thereby changes the future likelihood of selecting the action(s) coded by striatal neurons. But this hypothesis faces serious challenges. First, cortico-striatal plasticity is inexplicably complex, depending on spike timing, dopamine level, and dopamine receptor type. Second, there is a credit assignment problem-action selection signals occur long before the consequent dopamine reinforcement signal. Third, the two types of striatal output neuron have apparently opposite effects on action selection. Whether these factors rule out the interface hypothesis and how they interact to produce reinforcement learning is unknown. We present a computational framework that addresses these challenges. We first predict the expected activity changes over an operant task for both types of action-coding striatal neuron, and show they co-operate to promote action selection in learning and compete to promote action suppression in extinction. Separately, we derive a complete model of dopamine and spike-timing dependent cortico-striatal plasticity from in vitro data. We then show this model produces the predicted activity changes necessary for learning and extinction in an operant task, a remarkable convergence of a bottom-up data-driven plasticity model with the top-down behavioural requirements of learning theory. Moreover, we show the complex dependencies of cortico-striatal plasticity are not only sufficient but necessary for learning and extinction. Validating the model, we show it can account for behavioural data describing extinction, renewal, and reacquisition, and replicate in vitro experimental data on cortico-striatal plasticity. By bridging the levels between the single synapse and behaviour, our model shows how striatum acts as the action-reinforcement interface.

Finding minimal action sequences with a simple evaluation of actions.

Animals are able to discover the minimal number of actions that achieves an outcome (the minimal action sequence). In most accounts of this, actions are associated with a measure of behavior that is higher for actions that lead to the outcome with a shorter action sequence, and learning mechanisms find the actions associated with the highest measure. In this sense, previous accounts focus on more than the simple binary signal of "was the outcome achieved?"; they focus on "how well was the outcome achieved?" However, such mechanisms may not govern all types of behavioral development. In particular, in the process of action discovery (Redgrave and Gurney, 2006), actions are reinforced if they simply lead to a salient outcome because biological reinforcement signals occur too quickly to evaluate the consequences of an action beyond an indication of the outcome's occurrence. Thus, action discovery mechanisms focus on the simple evaluation of "was the outcome achieved?" and not "how well was the outcome achieved?" Notwithstanding this impoverishment of information, can the process of action discovery find the minimal action sequence? We address this question by implementing computational mechanisms, referred to in this paper as no-cost learning rules, in which each action that leads to the outcome is associated with the same measure of behavior. No-cost rules focus on "was the outcome achieved?" and are consistent with action discovery. No-cost rules discover the minimal action sequence in simulated tasks and execute it for a substantial amount of time. Extensive training, however, results in extraneous actions, suggesting that a separate process (which has been proposed in action discovery) must attenuate learning if no-cost rules participate in behavioral development. We describe how no-cost rules develop behavior, what happens when attenuation is disrupted, and relate the new mechanisms to wider computational and biological context.

Emergent structured transition from variation to repetition in a biologically-plausible model of learning in basal ganglia.

Often, when animals encounter an unexpected sensory event, they transition from executing a variety of movements to repeating the movement(s) that may have caused the event. According to a recent theory of action discovery (Redgrave and Gurney, 2006), repetition allows the animal to represent those movements, and the outcome, as an action for later recruitment. The transition from variation to repetition often follows a non-random, structured, pattern. While the structure of the pattern can be explained by sophisticated cognitive mechanisms, simpler mechanisms based on dopaminergic modulation of basal ganglia (BG) activity are thought to underlie action discovery (Redgrave and Gurney, 2006). In this paper we ask the question: can simple BG-mediated mechanisms account for a structured transition from variation to repetition, or are more sophisticated cognitive mechanisms always necessary? To address this question, we present a computational model of BG-mediated biasing of behavior. In our model, unlike most other models of BG function, the BG biases behavior through modulation of cortical response to excitation; many possible movements are represented by the cortical area; and excitation to the cortical area is topographically-organized. We subject the model to simple reaching tasks, inspired by behavioral studies, in which a location to which to reach must be selected. Locations within a target area elicit a reinforcement signal. A structured transition from variation to repetition emerges from simple BG-mediated biasing of cortical response to excitation. We show how the structured pattern influences behavior in simple and complicated tasks. We also present analyses that describe the structured transition from variation to repetition due to BG-mediated biasing and from biasing that would be expected from a type of cognitive biasing, allowing us to compare behavior resulting from these types of biasing and make connections with future behavioral experiments.

The discovery of novel actions is affected by very brief reinforcement delays and reinforcement modality.

ABSTRACT The authors investigated the ability of human participants to discover novel actions under conditions of delayed reinforcement. Participants used a joystick to search for a target indicated by visual or auditory reinforcement. Reinforcement delays of 75-150 ms were found to significantly impair action acquisition. They also found an effect of modality, with acquisition superior with auditory feedback. The duration at which delay was found to impede action discovery is, to the authors' knowledge, shorter than that previously reported from work with operant and causal learning paradigms. The sensitivity to delay reported, and the difference between modalities, is consistent with accounts of action discovery that emphasize the importance of a time stamp in the motor record for solving the credit assignment problem.

The basal ganglia optimize decision making over general perceptual hypotheses.

The basal ganglia are a subcortical group of interconnected nuclei involved in mediating action selection within cortex. A recent proposal is that this selection leads to optimal decision making over multiple alternatives because the basal ganglia anatomy maps onto a network implementation of an optimal statistical method for hypothesis testing, assuming that cortical activity encodes evidence for constrained gaussian-distributed alternatives. This letter demonstrates that this model of the basal ganglia extends naturally to encompass general Bayesian sequential analysis over arbitrary probability distributions, which raises the proposal to a practically realizable theory over generic perceptual hypotheses. We also show that the evidence in this model can represent either log likelihoods, log-likelihood ratios, or log odds, all leading proposals for the cortical processing of sensory data. For these reasons, we claim that the basal ganglia optimize decision making over general perceptual hypotheses represented in cortex. The relation of this theory to cortical encoding, cortico-basal ganglia anatomy, and reinforcement learning is discussed.

Additive factors do not imply discrete processing stages: a worked example using models of the stroop task.

Previously, it has been shown experimentally that the psychophysical law known as Piéron's Law holds for color intensity and that the size of the effect is additive with that of Stroop condition (Stafford et al., 2011). According to the additive factors method (Donders, 1868-1869/1969; Sternberg, 1998), additivity is assumed to indicate independent and discrete processing stages. We present computational modeling work, using an existing Parallel Distributed Processing model of the Stroop task (Cohen et al., 1990) and a standard model of decision making (Ratcliff, 1978). This demonstrates that additive factors can be successfully accounted for by existing single stage models of the Stroop effect. Consequently, it is not valid to infer either discrete stages or separate loci of effects from additive factors. Further, our modeling work suggests that information binding may be a more important architectural property for producing additive factors than discrete stages.

Piéron's Law holds during stroop conflict: insights into the architecture of decision making.

Piéron's Law describes the relationship between stimulus intensity and reaction time. Previously (Stafford & Gurney, 2004), we have shown that Piéron's Law is a necessary consequence of rise-to-threshold decision making and thus will arise from optimal simple decision-making algorithms (e.g., Bogacz, Brown, Moehlis, Holmes, & Cohen, 2006). Here, we manipulate the color saturation of a Stroop stimulus. Our results show that Piéron's Law holds for color intensity and color-naming reaction time, extending the domain of this law, in line with our suggestion of the generality of the processes that can give rise to Piéron's Law. In addition, we find that Stroop condition does not interact with the effect of color saturation; Stroop interference and facilitation remain constant at all levels of color saturation. An analysis demonstrates that this result cannot be accounted for by single-stage decision-making algorithms which combine all the evidence pertaining to a decision into a common metric. This shows that human decision making is not information-optimal and suggests that the generalization of current models of simple perceptual decision making to more complex decisions is not straightforward.

Spiking neural network simulation: memory-optimal synaptic event scheduling.

Spiking neural network simulations incorporating variable transmission delays require synaptic events to be scheduled prior to delivery. Conventional methods have memory requirements that scale with the total number of synapses in a network. We introduce novel scheduling algorithms for both discrete and continuous event delivery, where the memory requirement scales instead with the number of neurons. Superior algorithmic performance is demonstrated using large-scale, benchmarking network simulations.

Biologically constrained action selection improves cognitive control in a model of the Stroop task.

The Stroop task is a paradigmatic psychological task for investigating stimulus conflict and the effect this has on response selection. The model of Cohen et al. (Cohen et al. 1990 Psychol. Rev. 97, 332-361) has hitherto provided the best account of performance in the Stroop task, but there remains certain key data that it fails to match. We show that this failure is due to the mechanism used to perform final response selection-one based on the diffusion model of choice behaviour (Ratcliff 1978 Psychol. Rev. 85, 59-108). We adapt the model to use a selection mechanism which is based on the putative human locus of final response selection, the basal ganglia/thalamo-cortical complex (Redgrave et al. 1999 Neuroscience 89, 1009-1023). This improves the match to the core human data and, additionally, makes it possible for the model to accommodate, in a principled way, additional mechanisms of cognitive control that enable better fits to the data. This work prompts a critique of the diffusion model as a mechanism of response selection, and the features that any response mechanism must possess to provide adaptive action selection. We conclude that the consideration of biologically constrained solutions to the action selection problem is vital to the understanding and improvement of cognitive models of response selection.

A physiologically plausible model of action selection and oscillatory activity in the basal ganglia.

The basal ganglia (BG) have long been implicated in both motor function and dysfunction. It has been proposed that the BG form a centralized action selection circuit, resolving conflict between multiple neural systems competing for access to the final common motor pathway. We present a new spiking neuron model of the BG circuitry to test this proposal, incorporating all major features and many physiologically plausible details. We include the following: effects of dopamine in the subthalamic nucleus (STN) and globus pallidus (GP), transmission delays between neurons, and specific distributions of synaptic inputs over dendrites. All main parameters were derived from experimental studies. We find that the BG circuitry supports motor program selection and switching, which deteriorates under dopamine-depleted and dopamine-excessive conditions in a manner consistent with some pathologies associated with those dopamine states. We also validated the model against data describing oscillatory properties of BG. We find that the same model displayed detailed features of both gamma-band (30-80 Hz) and slow (approximately 1 Hz) oscillatory phenomena reported by Brown et al. (2002) and Magill et al. (2001), respectively. Only the parameters required to mimic experimental conditions (e.g., anesthetic) or manipulations (e.g., lesions) were changed. From the results, we derive the following novel predictions about the STN-GP feedback loop: (1) the loop is functionally decoupled by tonic dopamine under normal conditions and recoupled by dopamine depletion; (2) the loop does not show pacemaking activity under normal conditions in vivo (but does after combined dopamine depletion and cortical lesion); (3) the loop has a resonant frequency in the gamma-band.

Empirically inspired simulated electro-mechanical model of the rat mystacial follicle-sinus complex.

In whiskered animals, activity is evoked in the primary sensory afferent cells (trigeminal nerve) by mechanical stimulation of the whiskers. In some cell populations this activity is correlated well with continuous stimulus parameters such as whisker deflection magnitude, but in others it is observed to represent events such as whisker-stimulator contact or detachment. The transduction process is mediated by the mechanics of the whisker shaft and follicle-sinus complex (FSC), and the mechanics and electro-chemistry of mechanoreceptors within the FSC. An understanding of this transduction process and the nature of the primary neural codes generated is crucial for understanding more central sensory processing in the thalamus and cortex. However, the details of the peripheral processing are currently poorly understood. To overcome this deficiency in our knowledge, we constructed a simulated electro-mechanical model of the whisker-FSC-mechanoreceptor system in the rat and tested it against a variety of data drawn from the literature. The agreement was good enough to suggest that the model captures many of the key features of the peripheral whisker system in the rat.

The role of response mechanisms in determining reaction time performance: Piéron's law revisited.

A response mechanism takes evaluations of the importance of potential actions and selects the most suitable. Response mechanism function is a nontrivial problem that has not received the attention it deserves within cognitive psychology. In this article, we make a case for the importance of considering response mechanism function as a constraint on cognitive processes and emphasized links with the wider problem of behavioral action selection. First, we show that, contrary to previous suggestions, a well-known model of the Stroop task (Cohen, Dunbar, & McClelland, 1990) relies on the response mechanism for a key feature of its results-the interference-facilitation asymmetry. Second, we examine a variety of response mechanisms (including that in the model of Cohen et al., 1990) and show that they all follow a law analogous to Piéron's law in relating their input to reaction time. In particular, this is true of a decision mechanism not designed to explain RT data but based on a proposed solution to the general problem of action selection and grounded in the neurobiology of the vertebrate basal ganglia Finally, we show that the dynamics of simple artificial neurons also support a Piéron-like law.

A basal ganglia inspired model of action selection evaluated in a robotic survival task.

The basal ganglia system has been proposed as a possible neural substrate for action selection in the vertebrate brain. We describe a robotic implementation of a model of the basal ganglia and demonstrate the capacity of this system to generate adaptive switching between several acts when embedded in a robot that has to "survive" in a laboratory environment. A comparison between this brain-inspired selection mechanism and classical "winner-takes-all" selection highlights some adaptive properties specific to the model, such as avoidance of dithering and energy-saving. These properties derive, in part, from the capacity of simulated basal ganglia-thalamo-cortical loops to generate appropriate "behavioral persistence".