Selective particle attention: Rapidly and flexibly selecting features for deep reinforcement learning.

Deep Reinforcement Learning (RL) is often criticised for being data inefficient and inflexible to changes in task structure. Part of the reason for these issues is that Deep RL typically learns end-to-end using backpropagation, which results in task-specific representations. One approach for circumventing these problems is to apply Deep RL to existing representations that have been learned in a more task-agnostic fashion. However, this only partially solves the problem as the Deep RL algorithm learns a function of all pre-existing representations and is therefore still susceptible to data inefficiency and a lack of flexibility. Biological agents appear to solve this problem by forming internal representations over many tasks and only selecting a subset of these features for decision-making based on the task at hand; a process commonly referred to as selective attention. We take inspiration from selective attention in biological agents and propose a novel algorithm called Selective Particle Attention (SPA), which selects subsets of existing representations for Deep RL. Crucially, these subsets are not learned through backpropagation, which is slow and prone to overfitting, but instead via a particle filter that rapidly and flexibly identifies key subsets of features using only reward feedback. We evaluate SPA on two tasks that involve raw pixel input and dynamic changes to the task structure, and show that it greatly increases the efficiency and flexibility of downstream Deep RL algorithms.

A complementary learning systems approach to temporal difference learning.

Complementary Learning Systems (CLS) theory suggests that the brain uses a 'neocortical' and a 'hippocampal' learning system to achieve complex behaviour. These two systems are complementary in that the 'neocortical' system relies on slow learning of distributed representations while the 'hippocampal' system relies on fast learning of pattern-separated representations. Both of these systems project to the striatum, which is a key neural structure in the brain's implementation of Reinforcement Learning (RL). Current deep RL approaches share similarities with a 'neocortical' system because they slowly learn distributed representations through backpropagation in Deep Neural Networks (DNNs). An ongoing criticism of such approaches is that they are data inefficient and lack flexibility. CLS theory suggests that the addition of a 'hippocampal' system could address these criticisms. In the present study we propose a novel algorithm known as Complementary Temporal Difference Learning (CTDL), which combines a DNN with a Self-Organizing Map (SOM) to obtain the benefits of both a 'neocortical' and a 'hippocampal' system. Key features of CTDL include the use of Temporal Difference (TD) error to update a SOM and the combination of a SOM and DNN to calculate action values. We evaluate CTDL on Grid World, Cart-Pole and Continuous Mountain Car tasks and show several benefits over the classic Deep Q-Network (DQN) approach. These results demonstrate (1) the utility of complementary learning systems for the evaluation of actions, (2) that the TD error signal is a useful form of communication between the two systems and (3) that our approach extends to both discrete and continuous state and action spaces.

Hippocampal α7 nicotinic ACh receptors contribute to modulation of depression-like behaviour in C57BL/6J mice.

BACKGROUND AND PURPOSE: Clinical studies have identified links between cholinergic signalling and depression in human subjects. Increased cholinergic signalling in hippocampus also increases behaviours related to anxiety and depression in mice, which can be reversed by ACh receptor antagonists. EXPERIMENTAL APPROACH: As the α7 subunit of the nicotinic ACh receptor (nAChR) is highly expressed in hippocampus, we determined whether blocking α7 nAChRs could reverse the effects of increased ACh signalling in anxiety- and depression-related behaviours in mice. KEY RESULTS: Administration of the α7 nAChR agonist GTS-21 had no effect in tail suspension or forced swim tests. Conversely, the α7 nAChR antagonist methyllycaconitine (MLA) induced significant antidepressant-like effects in male mice in these paradigms, consistent with previous studies, but this was not observed in female mice. MLA also decreased physostigmine-induced c-fos immunoreactivity (a marker of neuronal activity) in hippocampus. Local knockdown of α7 nAChRs in hippocampus had no effect on its own but decreased a subset of depression-like phenotypes induced by physostigmine in male mice. Few effects of α7 nAChR knockdown were observed in depression-like behaviors in female mice, possibly due to a limited response to physostigmine. There was no significant effect of hippocampal α7 nAChR knockdown on anxiety-like phenotypes in male mice. However, a modest increase in anxiety-like behavior was observed in female mice infused with a scrambled control vector in response to physostigmine administration, that was not seen after a7 nAChR knockdown in the hippocampus. CONCLUSIONS AND IMPLICATIONS: These results suggest that ACh signalling through α7 nAChRs in the hippocampus contributes to regulation of a subset of depression-like behaviours when ACh is increased, as can occur under stressful conditions. These studies also provide evidence for sex differences that may be relevant for treatments of mood disorders based on cholinergic signalling. LINKED ARTICLES: This article is part of a themed section on Nicotinic Acetylcholine Receptors. To view the other articles in this section visit http://onlinelibrary.wiley.com/doi/10.1111/bph.v175.11/issuetoc.

Multiple Nicotinic Acetylcholine Receptor Subtypes in the Mouse Amygdala Regulate Affective Behaviors and Response to Social Stress.

Electrophysiological and neurochemical studies implicate cholinergic signaling in the basolateral amygdala (BLA) in behaviors related to stress. Both animal studies and human clinical trials suggest that drugs that alter nicotinic acetylcholine receptor (nAChR) activity can affect behaviors related to mood and anxiety. Clinical studies also suggest that abnormalities in cholinergic signaling are associated with major depressive disorder, whereas pre-clinical studies have implicated both ß2 subunit-containing (ß2*) and α7 nAChRs in the effects of nicotine in models of anxiety- and depression-like behaviors. We therefore investigated whether nAChR signaling in the amygdala contributes to stress-mediated behaviors in mice. Local infusion of the non-competitive non-selective nAChR antagonist mecamylamine or viral-mediated downregulation of the ß2 or α7 nAChR subunit in the amygdala all induced robust anxiolytic- and antidepressant-like effects in several mouse behavioral models. Further, whereas α7 nAChR subunit knockdown was somewhat more effective at decreasing anxiety-like behavior, only ß2 subunit knockdown decreased resilience to social defeat stress and c-fos immunoreactivity in the BLA. In contrast, α7, but not ß2, subunit knockdown effectively reversed the effect of increased ACh signaling in a mouse model of depression. These results suggest that signaling through ß2* nAChRs is essential for baseline excitability of the BLA, and a decrease in signaling through ß2 nAChRs alters anxiety- and depression-like behaviors even in unstressed animals. In contrast, stimulation of α7 nAChRs by acetylcholine may mediate the increased depression-like behaviors observed during the hypercholinergic state observed in depressed individuals.

Expression of the 5-HT1A serotonin receptor in the hippocampus is required for social stress resilience and the antidepressant-like effects induced by the nicotinic partial agonist cytisine.

Nicotinic acetylcholine receptor (nAChR) blockers potentiate the effects of selective serotonin reuptake inhibitors (SSRIs) in some treatment-resistant patients; however, it is not known whether these effects are independent, or whether the two neurotransmitter systems act synergistically. We first determined that the SSRI fluoxetine and the nicotinic partial agonist cytisine have synergistic effects in a mouse model of antidepressant efficacy, whereas serotonin depletion blocked the effects of cytisine. Using a pharmacological approach, we found that the 5-HT1A agonist 8-OH-DPAT also potentiated the antidepressant-like effects of cytisine, suggesting that this subtype might mediate the interaction between the serotonergic and cholinergic systems. The 5-HT1A receptors are located both presynaptically and postsynaptically. We therefore knocked down 5-HT1A receptors in either the dorsal raphe (presynaptic autoreceptors) or the hippocampus (a brain area with high expression of 5-HT1A heteroreceptors sensitive to cholinergic effects on affective behaviors). Knockdown of 5-HT1A receptors in hippocampus, but not dorsal raphe, significantly decreased the antidepressant-like effect of cytisine. This study suggests that serotonin signaling through postsynaptic 5-HT1A receptors in the hippocampus is critical for the antidepressant-like effects of a cholinergic drug and begins to elucidate the molecular mechanisms underlying interactions between the serotonergic and cholinergic systems related to mood disorders.