Changes in corticostriatal connectivity during reinforcement learning in humans.

Many computational models assume that reinforcement learning relies on changes in synaptic efficacy between cortical regions representing stimuli and striatal regions involved in response selection, but this assumption has thus far lacked empirical support in humans. We recorded hemodynamic signals with fMRI while participants navigated a virtual maze to find hidden rewards. We fitted a reinforcement-learning algorithm to participants' choice behavior and evaluated the neural activity and the changes in functional connectivity related to trial-by-trial learning variables. Activity in the posterior putamen during choice periods increased progressively during learning. Furthermore, the functional connections between the sensorimotor cortex and the posterior putamen strengthened progressively as participants learned the task. These changes in corticostriatal connectivity differentiated participants who learned the task from those who did not. These findings provide a direct link between changes in corticostriatal connectivity and learning, thereby supporting a central assumption common to several computational models of reinforcement learning.

Reduced functional connectivity within the limbic cortico-striato-thalamo-cortical loop in unmedicated adults with obsessive-compulsive disorder.

Cortico-striato-thalamo-cortical (CSTC) loops project from the cortex to the striatum, then from the striatum to the thalamus via the globus pallidus, and finally from the thalamus back to the cortex again. These loops have been implicated in Obsessive-Compulsive Disorder (OCD) with particular focus on the limbic CSTC loop, which encompasses the orbitofrontal and anterior cingulate cortices, as well as the ventral striatum. Resting state functional-connectivity MRI (rs-fcMRI) studies, which examine temporal correlations in neural activity across brain regions at rest, have examined CSTC loop connectivity in patients with OCD and suggest hyperconnectivity within these loops in medicated adults with OCD. We used rs-fcMRI to examine functional connectivity within CSTC loops in unmedicated adults with OCD (n = 23) versus healthy controls (HCs) (n = 20). Contrary to prior rs-fcMRI studies in OCD patients on medications that report hyperconnectivity in the limbic CSTC loop, we found that compared with HCs, unmedicated OCD participants had reduced connectivity within the limbic CSTC loop. Exploratory analyses revealed that reduced connectivity within the limbic CSTC loop correlated with OCD symptom severity in the OCD group. Our finding of limbic loop hypoconnectivity in unmedicted OCD patients highlights the potential confounding effects of antidepressants on connectivity measures and the value of future examinations of the effects of pharmacological and/or behavioral treatments on limbic CSTC loop connectivity.

Adaptation to conflict via context-driven anticipatory signals in the dorsomedial prefrontal cortex.

Behavioral interference elicited by competing response tendencies adapts to contextual changes. Recent nonhuman primate research suggests a key mnemonic role of distinct prefrontal cells in supporting such context-driven behavioral adjustments by maintaining conflict information across trials, but corresponding prefrontal functions have yet to be probed in humans. Using event-related functional magnetic resonance imaging, we investigated the human neural substrates of contextual adaptations to conflict. We found that a neural system comprising the rostral dorsomedial prefrontal cortex and portions of the dorsolateral prefrontal cortex specifically encodes the history of previously experienced conflict and influences subsequent adaptation to conflict on a trial-by-trial basis. This neural system became active in anticipation of stimulus onsets during preparatory periods and interacted with a second neural system engaged during the processing of conflict. Our findings suggest that a dynamic interaction between a system that represents conflict history and a system that resolves conflict underlies the contextual adaptation to conflict.

The attenuation of dysfunctional emotional processing with stimulant medication: an fMRI study of adolescents with ADHD.

Functional neuroimaging studies of attention-deficit/hyperactivity disorder (ADHD) have focused on the neural correlates of cognitive control. However, for many youths with ADHD, emotional lability is an important clinical feature of the disorder. We aimed to identify the neural substrates associated with emotional lability that were distinct from impairments in cognitive control and to assess the effects that stimulants have on those substrates. We used functional magnetic resonance imaging (fMRI) to assess neural activity in adolescents with (N=15) and without (N=15) ADHD while they performed cognitive and emotional versions of the Stroop task that engage cognitive control and emotional processing, respectively. The participants with ADHD were scanned both on and off stimulant medication in a counterbalanced fashion. Controlling for differences in cognitive control, we found that during the emotional Stroop task, adolescents with ADHD as compared with controls demonstrated atypical activity in the medial prefrontal cortex (mPFC). Stimulants attenuated activity in the mPFC to levels comparable with controls.

Mathematical Characterization of Changes in Fear During Exposure Therapy.

BACKGROUND: During exposure therapy, patients report increases in fear that generally decrease within and across exposure sessions. Our main aim was to characterize these changes in fear ratings mathematically; a secondary aim was to test whether the resulting model would help to predict treatment outcome. METHODS: We applied tools of computational psychiatry to a previously published dataset in which 30 women with spider phobia were randomly assigned to virtual-reality exposures in a single context or in multiple contexts (n = 15 each). Patients provided fear ratings every minute during exposures. We characterized fear decrease within exposures and return of fear between exposures using a set of mathematical models; we selected the best model using Bayesian techniques. In the multiple-contexts group, we tested the predictions of the best model in a separate, test exposure, and we investigated the ability of model parameters to predict treatment outcome. RESULTS: The best model characterized fear decrease within exposures in both groups as an exponential decay with constant decay rate across exposures. The best model for each group had only two parameters but captured with remarkable accuracy the patterns of fear change, both at the group level and for individual subjects. The best model also made remarkably accurate predictions for the test exposure. One of the model's parameters helped predict treatment outcome. CONCLUSIONS: Individual patterns of fear change during exposure therapy can be characterized mathematically. This mathematical characterization helps predict treatment outcome.

People exert more effort to avoid losses than to obtain gains.

Loss aversion is a fundamental tenet of behavioral economics and has led to many real-world applications. These applications, and some laboratory studies, show that people perform better under loss-avoidance than under gain incentives. This increased performance under loss-avoidance incentives has ubiquitously been explained by the notion that loss aversion causes people to exert more effort to avoid losses than to obtain gains. Only limited work, however, has directly examined whether people indeed choose to exert more effort to avoid losses than to obtain gains. Our primary aim was therefore to test this proposition. In an experiment with adults (N = 32) and in a subsequent experiment with children and adolescents (N = 29), we found that participants indeed exerted more effort to avoid losses than to obtain numerically equivalent gains. The effect sizes were large, with the effect being evident for most individual participants. As a secondary aim, in the study with adults, we also investigated whether the greater effort to avoid losses related to loss aversion measured using a task involving choices between prospects. Unexpectedly, the greater effort to avoid losses persisted robustly even after controlling for the effects of loss aversion measured using the task involving choices between prospects. We discuss two possible interpretations for this finding: our effort task may have been a more sensitive assessment of loss aversion than the task involving choices between prospects; alternatively, the processes underlying how much effort people choose to exert may partially differ from those engaged by choices between prospects. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Reinforcement learning, conditioning, and the brain: Successes and challenges.

The field of reinforcement learning has greatly influenced the neuroscientific study of conditioning. This article provides an introduction to reinforcement learning followed by an examination of the successes and challenges using reinforcement learning to understand the neural bases of conditioning. Successes reviewed include (1) the mapping of positive and negative prediction errors to the firing of dopamine neurons and neurons in the lateral habenula, respectively; (2) the mapping of model-based and model-free reinforcement learning to associative and sensorimotor cortico-basal ganglia-thalamo-cortical circuits, respectively; and (3) the mapping of actor and critic to the dorsal and ventral striatum, respectively. Challenges reviewed consist of several behavioral and neural findings that are at odds with standard reinforcement-learning models, including, among others, evidence for hyperbolic discounting and adaptive coding. The article suggests ways of reconciling reinforcement-learning models with many of the challenging findings, and highlights the need for further theoretical developments where necessary. Additional information related to this study may be downloaded from http://cabn.psychonomic-journals.org/content/supplemental.

The neural bases of obsessive-compulsive disorder in children and adults.

Functional imaging studies have reported with remarkable consistency hyperactivity in the orbitofrontal cortex (OFC), anterior cingulate cortex (ACC), and caudate nucleus of patients with obsessive-compulsive disorder (OCD). These findings have often been interpreted as evidence that abnormalities in cortico-basal ganglia-thalamo-cortical loops involving the OFC and ACC are causally related to OCD. This interpretation remains controversial, however, because such hyperactivity may represent either a cause or a consequence of the symptoms. This article analyzes the evidence for a causal role of these loops in producing OCD in children and adults. The article first reviews the strong evidence for anatomical abnormalities in these loops in patients with OCD. These findings are not sufficient to establish causality, however, because anatomical alterations may themselves be a consequence rather than a cause of the symptoms. The article then reviews three lines of evidence that, despite their own limitations, permit stronger causal inferences: the development of OCD following brain injury, pediatric autoimmune neuropsychiatric disorders associated with streptococcal infection, and neurosurgical lesions that attenuate OCD. Converging evidence from these various lines of research supports a causal role for the cortico-basal ganglia-thalamo-cortical loops that involve the OFC and ACC in the pathogenesis of OCD in children and adults.

Dopaminergic Disturbances in Tourette Syndrome: An Integrative Account.

Tourette syndrome (TS) is thought to involve dopaminergic disturbances, but the nature of those disturbances remains controversial. Existing hypotheses suggest that TS involves 1) supersensitive dopamine receptors, 2) overactive dopamine transporters that cause low tonic but high phasic dopamine, 3) presynaptic dysfunction in dopamine neurons, or 4) dopaminergic hyperinnervation. We review evidence that contradicts the first two hypotheses; we also note that the last two hypotheses have traditionally been considered too narrowly, explaining only small subsets of findings. We review all studies that have used positron emission tomography and single-photon emission computerized tomography to investigate the dopaminergic system in TS. The seemingly diverse findings from those studies have typically been interpreted as pointing to distinct mechanisms, as evidenced by the various hypotheses concerning the nature of dopaminergic disturbances in TS. We show, however, that the hyperinnervation hypothesis provides a simple, parsimonious explanation for all such seemingly diverse findings. Dopaminergic hyperinnervation likely causes increased tonic and phasic dopamine. We have previously shown, using a computational model of the role of dopamine in basal ganglia, that increased tonic dopamine and increased phasic dopamine likely increase the propensities to express and learn tics, respectively. There is therefore a plausible mechanistic link between dopaminergic hyperinnervation and TS via increased tonic and phasic dopamine. To further bolster this argument, we review evidence showing that all medications that are effective for TS reduce signaling by tonic dopamine, phasic dopamine, or both.

The Roles of Phasic and Tonic Dopamine in Tic Learning and Expression.

Tourette syndrome (TS) prominently involves dopaminergic disturbances, but the precise nature of those disturbances has remained elusive. A substantial body of empirical work and recent computational models have characterized the specific roles of phasic and tonic dopamine (DA) in action learning and selection, respectively. Using insights from this work and models, we suggest that TS involves increases in both phasic and tonic DA, which produce increased propensities for tic learning and expression, respectively. We review the evidence from reinforcement-learning and habit-learning studies in TS, which supports the idea that TS involves increased phasic DA responses; we also review the evidence that tics engage the habit-learning circuitry. On the basis of these findings, we suggest that tics are exaggerated, maladaptive, and persistent motor habits reinforced by aberrant, increased phasic DA responses. Increased tonic DA amplifies the tendency to execute learned tics and also provides a fertile ground of motor hyperactivity for tic learning. We review evidence suggesting that antipsychotics may counter both the increased propensity for tic expression, by increasing excitability in the indirect pathway, and the increased propensity for tic learning, by shifting plasticity in the indirect pathway toward long-term potentiation (and possibly also through more complex mechanisms). Finally, we review evidence suggesting that low doses of DA agonists that effectively treat TS decrease both phasic and tonic DA, thereby also reducing the propensity for both tic learning and tic expression, respectively.

An Integrative Perspective on the Role of Dopamine in Schizophrenia.

We propose that schizophrenia involves a combination of decreased phasic dopamine responses for relevant stimuli and increased spontaneous phasic dopamine release. Using insights from computational reinforcement-learning models and basic-science studies of the dopamine system, we show that each of these two disturbances contributes to a specific symptom domain and explains a large set of experimental findings associated with that domain. Reduced phasic responses for relevant stimuli help to explain negative symptoms and provide a unified explanation for the following experimental findings in schizophrenia, most of which have been shown to correlate with negative symptoms: reduced learning from rewards; blunted activation of the ventral striatum, midbrain, and other limbic regions for rewards and positive prediction errors; blunted activation of the ventral striatum during reward anticipation; blunted autonomic responding for relevant stimuli; blunted neural activation for aversive outcomes and aversive prediction errors; reduced willingness to expend effort for rewards; and psychomotor slowing. Increased spontaneous phasic dopamine release helps to explain positive symptoms and provides a unified explanation for the following experimental findings in schizophrenia, most of which have been shown to correlate with positive symptoms: aberrant learning for neutral cues (assessed with behavioral and autonomic responses), and aberrant, increased activation of the ventral striatum, midbrain, and other limbic regions for neutral cues, neutral outcomes, and neutral prediction errors. Taken together, then, these two disturbances explain many findings in schizophrenia. We review evidence supporting their co-occurrence and consider their differential implications for the treatment of positive and negative symptoms.

Premonitory urges and tics in Tourette syndrome: computational mechanisms and neural correlates.

Tourette syndrome is characterized by open motor behaviors - tics - but another crucial aspect of the disorder is the presence of premonitory urges: uncomfortable sensations that typically precede tics and are temporarily alleviated by tics. We review the evidence implicating the somatosensory cortices and the insula in premonitory urges and the motor cortico-basal ganglia-thalamo-cortical loop in tics. We consider how these regions interact during tic execution, suggesting that the insula plays an important role as a nexus linking the sensory and emotional character of premonitory urges with their translation into tics. We also consider how these regions interact during tic learning, integrating the neural evidence with a computational perspective on how premonitory-urge alleviation reinforces tics.

Consciousness: converging insights from connectionist modeling and neuroscience.

Over the past decade, many findings in cognitive neuroscience have resulted in the view that selective attention, working memory and cognitive control involve competition between widely distributed representations. This competition is biased by top-down projections (notably from prefrontal cortex), which can selectively enhance some representations over others. This view has now been implemented in several connectionist models. In this review, we emphasize the relevance of these models to understanding consciousness. Interestingly, the models we review have striking similarities to others directly aimed at implementing 'global workspace theory'. All of these models embody a fundamental principle that has been used in many connectionist models over the past twenty years: global constraint satisfaction.

A Roadmap for the Development of Applied Computational Psychiatry.

BACKGROUND: Computational psychiatry is a burgeoning field that utilizes mathematical approaches to investigate psychiatric disorders, derive quantitative predictions, and integrate data across multiple levels of description. Computational psychiatry has already led to many new insights into the neurobehavioral mechanisms that underlie several psychiatric disorders, but its usefulness from a clinical standpoint is only now starting to be considered. METHODS: Examples of computational psychiatry are highlighted, and a phase-based pipeline for the development of clinical computational-psychiatry applications is proposed, similar to the phase-based pipeline used in drug development. It is proposed that each phase has unique endpoints and deliverables, which will be important milestones to move tasks, procedures, computational models, and algorithms from the laboratory to clinical practice. RESULTS: Application of computational approaches should be tested on healthy volunteers in Phase I, transitioned to target populations in Phase IB and Phase IIA, and thoroughly evaluated using randomized clinical trials in Phase IIB and Phase III. Successful completion of these phases should be the basis of determining whether computational models are useful tools for prognosis, diagnosis, or treatment of psychiatric patients. CONCLUSIONS: A new type of infrastructure will be necessary to implement the proposed pipeline. This infrastructure should consist of groups of investigators with diverse backgrounds collaborating to make computational psychiatry relevant for the clinic.

Computational psychiatry as a bridge from neuroscience to clinical applications.

Translating advances in neuroscience into benefits for patients with mental illness presents enormous challenges because it involves both the most complex organ, the brain, and its interaction with a similarly complex environment. Dealing with such complexities demands powerful techniques. Computational psychiatry combines multiple levels and types of computation with multiple types of data in an effort to improve understanding, prediction and treatment of mental illness. Computational psychiatry, broadly defined, encompasses two complementary approaches: data driven and theory driven. Data-driven approaches apply machine-learning methods to high-dimensional data to improve classification of disease, predict treatment outcomes or improve treatment selection. These approaches are generally agnostic as to the underlying mechanisms. Theory-driven approaches, in contrast, use models that instantiate prior knowledge of, or explicit hypotheses about, such mechanisms, possibly at multiple levels of analysis and abstraction. We review recent advances in both approaches, with an emphasis on clinical applications, and highlight the utility of combining them.

Evidence that conditioned avoidance responses are reinforced by positive prediction errors signaled by tonic striatal dopamine.

We conducted an experiment in which hedonia, salience and prediction error hypotheses predicted different patterns of dopamine (DA) release in the striatum during learning of conditioned avoidance responses (CARs). The data strongly favor the latter hypothesis. It predicts that during learning of the 2-way active avoidance CAR task, positive prediction errors generated when rats do not receive an anticipated footshock (which is better than expected) cause DA release that reinforces the instrumental avoidance action. In vivo microdialysis in the rat striatum showed that extracellular DA concentration increased during early CAR learning and decreased throughout training returning to baseline once the response was well learned. In addition, avoidance learning was proportional to the degree of DA release. Critically, exposure of rats to the same stimuli but in an unpredictable, unavoidable, and inescapable manner, did not produce alterations from baseline DA levels as predicted by the prediction error but not hedonic or salience hypotheses. In addition, rats with a partial lesion of substantia nigra DA neurons, which did not show increased DA levels during learning, failed to learn this task. These data represent clear and unambiguous evidence that it was the factor positive prediction error, and not hedonia or salience, which caused increase in the tonic level of striatal DA and which reinforced learning of the instrumental avoidance response.

Conscious and unconscious processes in cognitive control: a theoretical perspective and a novel empirical approach.

Controlled processing is often referred to as "voluntary" or "willful" and therefore assumed to depend entirely on conscious processes. Recent studies using subliminal-priming paradigms, however, have started to question this assumption. Specifically, these studies have shown that subliminally presented stimuli can induce adjustments in control. Such findings are not immediately reconcilable with the view that conscious and unconscious processes are separate, with each having its own neural substrates and modus operandi. We propose a different theoretical perspective that suggests that conscious and unconscious processes might be implemented by the same neural substrates and largely perform the same neural computations, with the distinction between the two arising mostly from the quality of representations (although not all brain regions may be capable of supporting conscious representations). Thus, stronger and more durable neuronal firing would give rise to conscious processes; weaker or less durable neuronal firing would remain below the threshold of consciousness but still be causally efficacious in affecting behavior. We show that this perspective naturally explains the findings that subliminally presented primes induce adjustments in cognitive control. We also highlight an important gap in this literature: whereas subliminal-priming paradigms demonstrate that an unconsciously presented prime is sufficient to induce adjustments in cognitive control, they are uninformative about what occurs under standard task conditions. In standard tasks, the stimuli themselves are consciously perceived; however, the extent to which the processes that lead to adjustments in control are conscious or unconscious remains unexplored. We propose a new paradigm suitable to investigate these issues and to test important predictions of our hypothesis that conscious and unconscious processes both engage the same control machinery, differing mostly in the quality of the representations.

A neurocomputational approach to obsessive-compulsive disorder.

A recent article shows that a change in a single parameter in a neural-network model of brain dynamics leads to repetitive behaviors that resist termination and towards which the network tends. These findings may have implications for obsessive-compulsive disorder and are consistent with evidence of glutamatergic hyperactivity in this disorder.

From reinforcement learning models to psychiatric and neurological disorders.

Over the last decade and a half, reinforcement learning models have fostered an increasingly sophisticated understanding of the functions of dopamine and cortico-basal ganglia-thalamo-cortical (CBGTC) circuits. More recently, these models, and the insights that they afford, have started to be used to understand important aspects of several psychiatric and neurological disorders that involve disturbances of the dopaminergic system and CBGTC circuits. We review this approach and its existing and potential applications to Parkinson's disease, Tourette's syndrome, attention-deficit/hyperactivity disorder, addiction, schizophrenia and preclinical animal models used to screen new antipsychotic drugs. The approach's proven explanatory and predictive power bodes well for the continued growth of computational psychiatry and computational neurology.