Transdiagnostic inflexible learning dynamics explain deficits in depression and schizophrenia.

Deficits in reward learning are core symptoms across many mental disorders. Recent work suggests that such learning impairments arise by a diminished ability to use reward history to guide behaviour, but the neuro-computational mechanisms through which these impairments emerge remain unclear. Moreover, limited work has taken a transdiagnostic approach to investigate whether the psychological and neural mechanisms that give rise to learning deficits are shared across forms of psychopathology. To provide insight into this issue, we explored probabilistic reward learning in patients diagnosed with major depressive disorder (n = 33) or schizophrenia (n = 24) and 33 matched healthy controls by combining computational modelling and single-trial EEG regression. In our task, participants had to integrate the reward history of a stimulus to decide whether it is worthwhile to gamble on it. Adaptive learning in this task is achieved through dynamic learning rates that are maximal on the first encounters with a given stimulus and decay with increasing stimulus repetitions. Hence, over the course of learning, choice preferences would ideally stabilize and be less susceptible to misleading information. We show evidence of reduced learning dynamics, whereby both patient groups demonstrated hypersensitive learning (i.e. less decaying learning rates), rendering their choices more susceptible to misleading feedback. Moreover, there was a schizophrenia-specific approach bias and a depression-specific heightened sensitivity to disconfirmational feedback (factual losses and counterfactual wins). The inflexible learning in both patient groups was accompanied by altered neural processing, including no tracking of expected values in either patient group. Taken together, our results thus provide evidence that reduced trial-by-trial learning dynamics reflect a convergent deficit across depression and schizophrenia. Moreover, we identified disorder distinct learning deficits.

Explore your experimental designs and theories before you exploit them!

In many areas of the social and behavioral sciences, the nature of the experiments and theories that best capture the underlying constructs are themselves areas of active inquiry. Integrative experiment design risks being prematurely exploitative, hindering exploration of experimental paradigms and of diverse theoretical accounts for target phenomena.

Adaptive Learning through Temporal Dynamics of State Representation.

People adjust their learning rate rationally according to local environmental statistics and calibrate such adjustments based on the broader statistical context. To date, no theory has captured the observed range of adaptive learning behaviors or the complexity of its neural correlates. Here, we attempt to do so using a neural network model that learns to map an internal context representation onto a behavioral response via supervised learning. The network shifts its internal context on receiving supervised signals that are mismatched to its output, thereby changing the "state" to which feedback is associated. A key feature of the model is that such state transitions can either increase learning or decrease learning depending on the duration over which the new state is maintained. Sustained state transitions that occur after changepoints facilitate faster learning and mimic network reset phenomena observed in the brain during rapid learning. In contrast, state transitions after one-off outlier events are short lived, thereby limiting the impact of outlying observations on future behavior. State transitions in our model provide the first mechanistic interpretation for bidirectional learning signals, such as the P300, that relate to learning differentially according to the source of surprising events and may also shed light on discrepant observations regarding the relationship between transient pupil dilations and learning. Together, our results demonstrate that dynamic latent state representations can afford normative inference and provide a coherent framework for understanding neural signatures of adaptive learning across different statistical environments.SIGNIFICANCE STATEMENT How humans adjust their sensitivity to new information in a changing world has remained largely an open question. Bridging insights from normative accounts of adaptive learning and theories of latent state representation, here we propose a feedforward neural network model that adjusts its learning rate online by controlling the speed of transitioning its internal state representations. Our model proposes a mechanistic framework for explaining learning under different statistical contexts, explains previously observed behavior and brain signals, and makes testable predictions for future experimental studies.

Noise Correlations for Faster and More Robust Learning.

Distributed population codes are ubiquitous in the brain and pose a challenge to downstream neurons that must learn an appropriate readout. Here we explore the possibility that this learning problem is simplified through inductive biases implemented by stimulus-independent noise correlations that constrain learning to task-relevant dimensions. We test this idea in a set of neural networks that learn to perform a perceptual discrimination task. Correlations among similarly tuned units were manipulated independently of an overall population signal-to-noise ratio to test how the format of stored information affects learning. Higher noise correlations among similarly tuned units led to faster and more robust learning, favoring homogenous weights assigned to neurons within a functionally similar pool, and could emerge through Hebbian learning. When multiple discriminations were learned simultaneously, noise correlations across relevant feature dimensions sped learning, whereas those across irrelevant feature dimensions slowed it. Our results complement the existing theory on noise correlations by demonstrating that when such correlations are produced without significant degradation of the signal-to-noise ratio, they can improve the speed of readout learning by constraining it to appropriate dimensions.SIGNIFICANCE STATEMENT Positive noise correlations between similarly tuned neurons theoretically reduce the representational capacity of the brain, yet they are commonly observed, emerge dynamically in complex tasks, and persist even in well-trained animals. Here we show that such correlations, when embedded in a neural population with a fixed signal-to-noise ratio, can improve the speed and robustness with which an appropriate readout is learned. In a simple discrimination task such correlations can emerge naturally through Hebbian learning. In more complex tasks that require multiple discriminations, correlations between neurons that similarly encode the task-relevant feature improve learning by constraining it to the appropriate task dimension.

Dynamic Representation of the Subjective Value of Information.

To improve future decisions, people should seek information based on the value of information (VOI), which depends on the current evidence and the reward structure of the upcoming decision. When additional evidence is supplied, people should update the VOI to adjust subsequent information seeking, but the neurocognitive mechanisms of this updating process remain unknown. We used a modified beads task to examine how the VOI is represented and updated in the human brain of both sexes. We theoretically derived, and empirically verified, a normative prediction that the VOI depends on decision evidence and is biased by reward asymmetry. Using fMRI, we found that the subjective VOI is represented in the right dorsolateral prefrontal cortex (DLPFC). Critically, this VOI representation was updated when additional evidence was supplied, showing that the DLPFC dynamically tracks the up-to-date VOI over time. These results provide new insights into how humans adaptively seek information in the service of decision-making.SIGNIFICANCE STATEMENT For adaptive decision-making, people should seek information based on what they currently know and the extent to which additional information could improve the decision outcome, formalized as the VOI. Doing so requires dynamic updating of VOI according to outcome values and newly arriving evidence. We formalize these principles using a normative model and show that information seeking in people adheres to them. Using fMRI, we show that the underlying subjective VOI is represented in the dorsolateral prefrontal cortex and, critically, that it is updated in real time according to newly arriving evidence. Our results reveal the computational and neural dynamics through which evidence and values are combined to inform constantly evolving information-seeking decisions.

All or nothing belief updating in patients with schizophrenia reduces precision and flexibility of beliefs.

Schizophrenia is characterized by abnormal perceptions and beliefs, but the computational mechanisms through which these abnormalities emerge remain unclear. One prominent hypothesis asserts that such abnormalities result from overly precise representations of prior knowledge, which in turn lead beliefs to become insensitive to feedback. In contrast, another prominent hypothesis asserts that such abnormalities result from a tendency to interpret prediction errors as indicating meaningful change, leading to the assignment of aberrant salience to noisy or misleading information. Here we examine behaviour of patients and control subjects in a behavioural paradigm capable of adjudicating between these competing hypotheses and characterizing belief updates directly on individual trials. We show that patients are more prone to completely ignoring new information and perseverating on previous responses, but when they do update, tend to do so completely. This updating strategy limits the integration of information over time, reducing both the flexibility and precision of beliefs and provides a potential explanation for how patients could simultaneously show over-sensitivity and under-sensitivity to feedback in different paradigms.

The stability flexibility tradeoff and the dark side of detail.

Learning in dynamic environments requires integrating over stable fluctuations to minimize the impact of noise (stability) but rapidly responding in the face of fundamental changes (flexibility). Achieving one of these goals often requires sacrificing the other to some degree, producing a stability-flexibility tradeoff. Individuals navigate this tradeoff in different ways; some people learn rapidly (emphasizing flexibility) and others rely more heavily on historical information (emphasizing stability). Despite the prominence of such individual differences in learning tasks, the degree to which they relate to broader characteristics of real-world behavior or pathologies has not been well explored. We relate individual differences in learning behavior to self-report measures thought to capture collectively the characteristics of the Autism spectrum. We show that young adults who learn most slowly tend to integrate more effective samples into their beliefs about the world making them more robust to noise (more stability) but are more likely to integrate information from previous contexts (less flexibility). We show that individuals who report paying more attention to detail tend to use high flexibility and low stability information processing strategies. We demonstrate the robustness of this inverse relationship between attention to detail and formation of stable beliefs in a heterogeneous population of children that includes a high proportion of Autism diagnoses. Together, our results highlight that attention to detail reflects an information processing policy that comes with a substantial downside, namely the ability to integrate data to overcome environmental noise.

Dissociable Forms of Uncertainty-Driven Representational Change Across the Human Brain.

Environmental change can lead decision makers to shift rapidly among different behavioral regimes. These behavioral shifts can be accompanied by rapid changes in the firing pattern of neural networks. However, it is unknown what the populations of neurons that participate in such "network reset" phenomena are representing. Here, we investigated the following: (1) whether and where rapid changes in multivariate activity patterns are observable with fMRI during periods of rapid behavioral change and (2) what types of representations give rise to these phenomena. We did so by examining fluctuations in multivoxel patterns of BOLD activity from male and female human subjects making sequential inferences about the state of a partially observable and discontinuously changing variable. We found that, within the context of this sequential inference task, the multivariate patterns of activity in a number of cortical regions contain representations that change more rapidly during periods of uncertainty following a change in behavioral context. In motor cortex, this phenomenon was indicative of discontinuous change in behavioral outputs, whereas in visual regions, the same basic phenomenon was evoked by tracking of salient environmental changes. In most other cortical regions, including dorsolateral prefrontal and anterior cingulate cortex, the phenomenon was most consistent with directly encoding the degree of uncertainty. However, in a few other regions, including orbitofrontal cortex, the phenomenon was best explained by representations of a shifting context that evolve more rapidly during periods of rapid learning. These representations may provide a dynamic substrate for learning that facilitates rapid disengagement from learned responses during periods of change.SIGNIFICANCE STATEMENT Brain activity patterns tend to change more rapidly during periods of uncertainty and behavioral adjustment, yet the computational role of such rapid transitions is poorly understood. Here, we identify brain regions with fMRI BOLD activity patterns that change more rapidly during periods of behavioral adjustment and use computational modeling to attribute the phenomenon to specific causes. We demonstrate that the phenomenon emerges in different brain regions for different computational reasons, the most common being the representation of uncertainty itself, but that, in a selective subset of regions including orbitofrontal cortex, the phenomenon was best explained as a shifting latent state signal that may serve to control the degree to which recent temporal context affects ongoing expectations.

Individual Neurons in the Cingulate Cortex Encode Action Monitoring, Not Selection, during Adaptive Decision-Making.

The cingulate cortex contributes to complex, adaptive behaviors, but the exact nature of its contributions remains unresolved. Proposals from previous studies, including evaluating past actions or selecting future ones, have been difficult to distinguish in part because of an incomplete understanding of the task-relevant variables that are encoded by individual cingulate neurons. In this study, we recorded from individual neurons in parts of both the anterior cingulate cortex (ACC) and posterior cingulate cortex (PCC) in 2 male rhesus monkeys performing a saccadic reward task. The task required them to use adaptive, feedback-driven strategies to infer the spatial location of a rewarded saccade target in the presence of different forms of uncertainty. We found that task-relevant, spatially selective feedback signals were encoded by the activity of individual neurons in both brain regions, with stronger selectivity for spatial choice and reward-target signals in PCC and stronger selectivity for feedback in ACC. Moreover, neurons in both regions were sensitive to sequential effects of feedback that partly reflected sequential behavioral patterns. However, neither brain region exhibited systematic modulations by the blockwise conditions that governed the reliability of the trial-by-trial feedback and drove adaptive behavioral patterns. There was also little evidence that single-neuron responses in either brain region directly predicted the extent to which feedback and contextual information were used to inform choices on the subsequent trial. Thus, certain cingulate neurons encode diverse, evaluative signals needed for adaptive, feedback-driven decision-making, but those signals may be integrated elsewhere in the brain to guide actions.SIGNIFICANCE STATEMENT Effective decision-making in dynamic environments requires adapting to changes in feedback and context. The anterior and posterior cingulate cortex have been implicated in adaptive decision-making, but the exact nature of their respective roles remains unresolved. Here we compare patterns of task-driven activity of subsets of individual neurons from parts of the two brain regions in monkeys performing a saccadic task with dynamically changing reward locations. We find evidence for regional specializations in neural representations of choice and feedback, including task-relevant modulations of activity that could be used for performance monitoring. However, we find little evidence that these neural representations are used directly to adjust choice behavior, which thus likely requires integration of these signals elsewhere in the brain.

Correction: A Mixture of Delta-Rules Approximation to Bayesian Inference in Change-Point Problems.

[This corrects the article DOI: 10.1371/journal.pcbi.1003150.].

A Control Theoretic Model of Adaptive Learning in Dynamic Environments.

To behave adaptively in environments that are noisy and nonstationary, humans and other animals must monitor feedback from their environment and adjust their predictions and actions accordingly. An understudied approach for modeling these adaptive processes comes from the engineering field of control theory, which provides general principles for regulating dynamical systems, often without requiring a generative model. The proportional-integral-derivative (PID) controller is one of the most popular models of industrial process control. The proportional term is analogous to the "delta rule" in psychology, adjusting estimates in proportion to each error in prediction. The integral and derivative terms augment this update to simultaneously improve accuracy and stability. Here, we tested whether the PID algorithm can describe how people sequentially adjust their predictions in response to new information. Across three experiments, we found that the PID controller was an effective model of participants' decisions in noisy, changing environments. In Experiment 1, we reanalyzed a change-point detection experiment and showed that participants' behavior incorporated elements of PID updating. In Experiments 2-3, we developed a task with gradual transitions that we optimized to detect PID-like adjustments. In both experiments, the PID model offered better descriptions of behavioral adjustments than both the classical delta-rule model and its more sophisticated variant, the Kalman filter. We further examined how participants weighted different PID terms in response to salient environmental events, finding that these control terms were modulated by reward, surprise, and outcome entropy. These experiments provide preliminary evidence that adaptive learning in dynamic environments resembles PID control.

Catecholaminergic Regulation of Learning Rate in a Dynamic Environment.

Adaptive behavior in a changing world requires flexibly adapting one's rate of learning to the rate of environmental change. Recent studies have examined the computational mechanisms by which various environmental factors determine the impact of new outcomes on existing beliefs (i.e., the 'learning rate'). However, the brain mechanisms, and in particular the neuromodulators, involved in this process are still largely unknown. The brain-wide neurophysiological effects of the catecholamines norepinephrine and dopamine on stimulus-evoked cortical responses suggest that the catecholamine systems are well positioned to regulate learning about environmental change, but more direct evidence for a role of this system is scant. Here, we report evidence from a study employing pharmacology, scalp electrophysiology and computational modeling (N = 32) that suggests an important role for catecholamines in learning rate regulation. We found that the P3 component of the EEG-an electrophysiological index of outcome-evoked phasic catecholamine release in the cortex-predicted learning rate, and formally mediated the effect of prediction-error magnitude on learning rate. P3 amplitude also mediated the effects of two computational variables-capturing the unexpectedness of an outcome and the uncertainty of a preexisting belief-on learning rate. Furthermore, a pharmacological manipulation of catecholamine activity affected learning rate following unanticipated task changes, in a way that depended on participants' baseline learning rate. Our findings provide converging evidence for a causal role of the human catecholamine systems in learning-rate regulation as a function of environmental change.

What do we GANE with age?

Mather and colleagues provide an impressive cross-level account of how arousal levels modulate behavior, and they support it with data ranging from receptor pharmacology to measures of cognitive function. Here we consider two related questions: (1) Why should the brain engage in different arousal levels? and (2) What are the predicted consequences of age-related changes in norepinephrine signaling for cognitive function?

The mitochondrial uncoupler DNP triggers brain cell mTOR signaling network reprogramming and CREB pathway up-regulation.

Mitochondrial metabolism is highly responsive to nutrient availability and ongoing activity in neuronal circuits. The molecular mechanisms by which brain cells respond to an increase in cellular energy expenditure are largely unknown. Mild mitochondrial uncoupling enhances cellular energy expenditure in mitochondria and can be induced with 2,4-dinitrophenol (DNP), a proton ionophore previously used for weight loss. We found that DNP treatment reduces mitochondrial membrane potential, increases intracellular Ca(2+) levels and reduces oxidative stress in cerebral cortical neurons. Gene expression profiling of the cerebral cortex of DNP-treated mice revealed reprogramming of signaling cascades that included suppression of the mammalian target of rapamycin (mTOR) and insulin--PI3K - MAPK pathways, and up-regulation of tuberous sclerosis complex 2, a negative regulator of mTOR. Genes encoding proteins involved in autophagy processes were up-regulated in response to DNP. CREB (cAMP-response element-binding protein) signaling, Arc and brain-derived neurotrophic factor, which play important roles in synaptic plasticity and adaptive cellular stress responses, were up-regulated in response to DNP, and DNP-treated mice exhibited improved performance in a test of learning and memory. Immunoblot analysis verified that key DNP-induced changes in gene expression resulted in corresponding changes at the protein level. Our findings suggest that mild mitochondrial uncoupling triggers an integrated signaling response in brain cells characterized by reprogramming of mTOR and insulin signaling, and up-regulation of pathways involved in adaptive stress responses, molecular waste disposal, and synaptic plasticity. Physiological bioenergetic challenges such as exercise and fasting can enhance neuroplasticity and protect neurons against injury and neurodegeneration. Here, we show that the mitochondrial uncoupling agent 2,4-dinitrophenol (DNP) elicits adaptive signaling responses in the cerebral cortex involving activation of Ca(2+) -CREB and autophagy pathways, and inhibition of mTOR and insulin signaling pathways. The molecular reprogramming induced by DNP, which is similar to that of exercise and fasting, is associated with improved learning and memory, suggesting potential therapeutic applications for DNP.

A healthy fear of the unknown: perspectives on the interpretation of parameter fits from computational models in neuroscience.

Fitting models to behavior is commonly used to infer the latent computational factors responsible for generating behavior. However, the complexity of many behaviors can handicap the interpretation of such models. Here we provide perspectives on problems that can arise when interpreting parameter fits from models that provide incomplete descriptions of behavior. We illustrate these problems by fitting commonly used and neurophysiologically motivated reinforcement-learning models to simulated behavioral data sets from learning tasks. These model fits can pass a host of standard goodness-of-fit tests and other model-selection diagnostics even when the models do not provide a complete description of the behavioral data. We show that such incomplete models can be misleading by yielding biased estimates of the parameters explicitly included in the models. This problem is particularly pernicious when the neglected factors are unknown and therefore not easily identified by model comparisons and similar methods. An obvious conclusion is that a parsimonious description of behavioral data does not necessarily imply an accurate description of the underlying computations. Moreover, general goodness-of-fit measures are not a strong basis to support claims that a particular model can provide a generalized understanding of the computations that govern behavior. To help overcome these challenges, we advocate the design of tasks that provide direct reports of the computational variables of interest. Such direct reports complement model-fitting approaches by providing a more complete, albeit possibly more task-specific, representation of the factors that drive behavior. Computational models then provide a means to connect such task-specific results to a more general algorithmic understanding of the brain.

A mixture of delta-rules approximation to bayesian inference in change-point problems.

Error-driven learning rules have received considerable attention because of their close relationships to both optimal theory and neurobiological mechanisms. However, basic forms of these rules are effective under only a restricted set of conditions in which the environment is stable. Recent studies have defined optimal solutions to learning problems in more general, potentially unstable, environments, but the relevance of these complex mathematical solutions to how the brain solves these problems remains unclear. Here, we show that one such Bayesian solution can be approximated by a computationally straightforward mixture of simple error-driven 'Delta' rules. This simpler model can make effective inferences in a dynamic environment and matches human performance on a predictive-inference task using a mixture of a small number of Delta rules. This model represents an important conceptual advance in our understanding of how the brain can use relatively simple computations to make nearly optimal inferences in a dynamic world.

Thalamocortical architectures for flexible cognition and efficient learning.

The brain exhibits a remarkable ability to learn and execute context-appropriate behaviors. How it achieves such flexibility, without sacrificing learning efficiency, is an important open question. Neuroscience, psychology, and engineering suggest that reusing and repurposing computations are part of the answer. Here, we review evidence that thalamocortical architectures may have evolved to facilitate these objectives of flexibility and efficiency by coordinating distributed computations. Recent work suggests that distributed prefrontal cortical networks compute with flexible codes, and that the mediodorsal thalamus provides regularization to promote efficient reuse. Thalamocortical interactions resemble hierarchical Bayesian computations, and their network implementation can be related to existing gating, synchronization, and hub theories of thalamic function. By reviewing recent findings and providing a novel synthesis, we highlight key research horizons integrating computation, cognition, and systems neuroscience.

Enhancement of mediodorsal thalamus rescues aberrant belief dynamics in a mouse model with schizophrenia-associated mutation.

Optimizing behavioral strategy requires belief updating based on new evidence, a process that engages higher cognition. In schizophrenia, aberrant belief dynamics may lead to psychosis, but the mechanisms underlying this process are unknown, in part, due to lack of appropriate animal models and behavior readouts. Here, we address this challenge by taking two synergistic approaches. First, we generate a mouse model bearing patient-derived point mutation in Grin2a (Grin2aY700X+/-), a gene that confers high-risk for schizophrenia and recently identified by large-scale exome sequencing. Second, we develop a computationally trackable foraging task, in which mice form and update belief-driven strategies in a dynamic environment. We found that Grin2aY700X+/- mice perform less optimally than their wild-type (WT) littermates, showing unstable behavioral states and a slower belief update rate. Using functional ultrasound imaging, we identified the mediodorsal (MD) thalamus as hypofunctional in Grin2aY700X+/- mice, and in vivo task recordings showed that MD neurons encoded dynamic values and behavioral states in WT mice. Optogenetic inhibition of MD neurons in WT mice phenocopied Grin2aY700X+/- mice, and enhancing MD activity rescued task deficits in Grin2aY700X+/- mice. Together, our study identifies the MD thalamus as a key node for schizophrenia-relevant cognitive dysfunction, and a potential target for future therapeutics.

Prefrontal cortex state representations shape human credit assignment.

People learn adaptively from feedback, but the rate of such learning differs drastically across individuals and contexts. Here, we examine whether this variability reflects differences in what is learned. Leveraging a neurocomputational approach that merges fMRI and an iterative reward learning task, we link the specificity of credit assignment-how well people are able to appropriately attribute outcomes to their causes-to the precision of neural codes in the prefrontal cortex (PFC). Participants credit task-relevant cues more precisely in social compared vto nonsocial contexts, a process that is mediated by high-fidelity (i.e., distinct and consistent) state representations in the PFC. Specifically, the medial PFC and orbitofrontal cortex work in concert to match the neural codes from feedback to those at choice, and the strength of these common neural codes predicts credit assignment precision. Together this work provides a window into how neural representations drive adaptive learning.