Reward-Mediated, Model-Free Reinforcement-Learning Mechanisms in Pavlovian and Instrumental Tasks Are Related.

Model-free and model-based computations are argued to distinctly update action values that guide decision-making processes. It is not known, however, if these model-free and model-based reinforcement learning mechanisms recruited in operationally based instrumental tasks parallel those engaged by pavlovian-based behavioral procedures. Recently, computational work has suggested that individual differences in the attribution of incentive salience to reward predictive cues, that is, sign- and goal-tracking behaviors, are also governed by variations in model-free and model-based value representations that guide behavior. Moreover, it is not appreciated if these systems that are characterized computationally using model-free and model-based algorithms are conserved across tasks for individual animals. In the current study, we used a within-subject design to assess sign-tracking and goal-tracking behaviors using a pavlovian conditioned approach task and then characterized behavior using an instrumental multistage decision-making (MSDM) task in male rats. We hypothesized that both pavlovian and instrumental learning processes may be driven by common reinforcement-learning mechanisms. Our data confirm that sign-tracking behavior was associated with greater reward-mediated, model-free reinforcement learning and that it was also linked to model-free reinforcement learning in the MSDM task. Computational analyses revealed that pavlovian model-free updating was correlated with model-free reinforcement learning in the MSDM task. These data provide key insights into the computational mechanisms mediating associative learning that could have important implications for normal and abnormal states.SIGNIFICANCE STATEMENT Model-free and model-based computations that guide instrumental decision-making processes may also be recruited in pavlovian-based behavioral procedures. Here, we used a within-subject design to test the hypothesis that both pavlovian and instrumental learning processes were driven by common reinforcement-learning mechanisms. Sign-tracking and goal-tracking behaviors were assessed in rats using a pavlovian conditioned approach task, and then instrumental behavior was characterized using an MSDM task. We report that sign-tracking behavior was associated with greater model-free, but not model-based, learning in the MSDM task. These data suggest that pavlovian and instrumental behaviors may be driven by conserved reinforcement-learning mechanisms.

The well-worn route revisited: Striatal and hippocampal system contributions to familiar route navigation.

Classic research has shown a division in the neuroanatomical structures that support flexible (e.g., short-cutting) and habitual (e.g., familiar route following) navigational behavior, with hippocampal-caudate systems associated with the former and putamen systems with the latter. There is, however, disagreement about whether the neural structures involved in navigation process particular forms of spatial information, such as associations between constellations of cues forming a cognitive map, versus single landmark-action associations, or alternatively, perform particular reinforcement learning algorithms that allow the use of different spatial strategies, so-called model-based (flexible) or model-free (habitual) forms of learning. We sought to test these theories by asking participants (N = 24) to navigate within a virtual environment through a previously learned, 9-junction route with distinctive landmarks at each junction while undergoing functional magnetic resonance imaging (fMRI). In a series of probe trials, we distinguished knowledge of individual landmark-action associations along the route versus knowledge of the correct sequence of landmark-action associations, either by having absent landmarks, or "out-of-sequence" landmarks. Under a map-based perspective, sequence knowledge would not require hippocampal systems, because there are no constellations of cues available for cognitive map formation. Within a learning-based model, however, responding based on knowledge of sequence would require hippocampal systems because prior context has to be utilized. We found that hippocampal-caudate systems were more active in probes requiring sequence knowledge, supporting the learning-based model. However, we also found greater putamen activation in probes where navigation based purely on sequence memory could be planned, supporting models of putamen function that emphasize its role in action sequencing.

Deuterium spin relaxation of fractionally deuterated ribonuclease H using paired 475 and 950 MHz NMR spectrometers.

Deuterium (2H) spin relaxation of 13CH2D methyl groups has been widely applied to investigate picosecond-to-nanosecond conformational dynamics in proteins by solution-state NMR spectroscopy. The B0 dependence of the 2H spin relaxation rates is represented by a linear relationship between the spectral density function at three discrete frequencies J(0), J(ωD) and J(2ωD). In this study, the linear relation between 2H relaxation rates at B0 fields separated by a factor of two and the interpolation of rates at intermediate frequencies are combined for a more robust approach for spectral density mapping. The general usefulness of the approach is demonstrated on a fractionally deuterated (55%) and alternate 13C-12C labeled sample of E. coli RNase H. Deuterium relaxation rate constants (R1, R1ρ, RQ, RAP) were measured for 57 well-resolved 13CH2D moieties in RNase H at 1H frequencies of 475 MHz, 500 MHz, 900 MHz, and 950 MHz. The spectral density mapping of the 475/950 MHz data combination was performed independently and jointly to validate the expected relationship between data recorded at B0 fields separated by a factor of two. The final analysis was performed by jointly analyzing 475/950 MHz rates with 700 MHz rates interpolated from 500/900 MHz data to yield six J(ωD) values for each methyl peak. The J(ω) profile for each peak was fit to the original (τM, Sf2, τf) or extended model-free function (τM, Sf2, Ss2, τf, τs) to obtain optimized dynamic parameters.

Brain perfusion estimation by Tikhonov model-free deconvolution in a long axial field of view PET/CT scanner exploring five different PET tracers.

PURPOSE: New total-body PET scanners with a long axial field of view (LAFOV) allow for higher temporal resolution due to higher sensitivity, which facilitates perfusion estimation by model-free deconvolution. Fundamental tracer kinetic theory predicts that perfusion can be estimated for all tracers despite their different fates given sufficiently high temporal resolution of 1 s or better, bypassing the need for compartment modelling. The aim of this study was to investigate whether brain perfusion could be estimated using model-free Tikhonov generalized deconvolution for five different PET tracers, [15O]H2O, [11C]PIB, [18F]FE-PE2I, [18F]FDG and [18F]FET. To our knowledge, this is the first example of a general model-free approach to estimate cerebral blood flow (CBF) from PET data. METHODS: Twenty-five patients underwent dynamic LAFOV PET scanning (Siemens, Quadra). PET images were reconstructed with an isotropic voxel resolution of 1.65 mm3. Time framing was 40 × 1 s during bolus passage followed by increasing framing up to 60 min. AIF was obtained from the descending aorta. Both voxel- and region-based calculations of perfusion in the thalamus were performed using the Tikhonov method. The residue impulse response function was used to estimate the extraction fraction of tracer leakage across the blood-brain barrier. RESULTS: CBF ranged from 37 to 69 mL blood min-1 100 mL of tissue-1 in the thalamus. Voxelwise calculation of CBF resulted in CBF maps in the physiologically normal range. The extraction fractions of [15O]H2O, [18F]FE-PE2I, [11C]PIB, [18F]FDG and [18F]FET in the thalamus were 0.95, 0.78, 0.62, 0.19 and 0.03, respectively. CONCLUSION: The high temporal resolution and sensitivity associated with LAFOV PET scanners allow for noninvasive perfusion estimation of multiple tracers. The method provides an estimation of the residue impulse response function, from which the fate of the tracer can be studied, including the extraction fraction, influx constant, volume of distribution and transit time distribution, providing detailed physiological insight into normal and pathologic tissue.

Neural evidence for age-related deficits in the representation of state spaces.

Under high cognitive demands, older adults tend to resort to simpler, habitual, or model-free decision strategies. This age-related shift in decision behavior has been attributed to deficits in the representation of the cognitive maps, or state spaces, necessary for more complex model-based decision-making. Yet, the neural mechanisms behind this shift remain unclear. In this study, we used a modified 2-stage Markov task in combination with computational modeling and single-trial EEG analyses to establish neural markers of age-related changes in goal-directed decision-making under different demands on the representation of state spaces. Our results reveal that the shift to simpler decision strategies in older adults is due to (i) impairments in the representation of the transition structure of the task and (ii) a diminished signaling of the reward value associated with decision options. In line with the diminished state space hypothesis of human aging, our findings suggest that deficits in goal-directed, model-based behavior in older adults result from impairments in the representation of state spaces of cognitive tasks.

Model-based and model-free mechanisms in methamphetamine use disorder.

People with methamphetamine use disorder (MUD) struggle to shift their behaviour from methamphetamine-orientated habits to goal-oriented choices. The model-based/model-free framework is well suited to understand this difficulty by unpacking the computational mechanisms that support experienced-based (model-free) and goal-directed (model-based) choices. We aimed to examine whether 1) participants with MUD differed from controls on behavioural proxies and/or computational mechanisms of model-based/model-free choices; 2) model-based/model-free decision-making correlated with MUD symptoms; and 3) model-based/model-free deficits improved over six weeks in the group with MUD. Participants with MUD and controls with similar age, IQ and socioeconomic status completed the Two-Step Task at treatment commencement (MUD n = 30, Controls n = 31) and six weeks later (MUD n = 23, Controls n = 26). We examined behavioural proxies of model-based/model-free decisions using mixed logistic regression, and their underlying mechanisms using computational modelling. At a behavioural level, participants with MUD were more likely to switch their choices following rewarded actions, although this pattern improved at follow up. At a computational level, groups were similar in their use of model-based mechanisms, but participants with MUD were less likely to apply model-free mechanisms and less likely to repeat rewarded actions. We did not find evidence that individual differences in model-based or model-free parameters were associated with greater severity of methamphetamine dependence, nor did we find that group differences in computational parameters changed between baseline and follow-up assessment. Decision-making challenges in people with MUD are likely related to difficulties in pursuing choices previously associated with positive outcomes.

Disorders of compulsivity: Deficits in arbitrating learning strategies.

While previous research has shown that compulsivity is related to an imbalance between goal-directed and habitual learning systems, very little is known about whether this effect is due to the impairment of a single system or the impairment of the arbitration mechanism that determines which system controls behaviour at any given moment; the current study aims to address this disagreement. Nineteen alcohol use disorder, 30 obsessive-compulsive disorder (OCD) and 20 major depressive disorder patients and corresponding sex- and age-matched controls performed two-choice, three-stage Markov decision-making paradigm. Model-based and mode-free reinforcement learning models were used to independently fitted their behavioural data. Alcohol use disorder and OCD patients showed less model-based strategy choice than healthy controls in task conditions where the model-based strategy was optimal. Only OCD patients showed higher behavioural control system switching in task conditions where model-free use was optimal. Major depressive disorder patients did not differ from the matched control in both. These findings suggest that dysfunction in arbitration control between dual systems may be the basis for diverse disorders involving compulsivity.

Distributional conformal prediction.

We propose a robust method for constructing conditionally valid prediction intervals based on models for conditional distributions such as quantile and distribution regression. Our approach can be applied to important prediction problems, including cross-sectional prediction, k-step-ahead forecasts, synthetic controls and counterfactual prediction, and individual treatment effects prediction. Our method exploits the probability integral transform and relies on permuting estimated ranks. Unlike regression residuals, ranks are independent of the predictors, allowing us to construct conditionally valid prediction intervals under heteroskedasticity. We establish approximate conditional validity under consistent estimation and provide approximate unconditional validity under model misspecification, under overfitting, and with time series data. We also propose a simple "shape" adjustment of our baseline method that yields optimal prediction intervals.

Human subjects exploit a cognitive map for credit assignment.

An influential reinforcement learning framework proposes that behavior is jointly governed by model-free (MF) and model-based (MB) controllers. The former learns the values of actions directly from past encounters, and the latter exploits a cognitive map of the task to calculate these prospectively. Considerable attention has been paid to how these systems interact during choice, but how and whether knowledge of a cognitive map contributes to the way MF and MB controllers assign credit (i.e., to how they revaluate actions and states following the receipt of an outcome) remains underexplored. Here, we examine such sophisticated credit assignment using a dual-outcome bandit task. We provide evidence that knowledge of a cognitive map influences credit assignment in both MF and MB systems, mediating subtly different aspects of apparent relevance. Specifically, we show MF credit assignment is enhanced for those rewards that are related to a choice, and this contrasted with choice-unrelated rewards that reinforced subsequent choices negatively. This modulation is only possible based on knowledge of task structure. On the other hand, MB credit assignment was boosted for outcomes that impacted on differences in values between offered bandits. We consider mechanistic accounts and the normative status of these findings. We suggest the findings extend the scope and sophistication of cognitive map-based credit assignment during reinforcement learning, with implications for understanding behavioral control.

An Adaptive Control Scheme Based on Non-Interference Nonlinearity Approximation for a Class of Nonlinear Cascaded Systems and Its Application to Flexible Joint Manipulators.

Control design for the nonlinear cascaded system is challenging due to its complicated system dynamics and system uncertainty, both of which can be considered some kind of system nonlinearity. In this paper, we propose a novel nonlinearity approximation scheme with a simplified structure, where the system nonlinearity is approximated by a steady component and an alternating component using only local tracking errors. The nonlinearity of each subsystem is estimated independently. On this basis, a model-free adaptive control for a class of nonlinear cascaded systems is proposed. A squared-error correction procedure is introduced to regulate the weight coefficients of the approximation components, which makes the whole adaptive system stable even with the unmodeled uncertainties. The effectiveness of the proposed controller is validated on a flexible joint system through numerical simulations and experiments. Simulation and experimental results show that the proposed controller can achieve better control performance than the radial basis function network control. Due to its simplicity and robustness, this method is suitable for engineering applications.

Large Range of a High-Precision, Independent, Sub-Mirror Three-Dimensional Co-Phase Error Sensing and Correction Method via a Mask and Population Algorithm.

The emergence of segmented mirrors is expected to solve the design, processing, manufacturing, testing, and launching of space telescopes of large apertures. However, with the increase in the number of sub-mirrors, the sensing and correction of co-phase errors in segmented mirrors will be very difficult. In this paper, an independent three-dimensional method for sub-mirror co-phase error sensing and correction method is proposed. The method is based on a wide spectral modulation transfer function (MTF), mask, population optimization algorithm, and online model-free correction. In this method, the sensing and correction process of each sub-mirror co-phase error is independent of each other, so the increase in the number of sub-mirrors will not increase the difficulty of the method. This method can sense and correct the co-phase errors of three dimensions of the sub-mirror, including piston, tip, and tilt, even without modeling the optical system, and has a wide detection range and high precision. And the efficiency is high because the sub-mirrors can be corrected simultaneously in parallel. Simulation results show that the proposed method can effectively sense and correct the co-phase errors of the sub-mirrors in the range [-50λ, 50λ] in three dimensions with high precision. The average RMSE value in 100 experiments of the true co-phase error values and the experimental co-phase error values of one of the six sub-mirrors is 2.358 × 10-7λ.

Data Reconstruction Using Smart Sensor Placement.

This paper deals with spatio-temporal field estimation with efficient sensor placement based on the QR decomposition. The proposed method also identifies the optimal number of sensors needed for field estimation that captures the most relevant features of the field of interest. To address the uncertainties inherent in spatio-temporal field estimation, a robust data-driven control method is utilized, providing resilience against unpredictable environmental and model changes. In particular, the approach uses the Kriged Kalman Filter (KKF) for uncertainty-aware field reconstruction. Unlike other reconstruction methods, the positional uncertainty originating from the data acquisition platform is integrated into the KKF estimator. Numerical results are presented to show the efficacy of the proposed dynamic sensor placement strategy together with the KKF field estimator.

A Randomization-Based, Model-Free Approach to Functional Neuroimaging: A Proof of Concept.

Functional neuroimaging analysis takes noisy multidimensional measurements as input and produces statistical inferences regarding the functional properties of brain regions as output. Such inferences are most commonly model-based, in that they assume a model of how neural activity translates to the measured signal (blood oxygenation level-dependent signal in the case of functional MRI). The use of models increases statistical sensitivity and makes it possible to ask fine-grained theoretical questions. However, this comes at the cost of making theoretical assumptions about the underlying data-generating process. An advantage of model-free approaches is that they can be used in cases where model assumptions are known not to hold. To this end, we introduce a randomization-based, model-free approach to functional neuroimaging. TWISTER randomization makes it possible to infer functional selectivity from correlations between experimental runs. We provide a proof of concept in the form of a visuomotor mapping experiment and discuss the possible strengths and limitations of this new approach in light of our empirical results.

Experimental evidence of size-selective harvest and environmental stochasticity effects on population demography, fluctuations and non-linearity.

Theory and analyses of fisheries data sets indicate that harvesting can alter population structure and destabilise non-linear processes, which increases population fluctuations. We conducted a factorial experiment on the population dynamics of Daphnia magna in relation to size-selective harvesting and stochasticity of food supply. Harvesting and stochasticity treatments both increased population fluctuations. Timeseries analysis indicated that fluctuations in control populations were non-linear, and non-linearity increased substantially in response to harvesting. Both harvesting and stochasticity induced population juvenescence, but harvesting did so via the depletion of adults, whereas stochasticity increased the abundance of juveniles. A fitted fisheries model indicated that harvesting shifted populations towards higher reproductive rates and larger-magnitude damped oscillations that amplify demographic noise. These findings provide experimental evidence that harvesting increases the non-linearity of population fluctuations and that both harvesting and stochasticity increase population variability and juvenescence.

Depression: a decision-theoretic analysis.

The manifold symptoms of depression are common and often transient features of healthy life that are likely to be adaptive in difficult circumstances. It is when these symptoms enter a seemingly self-propelling spiral that the maladaptive features of a disorder emerge. We examine this malignant transformation from the perspective of the computational neuroscience of decision making, investigating how dysfunction of the brain's mechanisms of evaluation might lie at its heart. We start by considering the behavioral implications of pessimistic evaluations of decision variables. We then provide a selective review of work suggesting how such pessimism might arise via specific failures of the mechanisms of evaluation or state estimation. Finally, we analyze ways that miscalibration between the subject and environment may be self-perpetuating. We employ the formal framework of Bayesian decision theory as a foundation for this study, showing how most of the problems arise from one of its broad algorithmic facets, namely model-based reasoning.

Distinct reinforcement learning profiles distinguish between language and attentional neurodevelopmental disorders.

BACKGROUND: Theoretical models posit abnormalities in cortico-striatal pathways in two of the most common neurodevelopmental disorders (Developmental dyslexia, DD, and Attention deficit hyperactive disorder, ADHD), but it is still unclear what distinct cortico-striatal dysfunction might distinguish language disorders from others that exhibit very different symptomatology. Although impairments in tasks that depend on the cortico-striatal network, including reinforcement learning (RL), have been implicated in both disorders, there has been little attempt to dissociate between different types of RL or to compare learning processes in these two types of disorders. The present study builds upon prior research indicating the existence of two learning manifestations of RL and evaluates whether these processes can be differentiated in language and attention deficit disorders. We used a two-step RL task shown to dissociate model-based from model-free learning in human learners. RESULTS: Our results show that, relative to neurotypicals, DD individuals showed an impairment in model-free but not in model-based learning, whereas in ADHD the ability to use both model-free and model-based learning strategies was significantly compromised. CONCLUSIONS: Thus, learning impairments in DD may be linked to a selective deficit in the ability to form action-outcome associations based on previous history, whereas in ADHD some learning deficits may be related to an incapacity to pursue rewards based on the tasks' structure. Our results indicate how different patterns of learning deficits may underlie different disorders, and how computation-minded experimental approaches can differentiate between them.

Non-parametric inference about mean functionals of non-ignorable non-response data without identifying the joint distribution.

We consider identification and inference about mean functionals of observed covariates and an outcome variable subject to non-ignorable missingness. By leveraging a shadow variable, we establish a necessary and sufficient condition for identification of the mean functional even if the full data distribution is not identified. We further characterize a necessary condition for n-estimability of the mean functional. This condition naturally strengthens the identifying condition, and it requires the existence of a function as a solution to a representer equation that connects the shadow variable to the mean functional. Solutions to the representer equation may not be unique, which presents substantial challenges for non-parametric estimation, and standard theories for non-parametric sieve estimators are not applicable here. We construct a consistent estimator of the solution set and then adapt the theory of extremum estimators to find from the estimated set a consistent estimator of an appropriately chosen solution. The estimator is asymptotically normal, locally efficient and attains the semi-parametric efficiency bound under certain regularity conditions. We illustrate the proposed approach via simulations and a real data application on home pricing.

Computational and behavioral markers of model-based decision making in childhood.

Human decision-making is underpinned by distinct systems that differ in flexibility and associated cognitive cost. A widely accepted dichotomy distinguishes between a cheap but rigid model-free system and a flexible but costly model-based system. Typically, humans use a hybrid of both types of decision-making depending on environmental demands. However, children's use of a model-based system during decision-making has not yet been shown. While prior developmental work has identified simple building blocks of model-based reasoning in young children (1-4 years old), there has been little evidence of this complex cognitive system influencing behavior before adolescence. Here, by using a modified task to make engagement in cognitively costly strategies more rewarding, we show that children aged 5-11-years (N = 85), including the youngest children, displayed multiple indicators of model-based decision making, and that the degree of its use increased throughout childhood. Unlike adults (N = 24), however, children did not display adaptive arbitration between model-free and model-based decision-making. Our results demonstrate that throughout childhood, children can engage in highly sophisticated and costly decision-making strategies. However, the flexible arbitration between decision-making strategies might be a critically late-developing component in human development.

A structural approach to understanding enzymatic regulation of chemical reaction networks.

In living cells, chemical reactions are connected by sharing their products and substrates, and form complex systems, i.e. chemical reaction network. One of the largest missions in modern biology is to understand behaviors of such systems logically based on information of network structures. However, there are series of obstacles to study dynamical behaviors of complex network systems in biology. For example, network structure does not provide sufficient information to determine details of the dynamical behaviors. In this review, I will introduce a novel mathematical theory, structural sensitivity analysis, by which the responses of reaction systems upon the changes in enzyme activities/amounts are determined from network structure alone. The patterns of responses exhibit characteristic features, localization and hierarchy, depending on the topology of the network. The theory also shows that ranges of enzymatic regulations are governed by a mathematical law characterized by local topology of substructures. These findings imply that the network topology is one of the origins of biological robustness.

Model-free decision making is prioritized when learning to avoid harming others.

Moral behavior requires learning how our actions help or harm others. Theoretical accounts of learning propose a key division between "model-free" algorithms that cache outcome values in actions and "model-based" algorithms that map actions to outcomes. Here, we tested the engagement of these mechanisms and their neural basis as participants learned to avoid painful electric shocks for themselves and a stranger. We found that model-free decision making was prioritized when learning to avoid harming others compared to oneself. Model-free prediction errors for others relative to self were tracked in the thalamus/caudate. At the time of choice, neural activity consistent with model-free moral learning was observed in subgenual anterior cingulate cortex (sgACC), and switching after harming others was associated with stronger connectivity between sgACC and dorsolateral prefrontal cortex. Finally, model-free moral learning varied with individual differences in moral judgment. Our findings suggest moral learning favors efficiency over flexibility and is underpinned by specific neural mechanisms.