Distinct prediction errors in mesostriatal circuits of the human brain mediate learning about the values of both states and actions: evidence from high-resolution fMRI.

Prediction-error signals consistent with formal models of "reinforcement learning" (RL) have repeatedly been found within dopaminergic nuclei of the midbrain and dopaminoceptive areas of the striatum. However, the precise form of the RL algorithms implemented in the human brain is not yet well determined. Here, we created a novel paradigm optimized to dissociate the subtypes of reward-prediction errors that function as the key computational signatures of two distinct classes of RL models-namely, "actor/critic" models and action-value-learning models (e.g., the Q-learning model). The state-value-prediction error (SVPE), which is independent of actions, is a hallmark of the actor/critic architecture, whereas the action-value-prediction error (AVPE) is the distinguishing feature of action-value-learning algorithms. To test for the presence of these prediction-error signals in the brain, we scanned human participants with a high-resolution functional magnetic-resonance imaging (fMRI) protocol optimized to enable measurement of neural activity in the dopaminergic midbrain as well as the striatal areas to which it projects. In keeping with the actor/critic model, the SVPE signal was detected in the substantia nigra. The SVPE was also clearly present in both the ventral striatum and the dorsal striatum. However, alongside these purely state-value-based computations we also found evidence for AVPE signals throughout the striatum. These high-resolution fMRI findings suggest that model-free aspects of reward learning in humans can be explained algorithmically with RL in terms of an actor/critic mechanism operating in parallel with a system for more direct action-value learning.

Distinct Contributions of Ventromedial and Dorsolateral Subregions of the Human Substantia Nigra to Appetitive and Aversive Learning.

The role of neurons in the substantia nigra (SN) and ventral tegmental area (VTA) of the midbrain in contributing to the elicitation of reward prediction errors during appetitive learning has been well established. Less is known about the differential contribution of these midbrain regions to appetitive versus aversive learning, especially in humans. Here we scanned human participants with high-resolution fMRI focused on the SN and VTA while they participated in a sequential Pavlovian conditioning paradigm involving an appetitive outcome (a pleasant juice), as well as an aversive outcome (an unpleasant bitter and salty flavor). We found a degree of regional specialization within the SN: Whereas a region of ventromedial SN correlated with a temporal difference reward prediction error during appetitive Pavlovian learning, a dorsolateral area correlated instead with an aversive expected value signal in response to the most distal cue, and to a reward prediction error in response to the most proximal cue to the aversive outcome. Furthermore, participants' affective reactions to both the appetitive and aversive conditioned stimuli more than 1 year after the fMRI experiment was conducted correlated with activation in the ventromedial and dorsolateral SN obtained during the experiment, respectively. These findings suggest that, whereas the human ventromedial SN contributes to long-term learning about rewards, the dorsolateral SN may be particularly important for long-term learning in aversive contexts. SIGNIFICANCE STATEMENT: The role of the substantia nigra (SN) and ventral tegmental area (VTA) in appetitive learning is well established, but less is known about their contribution to aversive compared with appetitive learning, especially in humans. We used high-resolution fMRI to measure activity in the SN and VTA while participants underwent higher-order Pavlovian learning. We found a regional specialization within the SN: a ventromedial area was selectively engaged during appetitive learning, and a dorsolateral area during aversive learning. Activity in these areas predicted affective reactions to appetitive and aversive conditioned stimuli over 1 year later. These findings suggest that, whereas the human ventromedial SN contributes to long-term learning about rewards, the dorsolateral SN may be particularly important for long-term learning in aversive contexts.

Advance care planning in chronic liver disease.

The Danish Health Authority recommends that all patients with life threatening disease, regardless of the diagnosis, are offered palliative care with respect for individual goals of care. Only few studies have investigated the evidence of ACP in patients with decompensated liver cirrhosis. This review defines ways to identify patients with decompensated liver cirrhosis in need of palliative care and how to analyse the goals of care. We present a strategy for ACP-conversations and how to implement these in the daily clinical work.

Integration of reinforcement learning and optimal decision-making theories of the basal ganglia.

This article seeks to integrate two sets of theories describing action selection in the basal ganglia: reinforcement learning theories describing learning which actions to select to maximize reward and decision-making theories proposing that the basal ganglia selects actions on the basis of sensory evidence accumulated in the cortex. In particular, we present a model that integrates the actor-critic model of reinforcement learning and a model assuming that the cortico-basal-ganglia circuit implements a statistically optimal decision-making procedure. The values of cortico-striatal weights required for optimal decision making in our model differ from those provided by standard reinforcement learning models. Nevertheless, we show that an actor-critic model converges to the weights required for optimal decision making when biologically realistic limits on synaptic weights are introduced. We also describe the model's predictions concerning reaction times and neural responses during learning, and we discuss directions required for further integration of reinforcement learning and optimal decision-making theories.

PR interval prolongation and 1-year mortality among emergency department patients: a multicentre transnational cohort study.

OBJECTIVES: Emerging evidence supports that PR interval prolongation is associated with increased mortality. However, most previous studies have limited confounder control, and clinical impact in a population of acute ill patients is unknown. The aim of this study was to investigate whether 1-year all-cause mortality was increased in patients presenting with PR interval prolongation in the emergency department (ED). DESIGN AND SETTING: We conducted a register-based cohort study in two Swedish and two Danish EDs. We included all adult patients with an ECG performed at arrival to the Danish EDs during March 2013 to May 2014 and Swedish EDs during January 2010 to January 2011. Using propensity score matching, we analysed HR for 1-year all-cause mortality comparing patients with PR interval prolongation (>200 ms) and normal PR interval (120-200 ms). PARTICIPANTS AND RESULTS: We included 106 124 patients. PR interval prolongation occurred in 8.9% (95% CI 8.7% to 9.0%); these patients were older and had more comorbidity than those with a normal PR interval. The absolute 1-year risk of death was 13% (95% CI 12.3% to 13.7%) for patients with PR interval prolongation and 7.9% (95% CI 7.7% to 8.0%) for those without. After confounder adjustments by propensity score matching, PR interval prolongation showed no association with 1-year mortality with a HR of 1.00 (95% CI 0.93% to 1.08%). CONCLUSION: PR interval prolongation does not constitute an independent risk factor for 1-year mortality in ED patients.

Posterior weighted reinforcement learning with state uncertainty.

Reinforcement learning models generally assume that a stimulus is presented that allows a learner to unambiguously identify the state of nature, and the reward received is drawn from a distribution that depends on that state. However, in any natural environment, the stimulus is noisy. When there is state uncertainty, it is no longer immediately obvious how to perform reinforcement learning, since the observed reward cannot be unambiguously allocated to a state of the environment. This letter addresses the problem of incorporating state uncertainty in reinforcement learning models. We show that simply ignoring the uncertainty and allocating the reward to the most likely state of the environment results in incorrect value estimates. Furthermore, using only the information that is available before observing the reward also results in incorrect estimates. We therefore introduce a new technique, posterior weighted reinforcement learning, in which the estimates of state probabilities are updated according to the observed rewards (e.g., if a learner observes a reward usually associated with a particular state, this state becomes more likely). We show analytically that this modified algorithm can converge to correct reward estimates and confirm this with numerical experiments. The algorithm is shown to be a variant of the expectation-maximization algorithm, allowing rigorous convergence analyses to be carried out. A possible neural implementation of the algorithm in the cortico-basal-ganglia-thalamic network is presented, and experimental predictions of our model are discussed.

Real-world 10-year experiences with intravitreal treatment with ranibizumab and aflibercept for neovascular age-related macular degeneration.

PURPOSE: To report 10-year, real-world experiences with intravitreal therapy (IVT) using vascular endothelial growth factor inhibitors for neovascular age-related macular degeneration (nAMD). METHODS: Retrospective single-centre review of IVT-log 2007-2019 with a treatment-as-needed regimen and ETDRS visual acuity charts. RESULTS: The 4,678 treatment-naïve eyes of 3,668 patients received a mean of 5.4 IVT in the first year and 4.0-4.3 IVT yearly thereafter. Baseline mean best corrected visual acuity (BCVA) was 57.9 (±16.4) letters (6/18) that improved a mean +2.1 (±0.2) letters at the first follow-up visit and gradually declined to -5.0 (±2.2) letters after 10 years. At baseline, there were 29% with BCVA ≥6/12. This proportion increased to 31-37% until year 9. There were 8% with BCVA loss of ≥3 lines at the first follow-up visit increasing to 34% after 10 years. Poorer baseline BCVA was associated with larger increase in BCVA (p < 0.0001, multiple linear regression). The 2,566 (55%) discontinued eyes had a mean baseline BCVA of 56.9 (±16.4) letters compared with 61.5 (±15.9) letters for eyes remaining in treatment. In year 0-7, the discontinued eyes lost an additional mean 2-4 letters (last observation carried forward) but were similar thereafter. There were 12.6% (74 of 585 eligible eyes) that were still in treatment after 10 years. At baseline, 10% had bilateral nAMD. Of patients with unilateral presentation, 17% had received fellow-eye IVT after 5 years. CONCLUSION: A treatment-as-needed regimen stabilized BCVA in active nAMD up to 10 years in most eyes. Baseline BCVA was the most important prognostic factor.

Uncovering the spatio-temporal dynamics of value-based decision-making in the human brain: a combined fMRI-EEG study.

While there is a growing body of functional magnetic resonance imaging (fMRI) evidence implicating a corpus of brain regions in value-based decision-making in humans, the limited temporal resolution of fMRI cannot address the relative temporal precedence of different brain regions in decision-making. To address this question, we adopted a computational model-based approach to electroencephalography (EEG) data acquired during a simple binary choice task. fMRI data were also acquired from the same participants for source localization. Post-decision value signals emerged 200 ms post-stimulus in a predominantly posterior source in the vicinity of the intraparietal sulcus and posterior temporal lobe cortex, alongside a weaker anterior locus. The signal then shifted to a predominantly anterior locus 850 ms following the trial onset, localized to the ventromedial prefrontal cortex and lateral prefrontal cortex. Comparison signals between unchosen and chosen options emerged late in the trial at 1050 ms in dorsomedial prefrontal cortex, suggesting that such comparison signals may not be directly associated with the decision itself but rather may play a role in post-decision action selection. Taken together, these results provide us new insights into the temporal dynamics of decision-making in the brain, suggesting that for a simple binary choice task, decisions may be encoded predominantly in posterior areas such as intraparietal sulcus, before shifting anteriorly.

Initiation and termination of integration in a decision process.

Experimental data indicate that simple motor decisions in vertebrates are preceded by integration of evidence in certain cortical areas, and that the competition between them is resolved in the basal ganglia. While the occurrence of cortical integration is well established, it is not yet clear exactly how the integration occurs. Several models have been proposed, including the race model, the feed forward inhibition (FFI) model and the leaky competing accumulator (LCA) model. In this paper we establish qualitative and quantitative differences between the above mentioned models, with respect to how they are able to initiate the integration process without integrating noise prior to stimulus onset, as well as the models' ability to terminate the integration after a decision has been made, to ensure the possibility of subsequent decisions. Our results show that the LCA model has advantages over the race model and the FFI model in both respects, leading to shorter decision times and an effective termination process.