Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration.

When facing an unfamiliar environment, animals need to explore to gain new knowledge about which actions provide reward, but also put the newly acquired knowledge to use as quickly as possible. Optimal reinforcement learning strategies should therefore assess the uncertainties of these action-reward associations and utilise them to inform decision making. We propose a novel model whereby direct and indirect striatal pathways act together to estimate both the mean and variance of reward distributions, and mesolimbic dopaminergic neurons provide transient novelty signals, facilitating effective uncertainty-driven exploration. We utilised electrophysiological recording data to verify our model of the basal ganglia, and we fitted exploration strategies derived from the neural model to data from behavioural experiments. We also compared the performance of directed exploration strategies inspired by our basal ganglia model with other exploration algorithms including classic variants of upper confidence bound (UCB) strategy in simulation. The exploration strategies inspired by the basal ganglia model can achieve overall superior performance in simulation, and we found qualitatively similar results in fitting model to behavioural data compared with the fitting of more idealised normative models with less implementation level detail. Overall, our results suggest that transient dopamine levels in the basal ganglia that encode novelty could contribute to an uncertainty representation which efficiently drives exploration in reinforcement learning.

Neural signature of flexible coding in prefrontal cortex.

The ability of prefrontal cortex to quickly encode novel associations is crucial for adaptive behavior and central to working memory. Fast Hebbian changes in synaptic strength permit forming new associations, but neuronal signatures of this have been elusive. We devised a trialwise index of pattern similarity to look for rapid changes in population codes. Based on a computational model of working memory, we hypothesized that synaptic strength-and consequently, the tuning of neurons-could change if features of a subsequent stimulus need to be "reassociated," i.e., if bindings between features need to be broken to encode the new item. As a result, identical stimuli might elicit different neural responses. As predicted, neural response similarity dropped following rebinding, but only in prefrontal cortex. The history-dependent changes were expressed on top of traditional, fixed selectivity and were not explainable by carryover of previous firing into the current trial or by neural adaptation.

Reward insensitivity is associated with dopaminergic deficit in rapid eye movement sleep behaviour disorder.

Idiopathic rapid eye movement sleep behaviour disorder (iRBD) has now been established as an important marker of the prodromal stage of Parkinson's disease and related synucleinopathies. However, although dopamine transporter single photon emission computed tomography (SPECT) has been used to demonstrate the presence of nigro-striatal deficit in iRBD, quantifiable correlates of this are currently lacking. Sensitivity to rewarding stimuli is reduced in some people with Parkinson's disease, potentially contributing to aspects of the neuropsychiatric phenotype in these individuals. Furthermore, a role for dopaminergic degeneration is suggested by the fact that reward insensitivity can be improved by dopaminergic medications. Patients with iRBD present a unique opportunity to study the relationship between reward sensitivity and early dopaminergic deficit in the unmedicated state. Here, we investigate whether a non-invasive, objective measure of reward sensitivity might be a marker of dopaminergic status in prodromal Parkinson's disease by comparing with SPECT/CT measurement of dopaminergic loss in the basal ganglia. Striatal dopaminergic deficits in iRBD are associated with progression to Parkinsonian disorders. Therefore, identification of a clinically measurable correlate of this degenerative process might provide a basis for the development of novel risk stratification tools. Using a recently developed incentivized eye-tracking task, we quantified reward sensitivity in a cohort of 41 patients with iRBD and compared this with data from 40 patients with Parkinson's disease and 41 healthy controls. Patients with iRBD also underwent neuroimaging with dopamine transporter SPECT/CT. Overall, reward sensitivity, indexed by pupillary response to monetary incentives, was reduced in iRBD cases compared with controls and was not significantly different to that in patients with Parkinson's disease. However, in iRBD patients with normal dopamine transporter SPECT/CT imaging, reward sensitivity was not significantly different from healthy controls. Across all iRBD cases, a positive association was observed between reward sensitivity and dopaminergic SPECT/CT signal in the putamen. These findings demonstrate a direct relationship between dopaminergic deficit and reward sensitivity in patients with iRBD and suggest that measurement of pupillary responses could be of value in models of risk stratification and disease progression in these individuals.

Human lesions and animal studies link the claustrum to perception, salience, sleep and pain.

The claustrum is the most densely interconnected region in the human brain. Despite the accumulating data from clinical and experimental studies, the functional role of the claustrum remains unknown. Here, we systematically review claustrum lesion studies and discuss their functional implications. Claustral lesions are associated with an array of signs and symptoms, including changes in cognitive, perceptual and motor abilities; electrical activity; mental state; and sleep. The wide range of symptoms observed following claustral lesions do not provide compelling evidence to support prominent current theories of claustrum function such as multisensory integration or salience computation. Conversely, the lesions studies support the hypothesis that the claustrum regulates cortical excitability. We argue that the claustrum is connected to, or part of, multiple brain networks that perform both fundamental and higher cognitive functions. As a multifunctional node in numerous networks, this may explain the manifold effects of claustrum damage on brain and behaviour.

Neural Reinstatement Tracks Spread of Attention between Object Features in Working Memory.

Attention can be allocated in working memory (WM) to select and privilege relevant content. It is unclear whether attention selects individual features or whole objects in WM. Here, we used behavioral measures, eye-tracking, and EEG to test the hypothesis that attention spreads between an object's features in WM. Twenty-six participants completed a WM task that asked them to recall the angle of one of two oriented, colored bars after a delay while EEG and eye-tracking data were collected. During the delay, an orthogonal "incidental task" cued the color of one item for a match/mismatch judgment. On congruent trials (50%), the cued item was probed for subsequent orientation recall; on incongruent trials (50%), the other memory item was probed. As predicted, selecting the color of an object in WM brought other features of the cued object into an attended state as revealed by EEG decoding, oscillatory α-power, gaze bias, and improved orientation recall performance. Together, the results show that attentional selection spreads between an object's features in WM, consistent with object-based attentional selection. Analyses of neural processing at recall revealed that the selected object was automatically compared with the probe, whether it was the target for recall or not. This provides a potential mechanism for the observed benefits of nonpredictive cueing in WM, where a selected item is prioritized for subsequent decision-making.

Motivation improves working memory by two processes: Prioritisation and retrieval thresholds.

Motivation can improve performance when the potential rewards outweigh the cost of effort expended. In working memory (WM), people can prioritise rewarded items at the expense of unrewarded items, suggesting a fixed memory capacity. But can capacity itself change with motivation? Across four experiments (N = 30-34) we demonstrate motivational improvements in WM even when all items were rewarded. However, this was not due to better memory precision, but rather better selection of the probed item within memory. Motivational improvements operated independently of encoding, maintenance, or attention shifts between items in memory. Moreover, motivation slowed responses. This contrasted with the benefits of rewarding items unequally, which allowed prioritisation of one item over another. We conclude that motivation can improve memory recall, not via precision or capacity, but via speed-accuracy trade-offs when selecting the item to retrieve.

Dopamine increases risky choice while D2 blockade shortens decision time.

Dopamine is crucially involved in decision-making and overstimulation within dopaminergic pathways can lead to impulsive behaviour, including a desire to take risks and reduced deliberation before acting. These behavioural changes are side effects of treatment with dopaminergic drugs in Parkinson disease, but their likelihood of occurrence is difficult to predict and may be influenced by the individual's baseline endogenous dopamine state, and indeed correlate with sensation-seeking personality traits. We here collected data on a standard gambling task in healthy volunteers given either placebo, 2.5 mg of the dopamine antagonist haloperidol or 100/25 mg of the dopamine precursor levodopa in a within-subject design. We found an increase in risky choices on levodopa. Choices were, however, made faster on haloperidol with no effect of levodopa on deliberation time. Shortened deliberation times on haloperidol occurred in low sensation-seekers only, suggesting a correlation between sensation-seeking personality trait and baseline dopamine levels. We hypothesise that levodopa increases risk-taking behaviour via overstimulation at both D1 and D2 receptor level, while a single low dose of haloperidol, as previously reported (Frank and O'Reilly 2006), may block D2 receptors pre- and post-synaptically and may paradoxically lead to higher striatal dopamine acting on remaining striatal D1 receptors, causing speedier decision without influencing risk tolerance. These effects could also fit with a recently proposed computational model of the basal ganglia (Moeller and Bogacz 2019; Moeller et al. 2021). Furthermore, our data suggest that the actual dopaminergic drug effect may be dependent on the individual's baseline dopamine state, which may influence our therapeutic decision as clinicians in the future.

Modulation of the pupillary response by the content of visual working memory.

Studies of selective attention during perception have revealed modulation of the pupillary response according to the brightness of task-relevant (attended) vs. -irrelevant (unattended) stimuli within a visual display. As a strong test of top-down modulation of the pupil response by selective attention, we asked whether changes in pupil diameter follow internal shifts of attention to memoranda of visual stimuli of different brightness maintained in working memory, in the absence of any visual stimulation. Across 3 studies, we reveal dilation of the pupil when participants orient attention to the memorandum of a dark grating relative to that of a bright grating. The effect occurs even when the attention-orienting cue is independent of stimulus brightness, and even when stimulus brightness is merely incidental and not required for the working-memory task of judging stimulus orientation. Furthermore, relative dilation and constriction of the pupil occurred dynamically and followed the changing temporal expectation that 1 or the other stimulus would be probed across the retention delay. The results provide surprising and consistent evidence that pupil responses are under top-down control by cognitive factors, even when there is no direct adaptive gain for such modulation, since no visual stimuli were presented or anticipated. The results also strengthen the view of sensory recruitment during working memory, suggesting even activation of sensory receptors. The thought-provoking corollary to our findings is that the pupils provide a reliable measure of what is in the focus of mind, thus giving a different meaning to old proverbs about the eyes being a window to the mind.

Reward-Based Improvements in Motor Control Are Driven by Multiple Error-Reducing Mechanisms.

Reward has a remarkable ability to invigorate motor behavior, enabling individuals to select and execute actions with greater precision and speed. However, if reward is to be exploited in applied settings, such as rehabilitation, a thorough understanding of its underlying mechanisms is required. In a series of experiments, we first demonstrate that reward simultaneously improves the selection and execution components of a reaching movement. Specifically, reward promoted the selection of the correct action in the presence of distractors, while also improving execution through increased speed and maintenance of accuracy. These results led to a shift in the speed-accuracy functions for both selection and execution. In addition, punishment had a similar impact on action selection and execution, although it enhanced execution performance across all trials within a block, that is, its impact was noncontingent to trial value. Although the reward-driven enhancement of movement execution has been proposed to occur through enhanced feedback control, an untested possibility is that it is also driven by increased arm stiffness, an energy-consuming process that enhances limb stability. Computational analysis revealed that reward led to both an increase in feedback correction in the middle of the movement and a reduction in motor noise near the target. In line with our hypothesis, we provide novel evidence that this noise reduction is driven by a reward-dependent increase in arm stiffness. Therefore, reward drives multiple error-reduction mechanisms which enable individuals to invigorate motor performance without compromising accuracy.SIGNIFICANCE STATEMENT While reward is well-known for enhancing motor performance, how the nervous system generates these improvements is unclear. Despite recent work indicating that reward leads to enhanced feedback control, an untested possibility is that it also increases arm stiffness. We demonstrate that reward simultaneously improves the selection and execution components of a reaching movement. Furthermore, we show that punishment has a similar positive impact on performance. Importantly, by combining computational and biomechanical approaches, we show that reward leads to both improved feedback correction and an increase in stiffness. Therefore, reward drives multiple error-reduction mechanisms which enable individuals to invigorate performance without compromising accuracy. This work suggests that stiffness control plays a vital, and underappreciated, role in the reward-based imporvemenets in motor control.

Hunger improves reinforcement-driven but not planned action.

Human decisions can be reflexive or planned, being governed respectively by model-free and model-based learning systems. These two systems might differ in their responsiveness to our needs. Hunger drives us to specifically seek food rewards, but here we ask whether it might have more general effects on these two decision systems. On one hand, the model-based system is often considered flexible and context-sensitive, and might therefore be modulated by metabolic needs. On the other hand, the model-free system's primitive reinforcement mechanisms may have closer ties to biological drives. Here, we tested participants on a well-established two-stage sequential decision-making task that dissociates the contribution of model-based and model-free control. Hunger enhanced overall performance by increasing model-free control, without affecting model-based control. These results demonstrate a generalized effect of hunger on decision-making that enhances reliance on primitive reinforcement learning, which in some situations translates into adaptive benefits.

A new toolbox to distinguish the sources of spatial memory error.

Studying the sources of errors in memory recall has proven invaluable for understanding the mechanisms of working memory (WM). While one-dimensional memory features (e.g., color, orientation) can be analyzed using existing mixture modeling toolboxes to separate the influence of imprecision, guessing, and misbinding (the tendency to confuse features that belong to different memoranda), such toolboxes are not currently available for two-dimensional spatial WM tasks. Here we present a method to isolate sources of spatial error in tasks where participants have to report the spatial location of an item in memory, using two-dimensional mixture models. The method recovers simulated parameters well and is robust to the influence of response distributions and biases, as well as number of nontargets and trials. To demonstrate the model, we fit data from a complex spatial WM task and show the recovered parameters correspond well with previous spatial WM findings and with recovered parameters on a one-dimensional analogue of this task, suggesting convergent validity for this two-dimensional modeling approach. Because the extra dimension allows greater separation of memoranda and responses, spatial tasks turn out to be much better for separating misbinding from imprecision and guessing than one-dimensional tasks. Code for these models is freely available in the MemToolbox2D package and is integrated to work with the commonly used MATLAB package MemToolbox.

Dopamine Alters the Fidelity of Working Memory Representations according to Attentional Demands.

Capacity limitations in working memory (WM) necessitate the need to effectively control its contents. Here, we examined the effect of cabergoline, a dopamine D2 receptor agonist, on WM using a continuous report paradigm that allowed us to assess the fidelity with which items are stored. We assessed recall performance under three different gating conditions: remembering only one item, being cued to remember one target among distractors, and having to remember all items. Cabergoline had differential effects on recall performance according to whether distractors had to be ignored and whether mnemonic resources could be deployed exclusively to the target. Compared with placebo, cabergoline improved mnemonic performance when there were no distractors but significantly reduced performance when distractors were presented in a precue condition. No significant difference in performance was observed under cabergoline when all items had to be remembered. By applying a stochastic model of response selection, we established that the causes of drug-induced changes in performance were due to changes in the precision with which items were stored in WM. However, there was no change in the extent to which distractors were mistaken for targets. Thus, D2 agonism causes changes in the fidelity of mnemonic representations without altering interference between memoranda.

Magnetic Oculomotor Prosthetics for Acquired Nystagmus.

PURPOSE: Acquired nystagmus, a highly symptomatic consequence of damage to the substrates of oculomotor control, often is resistant to pharmacotherapy. Although heterogeneous in its neural cause, its expression is unified at the effector-the eye muscles themselves-where physical damping of the oscillation offers an alternative approach. Because direct surgical fixation would immobilize the globe, action at a distance is required to damp the oscillation at the point of fixation, allowing unhindered gaze shifts at other times. Implementing this idea magnetically, herein we describe the successful implantation of a novel magnetic oculomotor prosthesis in a patient. DESIGN: Case report of a pilot, experimental intervention. PARTICIPANT: A 49-year-old man with longstanding, medication-resistant, upbeat nystagmus resulting from a paraneoplastic syndrome caused by stage 2A, grade I, nodular sclerosing Hodgkin's lymphoma. METHODS: We designed a 2-part, titanium-encased, rare-earth magnet oculomotor prosthesis, powered to damp nystagmus without interfering with the larger forces involved in saccades. Its damping effects were confirmed when applied externally. We proceeded to implant the device in the patient, comparing visual functions and high-resolution oculography before and after implantation and monitoring the patient for more than 4 years after surgery. MAIN OUTCOME MEASURES: We recorded Snellen visual acuity before and after intervention, as well as the amplitude, drift velocity, frequency, and intensity of the nystagmus in each eye. RESULTS: The patient reported a clinically significant improvement of 1 line of Snellen acuity (from 6/9 bilaterally to 6/6 on the left and 6/5-2 on the right), reflecting an objectively measured reduction in the amplitude, drift velocity, frequency, and intensity of the nystagmus. These improvements were maintained throughout a follow-up of 4 years and enabled him to return to paid employment. CONCLUSIONS: This work opens a new field of implantable therapeutic devices-oculomotor prosthetics-designed to modify eye movements dynamically by physical means in cases where a purely neural approach is ineffective. Applied to acquired nystagmus refractory to all other interventions, it is shown successfully to damp pathologic eye oscillations while allowing normal saccadic shifts of gaze.

Distinct Motivational Effects of Contingent and Noncontingent Rewards.

When rewards are available, people expend more energy, increasing their motivational vigor. In theory, incentives might drive behavior for two distinct reasons: First, they increase expected reward; second, they increase the difference in subjective value between successful and unsuccessful performance, which increases contingency-the degree to which action determines outcome. Previous studies of motivational vigor have never compared these directly. Here, we indexed motivational vigor by measuring the speed of eye movements toward a target after participants heard a cue indicating how outcomes would be determined. Eye movements were faster when the cue indicated that monetary rewards would be contingent on performance than when the cue indicated that rewards would be random. But even when the cue indicated that a reward was guaranteed regardless of speed, movement was still faster than when no reward was available. Motivation by contingent and certain rewards was uncorrelated across individuals, which suggests that there are two separable, independent components of motivation. Contingent motivation generated autonomic arousal, and unlike noncontingent motivation, was effective with penalties as well as rewards.

Goal commitment is supported by vmPFC through selective attention.

When striking a balance between commitment to a goal and flexibility in the face of better options, people often demonstrate strong goal perseveration. Here, using functional MRI (n = 30) and lesion patient (n = 26) studies, we argue that the ventromedial prefrontal cortex (vmPFC) drives goal commitment linked to changes in goal-directed selective attention. Participants performed an incremental goal pursuit task involving sequential decisions between persisting with a goal versus abandoning progress for better alternative options. Individuals with stronger goal perseveration showed higher goal-directed attention in an interleaved attention task. Increasing goal-directed attention also affected abandonment decisions: while pursuing a goal, people lost their sensitivity to valuable alternative goals while remaining more sensitive to changes in the current goal. In a healthy population, individual differences in both commitment biases and goal-oriented attention were predicted by baseline goal-related activity in the vmPFC. Among lesion patients, vmPFC damage reduced goal commitment, leading to a performance benefit.

The effects of pramipexole on motivational vigour during a saccade task: a placebo-controlled study in healthy adults.

Motivation allows us to energise actions when we expect reward and is reduced in depression. This effect, termed motivational vigour, has been proposed to rely on central dopamine, with dopaminergic agents showing promise in the treatment of depression. This suggests that dopaminergic agents might act to reduce depression by increasing the effects of reward or by helping energise actions. The aim of the current study was to investigate whether the dopamine agonist pramipexole enhanced motivational vigour during a rewarded saccade task. In addition, we asked whether the effects of pramipexole on vigour differ between reward contingent on performance and guaranteed reward. Healthy adult participants were randomised to receive either pramipexole (n = 19) or placebo (controls n = 18) for 18 days. The vigour of saccades was measured twice, once before the administration of study medication (Time 1) and after taking it for 12-15 days (Time 2). To separate motivation by contingency vs. reward, saccadic vigour was separately measured when (1) rewards were contingent on performance (2) delivered randomly with matched frequency, (3) when reward was guaranteed, (4) when reward was not present at all. Motivation increased response vigour, as expected. Relative to placebo, pramipexole also increased response vigour. However, there was no interaction, meaning that the effects of reward were not modulated by drug, and there was no differential drug effect on contingent vs. guaranteed rewards. The effect of pramipexole on vigour could not be explained by a speed/accuracy trade-off, nor by autonomic arousal as indexed by pupillary dilation. Chronic D2 stimulation increases general vigour, energising movements in healthy adults irrespective of extrinsic reward.

Remote digital cognitive assessment reveals cognitive deficits related to hippocampal atrophy in autoimmune limbic encephalitis: a cross-sectional validation study.

Background: Autoimmune limbic encephalitis (ALE) is a neurological disease characterised by inflammation of the limbic regions of the brain, mediated by pathogenic autoantibodies. Because cognitive deficits persist following acute treatment of ALE, the accurate assessment of long-term cognitive outcomes is important for clinical assessments and trials. However, evaluating cognition is costly and an unmet need exists for validated digital methods. Methods: In this cross-sectional validation study, we investigated whether a remote digital platform could identify previously characterised cognitive impairments in patients with chronic ALE and whether digital metrics would correlate with standard neuropsychological assessment and hippocampal volume. Patients with ALE who had a chronic and stable presentation and received a clinical diagnosis of ALE were recruited for this study. The cognitive performance of 21 patients with ALE and 54 age-matched healthy controls - enrolled via the University of Oxford (UK) Cognitive Neurology Lab testing programme - was assessed with a battery of 12 cognitive tasks from the Cognitron online platform. The platform was optimised with National Institute for Health and Care Research (NIHR) support to be deliverable remotely to elderly and patient groups. The primary outcome measure was behavioural performance and corresponding neuroimaging and neuropsychological assessment metrics. Findings: Between February 15, 2021, and April 21, 2022, 21 patients with ALE (mean age 63.01 years, 14 males) and 54 healthy controls (mean age 65.56 years, 23 males) completed the digital cognitive assessment. Patients with ALE performed significantly worse in memory, visuospatial abilities, executive function, and language. No impairments in digit & spatial span, target detection (attention) and emotion discrimination were observed. The global score on the online cognitive tasks correlated significantly with the established Addenbrooke's Cognitive Examination III (ACE) pen-and-paper test. Deficits in visuospatial processing and language were identified in ALE compared to controls using remote digital testing but not using the ACE, highlighting higher sensitivity of computerised testing to residual cognitive impairment. Finally, the hippocampal volumes of patients with ALE and healthy controls correlated with online cognitive scores. Interpretation: These findings demonstrate that subtle cognitive deficits in patients with chronic ALE, who often show full recovery in measures of disability and dependence on daily activities, are detectable using a remote online platform, which also relates to hippocampal atrophy. Such methods may facilitate the characterisation of cognitive profiles in complex neurological diseases. Future longitudinal studies designed to assess the utility of such digital methods for further clinical characterisation are needed. Funding: The Wellcome Trust, Medical Research Council, National Institute for Health Research, Rhodes Scholarship, and the Berrow Foundation Scholarship.

Human ventromedial prefrontal cortex is necessary for prosocial motivation.

Ventromedial prefrontal cortex (vmPFC) is vital for decision-making. Functional neuroimaging links vmPFC to processing rewards and effort, while parallel work suggests vmPFC involvement in prosocial behaviour. However, the necessity of vmPFC for these functions is unknown. Patients with rare focal vmPFC lesions (n = 25), patients with lesions elsewhere (n = 15) and healthy controls (n = 40) chose between rest and exerting effort to earn rewards for themselves or another person. vmPFC damage decreased prosociality across behavioural and computational measures. vmPFC patients earned less, discounted rewards by effort more, and exerted less force when another person benefited, compared to both control groups. Voxel-based lesion mapping revealed dissociations between vmPFC subregions. While medial damage led to antisocial behaviour, lateral damage increased prosocial behaviour relative to patients with damage elsewhere. vmPFC patients also showed reduced effort sensitivity overall, but reward sensitivity was limited to specific subregions. These results reveal multiple causal contributions of vmPFC to prosocial behaviour, effort and reward.

The role of the human hippocampus in decision-making under uncertainty.

The role of the hippocampus in decision-making is beginning to be more understood. Because of its prospective and inferential functions, we hypothesized that it might be required specifically when decisions involve the evaluation of uncertain values. A group of individuals with autoimmune limbic encephalitis-a condition known to focally affect the hippocampus-were tested on how they evaluate reward against uncertainty compared to reward against another key attribute: physical effort. Across four experiments requiring participants to make trade-offs between reward, uncertainty and effort, patients with acute limbic encephalitis demonstrated blunted sensitivity to reward and effort whenever uncertainty was considered, despite demonstrating intact uncertainty sensitivity. By contrast, the valuation of these two attributes (reward and effort) was intact on uncertainty-free tasks. Reduced sensitivity to changes in reward under uncertainty correlated with the severity of hippocampal damage. Together, these findings provide evidence for a context-sensitive role of the hippocampus in value-based decision-making, apparent specifically under conditions of uncertainty.