Distributional reinforcement learning in prefrontal cortex.

The prefrontal cortex is crucial for learning and decision-making. Classic reinforcement learning (RL) theories center on learning the expectation of potential rewarding outcomes and explain a wealth of neural data in the prefrontal cortex. Distributional RL, on the other hand, learns the full distribution of rewarding outcomes and better explains dopamine responses. In the present study, we show that distributional RL also better explains macaque anterior cingulate cortex neuronal responses, suggesting that it is a common mechanism for reward-guided learning.

A cognitive map for value-guided choice in ventromedial prefrontal cortex.

The prefrontal cortex is crucial for economic decision-making and representing the value of options. However, how such representations facilitate flexible decisions remains unknown. We reframe economic decision-making in prefrontal cortex in line with representations of structure within the medial temporal lobe because such cognitive map representations are known to facilitate flexible behaviour. Specifically, we framed choice between different options as a navigation process in value space. Here we show that choices in a 2D value space defined by reward magnitude and probability were represented with a grid-like code, analogous to that found in spatial navigation. The grid-like code was present in ventromedial prefrontal cortex (vmPFC) local field potential theta frequency and the result replicated in an independent dataset. Neurons in vmPFC similarly contained a grid-like code, in addition to encoding the linear value of the chosen option. Importantly, both signals were modulated by theta frequency - occurring at theta troughs but on separate theta cycles. Furthermore, we found sharp-wave ripples - a key neural signature of planning and flexible behaviour - in vmPFC, which were modulated by accuracy and reward. These results demonstrate that multiple cognitive map-like computations are deployed in vmPFC during economic decision-making, suggesting a new framework for the implementation of choice in prefrontal cortex.

Human-Robot Interaction With Robust Prediction of Movement Intention Surpasses Manual Control.

Physical human-robot interaction (pHRI) enables a user to interact with a physical robotic device to advance beyond the current capabilities of high-payload and high-precision industrial robots. This paradigm opens up novel applications where a the cognitive capability of a user is combined with the precision and strength of robots. Yet, current pHRI interfaces suffer from low take-up and a high cognitive burden for the user. We propose a novel framework that robustly and efficiently assists users by reacting proactively to their commands. The key insight is to include context- and user-awareness in the controller, improving decision-making on how to assist the user. Context-awareness is achieved by inferring the candidate objects to be grasped in a task or scene and automatically computing plans for reaching them. User-awareness is implemented by facilitating the motion toward the most likely object that the user wants to grasp, as well as dynamically recovering from incorrect predictions. Experimental results in a virtual environment of two degrees of freedom control show the capability of this approach to outperform manual control. By robustly predicting user intention, the proposed controller allows subjects to achieve superhuman performance in terms of accuracy and, thereby, usability.

A causal role of estradiol in human reinforcement learning.

The sex hormone estradiol is hypothesized to play a key role in human cognition, and reward processing specifically, via increased dopamine D1-receptor signalling. However, the effect of estradiol on reward processing in men has never been established. To fill this gap, we performed a double-blind placebo-controlled study in which men (N = 100) received either a single dose of estradiol (2 mg) or a placebo. Subjects performed a probabilistic reinforcement learning task where they had to choose between two options with varying reward probabilities to maximize monetary reward. Results showed that estradiol administration increased reward sensitivity compared to placebo. This effect was observed in subjects' choices, how much weight they assigned to their previous choices, and subjective reports about the reward probabilities. Furthermore, effects of estradiol were moderated by reward sensitivity, as measured through the BIS/BAS questionnaire. Using reinforcement learning models, we found that behavioral effects of estradiol were reflected in increased learning rates. These results demonstrate a causal role of estradiol within the framework of reinforcement learning, by enhancing reward sensitivity and learning. Furthermore, they provide preliminary evidence for dopamine-related genetic variants moderating the effect of estradiol on reward processing.

Active inference on discrete state-spaces: A synthesis.

Active inference is a normative principle underwriting perception, action, planning, decision-making and learning in biological or artificial agents. From its inception, its associated process theory has grown to incorporate complex generative models, enabling simulation of a wide range of complex behaviours. Due to successive developments in active inference, it is often difficult to see how its underlying principle relates to process theories and practical implementation. In this paper, we try to bridge this gap by providing a complete mathematical synthesis of active inference on discrete state-space models. This technical summary provides an overview of the theory, derives neuronal dynamics from first principles and relates this dynamics to biological processes. Furthermore, this paper provides a fundamental building block needed to understand active inference for mixed generative models; allowing continuous sensations to inform discrete representations. This paper may be used as follows: to guide research towards outstanding challenges, a practical guide on how to implement active inference to simulate experimental behaviour, or a pointer towards various in-silico neurophysiological responses that may be used to make empirical predictions.

Integration of audiovisual spatial signals is not consistent with maximum likelihood estimation.

Multisensory perception is regarded as one of the most prominent examples where human behaviour conforms to the computational principles of maximum likelihood estimation (MLE). In particular, observers are thought to integrate auditory and visual spatial cues weighted in proportion to their relative sensory reliabilities into the most reliable and unbiased percept consistent with MLE. Yet, evidence to date has been inconsistent. The current pre-registered, large-scale (N = 36) replication study investigated the extent to which human behaviour for audiovisual localization is in line with maximum likelihood estimation. The acquired psychophysics data show that while observers were able to reduce their multisensory variance relative to the unisensory variances in accordance with MLE, they weighed the visual signals significantly stronger than predicted by MLE. Simulations show that this dissociation can be explained by a greater sensitivity of standard estimation procedures to detect deviations from MLE predictions for sensory weights than for audiovisual variances. Our results therefore suggest that observers did not integrate audiovisual spatial signals weighted exactly in proportion to their relative reliabilities for localization. These small deviations from the predictions of maximum likelihood estimation may be explained by observers' uncertainty about the world's causal structure as accounted for by Bayesian causal inference.