Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents.

How do societies learn and maintain social norms? Here we use multiagent reinforcement learning to investigate the learning dynamics of enforcement and compliance behaviors. Artificial agents populate a foraging environment and need to learn to avoid a poisonous berry. Agents learn to avoid eating poisonous berries better when doing so is taboo, meaning the behavior is punished by other agents. The taboo helps overcome a credit assignment problem in discovering delayed health effects. Critically, introducing an additional taboo, which results in punishment for eating a harmless berry, further improves overall returns. This "silly rule" counterintuitively has a positive effect because it gives agents more practice in learning rule enforcement. By probing what individual agents have learned, we demonstrate that normative behavior relies on a sequence of learned skills. Learning rule compliance builds upon prior learning of rule enforcement by other agents. Our results highlight the benefit of employing a multiagent reinforcement learning computational model focused on learning to implement complex actions.

The relationship between self-reported physical frailty and sensor-based physical activity measures in older adults - a multicentric cross-sectional study.

BACKGROUND: The decline in everyday life physical activity reflects and contributes to the frailty syndrome. While especially self-reported frailty assessments have the advantage of reaching large groups at low costs, little is known about the relationship between the self-report and objective measured daily physical activity behavior. The main objective was to evaluate whether and to what extent a self-reported assessment of frailty is associated with daily physical activity patterns. METHODS: Daily activity data were obtained from 88 elderly participants (mean 80.6 ± 9.1 years) over up to 21 days. Acceleration data were collected via smartwatch. According to the results of a self-report frailty questionnaire, participants were retrospectively split up into three groups, F (frail, n = 43), P (pre-frail, n = 33), and R (robust, n = 12). Gait- and activity-related measures were derived from the built-in step detector and acceleration sensor and comprised, i.a., standard deviation of 5-s-mean amplitude deviation (MADstd), median MAD (MADmedian), and the 95th percentile of cadence (STEP95). Parameters were fed into a PCA and component scores were used to derive behavioral clusters. RESULTS: The PCA suggested two components, one describing gait and one upper limb activity. Mainly gait related parameters showed meaningful associations with the self-reported frailty score (STEP95: R2 = 0.25), while measures of upper limb activity had lower coefficients (MADmedian: R2 = 0.07). Cluster analysis revealed two clusters with low and relatively high activity in both dimensions (cluster 2 and 3). Interestingly, a third cluster (cluster 1) was characterized by high activity and low extent of ambulation. Comparisons between the clusters showed significant differences between activity, gait, age, sex, number of chronic diseases, health status, and walking aid. Particularly, cluster 1 contained a higher number of female participants, whose self-reports tended towards a low health status, the frequent use of a walking aid, and a higher score related to frailty questions. CONCLUSIONS: The results demonstrate that subjective frailty assessments may be a simple first screening approach. However, especially older women using walking aids may classify themselves as frail despite still being active. Therefore, the results of self-reports may be particularly biased in older women.

What is the simplest model that can account for high-fidelity imitation?

What inductive biases must be incorporated into multi-agent artificial intelligence models to get them to capture high-fidelity imitation? We think very little is needed. In the right environments, both instrumental- and ritual-stance imitation can emerge from generic learning mechanisms operating on non-deliberative decision architectures. In this view, imitation emerges from trial-and-error learning and does not require explicit deliberation.

The Neural Representation of Prospective Choice during Spatial Planning and Decisions.

We are remarkably adept at inferring the consequences of our actions, yet the neuronal mechanisms that allow us to plan a sequence of novel choices remain unclear. We used functional magnetic resonance imaging (fMRI) to investigate how the human brain plans the shortest path to a goal in novel mazes with one (shallow maze) or two (deep maze) choice points. We observed two distinct anterior prefrontal responses to demanding choices at the second choice point: one in rostrodorsal medial prefrontal cortex (rd-mPFC)/superior frontal gyrus (SFG) that was also sensitive to (deactivated by) demanding initial choices and another in lateral frontopolar cortex (lFPC), which was only engaged by demanding choices at the second choice point. Furthermore, we identified hippocampal responses during planning that correlated with subsequent choice accuracy and response time, particularly in mazes affording sequential choices. Psychophysiological interaction (PPI) analyses showed that coupling between the hippocampus and rd-mPFC increases during sequential (deep versus shallow) planning and is higher before correct versus incorrect choices. In short, using a naturalistic spatial planning paradigm, we reveal how the human brain represents sequential choices during planning without extensive training. Our data highlight a network centred on the cortical midline and hippocampus that allows us to make prospective choices while maintaining initial choices during planning in novel environments.

Context-specific activation of hippocampus and SN/VTA by reward is related to enhanced long-term memory for embedded objects.

Animal studies indicate that hippocampal representations of environmental context modulate reward-related processing in the substantia nigra and ventral tegmental area (SN/VTA), a major origin of dopamine in the brain. Using functional magnetic resonance imaging (fMRI) in humans, we investigated the neural specificity of context-reward associations under conditions where the presence of perceptually similar neutral contexts imposed high demands on a putative hippocampal function, pattern separation. The design also allowed us to investigate how contextual reward enhances long-term memory for embedded neutral objects. SN/VTA activity underpinned specific context-reward associations in the face of perceptual similarity. A reward-related enhancement of long-term memory was restricted to the condition where the rewarding and the neutral contexts were perceptually similar, and in turn was linked to co-activation of the hippocampus (subfield DG/CA3) and SN/VTA. Thus, an ability of contextual reward to enhance memory for focal objects is closely linked to context-related engagement of hippocampal-SN/VTA circuitry.

Basal Ganglia Activity Mirrors a Benefit of Action and Reward on Long-Lasting Event Memory.

The expectation of reward is known to enhance a consolidation of long-term memory for events. We tested whether this effect is driven by positive valence or action requirements tied to expected reward. Using a functional magnetic resonance imaging (fMRI) paradigm in young adults, novel images predicted gain or loss outcomes, which in turn were either obtained or avoided by action or inaction. After 24 h, memory for these images reflected a benefit of action as well as a congruence of action requirements and valence, namely, action for reward and inaction for avoidance. fMRI responses in the hippocampus, a region known to be critical for long-term memory function, reflected the anticipation of inaction. In contrast, activity in the putamen mirrored the congruence of action requirement and valence, whereas other basal ganglia regions mirrored overall action benefits on long-lasting memory. The findings indicate a novel type of functional division between the hippocampus and the basal ganglia in the motivational regulation of long-term memory consolidation, which favors remembering events that are worth acting for.

Is choice-induced preference change long lasting?

The idea that decisions alter preferences has had a considerable influence on the field of psychology and underpins cognitive dissonance theory. Yet it is unknown whether choice-induced changes in preferences are long lasting or are transient manifestations seen in the immediate aftermath of decisions. In the research reported here, we investigated whether these changes in preferences are fleeting or stable. Participants rated vacation destinations before making hypothetical choices between destinations, immediately afterward, and 2.5 to 3 years later. We found that choices altered preferences both immediately after being made and after the delay. These changes could not be accounted for by participants' preexisting preferences, and they occurred only when participants made the choices themselves. Our findings provide evidence that making a decision can lead to enduring change in preferences.

Human-centred mechanism design with Democratic AI.

Building artificial intelligence (AI) that aligns with human values is an unsolved problem. Here we developed a human-in-the-loop research pipeline called Democratic AI, in which reinforcement learning is used to design a social mechanism that humans prefer by majority. A large group of humans played an online investment game that involved deciding whether to keep a monetary endowment or to share it with others for collective benefit. Shared revenue was returned to players under two different redistribution mechanisms, one designed by the AI and the other by humans. The AI discovered a mechanism that redressed initial wealth imbalance, sanctioned free riders and successfully won the majority vote. By optimizing for human preferences, Democratic AI offers a proof of concept for value-aligned policy innovation.

What can classic Atari video games tell us about the human brain?

In this issue of Neuron, Cross et. al (2021) use a deep reinforcement learning algorithm to understand human neural activation evoked by playing different video games. The results shed light on the neural principles underlying reward-guided decisions in naturalistic domains.

What is a "Good Life": Protocol for a qualitative study to explore the viewpoint of older persons.

INTRODUCTION: Over the last fifteen years, Living Labs have been on the rise in Europe to bridge the gap between service providers, and the needs of end-users, and to speed up innovation, particularly in the field of healthcare and ageing. Ageing tends to be considered by institutions as a set of risks to be managed for older persons, illustrated in particular via the concepts of "ageing well" or "successful ageing". In this context, this project aims to define the meaning and the conditions for a good life from the point of view of older persons themselves, thereby improving institutions' recognition and support of older persons' ways of living well, rather than imposing a general definition of "successful ageing" based on functional capacity. METHODS AND ANALYSIS: This qualitative study is designed as an action research underpinned by a Living-Lab approach to co-creation. The aims are to: define the conditions for a good life as accurately as possible with older persons (Step 1); share these findings with different healthcare and service providers to adjust existing services or create new ones (Step 2); and disseminate them more broadly within the regions under study and across the scientific community (Step 3). During Step 1, the features of a "good life" will be analysed in a socio-anthropological study based on semi-directed interviews and observations made in the homes of 70 elderly people living in a wide range of accommodation types and regions. In accordance with French legislation, and as confirmed by our formal Ethics Committee, this study does not require approval. The dissemination stage is integrated into the design of this action research, and notably will provide for the appropriation of research findings by the partners of this study, by setting up creativity sessions (Step 2) and by sharing the general findings through panel discussions bringing together regional and national stakeholders (Step 3).

Human hippocampal theta oscillations reflect sequential dependencies during spatial planning.

Movement-related theta oscillations in rodent hippocampus coordinate 'forward sweeps' of location-specific neural activity that could be used to evaluate spatial trajectories online. This raises the possibility that increases in human hippocampal theta power accompany the evaluation of upcoming spatial choices. To test this hypothesis, we measured neural oscillations during a spatial planning task that closely resembles a perceptual decision-making paradigm. In this task, participants searched visually for the shortest path between a start and goal location in novel mazes that contained multiple choice points, and were subsequently asked to make a spatial decision at one of those choice points. We observed ~4-8 Hz hippocampal/medial temporal lobe theta power increases specific to sequential planning that were negatively correlated with subsequent decision speed, where decision speed was inversely correlated with choice accuracy. These results implicate the hippocampal theta rhythm in decision tree search during planning in novel environments.

Big-Loop Recurrence within the Hippocampal System Supports Integration of Information across Episodes.

Recent evidence challenges the widely held view that the hippocampus is specialized for episodic memory, by demonstrating that it also underpins the integration of information across experiences. Contemporary computational theories propose that these two contrasting functions can be accomplished by big-loop recurrence, whereby the output of the system is recirculated back into the hippocampus. We use ultra-high-resolution fMRI to provide support for this hypothesis, by showing that retrieved information is presented as a new input on the superficial entorhinal cortex-driven by functional connectivity between the deep and superficial entorhinal layers. Further, the magnitude of this laminar connectivity correlated with inferential performance, demonstrating its importance for behavior. Our findings offer a novel perspective on information processing within the hippocampus and support a unifying framework in which the hippocampus captures higher-order structure across experiences, by creating a dynamic memory space from separate episodic codes for individual experiences.

Retrieval-Based Model Accounts for Striking Profile of Episodic Memory and Generalization.

A fundamental theoretical tension exists between the role of the hippocampus in generalizing across a set of related episodes, and in supporting memory for individual episodes. Whilst the former requires an appreciation of the commonalities across episodes, the latter emphasizes the representation of the specifics of individual experiences. We developed a novel version of the hippocampal-dependent paired associate inference (PAI) paradigm, which afforded us the unique opportunity to investigate the relationship between episodic memory and generalization in parallel. Across four experiments, we provide surprising evidence that the overlap between object pairs in the PAI paradigm results in a marked loss of episodic memory. Critically, however, we demonstrate that superior generalization ability was associated with stronger episodic memory. Through computational simulations we show that this striking profile of behavioral findings is best accounted for by a mechanism by which generalization occurs at the point of retrieval, through the recombination of related episodes on the fly. Taken together, our study offers new insights into the intricate relationship between episodic memory and generalization, and constrains theories of the mechanisms by which the hippocampus supports generalization.

Stimulus Novelty Energizes Actions in the Absence of Explicit Reward.

Novelty seeking has been tied to impulsive choice and biased value based choice. It has been postulated that novel stimuli should trigger more vigorous approach and exploration. However, it is unclear whether stimulus novelty can enhance simple motor actions in the absence of explicit reward, a necessary condition for energizing approach and exploration in an entirely unfamiliar situation. In this study human subjects were cued to omit or perform actions in form of button presses by novel or familiar images. We found that subjects' motor actions were faster when cued by a novel compared to a familiar image. This facilitation by novelty was strongest when the delay between cue and action was short, consistent with a link between novelty and impulsive choices. The facilitation of reaction times by novelty was correlated across subjects with trait novelty seeking as measured in the Tridimensional Personality Questionnaire. However, this li between high novelty-seeking and action facilitation was driven by trials with a long delay between cue and action. This prolonged time window of energization following novelty could hint at a mechanistic underpinning of enhanced vigour for approach and exploration frequently postulated for novelty seeking humans. In conclusion, we show that stimulus novelty enhances the speed of a cued motor action. We suggest this is likely to reflect an adaptation to changing environments but may also provide a source of maladaptive choice and impulsive behaviour.

Selective perturbation of cognitive conflict in the human brain-A combined fMRI and rTMS study.

We investigated if single and double conflicts are processed separately in different brain regions and if they are differentially vulnerable to TMS perturbation. Fifteen human volunteers performed a single (Flanker or Simon) conflict task or a double (Flanker and Simon) conflict task in a combined functional Magnetic Resonance Imaging (fMRI) and Transcranial Magnetic Stimulation (TMS) study. The fMRI approach aimed at localizing brain regions involved in interference resolution induced by single Flanker (stimulus-stimulus, S-S) and Simon (stimulus-response, S-R) conflicts as well as regions involved in the double conflict condition. The data revealed a distinct activation in the right intraparietal sulcus (IPS) for Flanker interference and in the right middle frontal gyrus (MFG) for the double interference condition. The causal functional role of these brain regions was then examined in the same volunteers by using offline TMS over right IPS and right MFG. TMS perturbation of the right IPS increased the Flanker effect, but had no effect in the Simon or double conflict condition. In contrast, perturbation of the right MFG had no effect on any of the conflict types. These findings suggest a causal role of the right IPS in the processing of the single conflict of Flanker (stimulus-stimulus) interference.

Action and valence modulate choice and choice-induced preference change.

Choices are not only communicated via explicit actions but also passively through inaction. In this study we investigated how active or passive choice impacts upon the choice process itself as well as a preference change induced by choice. Subjects were tasked to select a preference for unfamiliar photographs by action or inaction, before and after they gave valuation ratings for all photographs. We replicate a finding that valuation increases for chosen items and decreases for unchosen items compared to a control condition in which the choice was made post re-evaluation. Whether choice was expressed actively or passively affected the dynamics of revaluation differently for positive and negatively valenced items. Additionally, the choice itself was biased towards action such that subjects tended to choose a photograph obtained by action more often than a photographed obtained through inaction. These results highlight intrinsic biases consistent with a tight coupling of action and reward and add to an emerging understanding of how the mode of action itself, and not just an associated outcome, modulates the decision making process.

How beliefs about self-creation inflate value in the human brain.

Humans have a tendency to overvalue their own ideas and creations. Understanding how these errors in judgement emerge is important for explaining suboptimal decisions, as when individuals and groups choose self-created alternatives over superior or equal ones. We show that such overvaluation is a reconstructive process that emerges when participants believe they have created an item, regardless of whether this belief is true or false. This overvaluation is observed both when false beliefs of self-creation are elicited (Experiment 1) or implanted (Experiment 2). Using brain imaging data we highlight the brain processes mediating an interaction between value and belief of self-creation. Specifically, following the creation manipulation there is an increased functional connectivity during valuation between the right caudate nucleus, where we show BOLD activity correlated with subjective value, and the left amygdala, where we show BOLD activity is linked to subjective belief. Our study highlights psychological and neurobiological processes through which false beliefs alter human valuation and in doing so throw light on a common source of error in judgements of value.