Dynamics of personality: The Zurich model of motivation revived, extended, and applied to personality.

Personality researchers are increasingly interested in the dynamics of personality, that is, the proximal causal mechanisms underlying personality and behavior. Here, we review the Zurich Model of Social Motivation concerning its potential to explain central aspects of personality. It is a cybernetic model that provides a nomothetic structure of the causal relationships among needs for security, arousal, and power, and uses them to explain an individual's approach-avoidance or "proximity-distance" behavior. We review core features of the model and extend them by adding features based on recent behavioral and neuroscientific evidence. We close by discussing the model considering contemporary issues in personality science such as the dynamics of personality, five-factor personality traits and states, and personality growth.

General differential Hebbian learning: Capturing temporal relations between events in neural networks and the brain.

Learning in biologically relevant neural-network models usually relies on Hebb learning rules. The typical implementations of these rules change the synaptic strength on the basis of the co-occurrence of the neural events taking place at a certain time in the pre- and post-synaptic neurons. Differential Hebbian learning (DHL) rules, instead, are able to update the synapse by taking into account the temporal relation, captured with derivatives, between the neural events happening in the recent past. The few DHL rules proposed so far can update the synaptic weights only in few ways: this is a limitation for the study of dynamical neurons and neural-network models. Moreover, empirical evidence on brain spike-timing-dependent plasticity (STDP) shows that different neurons express a surprisingly rich repertoire of different learning processes going far beyond existing DHL rules. This opens up a second problem of how capturing such processes with DHL rules. Here we propose a general DHL (G-DHL) rule generating the existing rules and many others. The rule has a high expressiveness as it combines in different ways the pre- and post-synaptic neuron signals and derivatives. The rule flexibility is shown by applying it to various signals of artificial neurons and by fitting several different STDP experimental data sets. To these purposes, we propose techniques to pre-process the neural signals and capture the temporal relations between the neural events of interest. We also propose a procedure to automatically identify the rule components and parameters that best fit different STDP data sets, and show how the identified components might be used to heuristically guide the search of the biophysical mechanisms underlying STDP. Overall, the results show that the G-DHL rule represents a useful means to study time-sensitive learning processes in both artificial neural networks and brain.

Exploration and learning in capuchin monkeys (Sapajus spp.): the role of action-outcome contingencies.

Animals have a strong propensity to explore the environment. Spontaneous exploration has a great biological significance since it allows animals to discover and learn the relation between specific behaviours and their consequences. The role of the contingency between action and outcome for learning has been mainly investigated in instrumental learning settings and much less in free exploration contexts. We tested 16 capuchin monkeys (Sapajus spp.) with a mechatronic platform that allowed complex modules to be manipulated and to produce different outcomes. Experimental subjects could manipulate the modules and discover the contingencies between their own specific actions and the outcomes produced (i.e., the opening and lighting of a box). By contrast, Control subjects could operate on the modules, but the outcomes experienced were those performed by their paired Experimental subjects ("yoked-control" paradigm). In the exploration phase, in which no food reward was present, Experimental subjects spent more time on the board and manipulated the modules more than Yoked subjects. Experimental subjects outperformed Yoked subjects in the following test phase, where success required recalling the effective action so to open the box, now baited with food. These findings demonstrate that the opportunity to experience action-outcome contingencies in the absence of extrinsic rewards promotes capuchins' exploration and facilitates learning processes. Thus, this intrinsically motivated learning represents a powerful mechanism allowing the acquisition of skills and cognitive competence that the individual can later exploit for adaptive purposes.

Development of goal-directed action selection guided by intrinsic motivations: an experiment with children.

Action selection is extremely important, particularly when the accomplishment of competitive tasks may require access to limited motor resources. The spontaneous exploration of the world plays a fundamental role in the development of this capacity, providing subjects with an increasingly diverse set of opportunities to acquire, practice and refine the understanding of action-outcome connection. The computational modeling literature proposed a number of specific mechanisms for autonomous agents to discover and target interesting outcomes: intrinsic motivations hold a central importance among those mechanisms. Unfortunately, the study of the acquisition of action-outcome relation was mostly carried out with experiments involving extrinsic tasks, either based on rewards or on predefined task goals. This work presents a new experimental paradigm to study the effect of intrinsic motivation on action-outcome relation learning and action selection during free exploration of the world. Three- and four-year-old children were observed during the free exploration of a new toy: half of them were allowed to develop the knowledge concerning its functioning; the other half were not allowed to learn anything. The knowledge acquired during the free exploration of the toy was subsequently assessed and compared.

The Status Importance Scale: Development and validation of a self-report questionnaire for measuring how much people care about status.

Although substantial research indicates that considerations about status can lead to anxiety and other negative outcomes, a valid measure of the importance individuals attribute to status is lacking. This paper introduces the Status Importance Scale (SIS), a mono-factorial 10-item self-report questionnaire that quantifies how important a person deems status to be. Five studies validate the scale showing that it has excellent internal reliability and acceptable test-retest reliability, it correlates with several related measures (supporting convergent validity), it shows little correlation with theoretically unrelated constructs (supporting discriminant validity), it is the best predictor of conspicuous consumption compared with other potential candidates (supporting concurrent validity), and it can help predicting which activities one gives importance to (further supporting concurrent validity). Finally, as hypothesized by previous literature, the last study reveals that the SIS can predict status anxiety. The SIS can contribute to research regarding important phenomena such as the detrimental psychological effects of income inequality.

A mechatronic platform for behavioral analysis on nonhuman primates.

In this work we present a new mechatronic platform for measuring behavior of nonhuman primates, allowing high reprogrammability and providing several possibilities of interactions. The platform is the result of a multidisciplinary design process, which has involved bio-engineers, developmental neuroscientists, primatologists, and roboticians to identify its main requirements and specifications. Although such a platform has been designed for the behavioral analysis of capuchin monkeys (Cebus apella), it can be used for behavioral studies on other nonhuman primates and children. First, a state-of-the-art principal approach used in nonhuman primate behavioral studies is reported. Second, the main advantages of the mechatronic approach are presented. In this section, the platform is described in all its parts and the possibility to use it for studies on learning mechanism based on intrinsic motivation discussed. Third, a pilot study on capuchin monkeys is provided and preliminary data are presented and discussed.

The Relationship between Acceptance and Sleep-Wake Quality before, during, and after the First Italian COVID-19 Lockdown.

Several studies have reported that the COVID-19 pandemic has had deleterious effects on sleep quality and mood, but the mechanisms underlying these effects are not clearly understood. Recently, it has been shown that the acceptance component of mindfulness reduces anxiety, and, in turn, lower anxiety improves sleep quality. The purpose of this cross-sectional study was to assess changes in mindfulness traits, sleep-wake quality, and general distress, before, during, and after the first COVID-19 wave, testing the model in which acceptance influences sleep through anxiety in each period. A total of 250 participants were recruited before (Pre-Lockdown group: 69 participants, 29 females, 33.04 ± 12.94 years), during (Lockdown group: 78 participants, 59 females, 29.174 ± 8.50 years), and after (After-Lockdown group: 103 participants, 86 females, 30.29 ± 9.46 years) the first Italian lockdown. In each group, self-report questionnaires, assessing mindfulness facets, distress, and sleep-wake quality, were administered and assessed. The Lockdown group reported lower acceptance and higher depression, while the After-Lockdown group reported lower sleep-wake quality and higher anxiety. The results of the path analysis confirmed that higher acceptance reduced anxiety and higher anxiety decreased sleep-wake quality in all groups. Our results confirm that acceptance influences sleep through the mediating role of anxiety.

Attentional Bias for Sleep-Related Words as a Function of Severity of Insomnia Symptoms.

Attentional bias to sleep-related information is thought to be a core feature for developing and/or maintaining insomnia. This study used a hallmark measure of attentional bias, the dot-probe task, to determine whether this bias toward sleep-related stimuli was a function of the severity of insomnia symptoms. A sample of 231 volunteers (175 females; mean age of 26.91 ± 8.05 years) participated in this online study, filling out the Insomnia Severity Index (ISI) and performing a visual dot-probe task. After categorizing individuals based on the ISI score into normal, subclinical, and moderate/severe sleep groups, we only found a marginally significant interaction between sleep groups and the type of stimuli on RTs, suggesting that subclinical and moderate/severe sleep groups reported slower RTs for sleep-related words than for neutral words. When we calculated the attentional bias score (ABS), we found that ABS significantly differed from zero in the moderate/severe sleep group only, suggesting a disengagement for sleep-related information as a function of the severity of insomnia symptoms. This finding seems to suggest that insomnia is related to greater difficulties in shifting away from sleep-related stimuli.

Accept Anxiety to Improve Sleep: The Impact of the COVID-19 Lockdown on the Relationships between Mindfulness, Distress, and Sleep Quality.

It has been recently proposed that mindfulness can improve sleep quality through the mediating role on psychological distress and that acceptance may play a pivotal role in mindfulness beneficial effects. The aim of the present work was to understand the effects of the COVID-19 lockdown on dispositional mindfulness, sleep, and distress, and on their relationships. In particular, we wanted to test the hypothesis that the detrimental effects of lockdown on sleep depended on mindfulness and distress (including anxiety and depression) and that the acceptance facet of mindfulness played the leading role. A longitudinal study based on self-report questionnaires was conducted on 39 Italian adults (M age = 35.03, SD = 14.02; 21 men) assessing mindfulness, distress, and sleep quality before (23 December 2019-8 March 2020) and during (27 April 2020-10 May 2020) the first Italian COVID-19 lockdown. Lockdown decreased mindfulness while increasing distress and sleep problems. Path analysis showed that the effects of lockdown on sleep were fully mediated by mindfulness and distress. Furthermore, a more detailed analysis showed that these effects were mainly dependent on the acceptance component of mindfulness working through anxiety. The present study confirms, in the context of the COVID-19 lockdown, a model according to which mindfulness, and specifically acceptance, influences sleep through the mediating role of distress.

Goal-Directed Behavior and Instrumental Devaluation: A Neural System-Level Computational Model.

Devaluation is the key experimental paradigm used to demonstrate the presence of instrumental behaviors guided by goals in mammals. We propose a neural system-level computational model to address the question of which brain mechanisms allow the current value of rewards to control instrumental actions. The model pivots on and shows the computational soundness of the hypothesis for which the internal representation of instrumental manipulanda (e.g., levers) activate the representation of rewards (or "action-outcomes", e.g., foods) while attributing to them a value which depends on the current internal state of the animal (e.g., satiation for some but not all foods). The model also proposes an initial hypothesis of the integrated system of key brain components supporting this process and allowing the recalled outcomes to bias action selection: (a) the sub-system formed by the basolateral amygdala and insular cortex acquiring the manipulanda-outcomes associations and attributing the current value to the outcomes; (b) three basal ganglia-cortical loops selecting respectively goals, associative sensory representations, and actions; (c) the cortico-cortical and striato-nigro-striatal neural pathways supporting the selection, and selection learning, of actions based on habits and goals. The model reproduces and explains the results of several devaluation experiments carried out with control rats and rats with pre- and post-training lesions of the basolateral amygdala, the nucleus accumbens core, the prelimbic cortex, and the dorso-medial striatum. The results support the soundness of the hypotheses of the model and show its capacity to integrate, at the system-level, the operations of the key brain structures underlying devaluation. Based on its hypotheses and predictions, the model also represents an operational framework to support the design and analysis of new experiments on the motivational aspects of goal-directed behavior.

Corticolimbic catecholamines in stress: a computational model of the appraisal of controllability.

Appraisal of a stressful situation and the possibility to control or avoid it is thought to involve frontal-cortical mechanisms. The precise mechanism underlying this appraisal and its translation into effective stress coping (the regulation of physiological and behavioural responses) are poorly understood. Here, we propose a computational model which involves tuning motivational arousal to the appraised stressing condition. The model provides a causal explanation of the shift from active to passive coping strategies, i.e. from a condition characterised by high motivational arousal, required to deal with a situation appraised as stressful, to a condition characterised by emotional and motivational withdrawal, required when the stressful situation is appraised as uncontrollable/unavoidable. The model is motivated by results acquired via microdialysis recordings in rats and highlights the presence of two competing circuits dominated by different areas of the ventromedial prefrontal cortex: these are shown having opposite effects on several subcortical areas, affecting dopamine outflow in the striatum, and therefore controlling motivation. We start by reviewing published data supporting structure and functioning of the neural model and present the computational model itself with its essential neural mechanisms. Finally, we show the results of a new experiment, involving the condition of repeated inescapable stress, which validate most of the model's predictions.

Nonhuman gamblers: lessons from rodents, primates, and robots.

The search for neuronal and psychological underpinnings of pathological gambling in humans would benefit from investigating related phenomena also outside of our species. In this paper, we present a survey of studies in three widely different populations of agents, namely rodents, non-human primates, and robots. Each of these populations offers valuable and complementary insights on the topic, as the literature demonstrates. In addition, we highlight the deep and complex connections between relevant results across these different areas of research (i.e., cognitive and computational neuroscience, neuroethology, cognitive primatology, neuropsychiatry, evolutionary robotics), to make the case for a greater degree of methodological integration in future studies on pathological gambling.

Keep focussing: striatal dopamine multiple functions resolved in a single mechanism tested in a simulated humanoid robot.

The effects of striatal dopamine (DA) on behavior have been widely investigated over the past decades, with "phasic" burst firings considered as the key expression of a reward prediction error responsible for reinforcement learning. Less well studied is "tonic" DA, where putative functions include the idea that it is a regulator of vigor, incentive salience, disposition to exert an effort and a modulator of approach strategies. We present a model combining tonic and phasic DA to show how different outflows triggered by either intrinsically or extrinsically motivating stimuli dynamically affect the basal ganglia by impacting on a selection process this system performs on its cortical input. The model, which has been tested on the simulated humanoid robot iCub interacting with a mechatronic board, shows the putative functions ascribed to DA emerging from the combination of a standard computational mechanism coupled to a differential sensitivity to the presence of DA across the striatum.

Phasic dopamine as a prediction error of intrinsic and extrinsic reinforcements driving both action acquisition and reward maximization: a simulated robotic study.

An important issue of recent neuroscientific research is to understand the functional role of the phasic release of dopamine in the striatum, and in particular its relation to reinforcement learning. The literature is split between two alternative hypotheses: one considers phasic dopamine as a reward prediction error similar to the computational TD-error, whose function is to guide an animal to maximize future rewards; the other holds that phasic dopamine is a sensory prediction error signal that lets the animal discover and acquire novel actions. In this paper we propose an original hypothesis that integrates these two contrasting positions: according to our view phasic dopamine represents a TD-like reinforcement prediction error learning signal determined by both unexpected changes in the environment (temporary, intrinsic reinforcements) and biological rewards (permanent, extrinsic reinforcements). Accordingly, dopamine plays the functional role of driving both the discovery and acquisition of novel actions and the maximization of future rewards. To validate our hypothesis we perform a series of experiments with a simulated robotic system that has to learn different skills in order to get rewards. We compare different versions of the system in which we vary the composition of the learning signal. The results show that only the system reinforced by both extrinsic and intrinsic reinforcements is able to reach high performance in sufficiently complex conditions.

Novelty or surprise?

Novelty and surprise play significant roles in animal behavior and in attempts to understand the neural mechanisms underlying it. They also play important roles in technology, where detecting observations that are novel or surprising is central to many applications, such as medical diagnosis, text processing, surveillance, and security. Theories of motivation, particularly of intrinsic motivation, place novelty and surprise among the primary factors that arouse interest, motivate exploratory or avoidance behavior, and drive learning. In many of these studies, novelty and surprise are not distinguished from one another: the words are used more-or-less interchangeably. However, while undeniably closely related, novelty and surprise are very different. The purpose of this article is first to highlight the differences between novelty and surprise and to discuss how they are related by presenting an extensive review of mathematical and computational proposals related to them, and then to explore the implications of this for understanding behavioral and neuroscience data. We argue that opportunities for improved understanding of behavior and its neural basis are likely being missed by failing to distinguish between novelty and surprise.

Which is the best intrinsic motivation signal for learning multiple skills?

Humans and other biological agents are able to autonomously learn and cache different skills in the absence of any biological pressure or any assigned task. In this respect, Intrinsic Motivations (i.e., motivations not connected to reward-related stimuli) play a cardinal role in animal learning, and can be considered as a fundamental tool for developing more autonomous and more adaptive artificial agents. In this work, we provide an exhaustive analysis of a scarcely investigated problem: which kind of IM reinforcement signal is the most suitable for driving the acquisition of multiple skills in the shortest time? To this purpose we implemented an artificial agent with a hierarchical architecture that allows to learn and cache different skills. We tested the system in a setup with continuous states and actions, in particular, with a kinematic robotic arm that has to learn different reaching tasks. We compare the results of different versions of the system driven by several different intrinsic motivation signals. The results show (a) that intrinsic reinforcements purely based on the knowledge of the system are not appropriate to guide the acquisition of multiple skills, and (b) that the stronger the link between the IM signal and the competence of the system, the better the performance.

A spiking neuron model of the cortico-basal ganglia circuits for goal-directed and habitual action learning.

Dual-system theories postulate that actions are supported either by a goal-directed or by a habit-driven response system. Neuroimaging and anatomo-functional studies have provided evidence that the prefrontal cortex plays a fundamental role in the first type of action control, while internal brain areas such as the basal ganglia are more active during habitual and overtrained responses. Additionally, it has been shown that areas of the cortex and the basal ganglia are connected through multiple parallel "channels", which are thought to function as an action selection mechanism resolving competitions between alternative options available in a given context. In this paper we propose a multi-layer network of spiking neurons that implements in detail the thalamo-cortical circuits that are believed to be involved in action learning and execution. A key feature of this model is that neurons are organized in small pools in the motor cortex and form independent loops with specific pools of the basal ganglia where inhibitory circuits implement a multistep selection mechanism. The described model has been validated utilizing it to control the actions of a virtual monkey that has to learn to turn on briefly flashing lights by pressing corresponding buttons on a board. When the animal is able to fluently execute the task the button-light associations are remapped so that it has to suppress its habitual behavior in order to execute goal-directed actions. The model nicely shows how sensory-motor associations for action sequences are formed at the cortico-basal ganglia level and how goal-directed decisions may override automatic motor responses.

Intrinsically motivated action-outcome learning and goal-based action recall: a system-level bio-constrained computational model.

Reinforcement (trial-and-error) learning in animals is driven by a multitude of processes. Most animals have evolved several sophisticated systems of 'extrinsic motivations' (EMs) that guide them to acquire behaviours allowing them to maintain their bodies, defend against threat, and reproduce. Animals have also evolved various systems of 'intrinsic motivations' (IMs) that allow them to acquire actions in the absence of extrinsic rewards. These actions are used later to pursue such rewards when they become available. Intrinsic motivations have been studied in Psychology for many decades and their biological substrates are now being elucidated by neuroscientists. In the last two decades, investigators in computational modelling, robotics and machine learning have proposed various mechanisms that capture certain aspects of IMs. However, we still lack models of IMs that attempt to integrate all key aspects of intrinsically motivated learning and behaviour while taking into account the relevant neurobiological constraints. This paper proposes a bio-constrained system-level model that contributes a major step towards this integration. The model focusses on three processes related to IMs and on the neural mechanisms underlying them: (a) the acquisition of action-outcome associations (internal models of the agent-environment interaction) driven by phasic dopamine signals caused by sudden, unexpected changes in the environment; (b) the transient focussing of visual gaze and actions on salient portions of the environment; (c) the subsequent recall of actions to pursue extrinsic rewards based on goal-directed reactivation of the representations of their outcomes. The tests of the model, including a series of selective lesions, show how the focussing processes lead to a faster learning of action-outcome associations, and how these associations can be recruited for accomplishing goal-directed behaviours. The model, together with the background knowledge reviewed in the paper, represents a framework that can be used to guide the design and interpretation of empirical experiments on IMs, and to computationally validate and further develop theories on them.

Representations in dynamical embodied agents: re-analyzing a minimally cognitive model agent.

Understanding the role of ''representations'' in cognitive science is a fundamental problem facing the emerging framework of embodied, situated, dynamical cognition. To make progress, I follow the approach proposed by an influential representational skeptic, Randall Beer: building artificial agents capable of minimally cognitive behaviors and assessing whether their internal states can be considered to involve representations. Hence, I operationalize the concept of representing as ''standing in,'' and I look for representations in embodied agents involved in simple categorization tasks. In a first experiment, no representation can be found, but the relevance of the task is undermined by the fact that agents with no internal states can reach high performance. A simple modification makes the task more "representationally hungry," and in this case, agents' internal states are found to qualify as representations. I conclude by discussing the benefits of reconciling the embodied-dynamical approach with the notion of representation.