The consequences of AI training on human decision-making.

AI is now an integral part of everyday decision-making, assisting us in both routine and high-stakes choices. These AI models often learn from human behavior, assuming this training data is unbiased. However, we report five studies that show that people change their behavior to instill desired routines into AI, indicating this assumption is invalid. To show this behavioral shift, we recruited participants to play the ultimatum game, where they were asked to decide whether to accept proposals of monetary splits made by either other human participants or AI. Some participants were informed their choices would be used to train an AI proposer, while others did not receive this information. Across five experiments, we found that people modified their behavior to train AI to make fair proposals, regardless of whether they could directly benefit from the AI training. After completing this task once, participants were invited to complete this task again but were told their responses would not be used for AI training. People who had previously trained AI persisted with this behavioral shift, indicating that the new behavioral routine had become habitual. This work demonstrates that using human behavior as training data has more consequences than previously thought since it can engender AI to perpetuate human biases and cause people to form habits that deviate from how they would normally act. Therefore, this work underscores a problem for AI algorithms that aim to learn unbiased representations of human preferences.

Toward a Rational and Mechanistic Account of Mental Effort.

In spite of its familiar phenomenology, the mechanistic basis for mental effort remains poorly understood. Although most researchers agree that mental effort is aversive and stems from limitations in our capacity to exercise cognitive control, it is unclear what gives rise to those limitations and why they result in an experience of control as costly. The presence of these control costs also raises further questions regarding how best to allocate mental effort to minimize those costs and maximize the attendant benefits. This review explores recent advances in computational modeling and empirical research aimed at addressing these questions at the level of psychological process and neural mechanism, examining both the limitations to mental effort exertion and how we manage those limited cognitive resources. We conclude by identifying remaining challenges for theoretical accounts of mental effort as well as possible applications of the available findings to understanding the causes of and potential solutions for apparent failures to exert the mental effort required of us.

Computational and behavioral markers of model-based decision making in childhood.

Human decision-making is underpinned by distinct systems that differ in flexibility and associated cognitive cost. A widely accepted dichotomy distinguishes between a cheap but rigid model-free system and a flexible but costly model-based system. Typically, humans use a hybrid of both types of decision-making depending on environmental demands. However, children's use of a model-based system during decision-making has not yet been shown. While prior developmental work has identified simple building blocks of model-based reasoning in young children (1-4 years old), there has been little evidence of this complex cognitive system influencing behavior before adolescence. Here, by using a modified task to make engagement in cognitively costly strategies more rewarding, we show that children aged 5-11-years (N = 85), including the youngest children, displayed multiple indicators of model-based decision making, and that the degree of its use increased throughout childhood. Unlike adults (N = 24), however, children did not display adaptive arbitration between model-free and model-based decision-making. Our results demonstrate that throughout childhood, children can engage in highly sophisticated and costly decision-making strategies. However, the flexible arbitration between decision-making strategies might be a critically late-developing component in human development.

Planning Complexity Registers as a Cost in Metacontrol.

Decision-making algorithms face a basic tradeoff between accuracy and effort (i.e., computational demands). It is widely agreed that humans can choose between multiple decision-making processes that embody different solutions to this tradeoff: Some are computationally cheap but inaccurate, whereas others are computationally expensive but accurate. Recent progress in understanding this tradeoff has been catalyzed by formalizing it in terms of model-free (i.e., habitual) versus model-based (i.e., planning) approaches to reinforcement learning. Intuitively, if two tasks offer the same rewards for accuracy but one of them is much more demanding, we might expect people to rely on habit more in the difficult task: Devoting significant computation to achieve slight marginal accuracy gains would not be "worth it." We test and verify this prediction in a sequential reinforcement learning task. Because our paradigm is amenable to formal analysis, it contributes to the development of a computational model of how people balance the costs and benefits of different decision-making processes in a task-specific manner; in other words, how we decide when hard thinking is worth it.

Cost-Benefit Arbitration Between Multiple Reinforcement-Learning Systems.

Human behavior is sometimes determined by habit and other times by goal-directed planning. Modern reinforcement-learning theories formalize this distinction as a competition between a computationally cheap but inaccurate model-free system that gives rise to habits and a computationally expensive but accurate model-based system that implements planning. It is unclear, however, how people choose to allocate control between these systems. Here, we propose that arbitration occurs by comparing each system's task-specific costs and benefits. To investigate this proposal, we conducted two experiments showing that people increase model-based control when it achieves greater accuracy than model-free control, and especially when the rewards of accurate performance are amplified. In contrast, they are insensitive to reward amplification when model-based and model-free control yield equivalent accuracy. This suggests that humans adaptively balance habitual and planned action through on-line cost-benefit analysis.

When Does Model-Based Control Pay Off?

Many accounts of decision making and reinforcement learning posit the existence of two distinct systems that control choice: a fast, automatic system and a slow, deliberative system. Recent research formalizes this distinction by mapping these systems to "model-free" and "model-based" strategies in reinforcement learning. Model-free strategies are computationally cheap, but sometimes inaccurate, because action values can be accessed by inspecting a look-up table constructed through trial-and-error. In contrast, model-based strategies compute action values through planning in a causal model of the environment, which is more accurate but also more cognitively demanding. It is assumed that this trade-off between accuracy and computational demand plays an important role in the arbitration between the two strategies, but we show that the hallmark task for dissociating model-free and model-based strategies, as well as several related variants, do not embody such a trade-off. We describe five factors that reduce the effectiveness of the model-based strategy on these tasks by reducing its accuracy in estimating reward outcomes and decreasing the importance of its choices. Based on these observations, we describe a version of the task that formally and empirically obtains an accuracy-demand trade-off between model-free and model-based strategies. Moreover, we show that human participants spontaneously increase their reliance on model-based control on this task, compared to the original paradigm. Our novel task and our computational analyses may prove important in subsequent empirical investigations of how humans balance accuracy and demand.

Cognitive effort avoidance and detection in people with schizophrenia.

Many people with schizophrenia exhibit avolition, a difficulty initiating and maintaining goal-directed behavior, considered to be a key negative symptom of the disorder. Recent evidence indicates that patients with higher levels of negative symptoms differ from healthy controls in showing an exaggerated cost of the physical effort needed to obtain a potential reward. We examined whether patients show an exaggerated avoidance of cognitive effort, using the demand selection task developed by Kool, McGuire, Rosen, and Botvinick (Journal of Experimental Psychology. General, 139, 665-682, 2010). A total of 83 people with schizophrenia or schizoaffective disorder and 71 healthy volunteers participated in three experiments where instructions varied. In the standard task (Experiment 1), neither controls nor patients showed expected cognitive demand avoidance. With enhanced instructions (Experiment 2), controls demonstrated greater demand avoidance than patients. In Experiment 3, patients showed nonsignificant reductions in demand avoidance, relative to controls. In a control experiment, patients showed significantly reduced ability to detect the effort demands associated with different response alternatives. In both groups, the ability to detect effort demands was associated with increased effort avoidance. In both groups, increased cognitive effort avoidance was associated with higher IQ and general neuropsychological ability. No significant correlations between demand avoidance and negative symptom severity were observed. Thus, it appears that individual differences in general intellectual ability and effort detection are related to cognitive effort avoidance and likely account for the subtle reduction in effort avoidance observed in schizophrenia.

Mechanisms of motivation-cognition interaction: challenges and opportunities.

Recent years have seen a rejuvenation of interest in studies of motivation-cognition interactions arising from many different areas of psychology and neuroscience. The present issue of Cognitive, Affective, & Behavioral Neuroscience provides a sampling of some of the latest research from a number of these different areas. In this introductory article, we provide an overview of the current state of the field, in terms of key research developments and candidate neural mechanisms receiving focused investigation as potential sources of motivation-cognition interaction. However, our primary goal is conceptual: to highlight the distinct perspectives taken by different research areas, in terms of how motivation is defined, the relevant dimensions and dissociations that are emphasized, and the theoretical questions being targeted. Together, these distinctions present both challenges and opportunities for efforts aiming toward a more unified and cross-disciplinary approach. We identify a set of pressing research questions calling for this sort of cross-disciplinary approach, with the explicit goal of encouraging integrative and collaborative investigations directed toward them.

The construction and use of cognitive maps in model-based control.

When making decisions, we sometimes rely on habit and at other times plan toward goals. Planning requires the construction and use of an internal representation of the environment, a cognitive map. How are these maps constructed, and how do they guide goal-directed decisions? We coupled a sequential decision-making task with a behavioral representational similarity analysis approach to examine how relationships between choice options change when people build a cognitive map of the task structure. We found that participants who encoded stronger higher-order relationships among choice options showed increased planning and better performance. These higher-order relationships were more strongly encoded among objects encountered in high-reward contexts, indicating a role for motivation during cognitive map construction. In contrast, lower-order relationships such as simple visual co-occurrence of objects did not predict goal-directed planning. These results show that the construction of cognitive maps is an active process, with motivation dictating the degree to which higher-order relationships are encoded and used for planning. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Humans do not avoid reactively implementing cognitive control.

The ability to exert cognitive control allows us to achieve goals in the face of distraction and competing actions. However, control is costly-people generally aim to minimize its demands. Because control takes many forms, it is important to understand whether such costs apply universally. Specifically, reactive control, which is recruited in response to stimulus or contextual features, is theorized to be deployed automatically, and not depend on attentional resources. Here, we investigated whether people avoided implementing reactive control in three experiments. In all, participants performed a Stroop task in which certain items were mostly incongruent (MI), that is, associated with a high likelihood of conflict (triggering a focused control setting). Other items were mostly congruent, that is, associated with a low likelihood of conflict (triggering a relaxed control setting). Experiment 1 demonstrated that these control settings transfer to a subsequent unbiased transfer phase. In Experiments 2-3, we used a demand selection task to investigate whether people would avoid choice options that yielded items that were previously MI. In all, participants continued to retrieve focused control settings for previously MI items, but they did not avoid them in the demand selection task. Critically, we only found demand avoidance when there was an objective difference in demand between options. These findings are consistent with the idea that implementing reactive control does not register as costly. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Evidence for shallow cognitive maps in schizophrenia.

Individuals with schizophrenia can have marked deficits in goal-directed decision making. Prominent theories differ in whether schizophrenia (SZ) affects the ability to exert cognitive control, or the motivation to exert control. An alternative explanation is that schizophrenia negatively impacts the formation of cognitive maps, the internal representations of the way the world is structured, necessary for the formation of effective action plans. That is, deficits in decision-making could also arise when goal-directed control and motivation are intact, but used to plan over ill-formed maps. Here, we test the hypothesis that individuals with SZ are impaired in the construction of cognitive maps. We combine a behavioral representational similarity analysis technique with a sequential decision-making task. This enables us to examine how relationships between choice options change when individuals with SZ and healthy age-matched controls build a cognitive map of the task structure. Our results indicate that SZ affects how people represent the structure of the task, focusing more on simpler visual features and less on abstract, higher-order, planning-relevant features. At the same time, we find that SZ were able to display similar performance on this task compared to controls, emphasizing the need for a distinction between cognitive map formation and changes in goal-directed control in understanding cognitive deficits in schizophrenia.

Neural representation of reward probability: evidence from the illusion of control.

To support reward-based decision-making, the brain must encode potential outcomes both in terms of their incentive value and their probability of occurrence. Recent research has made it clear that the brain bears multiple representations of reward magnitude, meaning that a single choice option may be represented differently-and even inconsistently-in different brain areas. There are some hints that the same may be true for reward probability. Preliminary evidence hints that, even as systematic distortions of probability are expressed in behavior, these may not always be uniformly reflected at the neural level: Some neural representations of probability may be immune from such distortions. This study provides new evidence consistent with this possibility. Participants in a behavioral experiment displayed a classic "illusion of control," providing higher estimates of reward probability for gambles they had chosen than for identical gambles that were imposed on them. However, an fMRI study of the same task revealed that neural prediction error signals, arising when gamble outcomes were revealed, were unaffected by the illusion of control. The resulting behavioral-neural dissociation reinforces the case for multiple, inconsistent internal representations of reward probability, while also prompting a reinterpretation of the illusion of control effect itself.

The intrinsic cost of cognitive control.

Kurzban and colleagues carry forward an important contemporary movement in cognitive control research, tending away from resource-based models and toward a framework focusing on motivation or value. However, their specific proposal, centering on opportunity costs, appears problematic. We favor a simpler view, according to which the exertion of cognitive control carries intrinsic subjective costs.

Need for cognition moderates the relief of avoiding cognitive effort.

When making decisions, humans aim to maximize rewards while minimizing costs. The exertion of mental or physical effort has been proposed to be one those costs, translating into avoidance of behaviors carrying effort demands. This motivational framework also predicts that people should experience positive affect when anticipating demand that is subsequently avoided (i.e., a "relief effect"), but evidence for this prediction is scarce. Here, we follow up on a previous study [1] that provided some initial evidence that people more positively evaluated outcomes if it meant they could avoid performing an additional demanding task. However, the results from this study did not provide conclusive evidence that this effect was driven by effort avoidance. Here, we report two experiments that are able to do this. Participants performed a gambling task, and if they did not receive reward they would have to perform an orthogonal effort task. Prior to the gamble, a cue indicated whether this effort task would be easy or hard. We probed hedonic responses to the reward-related feedback, as well as after the subsequent effort task feedback. Participants reported lower hedonic responses for no-reward outcomes when high vs. low effort was anticipated (and later exerted). They also reported higher hedonic responses for reward outcomes when high vs. low effort was anticipated (and avoided). Importantly, this relief effect was smaller in participants with high need for cognition. These results suggest that avoidance of high effort tasks is rewarding, but that the size off this effect depends on the individual disposition to engage with and expend cognitive effort. They also raise the important question of whether this disposition alters the cost of effort per se, or rather offset this cost during cost-benefit analyses.

A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning.

The Gumbel-max trick is a method to draw a sample from a categorical distribution, given by its unnormalized (log-)probabilities. Over the past years, the machine learning community has proposed several extensions of this trick to facilitate, e.g., drawing multiple samples, sampling from structured domains, or gradient estimation for error backpropagation in neural network optimization. The goal of this survey article is to present background about the Gumbel-max trick, and to provide a structured overview of its extensions to ease algorithm selection. Moreover, it presents a comprehensive outline of (machine learning) literature in which Gumbel-based algorithms have been leveraged, reviews commonly-made design choices, and sketches a future perspective.

Reasoning supports utilitarian resolutions to moral dilemmas across diverse measures.

Sacrificial moral dilemmas elicit a strong conflict between the motive to not personally harm someone and the competing motive to achieving the greater good, which is often described as the "utilitarian" response. Some prior research suggests that reasoning abilities and deliberative cognitive style are associated with endorsement of utilitarian solutions, but, as has more recently been emphasized, both conceptual and methodological issues leave open the possibility that utilitarian responses are due instead to a reduced emotional response to harm. Across 8 studies, using self-report, behavioral performance, and neuroanatomical measures, we show that individual differences in reasoning ability and cognitive style of thinking are positively associated with a preference for utilitarian solutions, but bear no relationship to harm-relevant concerns. These findings support the dual-process model of moral decision making and highlight the utility of process dissociation methods. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Metacontrol of decision-making strategies in human aging.

Humans employ different strategies when making decisions. Previous research has reported reduced reliance on model-based strategies with aging, but it remains unclear whether this is due to cognitive or motivational factors. Moreover, it is not clear how aging affects the metacontrol of decision making, that is the dynamic adaptation of decision-making strategies to varying situational demands. In this cross-sectional study, we tested younger and older adults in a sequential decision-making task that dissociates model-free and model-based strategies. In contrast to previous research, model-based strategies led to higher payoffs. Moreover, we manipulated the costs and benefits of model-based strategies by varying reward magnitude and the stability of the task structure. Compared to younger adults, older adults showed reduced model-based decision making and less adaptation of decision-making strategies. Our findings suggest that aging affects the metacontrol of decision-making strategies and that reduced model-based strategies in older adults are due to limited cognitive abilities.

The transdiagnostic structure of mental effort avoidance.

The law of least mental effort states that, everything else being equal, the brain tries to minimize mental effort expenditure during task performance by avoiding decisions that require greater cognitive demands. Prior studies have shown associations between disruptions in effort expenditure and specific psychiatric illnesses (e.g., schizophrenia and depression) or clinically-related symptoms and traits (e.g., anhedonia and apathy), yet no research has explored this issue transdiagnostically. Specifically, this research has largely focused on a single diagnostic category, symptom, or trait. However, abnormalities in effort expression could be related to several different psychiatrically-relevant constructs that cut across diagnostic boundaries. Therefore, we examined the relationship between avoidance of mental effort and a diverse set of clinically-related symptoms and traits, and transdiagnostic latent factors in a large sample (n = 811). Only lack of perseverance, a dimension of impulsiveness, was associated with increased avoidance of mental effort. In contrast, several constructs were associated with less mental effort avoidance, including positive urgency, distress intolerance, obsessive-compulsive symptoms, disordered eating, and a factor consisting of compulsive behavior and intrusive thoughts. These findings demonstrate that deviations from normative effort expenditure are associated with a number of constructs that are common to several forms of psychiatric illness.

Incentives Boost Model-Based Control Across a Range of Severity on Several Psychiatric Constructs.

BACKGROUND: Human decision making exhibits a mixture of model-based and model-free control. Recent evidence indicates that arbitration between these two modes of control ("metacontrol") is based on their relative costs and benefits. While model-based control may increase accuracy, it requires greater computational resources, so people invoke model-based control only when potential rewards exceed those of model-free control. We used a sequential decision task, while concurrently manipulating performance incentives, to ask if symptoms and traits of psychopathology decrease or increase model-based control in response to incentives. METHODS: We recruited a nonpatient population of 839 online participants using Amazon Mechanical Turk who completed transdiagnostic self-report measures encompassing symptoms, traits, and factors. We fit a dual-controller reinforcement learning model and obtained a computational measure of model-based control separately for small incentives and large incentives. RESULTS: None of the constructs were related to a failure of large incentives to boost model-based control. In fact, for the sensation seeking trait and anxious-depression factor, higher scores were associated with a larger incentive effect, whereby greater levels of these constructs were associated with larger increases in model-based control. Many constructs showed decreases in model-based control as a function of severity, but a social withdrawal factor was positively correlated; alcohol use and social anxiety were unrelated to model-based control. CONCLUSIONS: Our results demonstrate that model-based control can reliably be improved independent of construct severity for most measures. This suggests that incentives may be a useful intervention for boosting model-based control across a range of symptom and trait severity.

Stress modulation of visuomotor binding.

The primate cortex represents perceived and produced events in a distributed way, which calls for a mechanism that integrates their features into coherent structures. Animal, drug, and patient studies suggest that the local binding of visual features is under muscarinic-cholinergic control, whereas visuomotor binding seems to be driven by dopaminergic pathways. Consistent with this picture, we present evidence that the binding of visual features and actions is modulated by stress, induced by the cold pressure test (CPT), which causes an excessive dopamine turnover in prefrontal cortex. The impact of stress was restricted to the task-relevant visuomotor binding, supporting claims that dopamine affects the maintenance of task-relevant information in working memory. The outcome pattern, including the impact of the personality trait extraversion, suggests that the relation between dopamine level and visuomotor performance follows an inverted U-shaped function, with strongest binding being associated with average dopamine levels.