A Genetically Defined Compartmentalized Striatal Direct Pathway for Negative Reinforcement.

The striosome compartment within the dorsal striatum has been implicated in reinforcement learning and regulation of motivation, but how striosomal neurons contribute to these functions remains elusive. Here, we show that a genetically identified striosomal population, which expresses the Teashirt family zinc finger 1 (Tshz1) and belongs to the direct pathway, drives negative reinforcement and is essential for aversive learning in mice. Contrasting a "conventional" striosomal direct pathway, the Tshz1 neurons cause aversion, movement suppression, and negative reinforcement once activated, and they receive a distinct set of synaptic inputs. These neurons are predominantly excited by punishment rather than reward and represent the anticipation of punishment or the motivation for avoidance. Furthermore, inhibiting these neurons impairs punishment-based learning without affecting reward learning or movement. These results establish a major role of striosomal neurons in behaviors reinforced by punishment and moreover uncover functions of the direct pathway unaccounted for in classic models.

A cognitive pathway to punishment insensitivity.

Individuals differ in their sensitivity to the adverse consequences of their actions, leading some to persist in maladaptive behaviors. Two pathways have been identified for this insensitivity: a motivational pathway based on excessive reward valuation and a behavioral pathway based on autonomous stimulus-response mechanisms. Here, we identify a third, cognitive pathway based on differences in punishment knowledge and use of that knowledge to suppress behavior. We show that distinct phenotypes of punishment sensitivity emerge from differences in what people learn about their actions. Exposed to identical punishment contingencies, some people (sensitive phenotype) form correct causal beliefs that they use to guide their behavior, successfully obtaining rewards and avoiding punishment, whereas others form incorrect but internally coherent causal beliefs that lead them to earn punishment they do not like. Incorrect causal beliefs were not inherently problematic because we show that many individuals benefit from information about why they are being punished, revaluing their actions and changing their behavior to avoid further punishment (unaware phenotype). However, one condition where incorrect causal beliefs were problematic was when punishment is infrequent. Under this condition, more individuals show punishment insensitivity and detrimental patterns of behavior that resist experience and information-driven updating, even when punishment is severe (compulsive phenotype). For these individuals, rare punishment acted as a "trap," inoculating maladaptive behavioral preferences against cognitive and behavioral updating.

How reputation does (and does not) drive people to punish without looking.

Punishing wrongdoers can confer reputational benefits, and people sometimes punish without careful consideration. But are these observations related? Does reputation drive people to people to "punish without looking"? And if so, is this because unquestioning punishment looks particularly virtuous? To investigate, we assigned "Actors" to decide whether to sign punitive petitions about politicized issues ("punishment"), after first deciding whether to read articles opposing these petitions ("looking"). To manipulate reputation, we matched Actors with copartisan "Evaluators," varying whether Evaluators observed i) nothing about Actors' behavior, ii) whether Actors punished, or iii) whether Actors punished and whether they looked. Across four studies of Americans (total n = 10,343), Evaluators rated Actors more positively, and financially rewarded them, if they chose to (vs. not to) punish. Correspondingly, making punishment observable to Evaluators (i.e., moving from our first to second condition) drove Actors to punish more overall. Furthermore, because some of these individuals did not look, making punishment observable increased rates of punishment without looking. Yet punishers who eschewed opposing perspectives did not appear particularly virtuous. In fact, Evaluators preferred Actors who punished with (vs. without) looking. Correspondingly, making looking observable (i.e., moving from our second to third condition) drove Actors to look more overall-and to punish without looking at comparable or diminished rates. We thus find that reputation can encourage reflexive punishment-but simply as a byproduct of generally encouraging punishment, and not as a specific reputational strategy. Indeed, rather than fueling unquestioning decisions, spotlighting punishers' decision-making processes may encourage reflection.

Emergence of specialized third-party enforcement.

The question of how cooperation evolves and is maintained among nonkin is central to the biological, social, and behavioral sciences. Previous research has focused on explaining how cooperation in social dilemmas can be maintained by direct and indirect reciprocity among the participants of the social dilemma. However, in complex human societies, both modern and ancient, cooperation is frequently maintained by means of specialized third-party enforcement. We provide an evolutionary-game-theoretic model that explains how specialized third-party enforcement of cooperation (specialized reciprocity) can emerge. A population consists of producers and enforcers. First, producers engage in a joint undertaking represented by a prisoner's dilemma. They are paired randomly and receive no information about their partner's history, which precludes direct and indirect reciprocity. Then, enforcers tax producers and may punish their clients. Finally, the enforcers are randomly paired and may try to grab resources from each other. In order to sustain producer cooperation, enforcers must punish defecting producers, but punishing is costly to enforcers. We show that the threat of potential intraenforcer conflict can incentivize enforcers to engage in costly punishment of producers, provided they are sufficiently informed to maintain a reputation system. That is, the "guards" are guarded by the guards themselves. We demonstrate the key mechanisms analytically and corroborate our results with numerical simulations.

Neural Representation of Valenced and Generic Probability and Uncertainty.

Representing the probability and uncertainty of outcomes facilitates adaptive behavior by allowing organisms to prepare in advance and devote attention to relevant events. Probability and uncertainty are often studied only for valenced (appetitive or aversive) outcomes, raising the question of whether the identified neural machinery also processes the probability and uncertainty of motivationally neutral outcomes. Here, we aimed to dissociate valenced from valence-independent (i.e., generic) probability (p; maximum at p = 1) and uncertainty (maximum at p = 0.5) signals using human neuroimaging. In a Pavlovian task (n = 41; 19 females), different cues predicted appetitive, aversive, or neutral liquids with different probabilities (p = 0, p = 0.5, p = 1). Cue-elicited motor responses accelerated, and pupil sizes increased primarily for cues that predicted valenced liquids with higher probability. For neutral liquids, uncertainty rather than probability tended to accelerate cue-induced responding and decrease pupil size. At the neural level, generic uncertainty signals were limited to the occipital cortex, while generic probability also activated the anterior ventromedial prefrontal cortex. These generic probability and uncertainty signals contrasted with cue-induced responses that only encoded the probability and uncertainty of valenced liquids in medial prefrontal, insular, and occipital cortices. Our findings show a behavioral and neural dissociation of generic and valenced signals. Thus, some parts of the brain keep track of motivational charge while others do not, highlighting the need and usefulness of characterizing the exact nature of learned representations.

Sex-based differences in fairness norm compliance and neural circuitry.

Human behavior often aligns with fairness norms, either voluntarily or under external pressure, like sanctions. Prior research has identified distinct neural activation patterns associated with voluntary and sanction-based compliance or non-compliance with fairness norms. However, an investigation gap exists into potential neural connectivity patterns and sex-based differences. To address this, we conducted a study using a monetary allocation game and functional magnetic resonance imaging to examine how neural activity and connectivity differ between sexes across three norm compliance conditions: voluntary, sanction-based, and voluntary post-sanctions. Fifty-five adults (27 females) participated, revealing that punishment influenced decisions, leading to strategic calculations and reduced generosity in voluntary compliance post-sanctions. Moreover, there were sex-based differences in neural activation and connectivity across the different compliance conditions. Specifically, the connectivity between the right dorsolateral prefrontal cortex and right dorsal anterior insular appeared to mediate intuitive preferences, with variations across norm compliance conditions and sexes. These findings imply potential sex-based differences in intuitive motivation for diverse norm compliance conditions. Our insights contribute to a better understanding of the neural pathways involved in fairness norm compliance and clarify sex-based differences, offering implications for future investigations into psychiatric and neurological disorders characterized by atypical socialization and mentalizing.

The shorter a man is, the more he defends fairness: relative height disadvantage promoting third-party punishment-evidence from inter-brain synchronization.

Third-party punishment occurs in interpersonal interactions to sustain social norms, and is strongly influenced by the characteristics of the interacting individuals. During social interactions, height is the striking physical appearance features first observed, height disadvantage may critically influence men's behavior and mental health. Herein, we explored the influence of height disadvantage on third-party punishment through time-frequency analysis and electroencephalography hyperscanning. Two participants were randomly designated as the recipient and third party after height comparison and instructed to complete third-party punishment task. Compared with when the third party's height is higher than the recipient's height, when the third party's height is lower, the punishment rate and transfer amount were significantly higher. Only for highly unfair offers, the theta power was significantly greater when the third party's height was lower. The inter-brain synchronization between the recipient and the third party was significantly stronger when the third party's height was lower. Compared with the fair and medium unfair offers, the inter-brain synchronization was strongest for highly unfair offers. Our findings indicate that the height disadvantage-induced anger and reputation concern promote third-party punishment and inter-brain synchronization. This study enriches research perspective and expands the application of the theory of Napoleon complex.

Hereditary and cortical morphological biomarker of sensitivity to reward in short-term withdrawal methamphetamine abusers.

Higher sensitivity to reward (SR) and weaker sensitivity to punishment (SP) construct the fundamental craving characteristics of methamphetamine abuse. However, few studies have appraised relationships between SR/SP (SR or SP) and cortical morphological alterations in methamphetamine abusers and whether hereditary factors take effects on SR/SP is unclear. Based on surface-based morphometric analysis, cortical discrepancy was investigated between 38 methamphetamine abusers and 37 healthy controls. Within methamphetamine abusers, correlation profiling was performed to discover associations among aberrant neuroimaging substrates, SR, SP, and craving. According to nine single nucleotide polymorphism sites of dopamine-related genes, we conducted univariate general linear model to find different effects of genotypes on cortical alterations and SR/SP/craving (SR, SP, or craving). Ultimately, mediation analyses were conducted among single nucleotide polymorphism sites, SR/SP/craving, and cortical morphological alterations to discover their association pathways. Compared to healthy controls, thinner cortices in inferior temporal gyrus, lateral orbitofrontal cortex, medial orbitofrontal cortex, inferior parietal lobule, and lateral occipital cortex in the left hemisphere were found in methamphetamine abusers (P < 0.05, family-wise error corrected). Cortical thickness in the inferior temporal gyrus was negatively correlated with SR scores. We found that rs1800497 A-containing genotypes had lower cortical thickness in the left inferior parietal lobule than the GG genotype. The rs5751876 had effects on SR scores. This study would provide convincing biomarkers for SR in methamphetamine abusers and offer potential genetic targets for personalizing relapse prevention.

Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents.

How do societies learn and maintain social norms? Here we use multiagent reinforcement learning to investigate the learning dynamics of enforcement and compliance behaviors. Artificial agents populate a foraging environment and need to learn to avoid a poisonous berry. Agents learn to avoid eating poisonous berries better when doing so is taboo, meaning the behavior is punished by other agents. The taboo helps overcome a credit assignment problem in discovering delayed health effects. Critically, introducing an additional taboo, which results in punishment for eating a harmless berry, further improves overall returns. This "silly rule" counterintuitively has a positive effect because it gives agents more practice in learning rule enforcement. By probing what individual agents have learned, we demonstrate that normative behavior relies on a sequence of learned skills. Learning rule compliance builds upon prior learning of rule enforcement by other agents. Our results highlight the benefit of employing a multiagent reinforcement learning computational model focused on learning to implement complex actions.

Circuit and Cell-Specific Contributions to Decision Making Involving Risk of Explicit Punishment in Male and Female Rats.

Decision making is a complex cognitive process that recruits a distributed network of brain regions, including the basolateral amygdala (BLA) and nucleus accumbens shell (NAcSh). Recent work suggests that communication between these structures, as well as activity of cells expressing dopamine (DA) D2 receptors (D2R) in the NAcSh, are necessary for some forms of decision making; however, the contributions of this circuit and cell population during decision making under risk of punishment are unknown. The current experiments addressed this question using circuit-specific and cell type-specific optogenetic approaches in rats during a decision making task involving risk of punishment. In experiment 1, Long-Evans rats received intra-BLA injections of halorhodopsin or mCherry (control) and in experiment 2, D2-Cre transgenic rats received intra-NAcSh injections of Cre-dependent halorhodopsin or mCherry. Optic fibers were implanted in the NAcSh in both experiments. Following training in the decision making task, BLA→NAcSh or D2R-expressing neurons were optogenetically inhibited during different phases of the decision process. Inhibition of the BLA→NAcSh during deliberation (the time between trial initiation and choice) increased preference for the large, risky reward (increased risk taking). Similarly, inhibition during delivery of the large, punished reward increased risk taking, but only in males. Inhibition of D2R-expressing neurons in the NAcSh during deliberation increased risk taking. In contrast, inhibition of these neurons during delivery of the small, safe reward decreased risk taking. These findings extend our knowledge of the neural dynamics of risk taking, revealing sex-dependent circuit recruitment and dissociable activity of selective cell populations during decision making.SIGNIFICANCE STATEMENT Until recently, the ability to dissect the neural substrates of decision making involving risk of punishment (risk taking) in a circuit-specific and cell-specific manner has been limited by the tools available for use in rats. Here, we leveraged the temporal precision of optogenetics, together with transgenic rats, to probe contributions of a specific circuit and cell population to different phases of risk-based decision making. Our findings reveal basolateral amygdala (BLA)→nucleus accumbens shell (NAcSh) is involved in evaluation of punished rewards in a sex-dependent manner. Further, NAcSh D2 receptor (D2R)-expressing neurons make unique contributions to risk taking that vary across the decision making process. These findings advance our understanding of the neural principles of decision making and provide insight into how risk taking may become compromised in neuropsychiatric diseases.

Cortical Grey Matter Mediates Increases in Model-Based Control and Learning from Positive Feedback from Adolescence to Adulthood.

Cognition and brain structure undergo significant maturation from adolescence into adulthood. Model-based (MB) control is known to increase across development, which is mediated by cognitive abilities. Here, we asked two questions unaddressed in previous developmental studies. First, what are the brain structural correlates of age-related increases in MB control? Second, how are age-related increases in MB control from adolescence to adulthood influenced by motivational context? A human developmental sample (n = 103; age, 12-50, male/female, 55:48) completed structural MRI and an established task to capture MB control. The task was modified with respect to outcome valence by including (1) reward and punishment blocks to manipulate the motivational context and (2) an additional choice test to assess learning from positive versus negative feedback. After replicating that an age-dependent increase in MB control is mediated by cognitive abilities, we demonstrate first-time evidence that gray matter density (GMD) in the parietal cortex mediates the increase of MB control with age. Although motivational context did not relate to age-related changes in MB control, learning from positive feedback improved with age. Meanwhile, negative feedback learning showed no age effects. We present a first report that an age-related increase in positive feedback learning was mediated by reduced GMD in the parietal, medial, and dorsolateral prefrontal cortex. Our findings indicate that brain maturation, putatively reflected in lower GMD, in distinct and partially overlapping brain regions could lead to a more efficient brain organization and might thus be a key developmental step toward age-related increases in planning and value-based choice.SIGNIFICANCE STATEMENT Changes in model-based decision-making are paralleled by extensive maturation in cognition and brain structure across development. Still, to date the neuroanatomical underpinnings of these changes remain unclear. Here, we demonstrate for the first time that parietal GMD mediates age-dependent increases in model-based control. Age-related increases in positive feedback learning were mediated by reduced GMD in the parietal, medial, and dorsolateral prefrontal cortex. A manipulation of motivational context did not have an impact on age-related changes in model-based control. These findings highlight that brain maturation in distinct and overlapping cortical regions constitutes a key developmental step toward improved value-based choices.

The paradigm of tax-reward and tax-punishment strategies in the advancement of public resource management dynamics.

In contemporary society, the effective utilization of public resources remains a subject of significant concern. A common issue arises from defectors seeking to obtain an excessive share of these resources for personal gain, potentially leading to resource depletion. To mitigate this tragedy and ensure sustainable development of resources, implementing mechanisms to either reward those who adhere to distribution rules or penalize those who do not, appears advantageous. We introduce two models: a tax-reward model and a tax-punishment model, to address this issue. Our analysis reveals that in the tax-reward model, the evolutionary trajectory of the system is influenced not only by the tax revenue collected but also by the natural growth rate of the resources. Conversely, the tax-punishment model exhibits distinct characteristics when compared with the tax-reward model, notably the potential for bistability. In such scenarios, the selection of initial conditions is critical, as it can determine the system's path. Furthermore, our study identifies instances where the system lacks stable points, exemplified by a limit cycle phenomenon, underscoring the complexity and dynamism inherent in managing public resources using these models.

Effects of psilocybin on uncertain punishment learning.

Psilocybin may provide a useful treatment for mood disorders including anxiety and depression but its mechanisms of action for these effects are not well understood. While recent preclinical work has begun to assess psilocybin's role in affective behaviors through innate anxiety or fear conditioning, there is scant evidence for its role in conflict between reward and punishment. The current study was designed to determine the impact of psilocybin on the learning of reward-punishment conflict associations, as well as its effects after learning, in male and female rats. We utilized a chained schedule of reinforcement that involved execution of safe and risky reward-guided actions under uncertain punishment. Different patterns of behavioral suppression by psilocybin emerged during learning versus after learning of risky action-reward associations. Psilocybin increased behavioral suppression in female rats as punishment associations were learned. After learning, psilocybin decreased behavioral suppression in both sexes. Thus, psilocybin produces divergent effects on action suppression during approach-avoidance conflict depending on when the conflict is experienced. This observation may have implications for its therapeutic mechanism of action.

Infralimbic cortex plays a similar role in the punishment and extinction of instrumental behavior.

Learning to stop responding is a fundamental process in instrumental learning. Animals may learn to stop responding under a variety of conditions that include punishment-where the response earns an aversive stimulus in addition to a reinforcer-and extinction-where a reinforced response now earns nothing at all. Recent research suggests that punishment and extinction may be related manifestations of a common retroactive interference process. In both paradigms, animals learn to stop performing a specific response in a specific context, suggesting direct inhibition of the response by the context. This process may depend on the infralimbic cortex (IL), which has been implicated in a variety of interference-based learning paradigms including extinction and habit learning. Despite the behavioral parallels between extinction and punishment, a corresponding role for IL in punishment has not been identified. Here we report that, in a simple arrangement where either punishment or extinction was conducted in a context that differed from the context in which the behavior was first acquired, IL inactivation reduced response suppression in the inhibitory context, but not responding when it "renewed" in the original context. In a more complex arrangement in which two responses were first trained in different contexts and then extinguished or punished in the opposite one, IL inactivation had no effect. The results advance our understanding of the effects of IL in retroactive interference and the behavioral mechanisms that can produce suppression of a response.

Understanding sex differences and the translational value of models of persistent substance use despite negative consequences.

Persistent substance use despite negative consequences is a key facet of substance use disorder. The last decade has seen the preclinical field adopt the use of punishment to model adverse consequences associated with substance use. This has largely involved the pairing of drug use with either electric foot shock or quinine, a bitter tastant. Whilst at face value, these punishers may model aspects of the physical and psychological consequences of substance use, such models are yet to assist the development of approved medications for treatment. This review discusses progress made with animal models of punishment to understand the behavioral consequences of persistent substance use despite negative consequences. We highlight the importance of examining sex differences, especially when the behavioral response to punishment changes following drug exposure. Finally, we critique the translational value these models provide for the substance use disorder field.

Personal Misconduct Elicits Harsher Professional Consequences for Artists (vs. Scientists): A Moral-Decoupling Process.

Recent years have brought increased accountability for personal misconduct, yet often, unequal consequences have resulted from similar offenses. Findings from a unique archival data set (N = 619; all university faculty) and three preregistered experiments (N = 2,594) show that the perceived artistic-versus-scientific nature of the offender's professional contributions influences the professional punishment received. In Study 1, analysis of four decades of university sexual-misconduct cases reveals that faculty in artistic (vs. scientific) fields have on average received more severe professional consequences. Study 2 demonstrates this experimentally, offering mediational evidence that greater difficulty morally decoupling art (vs. science) contributes to the phenomenon. Study 3 provides further evidence for this mechanism through experimental moderation. Finally, Study 4 shows that merely framing an individual's work as artistic versus scientific results in replication of these effects. Several potential alternative mechanisms to moral decoupling are tested but not supported. Practical and theoretical implications are discussed.

IR76b-expressing neurons in Drosophila melanogaster are necessary for associative reward learning of an amino acid mixture.

Learning where to find nutrients while at the same time avoiding toxic food is essential for survival of any animal. Using Drosophila melanogaster larvae as a study case, we investigate the role of gustatory sensory neurons expressing IR76b for associative learning of amino acids, the building blocks of proteins. We found surprising complexity in the neuronal underpinnings of sensing amino acids, and a functional division of sensory neurons. We found that the IR76b receptor is dispensable for amino acid learning, whereas the neurons expressing IR76b are specifically required for the rewarding but not the punishing effect of amino acids. This unexpected dissociation in neuronal processing of amino acids for different behavioural functions provides a study case for functional divisions of labour in gustatory systems.

Reinforcement learning and the reward positivity with aversive outcomes.

The reinforcement learning (RL) theory of the reward positivity (RewP), an event-related potential (ERP) component that measures reward responsivity, suggests that the RewP should be largest when positive outcomes are unexpected and has been supported by work using appetitive outcomes (e.g., money). However, the RewP can also be elicited by the absence of aversive outcomes (e.g., shock). The limited work to-date that has manipulated expectancy while using aversive outcomes has not supported the predictions of RL theory. Nonetheless, this work has been difficult to reconcile with the appetitive literature because the RewP was not observed as a reward signal in these studies, which used passive tasks that did not involve participant choice. Here, we tested the predictions of the RL theory by manipulating expectancy in an active/choice-based threat-of-shock doors task that was previously found to elicit the RewP as a reward signal. Moreover, we used principal components analysis to isolate the RewP from overlapping ERP components. Eighty participants viewed pairs of doors surrounded by a red or green border; shock delivery was expected (80%) following red-bordered doors and unexpected (20%) following green-bordered doors. The RewP was observed as a reward signal (i.e., no shock > shock) that was not potentiated for unexpected feedback. In addition, the RewP was larger overall for unexpected (vs expected) feedback. Therefore, the RewP appears to reflect the additive (not interactive) effects of reward and expectancy, challenging the RL theory of the RewP, at least when reward is defined as the absence of an aversive outcome.

Context Modulates Perceived Fairness in Altruistic Punishment: Neural Signatures from ERPs and EEG Oscillations.

Social norms and altruistic punitive behaviours are both based on the integration of information from multiple contexts. Individual behavioural performance can be altered by loss and gain contexts, which produce different mental states and subjective perceptions. In this study, we used event-related potential and time-frequency techniques to examine performance on a third-party punishment task and to explore the neural mechanisms underlying context-dependent differences in punishment decisions. The results indicated that individuals were more likely to reject unfairness in the context of loss (vs. gain) and to increase punishment as unfairness increased. In contrast, fairness appeared to cause an early increase in cognitive control signal enhancement, as indicated by the P2 amplitude and theta oscillations, and a later increase in emotional and motivational salience during decision-making in gain vs. loss contexts, as indicated by the medial frontal negativity and beta oscillations. In summary, individuals were more willing to sanction violations of social norms in the loss context than in the gain context and rejecting unfair losses induced more equity-related cognitive conflict than accepting unfair gains, highlighting the importance of context (i.e., gain vs. loss) in equity-related social decision-making processes.

Disrupting the Dorsolateral Prefrontal Cortex Attenuates the Difference in Decision-Making for Altruistic Punishment Between the Gain and Loss Contexts.

Altruistic punishment is a primary response to social norms violations; its neural mechanism has also attracted extensive research attention. In the present studies, we applied a low-frequency repetitive transcranial magnetic stimulation (rTMS) to the bilateral dorsolateral prefrontal cortex (DLPFC) while participants engaged in a modified Ultimatum Game (Study 1) and third-party punishment game (Study 2) to explore how the bilateral DLPFC disruption affects people's perception of violation of fairness norms and altruistic punishment decision in the gain and loss contexts. Typically, punishers intervene more often against and show more social outrage towards Dictators/Proposers who unfairly distribute losses than those who unfairly share gains. We found that disrupting the function of the left DLPFC in the second-party punishment and the bilateral DLPFC in the third-party punishment with rTMS effectively obliterated this difference, making participants punish unfairly shared gains as often as they usually would punish unfairly shared losses. In the altruistic punishment of maintaining the social fairness norms, the inhibition of the right DLPFC function will affect the deviation of individual information integration ability; the inhibition of the left DLPFC function will affect the assessment of the degree of violation of fairness norms and weaken impulse control, leading to attenuate the moderating effect of gain and loss contexts on altruistic punishment. Our findings emphasize that DLPFC is closely related to altruistic punishment and provide causal neuroscientific evidence.