Toward a Rational and Mechanistic Account of Mental Effort.

In spite of its familiar phenomenology, the mechanistic basis for mental effort remains poorly understood. Although most researchers agree that mental effort is aversive and stems from limitations in our capacity to exercise cognitive control, it is unclear what gives rise to those limitations and why they result in an experience of control as costly. The presence of these control costs also raises further questions regarding how best to allocate mental effort to minimize those costs and maximize the attendant benefits. This review explores recent advances in computational modeling and empirical research aimed at addressing these questions at the level of psychological process and neural mechanism, examining both the limitations to mental effort exertion and how we manage those limited cognitive resources. We conclude by identifying remaining challenges for theoretical accounts of mental effort as well as possible applications of the available findings to understanding the causes of and potential solutions for apparent failures to exert the mental effort required of us.

Compositional Sequence Generation in the Entorhinal-Hippocampal System.

Neurons in the medial entorhinal cortex exhibit multiple, periodically organized, firing fields which collectively appear to form an internal representation of space. Neuroimaging data suggest that this grid coding is also present in other cortical areas such as the prefrontal cortex, indicating that it may be a general principle of neural functionality in the brain. In a recent analysis through the lens of dynamical systems theory, we showed how grid coding can lead to the generation of a diversity of empirically observed sequential reactivations of hippocampal place cells corresponding to traversals of cognitive maps. Here, we extend this sequence generation model by describing how the synthesis of multiple dynamical systems can support compositional cognitive computations. To empirically validate the model, we simulate two experiments demonstrating compositionality in space or in time during sequence generation. Finally, we describe several neural network architectures supporting various types of compositionality based on grid coding and highlight connections to recent work in machine learning leveraging analogous techniques.

Subgoal- and Goal-related Reward Prediction Errors in Medial Prefrontal Cortex.

A longstanding view of the organization of human and animal behavior holds that behavior is hierarchically organized-in other words, directed toward achieving superordinate goals through the achievement of subordinate goals or subgoals. However, most research in neuroscience has focused on tasks without hierarchical structure. In past work, we have shown that negative reward prediction error (RPE) signals in medial prefrontal cortex (mPFC) can be linked not only to superordinate goals but also to subgoals. This suggests that mPFC tracks impediments in the progression toward subgoals. Using fMRI of human participants engaged in a hierarchical navigation task, here we found that mPFC also processes positive prediction errors at the level of subgoals, indicating that this brain region is sensitive to advances in subgoal completion. However, when subgoal RPEs were elicited alongside with goal-related RPEs, mPFC responses reflected only the goal-related RPEs. These findings suggest that information from different levels of hierarchy is processed selectively, depending on the task context.

Predictive representations can link model-based reinforcement learning to model-free mechanisms.

Humans and animals are capable of evaluating actions by considering their long-run future rewards through a process described using model-based reinforcement learning (RL) algorithms. The mechanisms by which neural circuits perform the computations prescribed by model-based RL remain largely unknown; however, multiple lines of evidence suggest that neural circuits supporting model-based behavior are structurally homologous to and overlapping with those thought to carry out model-free temporal difference (TD) learning. Here, we lay out a family of approaches by which model-based computation may be built upon a core of TD learning. The foundation of this framework is the successor representation, a predictive state representation that, when combined with TD learning of value predictions, can produce a subset of the behaviors associated with model-based learning, while requiring less decision-time computation than dynamic programming. Using simulations, we delineate the precise behavioral capabilities enabled by evaluating actions using this approach, and compare them to those demonstrated by biological organisms. We then introduce two new algorithms that build upon the successor representation while progressively mitigating its limitations. Because this framework can account for the full range of observed putatively model-based behaviors while still utilizing a core TD framework, we suggest that it represents a neurally plausible family of mechanisms for model-based evaluation.

Evidence integration in model-based tree search.

Research on the dynamics of reward-based, goal-directed decision making has largely focused on simple choice, where participants decide among a set of unitary, mutually exclusive options. Recent work suggests that the deliberation process underlying simple choice can be understood in terms of evidence integration: Noisy evidence in favor of each option accrues over time, until the evidence in favor of one option is significantly greater than the rest. However, real-life decisions often involve not one, but several steps of action, requiring a consideration of cumulative rewards and a sensitivity to recursive decision structure. We present results from two experiments that leveraged techniques previously applied to simple choice to shed light on the deliberation process underlying multistep choice. We interpret the results from these experiments in terms of a new computational model, which extends the evidence accumulation perspective to multiple steps of action.

Statistical learning of temporal community structure in the hippocampus.

The hippocampus is involved in the learning and representation of temporal statistics, but little is understood about the kinds of statistics it can uncover. Prior studies have tested various forms of structure that can be learned by tracking the strength of transition probabilities between adjacent items in a sequence. We test whether the hippocampus can learn higher-order structure using sequences that have no variance in transition probability and instead exhibit temporal community structure. We find that the hippocampus is indeed sensitive to this form of structure, as revealed by its representations, activity dynamics, and connectivity with other regions. These findings suggest that the hippocampus is a sophisticated learner of environmental regularities, able to uncover higher-order structure that requires sensitivity to overlapping associations.

Dorsal anterior cingulate and ventromedial prefrontal cortex have inverse roles in both foraging and economic choice.

Recent research has highlighted a distinction between sequential foraging choices and traditional economic choices between simultaneously presented options. This was partly motivated by observations in Kolling, Behrens, Mars, and Rushworth, Science, 336(6077), 95-98 (2012) (hereafter, KBMR) that these choice types are subserved by different circuits, with dorsal anterior cingulate (dACC) preferentially involved in foraging and ventromedial prefrontal cortex (vmPFC) preferentially involved in economic choice. To support this account, KBMR used fMRI to scan human subjects making either a foraging choice (between exploiting a current offer or swapping for potentially better rewards) or an economic choice (between two reward-probability pairs). This study found that dACC better tracked values pertaining to foraging, whereas vmPFC better tracked values pertaining to economic choice. We recently showed that dACC's role in these foraging choices is better described by the difficulty of choosing than by foraging value, when correcting for choice biases and testing a sufficiently broad set of foraging values (Shenhav, Straccia, Cohen, & Botvinick Nature Neuroscience, 17(9), 1249-1254, 2014). Here, we extend these findings in 3 ways. First, we replicate our original finding with a larger sample and a task modified to address remaining methodological gaps between our previous experiments and that of KBMR. Second, we show that dACC activity is best accounted for by choice difficulty alone (rather than in combination with foraging value) during both foraging and economic choices. Third, we show that patterns of vmPFC activity, inverted relative to dACC, also suggest a common function across both choice types. Overall, we conclude that both regions are similarly engaged by foraging-like and economic choice.

Cognitive effort avoidance and detection in people with schizophrenia.

Many people with schizophrenia exhibit avolition, a difficulty initiating and maintaining goal-directed behavior, considered to be a key negative symptom of the disorder. Recent evidence indicates that patients with higher levels of negative symptoms differ from healthy controls in showing an exaggerated cost of the physical effort needed to obtain a potential reward. We examined whether patients show an exaggerated avoidance of cognitive effort, using the demand selection task developed by Kool, McGuire, Rosen, and Botvinick (Journal of Experimental Psychology. General, 139, 665-682, 2010). A total of 83 people with schizophrenia or schizoaffective disorder and 71 healthy volunteers participated in three experiments where instructions varied. In the standard task (Experiment 1), neither controls nor patients showed expected cognitive demand avoidance. With enhanced instructions (Experiment 2), controls demonstrated greater demand avoidance than patients. In Experiment 3, patients showed nonsignificant reductions in demand avoidance, relative to controls. In a control experiment, patients showed significantly reduced ability to detect the effort demands associated with different response alternatives. In both groups, the ability to detect effort demands was associated with increased effort avoidance. In both groups, increased cognitive effort avoidance was associated with higher IQ and general neuropsychological ability. No significant correlations between demand avoidance and negative symptom severity were observed. Thus, it appears that individual differences in general intellectual ability and effort detection are related to cognitive effort avoidance and likely account for the subtle reduction in effort avoidance observed in schizophrenia.

Optimal behavioral hierarchy.

Human behavior has long been recognized to display hierarchical structure: actions fit together into subtasks, which cohere into extended goal-directed activities. Arranging actions hierarchically has well established benefits, allowing behaviors to be represented efficiently by the brain, and allowing solutions to new tasks to be discovered easily. However, these payoffs depend on the particular way in which actions are organized into a hierarchy, the specific way in which tasks are carved up into subtasks. We provide a mathematical account for what makes some hierarchies better than others, an account that allows an optimal hierarchy to be identified for any set of tasks. We then present results from four behavioral experiments, suggesting that human learners spontaneously discover optimal action hierarchies.

Mechanisms of motivation-cognition interaction: challenges and opportunities.

Recent years have seen a rejuvenation of interest in studies of motivation-cognition interactions arising from many different areas of psychology and neuroscience. The present issue of Cognitive, Affective, & Behavioral Neuroscience provides a sampling of some of the latest research from a number of these different areas. In this introductory article, we provide an overview of the current state of the field, in terms of key research developments and candidate neural mechanisms receiving focused investigation as potential sources of motivation-cognition interaction. However, our primary goal is conceptual: to highlight the distinct perspectives taken by different research areas, in terms of how motivation is defined, the relevant dimensions and dissociations that are emphasized, and the theoretical questions being targeted. Together, these distinctions present both challenges and opportunities for efforts aiming toward a more unified and cross-disciplinary approach. We identify a set of pressing research questions calling for this sort of cross-disciplinary approach, with the explicit goal of encouraging integrative and collaborative investigations directed toward them.

Neural representation of reward probability: evidence from the illusion of control.

To support reward-based decision-making, the brain must encode potential outcomes both in terms of their incentive value and their probability of occurrence. Recent research has made it clear that the brain bears multiple representations of reward magnitude, meaning that a single choice option may be represented differently-and even inconsistently-in different brain areas. There are some hints that the same may be true for reward probability. Preliminary evidence hints that, even as systematic distortions of probability are expressed in behavior, these may not always be uniformly reflected at the neural level: Some neural representations of probability may be immune from such distortions. This study provides new evidence consistent with this possibility. Participants in a behavioral experiment displayed a classic "illusion of control," providing higher estimates of reward probability for gambles they had chosen than for identical gambles that were imposed on them. However, an fMRI study of the same task revealed that neural prediction error signals, arising when gamble outcomes were revealed, were unaffected by the illusion of control. The resulting behavioral-neural dissociation reinforces the case for multiple, inconsistent internal representations of reward probability, while also prompting a reinterpretation of the illusion of control effect itself.

Prefrontal cortex, cognitive control, and the registration of decision costs.

Human choice behavior takes account of internal decision costs: people show a tendency to avoid making decisions in ways that are computationally demanding and subjectively effortful. Here, we investigate neural processes underlying the registration of decision costs. We report two functional MRI experiments that implicate lateral prefrontal cortex (LPFC) in this function. In Experiment 1, LPFC activity correlated positively with a self-report measure of costs as this measure varied over blocks of simple decisions. In Experiment 2, LPFC activity also correlated with individual differences in effort-based choice, taking on higher levels in subjects with a strong tendency to avoid cognitively demanding decisions. These relationships persisted even when effects of reaction time and error were partialled out, linking LPFC activity to subjectively experienced costs and not merely to response accuracy or time on task. In contrast to LPFC, dorsomedial frontal cortex--an area widely implicated in performance monitoring--showed no relationship to decision costs independent of overt performance. Previous work has implicated LPFC in executive control. Our results thus imply that costs may be registered based on the degree to which control mechanisms are recruited during decision-making.

Realizing the promise of AI: a new calling for cognitive science.

Rapid progress in artificial intelligence (AI) places a new spotlight on a long-standing question: how can we best develop AI to maximize its benefits to humanity? Answering this question in a satisfying and timely way represents an exciting challenge not only for AI research but also for all member disciplines of cognitive science.

Value representations in the rodent orbitofrontal cortex drive learning, not choice.

Humans and animals make predictions about the rewards they expect to receive in different situations. In formal models of behavior, these predictions are known as value representations, and they play two very different roles. Firstly, they drive choice: the expected values of available options are compared to one another, and the best option is selected. Secondly, they support learning: expected values are compared to rewards actually received, and future expectations are updated accordingly. Whether these different functions are mediated by different neural representations remains an open question. Here, we employ a recently developed multi-step task for rats that computationally separates learning from choosing. We investigate the role of value representations in the rodent orbitofrontal cortex, a key structure for value-based cognition. Electrophysiological recordings and optogenetic perturbations indicate that these representations do not directly drive choice. Instead, they signal expected reward information to a learning process elsewhere in the brain that updates choice mechanisms.

Errors of interpretation and modeling: a reply to Grinband et al.

Grinband et al., 2011 compare evidence that they have collected from a neuroimaging study of the Stroop task with a simulation model of performance and conflict in that task, and interpret the results as providing evidence against the theory that activity in dorsal medial frontal cortex (dMFC) reflects monitoring for conflict. Here, we discuss several errors in their methods and conclusions and show, contrary to their claims, that their findings are entirely consistent with previously published predictions of the conflict monitoring theory. Specifically, we point out that their argument rests on the assumption that conflict must be greater on all incongruent trials than on all congruent trials-an assumption that is theoretically and demonstrably incorrect. We also point out that their simulations are flawed and diverge substantially from previously published implementations of the conflict monitoring theory. When simulated appropriately, the conflict monitoring theory predicts precisely the patterns of results that Grinband et al. take to present serious challenges to the theory. Finally, we note that their proposal that dMFC activity reflects time on task is theoretically weak, pointing to a direct relationship between behavior (RT) and neural activity but failing to identify any intervening psychological construct to relate the two. The conflict monitoring theory provides such a construct, and a mechanistic implementation that continues to receive strong support from the neuroimaging literature, including the results reported by Grinband et al.

Us versus them: social identity shapes neural responses to intergroup competition and harm.

Intergroup competition makes social identity salient, which in turn affects how people respond to competitors' hardships. The failures of an in-group member are painful, whereas those of a rival out-group member may give pleasure-a feeling that may motivate harming rivals. The present study examined whether valuation-related neural responses to rival groups' failures correlate with likelihood of harming individuals associated with those rivals. Avid fans of the Red Sox and Yankees teams viewed baseball plays while undergoing functional magnetic resonance imaging. Subjectively negative outcomes (failure of the favored team or success of the rival team) activated anterior cingulate cortex and insula, whereas positive outcomes (success of the favored team or failure of the rival team, even against a third team) activated ventral striatum. The ventral striatum effect, associated with subjective pleasure, also correlated with self-reported likelihood of aggressing against a fan of the rival team (controlling for general aggression). Outcomes of social group competition can directly affect primary reward-processing neural systems, which has implications for intergroup harm.

Flexible modulation of sequence generation in the entorhinal-hippocampal system.

Exploration, consolidation and planning depend on the generation of sequential state representations. However, these algorithms require disparate forms of sampling dynamics for optimal performance. We theorize how the brain should adapt internally generated sequences for particular cognitive functions and propose a neural mechanism by which this may be accomplished within the entorhinal-hippocampal circuit. Specifically, we demonstrate that the systematic modulation along the medial entorhinal cortex dorsoventral axis of grid population input into the hippocampus facilitates a flexible generative process that can interpolate between qualitatively distinct regimes of sequential hippocampal reactivations. By relating the emergent hippocampal activity patterns drawn from our model to empirical data, we explain and reconcile a diversity of recently observed, but apparently unrelated, phenomena such as generative cycling, diffusive hippocampal reactivations and jumping trajectory events.

Toward an integrated account of object and action selection: a computational analysis and empirical findings from reaching-to-grasp and tool-use.

The act of reaching for and acting upon an object involves two forms of selection: selection of the object as a target, and selection of the action to be performed. While these two forms of selection are logically dissociable, and are evidently subserved by separable neural pathways, they must also be closely coordinated. We examine the nature of this coordination by developing and analyzing a computational model of object and action selection first proposed by Ward [Ward, R. (1999). Interactions between perception and action systems: a model for selective action. In G. W. Humphreys, J. Duncan, & A. Treisman (Eds.), Attention, Space and Action: Studies in Cognitive Neuroscience. Oxford: Oxford University Press]. An interesting tenet of this account, which we explore in detail, is that the interplay between object and action selection depends critically on top-down inputs representing the current task set or plan of action. A concrete manifestation of this, established through a series of simulations, is that the impact of distractor objects on reaching times can vary depending on the nature of the current action plan. In order to test the model's predictions in this regard, we conducted two experiments, one involving direct object manipulation, the other involving tool-use. In both experiments we observed the specific interaction between task set and distractor type predicted by the model. Our findings provide support for the computational model, and more broadly for an interactive account of object and action selection.

Empirical and computational support for context-dependent representations of serial order: reply to Bowers, Damian, and Davis (2009).

J. S. Bowers, M. F. Damian, and C. J. Davis critiqued the computational model of serial order memory put forth in M. Botvinick and D. C. Plaut, purporting to show that the model does not generalize in a way that people do. They attributed this supposed failure to the model's dependence on context-dependent representations, translating this argument into a general critique of all parallel distributed processing models. The authors reply here, addressing both Bowers et al.'s criticisms of the Botvinick and Plaut model and the former's assessment of parallel distributed processing models in general.