Human subjects exploit a cognitive map for credit assignment.

An influential reinforcement learning framework proposes that behavior is jointly governed by model-free (MF) and model-based (MB) controllers. The former learns the values of actions directly from past encounters, and the latter exploits a cognitive map of the task to calculate these prospectively. Considerable attention has been paid to how these systems interact during choice, but how and whether knowledge of a cognitive map contributes to the way MF and MB controllers assign credit (i.e., to how they revaluate actions and states following the receipt of an outcome) remains underexplored. Here, we examine such sophisticated credit assignment using a dual-outcome bandit task. We provide evidence that knowledge of a cognitive map influences credit assignment in both MF and MB systems, mediating subtly different aspects of apparent relevance. Specifically, we show MF credit assignment is enhanced for those rewards that are related to a choice, and this contrasted with choice-unrelated rewards that reinforced subsequent choices negatively. This modulation is only possible based on knowledge of task structure. On the other hand, MB credit assignment was boosted for outcomes that impacted on differences in values between offered bandits. We consider mechanistic accounts and the normative status of these findings. We suggest the findings extend the scope and sophistication of cognitive map-based credit assignment during reinforcement learning, with implications for understanding behavioral control.

Exploration heuristics decrease during youth.

Deciding between exploring new avenues and exploiting known choices is central to learning, and this exploration-exploitation trade-off changes during development. Exploration is not a unitary concept, and humans deploy multiple distinct mechanisms, but little is known about their specific emergence during development. Using a previously validated task in adults, changes in exploration mechanisms were investigated between childhood (8-9 y/o, N = 26; 16 females), early (12-13 y/o, N = 38; 21 females), and late adolescence (16-17 y/o, N = 33; 19 females) in ethnically and socially diverse schools from disadvantaged areas. We find an increased usage of a computationally light exploration heuristic in younger groups, effectively accommodating their limited neurocognitive resources. Moreover, this heuristic was associated with self-reported, attention-deficit/hyperactivity disorder symptoms in this population-based sample. This study enriches our mechanistic understanding about how exploration strategies mature during development.

Model based planners reflect on their model-free propensities.

Dual-reinforcement learning theory proposes behaviour is under the tutelage of a retrospective, value-caching, model-free (MF) system and a prospective-planning, model-based (MB), system. This architecture raises a question as to the degree to which, when devising a plan, a MB controller takes account of influences from its MF counterpart. We present evidence that such a sophisticated self-reflective MB planner incorporates an anticipation of the influences its own MF-proclivities exerts on the execution of its planned future actions. Using a novel bandit task, wherein subjects were periodically allowed to design their environment, we show that reward-assignments were constructed in a manner consistent with a MB system taking account of its MF propensities. Thus, in the task participants assigned higher rewards to bandits that were momentarily associated with stronger MF tendencies. Our findings have implications for a range of decision making domains that includes drug abuse, pre-commitment, and the tension between short and long-term decision horizons in economics.

Credit assignment to state-independent task representations and its relationship with model-based decision making.

Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment's structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challenge this assumption by providing evidence that a putative model-free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well-described 2-step reward decision task where stimulus identity but not spatial-motor aspects of the task predicted reward. We show that participants assigned value to spatial-motor representations despite it being outcome irrelevant. Strikingly, spatial-motor value associations affected behavior across all outcome-relevant features and stages of the task, consistent with credit assignment to low-level state-independent task representations. Individual difference analyses suggested that the impact of spatial-motor value formation was attenuated for individuals who showed greater deployment of goal-directed (model-based) strategies. Our findings highlight a need for a reconsideration of how model-free representations are formed and regulated according to the structure of the environment.

Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling.

A well-established notion in cognitive neuroscience proposes that multiple brain systems contribute to choice behaviour. These include: (1) a model-free system that uses values cached from the outcome history of alternative actions, and (2) a model-based system that considers action outcomes and the transition structure of the environment. The widespread use of this distinction, across a range of applications, renders it important to index their distinct influences with high reliability. Here we consider the two-stage task, widely considered as a gold standard measure for the contribution of model-based and model-free systems to human choice. We tested the internal/temporal stability of measures from this task, including those estimated via an established computational model, as well as an extended model using drift-diffusion. Drift-diffusion modeling suggested that both choice in the first stage, and RTs in the second stage, are directly affected by a model-based/free trade-off parameter. Both parameter recovery and the stability of model-based estimates were poor but improved substantially when both choice and RT were used (compared to choice only), and when more trials (than conventionally used in research practice) were included in our analysis. The findings have implications for interpretation of past and future studies based on the use of the two-stage task, as well as for characterising the contribution of model-based processes to choice behaviour.

Opportunities for emotion and mental health research in the resource-rationality framework.

We discuss opportunities in applying the resource-rationality framework toward answering questions in emotion and mental health research. These opportunities rely on characterization of individual differences in cognitive strategies; an endeavor that may be at odds with the normative approach outlined in the target article. We consider ways individual differences might enter the framework and the translational opportunities offered by each.

Economic irrationality is optimal during noisy decision making.

According to normative theories, reward-maximizing agents should have consistent preferences. Thus, when faced with alternatives A, B, and C, an individual preferring A to B and B to C should prefer A to C. However, it has been widely argued that humans can incur losses by violating this axiom of transitivity, despite strong evolutionary pressure for reward-maximizing choices. Here, adopting a biologically plausible computational framework, we show that intransitive (and thus economically irrational) choices paradoxically improve accuracy (and subsequent economic rewards) when decision formation is corrupted by internal neural noise. Over three experiments, we show that humans accumulate evidence over time using a "selective integration" policy that discards information about alternatives with momentarily lower value. This policy predicts violations of the axiom of transitivity when three equally valued alternatives differ circularly in their number of winning samples. We confirm this prediction in a fourth experiment reporting significant violations of weak stochastic transitivity in human observers. Crucially, we show that relying on selective integration protects choices against "late" noise that otherwise corrupts decision formation beyond the sensory stage. Indeed, we report that individuals with higher late noise relied more strongly on selective integration. These findings suggest that violations of rational choice theory reflect adaptive computations that have evolved in response to irreducible noise during neural information processing.

The standard Bayesian model is normatively invalid for biological brains.

We show that the benchmark Bayesian framework that Rahnev & Denison (R&D) used to assess optimality is actually suboptimal under realistic assumptions about how noise corrupts decision making in biological brains. This model is therefore invalid qua normative standard. We advise against generally forsaking optimality and argue that a biologically constrained definition of optimality could serve as an important driver for scientific progress.

Parallel attentive processing and pre-attentive guidance.

This commentary focuses on two related, open questions in Hulleman & Olivers' (H&O's) proposal: (1) the nature of the parallel attentive process that determines target presence within, and thus presumably the size of, the functional visual field, and (2) how the pre-attentive guidance mechanism must be conceived to also account for search performance in tasks that afford no reliable target-based guidance.

An appeal against the item's death sentence: Accounting for diagnostic data patterns with an item-based model of visual search.

We show that our item-based model, competitive guided search, accounts for the empirical patterns that Hulleman & Olivers (H&O) invoke against item-based models, and we highlight recently reported diagnostic data that challenge their approach. We advise against "forsaking the item" unless and until a full fixation-based model is shown to be superior to extant item-based models.

Decisions reduce sensitivity to subsequent information.

Behavioural studies over half a century indicate that making categorical choices alters beliefs about the state of the world. People seem biased to confirm previous choices, and to suppress contradicting information. These choice-dependent biases imply a fundamental bound of human rationality. However, it remains unclear whether these effects extend to lower level decisions, and only little is known about the computational mechanisms underlying them. Building on the framework of sequential-sampling models of decision-making, we developed novel psychophysical protocols that enable us to dissect quantitatively how choices affect the way decision-makers accumulate additional noisy evidence. We find robust choice-induced biases in the accumulation of abstract numerical (experiment 1) and low-level perceptual (experiment 2) evidence. These biases deteriorate estimations of the mean value of the numerical sequence (experiment 1) and reduce the likelihood to revise decisions (experiment 2). Computational modelling reveals that choices trigger a reduction of sensitivity to subsequent evidence via multiplicative gain modulation, rather than shifting the decision variable towards the chosen alternative in an additive fashion. Our results thus show that categorical choices alter the evidence accumulation mechanism itself, rather than just its outcome, rendering the decision-maker less sensitive to new information.

Old processes, new perspectives: Familiarity is correlated with (not independent of) recollection and is more (not equally) variable for targets than for lures.

According to dual-process models of memory, recognition is subserved by two processes: recollection and familiarity. Many variants of these models assume that recollection and familiarity make stochastically independent contributions to performance in recognition tasks and that the variance of the familiarity signal is equal for targets and for lures. Here, we challenge these 'common-currency' assumptions. Using a model-comparison approach, featuring the Continuous Dual Process (CDP; Wixted & Mickes, 2010) model as the protagonist, we show that when these assumptions are relaxed, the model's fits to individual participants' data improve. Furthermore, our analyses reveal that across items, recollection and familiarity show a positive correlation. Interestingly, this across-items correlation was dissociated from an across-participants correlation between the sensitivities of these processes. We also find that the familiarity signal is significantly more variable for targets than for lures. One striking theoretical implication of these findings is that familiarity-rather than recollection, as most models assume-may be the main contributor responsible for one of the most influential findings of recognition memory, that of subunit zROC slopes. Additionally, we show that erroneously adopting the common-currency assumptions, introduces severe biases to estimates of recollection and familiarity.

Post choice information integration as a causal determinant of confidence: Novel data and a computational account.

Confidence judgments are pivotal in the performance of daily tasks and in many domains of scientific research including the behavioral sciences, psychology and neuroscience. Positive resolution i.e., the positive correlation between choice-correctness and choice-confidence is a critical property of confidence judgments, which justifies their ubiquity. In the current paper, we study the mechanism underlying confidence judgments and their resolution by investigating the source of the inputs for the confidence-calculation. We focus on the intriguing debate between two families of confidence theories. According to single stage theories, confidence is based on the same information that underlies the decision (or on some other aspect of the decision process), whereas according to dual stage theories, confidence is affected by novel information that is collected after the decision was made. In three experiments, we support the case for dual stage theories by showing that post-choice perceptual availability manipulations exert a causal effect on confidence-resolution in the decision followed by confidence paradigm. These finding establish the role of RT2, the duration of the post-choice information-integration stage, as a prime dependent variable that theories of confidence should account for. We then present a novel list of robust empirical patterns ('hurdles') involving RT2 to guide further theorizing about confidence judgments. Finally, we present a unified computational dual stage model for choice, confidence and their latencies namely, the collapsing confidence boundary model (CCB). According to CCB, a diffusion-process choice is followed by a second evidence-integration stage towards a stochastic collapsing confidence boundary. Despite its simplicity, CCB clears the entire list of hurdles.

Efficiency in redundancy.

In engineering, redundancy is the duplication of vital systems for use in the event of failure. In studies of human cognition, redundancy often refers to the duplication of the signal. Scores of studies have shown the salutary effects of a combined auditory and visual signal over single modality, the advantage of processing complete faces over facial features, and more recently the advantage of two observers over one. But what if the signal (or the number of observers) is fixed and cannot be altered or augmented? Can people improve the efficiency of information processing by recruiting an additional, redundant system? Here we demonstrate that recruiting a second redundant system can, under reasonable assumptions about human capacity, result in improved performance. Recruiting a second redundant system may come with a higher energy cost, but may be worthwhile in high-stakes situations where processing information accurately is crucial.

Heuristics in risky decision-making relate to preferential representation of information.

When making choices, individuals differ from one another, as well as from normativity, in how they weigh different types of information. One explanation for this relates to idiosyncratic preferences in what information individuals represent when evaluating choice options. Here, we test this explanation with a simple risky-decision making task, combined with magnetoencephalography (MEG). We examine the relationship between individual differences in behavioral markers of information weighting and neural representation of stimuli pertinent to incorporating that information. We find that the extent to which individuals (N = 19) behaviorally weight probability versus reward information is related to how preferentially they neurally represent stimuli most informative for making probability and reward comparisons. These results are further validated in an additional behavioral experiment (N = 88) that measures stimulus representation as the latency of perceptual detection following priming. Overall, the results suggest that differences in the information individuals consider during choice relate to their risk-taking tendencies.

Competitive guided search: meeting the challenge of benchmark RT distributions.

Historically, visual search models were mainly evaluated based on their account of mean reaction times (RTs) and accuracy data. More recently, Wolfe, Palmer, and Horowitz (2010) have demonstrated that the shape of the entire RT distributions imposes important constraints on visual search theories and can falsify even successful models such as guided search, raising a challenge to computational theories of search. Competitive guided search is a novel model that meets this important challenge. The model is an adaptation of guided search, featuring a series of item selection and identification iterations with guidance towards targets. The main novelty of the model is its termination rule: A quit unit, which aborts the search upon selection, competes with items for selection and is inhibited by the saliency map of the visual display. As the trial proceeds, the quit unit both increases in strength and suffers less saliency-based inhibition and hence the conditional probability of quitting the trial accelerates. The model is fitted to data the data from three classical search task that have been traditionally considered to be governed by qualitatively different mechanisms, including a spatial configuration, a conjunction, and a feature search (Wolfe et al., 2010). The model is mathematically tractable and it accounts for the properties of RT distributions and for error rates in all three search tasks, providing a unifying theoretical framework for visual search.

Evidence integration and decision confidence are modulated by stimulus consistency.

Evidence integration is a normative algorithm for choosing between alternatives with noisy evidence, which has been successful in accounting for vast amounts of behavioural and neural data. However, this mechanism has been challenged by non-integration heuristics, and tracking decision boundaries has proven elusive. Here we first show that the decision boundaries can be extracted using a model-free behavioural method termed decision classification boundary, which optimizes choice classification based on the accumulated evidence. Using this method, we provide direct support for evidence integration over non-integration heuristics, show that the decision boundaries collapse across time and identify an integration bias whereby incoming evidence is modulated based on its consistency with preceding information. This consistency bias, which is a form of pre-decision confirmation bias, was supported in four cross-domain experiments, showing that choice accuracy and decision confidence are modulated by stimulus consistency. Strikingly, despite its seeming sub-optimality, the consistency bias fosters performance by enhancing robustness to integration noise.

The nature of Garner interference: The role of uncertainty, information, and variation in the breakdown in selective attention.

The popular measure of Garner Interference specifies the detriment to performance with the task-relevant attribute in the presence of a randomly varying distractor. But is irrelevant variation per se responsible for this breakdown of selective attention as the traditional account suggests? In this study we identified an overlooked alternative account - increased irrelevant information - which threatens the validity of the variation interpretation. We designed a new condition within the Garner paradigm, Roving Baseline, which allowed for dissociating the separate and combined contributions of information and variation at both macro and micro levels of analysis. A third account, increased number of stimuli or stimulus uncertainty, was also considered as well as the rival interpretations of configural processing and change detection. Our conceptual assay was complemented by a pair of dedicated experiments that included the novel Roving Baseline condition. The results of the theoretical analysis and of the experiments converged on supporting variability as the source of Garner interference. We found no evidence for an influence of information or of stimulus uncertainty. Our study thus adds further support for W. R. Garner's original intuition when designing the paradigm and the interference bearing his name.

Diversity of opinions promotes herding in uncertain crowds.

Classic and recent studies demonstrate how we fall for the 'tyranny of the majority' and conform to the dominant trend when uncertain. However, in many social interactions outside of the laboratory, there is rarely a clearly identified majority and discerning who to follow might be challenging. Here, we asked whether in such conditions herding behaviour depends on a key statistical property of social information: the variance of opinions in a group. We selected a task domain where opinions are widely variable and asked participants (N = 650) to privately estimate the price of eight anonymous paintings. Then, in groups of five, they discussed and agreed on a shared estimate for four paintings. Finally, they provided revised individual estimates for all paintings. As predicted (https://osf.io/s89w4), we observed that group members converged to each other and boosted their confidence following social interaction. We also found evidence supporting the hypothesis that the more diverse groups show greater convergence, suggesting that the variance of opinions promotes herding in uncertain crowds. Overall, these findings empirically examine how, in the absence of a clear majority, the distribution of opinions relates to subjective feelings of confidence and herding behaviour.