Mice exhibit stochastic and efficient action switching during probabilistic decision making.

In probabilistic and nonstationary environments, individuals must use internal and external cues to flexibly make decisions that lead to desirable outcomes. To gain insight into the process by which animals choose between actions, we trained mice in a task with time-varying reward probabilities. In our implementation of such a two-armed bandit task, thirsty mice use information about recent action and action­outcome histories to choose between two ports that deliver water probabilistically. Here we comprehensively modeled choice behavior in this task, including the trial-to-trial changes in port selection, i.e., action switching behavior. We find that mouse behavior is, at times, deterministic and, at others, apparently stochastic. The behavior deviates from that of a theoretically optimal agent performing Bayesian inference in a hidden Markov model (HMM). We formulate a set of models based on logistic regression, reinforcement learning, and sticky Bayesian inference that we demonstrate are mathematically equivalent and that accurately describe mouse behavior. The switching behavior of mice in the task is captured in each model by a stochastic action policy, a history-dependent representation of action value, and a tendency to repeat actions despite incoming evidence. The models parsimoniously capture behavior across different environmental conditionals by varying the stickiness parameter, and like the mice, they achieve nearly maximal reward rates. These results indicate that mouse behavior reaches near-maximal performance with reduced action switching and can be described by a set of equivalent models with a small number of relatively fixed parameters.

Reward-specific satiety affects subjective value signals in orbitofrontal cortex during multicomponent economic choice.

Sensitivity to satiety constitutes a basic requirement for neuronal coding of subjective reward value. Satiety from natural ongoing consumption affects reward functions in learning and approach behavior. More specifically, satiety reduces the subjective economic value of individual rewards during choice between options that typically contain multiple reward components. The unconfounded assessment of economic reward value requires tests at choice indifference between two options, which is difficult to achieve with sated rewards. By conceptualizing choices between options with multiple reward components ("bundles"), Revealed Preference Theory may offer a solution. Despite satiety, choices against an unaltered reference bundle may remain indifferent when the reduced value of a sated bundle reward is compensated by larger amounts of an unsated reward of the same bundle, and then the value loss of the sated reward is indicated by the amount of the added unsated reward. Here, we show psychophysically titrated choice indifference in monkeys between bundles of differently sated rewards. Neuronal chosen value signals in the orbitofrontal cortex (OFC) followed closely the subjective value change within recording periods of individual neurons. A neuronal classifier distinguishing the bundles and predicting choice substantiated the subjective value change. The choice between conventional single rewards confirmed the neuronal changes seen with two-reward bundles. Thus, reward-specific satiety reduces subjective reward value signals in OFC. With satiety being an important factor of subjective reward value, these results extend the notion of subjective economic reward value coding in OFC neurons.

Single-Dimensional Human Brain Signals for Two-Dimensional Economic Choice Options.

Rewarding choice options typically contain multiple components, but neural signals in single brain voxels are scalar and primarily vary up or down. In a previous study, we had designed reward bundles that contained the same two milkshakes with independently set amounts; we had used psychophysics and rigorous economic concepts to estimate two-dimensional choice indifference curves (ICs) that represented revealed stochastic preferences for these bundles in a systematic, integrated manner. All bundles on the same ICs were equally revealed preferred (and thus had same utility, as inferred from choice indifference); bundles on higher ICs (higher utility) were preferred to bundles on lower ICs (lower utility). In the current study, we used the established behavior for testing with functional magnetic resonance imaging (fMRI). We now demonstrate neural responses in reward-related brain structures of human female and male participants, including striatum, midbrain, and medial orbitofrontal cortex (mid-OFC) that followed the characteristic pattern of ICs: similar responses along ICs (same utility despite different bundle composition), but monotonic change across ICs (different utility). Thus, these brain structures integrated multiple reward components into a scalar signal, well beyond the known subjective value coding of single-component rewards.SIGNIFICANCE STATEMENT Rewards have several components, like the taste and size of an apple, but it is unclear how each component contributes to the overall value of the reward. While choice indifference curves (ICs) of economic theory provide behavioral approaches to this question, it is unclear whether brain responses capture the preference and utility integrated from multiple components. We report activations in striatum, midbrain, and orbitofrontal cortex (OFC) that follow choice ICs representing behavioral preferences over and above variations of individual reward components. In addition, the concept-driven approach encourages future studies on natural, multicomponent rewards that are prone to irrational choice of normal and brain-damaged individuals.

A universal transportin protein drives stochastic choice of olfactory neurons via specific nuclear import of a sox-2-activating factor.

Stochastic neuronal cell fate choice involving notch-independent mechanisms is a poorly understood biological process. The Caenorhabditis elegans AWC olfactory neuron pair asymmetrically differentiates into the default AWCOFF and induced AWCON subtypes in a stochastic manner. Stochastic choice of the AWCON subtype is established using gap junctions and SLO BK potassium channels to repress a calcium-activated protein kinase pathway. However, it is unknown how the potassium channel-repressed calcium signaling is translated into the induction of the AWCON subtype. Here, we identify a detailed working mechanism of how the homeodomain-like transcription factor NSY-7, previously described as a repressor in the maintenance of AWC asymmetry, couples SLO BK potassium channels to transactivation of sox-2 expression for the induction of the AWCON subtype through the identification of a unique imb-2 (transportin 1) allele. imb-2 loss-of-function mutants are not viable; however, we identify a viable imb-2 allele from an unbiased forward genetic screen that reveals a specific role of imb-2 in AWC olfactory neuron asymmetry. IMB-2 specifically drives nuclear import of NSY-7 within AWC neurons to transactivate the expression of the high mobility group (HMG)-box transcription factor SOX-2 for the specification of the AWCON subtype. This study provides mechanistic insight into how NSY-7 couples SLO BK potassium channels to transactivation of sox-2 expression for the induction of the AWCON subtype. Our findings also provide structure-function insight into a conserved amino acid residue of transportins in brain development and suggest its dysfunction may lead to human neurological disorders.

Balancing model-based and memory-free action selection under competitive pressure.

In competitive situations, winning depends on selecting actions that surprise the opponent. Such unpredictable action can be generated based on representations of the opponent's strategy and choice history (model-based counter-prediction) or by choosing actions in a memory-free, stochastic manner. Across five different experiments using a variant of a matching-pennies game with simulated and human opponents we found that people toggle between these two strategies, using model-based selection when recent wins signal the appropriateness of the current model, but reverting to stochastic selection following losses. Also, after wins, feedback-related, mid-frontal EEG activity reflected information about the opponent's global and local strategy, and predicted upcoming choices. After losses, this activity was nearly absent-indicating that the internal model is suppressed after negative feedback. We suggest that the mixed-strategy approach allows negotiating two conflicting goals: 1) exploiting the opponent's deviations from randomness while 2) remaining unpredictable for the opponent.

Variable time preference.

We re-examine behavioral patterns of intertemporal choice with recognition that time preferences may be inherently variable, focusing in particular on the explanatory power of an exponential discounting model with variable discount factors - the variable exponential model. We provide analytical results showing that this model can generate systematically different choice patterns from an exponential discounting model with a fixed discount factor. The variable exponential model accounts for the common behavioral pattern of decreasing impatience, which is typically attributed to hyperbolic discounting. The variable exponential model also generates violations of strong stochastic transitivity in choices involving intertemporal dominance. We present the results of two experiments designed to evaluate the variable exponential model in terms of quantitative fit to individual-level choice data. Data from these experiments reveal that allowing for a variable discount factor significantly improves the fit of the exponential model, and that a variable exponential model provides a better account of individual-level choice probabilities than hyperbolic discounting models. In a third experiment we find evidence of strong stochastic transitivity violations when intertemporal dominance is involved, in accordance with the variable exponential model. Overall, our analytical and experimental results indicate that exponential discounting can explain intertemporal choice behavior that was supposed to be beyond its descriptive scope if the discount factor is permitted to vary at random. Our results also highlight the importance of allowing for different sources of randomness in choice modeling.

Post-Anesthesia Evaluation Shows No Relationship Between Body Conditions and Salinity-Choice Propensities in Anguilla japonica Glass Eels.

Anguillid eels generally exhibit catadromous migration between oceanic spawning grounds and freshwater growth habitats, but some individuals remain in coastal or estuarine saline waters for growth. This migratory plasticity had been considered to be a conditional strategy based on individual energetic status during the glass eel stage. Several studies have examined whether salinity-based habitat selection is linked to individual body conditions, but while frozen specimens of European eels showed this relationship, anesthetized samples of American eels did not. Here, we report that freezing preservation under different salinity levels influences body-condition evaluation in Japanese eels. Behavioral tests of Japanese eels did not reveal significant differences in anesthetized body conditions between those choosing saltwater and those choosing freshwater. In conclusion, the body conditions of glass-eel-stage Japanese eels are unlikely to be associated with their salinity-choice propensity.

Parallels between Mammalian Mechanisms of Monoallelic Gene Expression.

Different types of monoallelic gene expression are present in mammals, some of which are highly flexible, whereas others are more rigid. These include allelic exclusion at antigen receptor loci, the expression of olfactory receptor genes, genomic imprinting, X-chromosome inactivation, and random monoallelic expression (MAE). Although these processes play diverse biological roles, and arose through different selective pressures, the underlying epigenetic mechanisms show striking resemblances. Regulatory transcriptional events are important in all systems, particularly in the specification of MAE. Combined with comparative studies between species, this suggests that the different MAE systems found in mammals may have evolved from analogous ancestral processes.