Joint modeling of choices and reaction times based on Bayesian contextual behavioral control.

In cognitive neuroscience and psychology, reaction times are an important behavioral measure. However, in instrumental learning and goal-directed decision making experiments, findings often rely only on choice probabilities from a value-based model, instead of reaction times. Recent advancements have shown that it is possible to connect value-based decision models with reaction time models. However, typically these models do not provide an integrated account of both value-based choices and reaction times, but simply link two types of models. Here, we propose a novel integrative joint model of both choices and reaction times by combining a computational account of Bayesian sequential decision making with a sampling procedure. This allows us to describe how internal uncertainty in the planning process shapes reaction time distributions. Specifically, we use a recent context-specific Bayesian forward planning model which we extend by a Markov chain Monte Carlo (MCMC) sampler to obtain both choices and reaction times. As we will show this makes the sampler an integral part of the decision making process and enables us to reproduce, using simulations, well-known experimental findings in value based-decision making as well as classical inhibition and switching tasks. Specifically, we use the proposed model to explain both choice behavior and reaction times in instrumental learning and automatized behavior, in the Eriksen flanker task and in task switching. These findings show that the proposed joint behavioral model may describe common underlying processes in these different decision making paradigms.

The ReCoDe addiction research consortium: Losing and regaining control over drug intake-Findings and future perspectives.

Substance use disorders (SUDs) are seen as a continuum ranging from goal-directed and hedonic drug use to loss of control over drug intake with aversive consequences for mental and physical health and social functioning. The main goals of our interdisciplinary German collaborative research centre on Losing and Regaining Control over Drug Intake (ReCoDe) are (i) to study triggers (drug cues, stressors, drug priming) and modifying factors (age, gender, physical activity, cognitive functions, childhood adversity, social factors, such as loneliness and social contact/interaction) that longitudinally modulate the trajectories of losing and regaining control over drug consumption under real-life conditions. (ii) To study underlying behavioural, cognitive and neurobiological mechanisms of disease trajectories and drug-related behaviours and (iii) to provide non-invasive mechanism-based interventions. These goals are achieved by: (A) using innovative mHealth (mobile health) tools to longitudinally monitor the effects of triggers and modifying factors on drug consumption patterns in real life in a cohort of 900 patients with alcohol use disorder. This approach will be complemented by animal models of addiction with 24/7 automated behavioural monitoring across an entire disease trajectory; i.e. from a naïve state to a drug-taking state to an addiction or resilience-like state. (B) The identification and, if applicable, computational modelling of key molecular, neurobiological and psychological mechanisms (e.g., reduced cognitive flexibility) mediating the effects of such triggers and modifying factors on disease trajectories. (C) Developing and testing non-invasive interventions (e.g., Just-In-Time-Adaptive-Interventions (JITAIs), various non-invasive brain stimulations (NIBS), individualized physical activity) that specifically target the underlying mechanisms for regaining control over drug intake. Here, we will report on the most important results of the first funding period and outline our future research strategy.

A limited role of NKCC1 in telencephalic glutamatergic neurons for developing hippocampal network dynamics and behavior.

NKCC1 is the primary transporter mediating chloride uptake in immature principal neurons, but its role in the development of in vivo network dynamics and cognitive abilities remains unknown. Here, we address the function of NKCC1 in developing mice using electrophysiological, optical, and behavioral approaches. We report that NKCC1 deletion from telencephalic glutamatergic neurons decreases in vitro excitatory actions of γ-aminobutyric acid (GABA) and impairs neuronal synchrony in neonatal hippocampal brain slices. In vivo, it has a minor impact on correlated spontaneous activity in the hippocampus and does not affect network activity in the intact visual cortex. Moreover, long-term effects of the developmental NKCC1 deletion on synaptic maturation, network dynamics, and behavioral performance are subtle. Our data reveal a neural network function of NKCC1 in hippocampal glutamatergic neurons in vivo, but challenge the hypothesis that NKCC1 is essential for major aspects of hippocampal development.

Forward planning driven by context-dependant conflict processing in anterior cingulate cortex.

Cognitive control and forward planning in particular is costly, and therefore must be regulated such that the amount of cognitive resources invested is adequate to the current situation. However, knowing in advance how beneficial forward planning will be in a given situation is hard. A way to know the exact value of planning would be to actually do it, which would ab initio defeat the purpose of regulating planning, i.e. the reduction of computational and time costs. One possible solution to this dilemma is that planning is regulated by learned associations between stimuli and the expected demand for planning. Such learning might be based on generalisation processes that cluster together stimulus states with similar control relevant properties into more general control contexts. In this way, the brain could infer the demand for planning, based on previous experience with situations that share some structural properties with the current situation. Here, we used a novel sequential task to test the hypothesis that people use control contexts to efficiently regulate their forward planning, using behavioural and functional magnetic resonance imaging data. Consistent with our hypothesis, reaction times increased with trial-by-trial conflict, where this increase was more pronounced in a context with a learned high demand for planning. Similarly, we found that fMRI activity in the dorsal anterior cingulate cortex (dACC) increased with conflict, and this increase was more pronounced in a context with generally high demand for planning. Taken together, the results indicate that the dACC integrates representations of planning demand at different levels of abstraction to regulate planning in an efficient and situation-appropriate way.

Computational Theories of Alcohol Use Disorder: Mapping Learning and Choice Mechanisms on Symptoms.

Alcohol use disorder (AUD) is characterized by a combination of symptoms including excessive craving, loss of control, and progressive neglect of alternative pleasures. A mechanistic understanding of what drives these symptoms is needed to improve diagnostic stratification and to develop new treatment and prevention strategies for AUD. To date, there is no consensus regarding a unifying mechanistic framework that accounts for the different symptoms of AUD. Reinforcement learning (RL) and economic choice theories may be key to elucidating the underlying processes of symptom development and maintenance in AUD. These algorithms may account for the different behavioral and physiological phenomena and are suited to dissect mechanisms linked to different symptoms of AUD. We here review different RL and economic choice models and how they map onto three symptoms of AUD: (1) cue-induced craving, (2) neglect of alternative rewards, and (3) consumption despite adverse consequences. For each symptom and theory, we describe findings from animal and human studies. In humans, we focus on empirical studies that investigated RL models in the context of treatment outcome in AUD. The review indicates important gaps to be addressed in the future by highlighting the challenges in transferring findings from RL and economic choice studies to clinical application. We also critically evaluate the potential and pitfalls of a symptom-oriented approach and highlight the importance of elucidating the role of learning and decision-making processes across diagnostic boundaries.

Meta-control of the exploration-exploitation dilemma emerges from probabilistic inference over a hierarchy of time scales.

Cognitive control is typically understood as a set of mechanisms that enable humans to reach goals that require integrating the consequences of actions over longer time scales. Importantly, using routine behaviour or making choices beneficial only at short time scales would prevent one from attaining these goals. During the past two decades, researchers have proposed various computational cognitive models that successfully account for behaviour related to cognitive control in a wide range of laboratory tasks. As humans operate in a dynamic and uncertain environment, making elaborate plans and integrating experience over multiple time scales is computationally expensive. Importantly, it remains poorly understood how uncertain consequences at different time scales are integrated into adaptive decisions. Here, we pursue the idea that cognitive control can be cast as active inference over a hierarchy of time scales, where inference, i.e., planning, at higher levels of the hierarchy controls inference at lower levels. We introduce the novel concept of meta-control states, which link higher-level beliefs with lower-level policy inference. Specifically, we conceptualize cognitive control as inference over these meta-control states, where solutions to cognitive control dilemmas emerge through surprisal minimisation at different hierarchy levels. We illustrate this concept using the exploration-exploitation dilemma based on a variant of a restless multi-armed bandit task. We demonstrate that beliefs about contexts and meta-control states at a higher level dynamically modulate the balance of exploration and exploitation at the lower level of a single action. Finally, we discuss the generalisation of this meta-control concept to other control dilemmas.

Dynamic integration of forward planning and heuristic preferences during multiple goal pursuit.

Selecting goals and successfully pursuing them in an uncertain and dynamic environment is an important aspect of human behaviour. In order to decide which goal to pursue at what point in time, one has to evaluate the consequences of one's actions over future time steps by forward planning. However, when the goal is still temporally distant, detailed forward planning can be prohibitively costly. One way to select actions at minimal computational costs is to use heuristics. It is an open question how humans mix heuristics with forward planning to balance computational costs with goal reaching performance. To test a hypothesis about dynamic mixing of heuristics with forward planning, we used a novel stochastic sequential two-goal task. Comparing participants' decisions with an optimal full planning agent, we found that at the early stages of goal-reaching sequences, in which both goals are temporally distant and planning complexity is high, on average 42% (SD = 19%) of participants' choices deviated from the agent's optimal choices. Only towards the end of the sequence, participant's behaviour converged to near optimal performance. Subsequent model-based analyses showed that participants used heuristic preferences when the goal was temporally distant and switched to forward planning when the goal was close.

Predicting change: Approximate inference under explicit representation of temporal structure in changing environments.

In our daily lives timing of our actions plays an essential role when we navigate the complex everyday environment. It is an open question though how the representations of the temporal structure of the world influence our behavior. Here we propose a probabilistic model with an explicit representation of state durations which may provide novel insights in how the brain predicts upcoming changes. We illustrate several properties of the behavioral model using a standard reversal learning design and compare its task performance to standard reinforcement learning models. Furthermore, using experimental data, we demonstrate how the model can be applied to identify participants' beliefs about the latent temporal task structure. We found that roughly one quarter of participants seem to have learned the latent temporal structure and used it to anticipate changes, whereas the remaining participants' behavior did not show signs of anticipatory responses, suggesting a lack of precise temporal expectations. We expect that the introduced behavioral model will allow, in future studies, for a systematic investigation of how participants learn the underlying temporal structure of task environments and how these representations shape behavior.

Addiction Research Consortium: Losing and regaining control over drug intake (ReCoDe)-From trajectories to mechanisms and interventions.

One of the major risk factors for global death and disability is alcohol, tobacco, and illicit drug use. While there is increasing knowledge with respect to individual factors promoting the initiation and maintenance of substance use disorders (SUDs), disease trajectories involved in losing and regaining control over drug intake (ReCoDe) are still not well described. Our newly formed German Collaborative Research Centre (CRC) on ReCoDe has an interdisciplinary approach funded by the German Research Foundation (DFG) with a 12-year perspective. The main goals of our research consortium are (i) to identify triggers and modifying factors that longitudinally modulate the trajectories of losing and regaining control over drug consumption in real life, (ii) to study underlying behavioral, cognitive, and neurobiological mechanisms, and (iii) to implicate mechanism-based interventions. These goals will be achieved by: (i) using mobile health (m-health) tools to longitudinally monitor the effects of triggers (drug cues, stressors, and priming doses) and modify factors (eg, age, gender, physical activity, and cognitive control) on drug consumption patterns in real-life conditions and in animal models of addiction; (ii) the identification and computational modeling of key mechanisms mediating the effects of such triggers and modifying factors on goal-directed, habitual, and compulsive aspects of behavior from human studies and animal models; and (iii) developing and testing interventions that specifically target the underlying mechanisms for regaining control over drug intake.

Ultra-fast accurate reconstruction of spiking activity from calcium imaging data.

Calcium imaging provides an indirect observation of the underlying neural dynamics and enables the functional analysis of neuronal populations. However, the recorded fluorescence traces are temporally smeared, thus making the reconstruction of exact spiking activity challenging. Most of the established methods to tackle this issue are limited in dealing with issues such as the variability in the kinetics of fluorescence transients, fast processing of long-term data, high firing rates, and measurement noise. We propose a novel, heuristic reconstruction method to overcome these limitations. By using both synthetic and experimental data, we demonstrate the four main features of this method: 1) it accurately reconstructs both isolated spikes and within-burst spikes, and the spike count per fluorescence transient, from a given noisy fluorescence trace; 2) it performs the reconstruction of a trace extracted from 1,000,000 frames in less than 2 s; 3) it adapts to transients with different rise and decay kinetics or amplitudes, both within and across single neurons; and 4) it has only one key parameter, which we will show can be set in a nearly automatic way to an approximately optimal value. Furthermore, we demonstrate the ability of the method to effectively correct for fast and rather complex, slowly varying drifts as frequently observed in in vivo data. NEW & NOTEWORTHY Reconstruction of spiking activities from calcium imaging data remains challenging. Most of the established reconstruction methods not only have limitations in adapting to systematic variations in the data and fast processing of large amounts of data, but their results also depend on the user's experience. To overcome these limitations, we present a novel, heuristic model-free-type method that enables an ultra-fast, accurate, near-automatic reconstruction from data recorded under a wide range of experimental conditions.

Active Inference, Belief Propagation, and the Bethe Approximation.

When modeling goal-directed behavior in the presence of various sources of uncertainty, planning can be described as an inference process. A solution to the problem of planning as inference was previously proposed in the active inference framework in the form of an approximate inference scheme based on variational free energy. However, this approximate scheme was based on the mean-field approximation, which assumes statistical independence of hidden variables and is known to show overconfidence and may converge to local minima of the free energy. To better capture the spatiotemporal properties of an environment, we reformulated the approximate inference process using the so-called Bethe approximation. Importantly, the Bethe approximation allows for representation of pairwise statistical dependencies. Under these assumptions, the minimizer of the variational free energy corresponds to the belief propagation algorithm, commonly used in machine learning. To illustrate the differences between the mean-field approximation and the Bethe approximation, we have simulated agent behavior in a simple goal-reaching task with different types of uncertainties. Overall, the Bethe agent achieves higher success rates in reaching goal states. We relate the better performance of the Bethe agent to more accurate predictions about the consequences of its own actions. Consequently, active inference based on the Bethe approximation extends the application range of active inference to more complex behavioral tasks.

Inferring Neuronal Dynamics from Calcium Imaging Data Using Biophysical Models and Bayesian Inference.

Calcium imaging has been used as a promising technique to monitor the dynamic activity of neuronal populations. However, the calcium trace is temporally smeared which restricts the extraction of quantities of interest such as spike trains of individual neurons. To address this issue, spike reconstruction algorithms have been introduced. One limitation of such reconstructions is that the underlying models are not informed about the biophysics of spike and burst generations. Such existing prior knowledge might be useful for constraining the possible solutions of spikes. Here we describe, in a novel Bayesian approach, how principled knowledge about neuronal dynamics can be employed to infer biophysical variables and parameters from fluorescence traces. By using both synthetic and in vitro recorded fluorescence traces, we demonstrate that the new approach is able to reconstruct different repetitive spiking and/or bursting patterns with accurate single spike resolution. Furthermore, we show that the high inference precision of the new approach is preserved even if the fluorescence trace is rather noisy or if the fluorescence transients show slow rise kinetics lasting several hundred milliseconds, and inhomogeneous rise and decay times. In addition, we discuss the use of the new approach for inferring parameter changes, e.g. due to a pharmacological intervention, as well as for inferring complex characteristics of immature neuronal circuits.

A Bayesian Attractor Model for Perceptual Decision Making.

Even for simple perceptual decisions, the mechanisms that the brain employs are still under debate. Although current consensus states that the brain accumulates evidence extracted from noisy sensory information, open questions remain about how this simple model relates to other perceptual phenomena such as flexibility in decisions, decision-dependent modulation of sensory gain, or confidence about a decision. We propose a novel approach of how perceptual decisions are made by combining two influential formalisms into a new model. Specifically, we embed an attractor model of decision making into a probabilistic framework that models decision making as Bayesian inference. We show that the new model can explain decision making behaviour by fitting it to experimental data. In addition, the new model combines for the first time three important features: First, the model can update decisions in response to switches in the underlying stimulus. Second, the probabilistic formulation accounts for top-down effects that may explain recent experimental findings of decision-related gain modulation of sensory neurons. Finally, the model computes an explicit measure of confidence which we relate to recent experimental evidence for confidence computations in perceptual decision tasks.

Modelling Odor Decoding in the Antennal Lobe by Combining Sequential Firing Rate Models with Bayesian Inference.

The olfactory information that is received by the insect brain is encoded in the form of spatiotemporal patterns in the projection neurons of the antennal lobe. These dense and overlapping patterns are transformed into a sparse code in Kenyon cells in the mushroom body. Although it is clear that this sparse code is the basis for rapid categorization of odors, it is yet unclear how the sparse code in Kenyon cells is computed and what information it represents. Here we show that this computation can be modeled by sequential firing rate patterns using Lotka-Volterra equations and Bayesian online inference. This new model can be understood as an 'intelligent coincidence detector', which robustly and dynamically encodes the presence of specific odor features. We found that the model is able to qualitatively reproduce experimentally observed activity in both the projection neurons and the Kenyon cells. In particular, the model explains mechanistically how sparse activity in the Kenyon cells arises from the dense code in the projection neurons. The odor classification performance of the model proved to be robust against noise and time jitter in the observed input sequences. As in recent experimental results, we found that recognition of an odor happened very early during stimulus presentation in the model. Critically, by using the model, we found surprising but simple computational explanations for several experimental phenomena.

Modeling the Evolution of Beliefs Using an Attentional Focus Mechanism.

For making decisions in everyday life we often have first to infer the set of environmental features that are relevant for the current task. Here we investigated the computational mechanisms underlying the evolution of beliefs about the relevance of environmental features in a dynamical and noisy environment. For this purpose we designed a probabilistic Wisconsin card sorting task (WCST) with belief solicitation, in which subjects were presented with stimuli composed of multiple visual features. At each moment in time a particular feature was relevant for obtaining reward, and participants had to infer which feature was relevant and report their beliefs accordingly. To test the hypothesis that attentional focus modulates the belief update process, we derived and fitted several probabilistic and non-probabilistic behavioral models, which either incorporate a dynamical model of attentional focus, in the form of a hierarchical winner-take-all neuronal network, or a diffusive model, without attention-like features. We used Bayesian model selection to identify the most likely generative model of subjects' behavior and found that attention-like features in the behavioral model are essential for explaining subjects' responses. Furthermore, we demonstrate a method for integrating both connectionist and Bayesian models of decision making within a single framework that allowed us to infer hidden belief processes of human subjects.

Voice identity recognition: functional division of the right STS and its behavioral relevance.

The human voice is the primary carrier of speech but also a fingerprint for person identity. Previous neuroimaging studies have revealed that speech and identity recognition is accomplished by partially different neural pathways, despite the perceptual unity of the vocal sound. Importantly, the right STS has been implicated in voice processing, with different contributions of its posterior and anterior parts. However, the time point at which vocal and speech processing diverge is currently unknown. Also, the exact role of the right STS during voice processing is so far unclear because its behavioral relevance has not yet been established. Here, we used the high temporal resolution of magnetoencephalography and a speech task control to pinpoint transient behavioral correlates: we found, at 200 msec after stimulus onset, that activity in right anterior STS predicted behavioral voice recognition performance. At the same time point, the posterior right STS showed increased activity during voice identity recognition in contrast to speech recognition whereas the left mid STS showed the reverse pattern. In contrast to the highly speech-sensitive left STS, the current results highlight the right STS as a key area for voice identity recognition and show that its anatomical-functional division emerges around 200 msec after stimulus onset. We suggest that this time point marks the speech-independent processing of vocal sounds in the posterior STS and their successful mapping to vocal identities in the anterior STS.

How the human brain exchanges information across sensory modalities to recognize other people.

Recognizing the identity of other individuals across different sensory modalities is critical for successful social interaction. In the human brain, face- and voice-sensitive areas are separate, but structurally connected. What kind of information is exchanged between these specialized areas during cross-modal recognition of other individuals is currently unclear. For faces, specific areas are sensitive to identity and to physical properties. It is an open question whether voices activate representations of face identity or physical facial properties in these areas. To address this question, we used functional magnetic resonance imaging in humans and a voice-face priming design. In this design, familiar voices were followed by morphed faces that matched or mismatched with respect to identity or physical properties. The results showed that responses in face-sensitive regions were modulated when face identity or physical properties did not match to the preceding voice. The strength of this mismatch signal depended on the level of certainty the participant had about the voice identity. This suggests that both identity and physical property information was provided by the voice to face areas. The activity and connectivity profiles differed between face-sensitive areas: (i) the occipital face area seemed to receive information about both physical properties and identity, (ii) the fusiform face area seemed to receive identity, and (iii) the anterior temporal lobe seemed to receive predominantly identity information from the voice. We interpret these results within a prediction coding scheme in which both identity and physical property information is used across sensory modalities to recognize individuals.

Dysfunction of the auditory thalamus in developmental dyslexia.

Developmental dyslexia, a severe and persistent reading and spelling impairment, is characterized by difficulties in processing speech sounds (i.e., phonemes). Here, we test the hypothesis that these phonological difficulties are associated with a dysfunction of the auditory sensory thalamus, the medial geniculate body (MGB). By using functional MRI, we found that, in dyslexic adults, the MGB responded abnormally when the task required attending to phonemes compared with other speech features. No other structure in the auditory pathway showed distinct functional neural patterns between the two tasks for dyslexic and control participants. Furthermore, MGB activity correlated with dyslexia diagnostic scores, indicating that the task modulation of the MGB is critical for performance in dyslexics. These results suggest that deficits in dyslexia are associated with a failure of the neural mechanism that dynamically tunes MGB according to predictions from cortical areas to optimize speech processing. This view on task-related MGB dysfunction in dyslexics has the potential to reconcile influential theories of dyslexia within a predictive coding framework of brain function.

From birdsong to human speech recognition: bayesian inference on a hierarchy of nonlinear dynamical systems.

Our knowledge about the computational mechanisms underlying human learning and recognition of sound sequences, especially speech, is still very limited. One difficulty in deciphering the exact means by which humans recognize speech is that there are scarce experimental findings at a neuronal, microscopic level. Here, we show that our neuronal-computational understanding of speech learning and recognition may be vastly improved by looking at an animal model, i.e., the songbird, which faces the same challenge as humans: to learn and decode complex auditory input, in an online fashion. Motivated by striking similarities between the human and songbird neural recognition systems at the macroscopic level, we assumed that the human brain uses the same computational principles at a microscopic level and translated a birdsong model into a novel human sound learning and recognition model with an emphasis on speech. We show that the resulting Bayesian model with a hierarchy of nonlinear dynamical systems can learn speech samples such as words rapidly and recognize them robustly, even in adverse conditions. In addition, we show that recognition can be performed even when words are spoken by different speakers and with different accents-an everyday situation in which current state-of-the-art speech recognition models often fail. The model can also be used to qualitatively explain behavioral data on human speech learning and derive predictions for future experiments.

Corrigendum: Interaction between habits as action sequences and goal-directed behavior under time pressure.

[This corrects the article DOI: 10.3389/fnins.2022.996957.].