Joint modeling of choices and reaction times based on Bayesian contextual behavioral control.

In cognitive neuroscience and psychology, reaction times are an important behavioral measure. However, in instrumental learning and goal-directed decision making experiments, findings often rely only on choice probabilities from a value-based model, instead of reaction times. Recent advancements have shown that it is possible to connect value-based decision models with reaction time models. However, typically these models do not provide an integrated account of both value-based choices and reaction times, but simply link two types of models. Here, we propose a novel integrative joint model of both choices and reaction times by combining a computational account of Bayesian sequential decision making with a sampling procedure. This allows us to describe how internal uncertainty in the planning process shapes reaction time distributions. Specifically, we use a recent context-specific Bayesian forward planning model which we extend by a Markov chain Monte Carlo (MCMC) sampler to obtain both choices and reaction times. As we will show this makes the sampler an integral part of the decision making process and enables us to reproduce, using simulations, well-known experimental findings in value based-decision making as well as classical inhibition and switching tasks. Specifically, we use the proposed model to explain both choice behavior and reaction times in instrumental learning and automatized behavior, in the Eriksen flanker task and in task switching. These findings show that the proposed joint behavioral model may describe common underlying processes in these different decision making paradigms.

Talking Different Languages: The Role of Plant-Plant Communication When an Invader Beats up a Strange Neighborhood.

Communication through airborne volatile organic compounds (VOCs) and root exudates plays a vital role in the multifarious interactions of plants. Common ragweed (Ambrosia artemesiifolia L.) is one of the most troublesome invasive alien species in agriculture. Below- and aboveground chemical interactions of ragweed with crops might be an important factor in the invasive species' success in agriculture. In laboratory experiments, we investigated the contribution of intra- and interspecific airborne VOCs and root exudates of ragweed to its competitiveness. Wheat, soybean, and maize were exposed to VOCs emitted from ragweed and vice versa, and the adaptation response was measured through plant morphological and physiological traits. We observed significant changes in plant traits of crops in response to ragweed VOCs, characterized by lower biomass production, lower specific leaf area, or higher chlorophyll contents. After exposure to ragweed VOCs, soybean and wheat produced significantly less aboveground dry mass, whereas maize did not. Ragweed remained unaffected when exposed to VOCs from the crops or a conspecific. All crops and ragweed significantly avoided root growth toward the root exudates of ragweed. The study shows that the plant response to either above- or belowground chemical cues is highly dependent on the identity of the neighbor, pointing out the complexity of plant-plant communication in plant communities.

Shorter planning depth and higher response noise during sequential decision-making in old age.

Forward planning is crucial to maximize outcome in complex sequential decision-making scenarios. In this cross-sectional study, we were particularly interested in age-related differences of forward planning. We presumed that especially older individuals would show a shorter planning depth to keep the costs of model-based decision-making within limits. To test this hypothesis, we developed a sequential decision-making task to assess forward planning in younger (age < 40 years; n = 25) and older (age > 60 years; n = 27) adults. By using reinforcement learning modelling, we inferred planning depths from participants' choices. Our results showed significantly shorter planning depths and higher response noise for older adults. Age differences in planning depth were only partially explained by well-known cognitive covariates such as working memory and processing speed. Consistent with previous findings, this indicates age-related shifts away from model-based behaviour in older adults. In addition to a shorter planning depth, our findings suggest that older adults also apply a variety of heuristical low-cost strategies.

Revealing human sensitivity to a latent temporal structure of changes.

Precisely timed behavior and accurate time perception plays a critical role in our everyday lives, as our wellbeing and even survival can depend on well-timed decisions. Although the temporal structure of the world around us is essential for human decision making, we know surprisingly little about how representation of temporal structure of our everyday environment impacts decision making. How does the representation of temporal structure affect our ability to generate well-timed decisions? Here we address this question by using a well-established dynamic probabilistic learning task. Using computational modeling, we found that human subjects' beliefs about temporal structure are reflected in their choices to either exploit their current knowledge or to explore novel options. The model-based analysis illustrates a large within-group and within-subject heterogeneity. To explain these results, we propose a normative model for how temporal structure is used in decision making, based on the semi-Markov formalism in the active inference framework. We discuss potential key applications of the presented approach to the fields of cognitive phenotyping and computational psychiatry.

An empirical evaluation of active inference in multi-armed bandits.

A key feature of sequential decision making under uncertainty is a need to balance between exploiting-choosing the best action according to the current knowledge, and exploring-obtaining information about values of other actions. The multi-armed bandit problem, a classical task that captures this trade-off, served as a vehicle in machine learning for developing bandit algorithms that proved to be useful in numerous industrial applications. The active inference framework, an approach to sequential decision making recently developed in neuroscience for understanding human and animal behaviour, is distinguished by its sophisticated strategy for resolving the exploration-exploitation trade-off. This makes active inference an exciting alternative to already established bandit algorithms. Here we derive an efficient and scalable approximate active inference algorithm and compare it to two state-of-the-art bandit algorithms: Bayesian upper confidence bound and optimistic Thompson sampling. This comparison is done on two types of bandit problems: a stationary and a dynamic switching bandit. Our empirical evaluation shows that the active inference algorithm does not produce efficient long-term behaviour in stationary bandits. However, in the more challenging switching bandit problem active inference performs substantially better than the two state-of-the-art bandit algorithms. The results open exciting venues for further research in theoretical and applied machine learning, as well as lend additional credibility to active inference as a general framework for studying human and animal behaviour.

Neuronal Sequence Models for Bayesian Online Inference.

Various imaging and electrophysiological studies in a number of different species and brain regions have revealed that neuronal dynamics associated with diverse behavioral patterns and cognitive tasks take on a sequence-like structure, even when encoding stationary concepts. These neuronal sequences are characterized by robust and reproducible spatiotemporal activation patterns. This suggests that the role of neuronal sequences may be much more fundamental for brain function than is commonly believed. Furthermore, the idea that the brain is not simply a passive observer but an active predictor of its sensory input, is supported by an enormous amount of evidence in fields as diverse as human ethology and physiology, besides neuroscience. Hence, a central aspect of this review is to illustrate how neuronal sequences can be understood as critical for probabilistic predictive information processing, and what dynamical principles can be used as generators of neuronal sequences. Moreover, since different lines of evidence from neuroscience and computational modeling suggest that the brain is organized in a functional hierarchy of time scales, we will also review how models based on sequence-generating principles can be embedded in such a hierarchy, to form a generative model for recognition and prediction of sensory input. We shortly introduce the Bayesian brain hypothesis as a prominent mathematical description of how online, i.e., fast, recognition, and predictions may be computed by the brain. Finally, we briefly discuss some recent advances in machine learning, where spatiotemporally structured methods (akin to neuronal sequences) and hierarchical networks have independently been developed for a wide range of tasks. We conclude that the investigation of specific dynamical and structural principles of sequential brain activity not only helps us understand how the brain processes information and generates predictions, but also informs us about neuroscientific principles potentially useful for designing more efficient artificial neuronal networks for machine learning tasks.

Predicting the cumulative medical load of COVID-19 outbreaks after the peak in daily fatalities.

The distinct ways the COVID-19 pandemic has been unfolding in different countries and regions suggest that local societal and governmental structures play an important role not only for the baseline infection rate, but also for short and long-term reactions to the outbreak. We propose to investigate the question of how societies as a whole, and governments in particular, modulate the dynamics of a novel epidemic using a generalization of the SIR model, the reactive SIR (short-term and long-term reaction) model. We posit that containment measures are equivalent to a feedback between the status of the outbreak and the reproduction factor. Short-term reaction to an outbreak corresponds in this framework to the reaction of governments and individuals to daily cases and fatalities. The reaction to the cumulative number of cases or deaths, and not to daily numbers, is captured in contrast by long-term reaction. We present the exact phase space solution of the controlled SIR model and use it to quantify containment policies for a large number of countries in terms of short and long-term control parameters. We find increased contributions of long-term control for countries and regions in which the outbreak was suppressed substantially together with a strong correlation between the strength of societal and governmental policies and the time needed to contain COVID-19 outbreaks. Furthermore, for numerous countries and regions we identified a predictive relation between the number of fatalities within a fixed period before and after the peak of daily fatality counts, which allows to gauge the cumulative medical load of COVID-19 outbreaks that should be expected after the peak. These results suggest that the proposed model is applicable not only for understanding the outbreak dynamics, but also for predicting future cases and fatalities once the effectiveness of outbreak suppression policies is established with sufficient certainty. Finally, we provide a web app (https://itp.uni-frankfurt.de/covid-19/) with tools for visualising the phase space representation of real-world COVID-19 data and for exporting the preprocessed data for further analysis.

Aphid feeding induces the relaxation of epigenetic control and the associated regulation of the defense response in Arabidopsis.

Environmentally induced changes in the epigenome help individuals to quickly adapt to fluctuations in the conditions of their habitats. We explored those changes in Arabidopsis thaliana plants subjected to multiple biotic and abiotic stresses, and identified transposable element (TE) activation in plants infested with the green peach aphid, Myzus persicae. We performed a genome-wide analysis mRNA expression, small RNA accumulation and DNA methylation Our results demonstrate that aphid feeding induces loss of methylation of hundreds of loci, mainly TEs. This loss of methylation has the potential to regulate gene expression and we found evidence that it is involved in the control of plant immunity genes. Accordingly, mutant plants deficient in DNA and H3K9 methylation (kyp) showed increased resistance to M. persicae infestation. Collectively, our results show that changes in DNA methylation play a significant role in the regulation of the plant transcriptional response and induction of defense response against aphid feeding.

Meta-control of the exploration-exploitation dilemma emerges from probabilistic inference over a hierarchy of time scales.

Cognitive control is typically understood as a set of mechanisms that enable humans to reach goals that require integrating the consequences of actions over longer time scales. Importantly, using routine behaviour or making choices beneficial only at short time scales would prevent one from attaining these goals. During the past two decades, researchers have proposed various computational cognitive models that successfully account for behaviour related to cognitive control in a wide range of laboratory tasks. As humans operate in a dynamic and uncertain environment, making elaborate plans and integrating experience over multiple time scales is computationally expensive. Importantly, it remains poorly understood how uncertain consequences at different time scales are integrated into adaptive decisions. Here, we pursue the idea that cognitive control can be cast as active inference over a hierarchy of time scales, where inference, i.e., planning, at higher levels of the hierarchy controls inference at lower levels. We introduce the novel concept of meta-control states, which link higher-level beliefs with lower-level policy inference. Specifically, we conceptualize cognitive control as inference over these meta-control states, where solutions to cognitive control dilemmas emerge through surprisal minimisation at different hierarchy levels. We illustrate this concept using the exploration-exploitation dilemma based on a variant of a restless multi-armed bandit task. We demonstrate that beliefs about contexts and meta-control states at a higher level dynamically modulate the balance of exploration and exploitation at the lower level of a single action. Finally, we discuss the generalisation of this meta-control concept to other control dilemmas.

Plant volatiles as cues and signals in plant communication.

Volatile organic compounds are important mediators of mutualistic interactions between plants and their physical and biological surroundings. Volatiles rapidly indicate competition or potential threat before these can take place, and they regulate and coordinate adaptation responses in neighbouring plants, fine-tuning them to match the exact stress encountered. Ecological specificity and context-dependency of plant-plant communication mediated by volatiles represent important factors that determine plant performance in specific environments. In this review, we synthesise the recent progress made in understanding the role of plant volatiles as mediators of plant interactions at the individual and community levels, highlighting the complexity of the plant receiver response to diverse volatile cues and signals and addressing how specific responses shape plant growth and survival. Finally, we outline the knowledge gaps and provide directions for future research. The complex dialogue between the emitter and receiver based on either volatile cues or signals determines the outcome of information exchange, which shapes the communication pattern between individuals at the community level and determines their ecological implications at other trophic levels.

Dynamic integration of forward planning and heuristic preferences during multiple goal pursuit.

Selecting goals and successfully pursuing them in an uncertain and dynamic environment is an important aspect of human behaviour. In order to decide which goal to pursue at what point in time, one has to evaluate the consequences of one's actions over future time steps by forward planning. However, when the goal is still temporally distant, detailed forward planning can be prohibitively costly. One way to select actions at minimal computational costs is to use heuristics. It is an open question how humans mix heuristics with forward planning to balance computational costs with goal reaching performance. To test a hypothesis about dynamic mixing of heuristics with forward planning, we used a novel stochastic sequential two-goal task. Comparing participants' decisions with an optimal full planning agent, we found that at the early stages of goal-reaching sequences, in which both goals are temporally distant and planning complexity is high, on average 42% (SD = 19%) of participants' choices deviated from the agent's optimal choices. Only towards the end of the sequence, participant's behaviour converged to near optimal performance. Subsequent model-based analyses showed that participants used heuristic preferences when the goal was temporally distant and switched to forward planning when the goal was close.

Who is my neighbor? Volatile cues in plant interactions.

One of the most important challenges for individual plants is coexistence with their neighbors. To compensate for their sessile lifestyle, plants developed complex and sophisticated chemical systems of communication among each other. Site-specific biotic and abiotic factors constantly alter the physiological activity of plants, which causes them to release various secondary metabolites in their environments. Volatile organic compounds (VOCs) are the most common cues that reflect a plant's current physiological status. In this sense, the identity of its immediate neighbors may have the greatest impact for a plant, as they share the same available resources. Plants constantly monitor and respond to these cues with great sensitivity and discrimination, resulting in specific changes in their growth pattern and adjusting their physiology, morphology, and phenotype accordingly. Those typical competition responses in receivers may increase their fitness as they can be elicited even before the competition takes place. Plant-plant interactions are dynamic and complex as they can include many different and important surrounding cues. A major challenge for all individual plants is detecting and actively responding only to "true" cues that point to real upcoming threat. Such selective responses to highly specific cues embedded in volatile bouquets are of great ecological importance in understanding plant-plant interactions. We have reviewed recent research on the role of VOCs in complex plant-plant interactions in plant-cross kingdom and highlighted their influence on organisms at higher trophic levels.

Neuronal message passing using Mean-field, Bethe, and Marginal approximations.

Neuronal computations rely upon local interactions across synapses. For a neuronal network to perform inference, it must integrate information from locally computed messages that are propagated among elements of that network. We review the form of two popular (Bayesian) message passing schemes and consider their plausibility as descriptions of inference in biological networks. These are variational message passing and belief propagation - each of which is derived from a free energy functional that relies upon different approximations (mean-field and Bethe respectively). We begin with an overview of these schemes and illustrate the form of the messages required to perform inference using Hidden Markov Models as generative models. Throughout, we use factor graphs to show the form of the generative models and of the messages they entail. We consider how these messages might manifest neuronally and simulate the inferences they perform. While variational message passing offers a simple and neuronally plausible architecture, it falls short of the inferential performance of belief propagation. In contrast, belief propagation allows exact computation of marginal posteriors at the expense of the architectural simplicity of variational message passing. As a compromise between these two extremes, we offer a third approach - marginal message passing - that features a simple architecture, while approximating the performance of belief propagation. Finally, we link formal considerations to accounts of neurological and psychiatric syndromes in terms of aberrant message passing.

Predicting change: Approximate inference under explicit representation of temporal structure in changing environments.

In our daily lives timing of our actions plays an essential role when we navigate the complex everyday environment. It is an open question though how the representations of the temporal structure of the world influence our behavior. Here we propose a probabilistic model with an explicit representation of state durations which may provide novel insights in how the brain predicts upcoming changes. We illustrate several properties of the behavioral model using a standard reversal learning design and compare its task performance to standard reinforcement learning models. Furthermore, using experimental data, we demonstrate how the model can be applied to identify participants' beliefs about the latent temporal task structure. We found that roughly one quarter of participants seem to have learned the latent temporal structure and used it to anticipate changes, whereas the remaining participants' behavior did not show signs of anticipatory responses, suggesting a lack of precise temporal expectations. We expect that the introduced behavioral model will allow, in future studies, for a systematic investigation of how participants learn the underlying temporal structure of task environments and how these representations shape behavior.

Airborne signals synchronize the defenses of neighboring plants in response to touch.

Plants activate defense-related pathways in response to subtle abiotic or biotic disturbances, changing their volatile profile rapidly. How such perturbations reach and potentially affect neighboring plants is less understood. We evaluated whether brief and light touching had a cascade effect on the profile of volatiles and gene expression of the focal plant and a neighboring untouched plant. Within minutes after contact, Zea mays showed an up-regulation of certain defense genes and increased the emission of specific volatiles that primed neighboring plants, making them less attractive for aphids. Exposure to volatiles from touched plants activated many of the same defense-related genes in non-touched neighboring plants, demonstrating a transcriptional mirroring effect for expression of genes up-regulated by brief contact. Perception of so-far-overlooked touch-induced volatile organic compounds was of ecological significance as these volatiles are directly involved in plant-plant communication as an effective trigger for rapid defense synchronization among nearby plants. Our findings shed new light on mechanisms of plant responses to mechanical contact at the molecular level and on the ecological role of induced volatiles as airborne signals in plant-plant interactions.

Deterministic response strategies in a trial-and-error learning task.

Trial-and-error learning is a universal strategy for establishing which actions are beneficial or harmful in new environments. However, learning stimulus-response associations solely via trial-and-error is often suboptimal, as in many settings dependencies among stimuli and responses can be exploited to increase learning efficiency. Previous studies have shown that in settings featuring such dependencies, humans typically engage high-level cognitive processes and employ advanced learning strategies to improve their learning efficiency. Here we analyze in detail the initial learning phase of a sample of human subjects (N = 85) performing a trial-and-error learning task with deterministic feedback and hidden stimulus-response dependencies. Using computational modeling, we find that the standard Q-learning model cannot sufficiently explain human learning strategies in this setting. Instead, newly introduced deterministic response models, which are theoretically optimal and transform stimulus sequences unambiguously into response sequences, provide the best explanation for 50.6% of the subjects. Most of the remaining subjects either show a tendency towards generic optimal learning (21.2%) or at least partially exploit stimulus-response dependencies (22.3%), while a few subjects (5.9%) show no clear preference for any of the employed models. After the initial learning phase, asymptotic learning performance during the subsequent practice phase is best explained by the standard Q-learning model. Our results show that human learning strategies in the presented trial-and-error learning task go beyond merely associating stimuli and responses via incremental reinforcement. Specifically during initial learning, high-level cognitive processes support sophisticated learning strategies that increase learning efficiency while keeping memory demands and computational efforts bounded. The good asymptotic fit of the Q-learning model indicates that these cognitive processes are successively replaced by the formation of stimulus-response associations over the course of learning.

Context-Dependent Risk Aversion: A Model-Based Approach.

Most research on risk aversion in behavioral science with human subjects has focused on a component of risk aversion that does not adapt itself to context. More recently, studies have explored risk aversion adaptation to changing circumstances in sequential decision-making tasks. It is an open question whether one can identify evidence, at the single subject level, for such risk aversion adaptation. We conducted a behavioral experiment on human subjects, using a sequential decision making task. We developed a model-based approach for estimating the adaptation of risk-taking behavior with single-trial resolution by modeling a subject's goals and internal representation of task contingencies. Using this model-based approach, we estimated the subject-specific adaptation of risk aversion depending on the current task context. We found striking inter-subject variations in the adaptation of risk-taking behavior. We show that these differences can be explained by differences in subjects' internal representations of task contingencies and goals. We discuss that the proposed approach can be adapted to a wide range of experimental paradigms and be used to analyze behavioral measures other than risk aversion.

Active Inference, Belief Propagation, and the Bethe Approximation.

When modeling goal-directed behavior in the presence of various sources of uncertainty, planning can be described as an inference process. A solution to the problem of planning as inference was previously proposed in the active inference framework in the form of an approximate inference scheme based on variational free energy. However, this approximate scheme was based on the mean-field approximation, which assumes statistical independence of hidden variables and is known to show overconfidence and may converge to local minima of the free energy. To better capture the spatiotemporal properties of an environment, we reformulated the approximate inference process using the so-called Bethe approximation. Importantly, the Bethe approximation allows for representation of pairwise statistical dependencies. Under these assumptions, the minimizer of the variational free energy corresponds to the belief propagation algorithm, commonly used in machine learning. To illustrate the differences between the mean-field approximation and the Bethe approximation, we have simulated agent behavior in a simple goal-reaching task with different types of uncertainties. Overall, the Bethe agent achieves higher success rates in reaching goal states. We relate the better performance of the Bethe agent to more accurate predictions about the consequences of its own actions. Consequently, active inference based on the Bethe approximation extends the application range of active inference to more complex behavioral tasks.

Aboveground mechanical stimuli affect belowground plant-plant communication.

Plants can detect the presence of their neighbours and modify their growth behaviour accordingly. But the extent to which this neighbour detection is mediated by abiotic stressors is not well known. In this study we tested the acclimation response of Zea mays L. seedlings through belowground interactions to the presence of their siblings exposed to brief mechano stimuli. Maize seedling simultaneously shared the growth solution of touched plants or they were transferred to the growth solution of previously touched plants. We tested the growth preferences of newly germinated seedlings toward the growth solution of touched (T_solution) or untouched plants (C_solution). The primary root of the newly germinated seedlings grew significantly less towards T_solution than to C_solution. Plants transferred to T_solution allocated more biomass to shoots and less to roots. While plants that simultaneously shared their growth solution with the touched plants produced more biomass. Results show that plant responses to neighbours can be modified by aboveground abiotic stress to those neighbours and suggest that these modifications are mediated by belowground interactions.