A Computational Analysis of Abnormal Belief Updating Processes and Their Association With Psychotic Experiences and Childhood Trauma in a UK Birth Cohort.

BACKGROUND: Psychotic experiences emerge from abnormalities in perception and belief formation and occur more commonly in those experiencing childhood trauma. However, which precise aspects of belief formation are atypical in psychosis is not well understood. We used a computational modeling approach to characterize belief updating in young adults in the general population, examine their relationship with psychotic outcomes and trauma, and determine the extent to which they mediate the trauma-psychosis relationship. METHODS: We used data from 3360 individuals from the Avon Longitudinal Study of Parents and Children birth cohort who completed assessments for psychotic outcomes, depression, anxiety, and two belief updating tasks at age 24 and had data available on traumatic events assessed from birth to late adolescence. Unadjusted and adjusted regression and counterfactual mediation methods were used for the analyses. RESULTS: Basic behavioral measures of belief updating (draws-to-decision and disconfirmatory updating) were not associated with psychotic experiences. However, computational modeling revealed an association between increased decision noise with both psychotic experiences and trauma exposure, although <3% of the trauma-psychotic experience association was mediated by decision noise. Belief updating measures were also associated with intelligence and sociodemographic characteristics, confounding most of the associations with psychotic experiences. There was little evidence that belief updating parameters were differentially associated with delusions compared with hallucinations or that they were differentially associated with psychotic outcomes compared with depression or anxiety. CONCLUSIONS: These findings challenge the hypothesis that atypical belief updating mechanisms (as indexed by the computational models and behavioral measures we used) underlie the development of psychotic phenomena.

Retrospective Inference as a Form of Bounded Rationality, and Its Beneficial Influence on Learning.

Probabilistic models of cognition typically assume that agents make inferences about current states by combining new sensory information with fixed beliefs about the past, an approach known as Bayesian filtering. This is computationally parsimonious, but, in general, leads to suboptimal beliefs about past states, since it ignores the fact that new observations typically contain information about the past as well as the present. This is disadvantageous both because knowledge of past states may be intrinsically valuable, and because it impairs learning about fixed or slowly changing parameters of the environment. For these reasons, in offline data analysis it is usual to infer on every set of states using the entire time series of observations, an approach known as (fixed-interval) Bayesian smoothing. Unfortunately, however, this is impractical for real agents, since it requires the maintenance and updating of beliefs about an ever-growing set of states. We propose an intermediate approach, finite retrospective inference (FRI), in which agents perform update beliefs about a limited number of past states (Formally, this represents online fixed-lag smoothing with a sliding window). This can be seen as a form of bounded rationality in which agents seek to optimize the accuracy of their beliefs subject to computational and other resource costs. We show through simulation that this approach has the capacity to significantly increase the accuracy of both inference and learning, using a simple variational scheme applied to both randomly generated Hidden Markov models (HMMs), and a specific application of the HMM, in the form of the widely used probabilistic reversal task. Our proposal thus constitutes a theoretical contribution to normative accounts of bounded rationality, which makes testable empirical predictions that can be explored in future work.

Modeling subjective belief states in computational psychiatry: interoceptive inference as a candidate framework.

The nascent field computational psychiatry has undergone exponential growth since its inception. To date, much of the published work has focused on choice behaviors, which are primarily modeled within a reinforcement learning framework. While this initial normative effort represents a milestone in psychiatry research, the reality is that many psychiatric disorders are defined by disturbances in subjective states (e.g., depression, anxiety) and associated beliefs (e.g., dysmorphophobia, paranoid ideation), which are not considered in normative models. In this paper, we present interoceptive inference as a candidate framework for modeling subjective-and associated belief-states in computational psychiatry. We first introduce the notion and significance of modeling subjective states in computational psychiatry. Next, we present the interoceptive inference framework, and in particular focus on the relationship between interoceptive inference (i.e., belief updating) and emotions. Lastly, we will use drug craving as an example of subjective states to demonstrate the feasibility of using interoceptive inference to model the psychopathology of subjective states.

Dopaminergic basis for signaling belief updates, but not surprise, and the link to paranoia.

Distinguishing between meaningful and meaningless sensory information is fundamental to forming accurate representations of the world. Dopamine is thought to play a central role in processing the meaningful information content of observations, which motivates an agent to update their beliefs about the environment. However, direct evidence for dopamine's role in human belief updating is lacking. We addressed this question in healthy volunteers who performed a model-based fMRI task designed to separate the neural processing of meaningful and meaningless sensory information. We modeled participant behavior using a normative Bayesian observer model and used the magnitude of the model-derived belief update following an observation to quantify its meaningful information content. We also acquired PET imaging measures of dopamine function in the same subjects. We show that the magnitude of belief updates about task structure (meaningful information), but not pure sensory surprise (meaningless information), are encoded in midbrain and ventral striatum activity. Using PET we show that the neural encoding of meaningful information is negatively related to dopamine-2/3 receptor availability in the midbrain and dexamphetamine-induced dopamine release capacity in the striatum. Trial-by-trial analysis of task performance indicated that subclinical paranoid ideation is negatively related to behavioral sensitivity to observations carrying meaningful information about the task structure. The findings provide direct evidence implicating dopamine in model-based belief updating in humans and have implications for understating the pathophysiology of psychotic disorders where dopamine function is disrupted.

Sequential inference as a mode of cognition and its correlates in fronto-parietal and hippocampal brain regions.

Normative models of human cognition often appeal to Bayesian filtering, which provides optimal online estimates of unknown or hidden states of the world, based on previous observations. However, in many cases it is necessary to optimise beliefs about sequences of states rather than just the current state. Importantly, Bayesian filtering and sequential inference strategies make different predictions about beliefs and subsequent choices, rendering them behaviourally dissociable. Taking data from a probabilistic reversal task we show that subjects' choices provide strong evidence that they are representing short sequences of states. Between-subject measures of this implicit sequential inference strategy had a neurobiological underpinning and correlated with grey matter density in prefrontal and parietal cortex, as well as the hippocampus. Our findings provide, to our knowledge, the first evidence for sequential inference in human cognition, and by exploiting between-subject variation in this measure we provide pointers to its neuronal substrates.

Neural signals encoding shifts in beliefs.

Dopamine is implicated in a diverse range of cognitive functions including cognitive flexibility, task switching, signalling novel or unexpected stimuli as well as advance information. There is also longstanding line of thought that links dopamine with belief formation and, crucially, aberrant belief formation in psychosis. Integrating these strands of evidence would suggest that dopamine plays a central role in belief updating and more specifically in encoding of meaningful information content in observations. The precise nature of this relationship has remained unclear. To directly address this question we developed a paradigm that allowed us to decompose two distinct types of information content, information-theoretic surprise that reflects the unexpectedness of an observation, and epistemic value that induces shifts in beliefs or, more formally, Bayesian surprise. Using functional magnetic-resonance imaging in humans we show that dopamine-rich midbrain regions encode shifts in beliefs whereas surprise is encoded in prefrontal regions, including the pre-supplementary motor area and dorsal cingulate cortex. By linking putative dopaminergic activity to belief updating these data provide a link to false belief formation that characterises hyperdopaminergic states associated with idiopathic and drug induced psychosis.

Dopamine, reward learning, and active inference.

Temporal difference learning models propose phasic dopamine signaling encodes reward prediction errors that drive learning. This is supported by studies where optogenetic stimulation of dopamine neurons can stand in lieu of actual reward. Nevertheless, a large body of data also shows that dopamine is not necessary for learning, and that dopamine depletion primarily affects task performance. We offer a resolution to this paradox based on an hypothesis that dopamine encodes the precision of beliefs about alternative actions, and thus controls the outcome-sensitivity of behavior. We extend an active inference scheme for solving Markov decision processes to include learning, and show that simulated dopamine dynamics strongly resemble those actually observed during instrumental conditioning. Furthermore, simulated dopamine depletion impairs performance but spares learning, while simulated excitation of dopamine neurons drives reward learning, through aberrant inference about outcome states. Our formal approach provides a novel and parsimonious reconciliation of apparently divergent experimental findings.

Evidence for surprise minimization over value maximization in choice behavior.

Classical economic models are predicated on the idea that the ultimate aim of choice is to maximize utility or reward. In contrast, an alternative perspective highlights the fact that adaptive behavior requires agents' to model their environment and minimize surprise about the states they frequent. We propose that choice behavior can be more accurately accounted for by surprise minimization compared to reward or utility maximization alone. Minimizing surprise makes a prediction at variance with expected utility models; namely, that in addition to attaining valuable states, agents attempt to maximize the entropy over outcomes and thus 'keep their options open'. We tested this prediction using a simple binary choice paradigm and show that human decision-making is better explained by surprise minimization compared to utility maximization. Furthermore, we replicated this entropy-seeking behavior in a control task with no explicit utilities. These findings highlight a limitation of purely economic motivations in explaining choice behavior and instead emphasize the importance of belief-based motivations.

Optimal inference with suboptimal models: addiction and active Bayesian inference.

When casting behaviour as active (Bayesian) inference, optimal inference is defined with respect to an agent's beliefs - based on its generative model of the world. This contrasts with normative accounts of choice behaviour, in which optimal actions are considered in relation to the true structure of the environment - as opposed to the agent's beliefs about worldly states (or the task). This distinction shifts an understanding of suboptimal or pathological behaviour away from aberrant inference as such, to understanding the prior beliefs of a subject that cause them to behave less 'optimally' than our prior beliefs suggest they should behave. Put simply, suboptimal or pathological behaviour does not speak against understanding behaviour in terms of (Bayes optimal) inference, but rather calls for a more refined understanding of the subject's generative model upon which their (optimal) Bayesian inference is based. Here, we discuss this fundamental distinction and its implications for understanding optimality, bounded rationality and pathological (choice) behaviour. We illustrate our argument using addictive choice behaviour in a recently described 'limited offer' task. Our simulations of pathological choices and addictive behaviour also generate some clear hypotheses, which we hope to pursue in ongoing empirical work.

The Dopaminergic Midbrain Encodes the Expected Certainty about Desired Outcomes.

Dopamine plays a key role in learning; however, its exact function in decision making and choice remains unclear. Recently, we proposed a generic model based on active (Bayesian) inference wherein dopamine encodes the precision of beliefs about optimal policies. Put simply, dopamine discharges reflect the confidence that a chosen policy will lead to desired outcomes. We designed a novel task to test this hypothesis, where subjects played a "limited offer" game in a functional magnetic resonance imaging experiment. Subjects had to decide how long to wait for a high offer before accepting a low offer, with the risk of losing everything if they waited too long. Bayesian model comparison showed that behavior strongly supported active inference, based on surprise minimization, over classical utility maximization schemes. Furthermore, midbrain activity, encompassing dopamine projection neurons, was accurately predicted by trial-by-trial variations in model-based estimates of precision. Our findings demonstrate that human subjects infer both optimal policies and the precision of those inferences, and thus support the notion that humans perform hierarchical probabilistic Bayesian inference. In other words, subjects have to infer both what they should do as well as how confident they are in their choices, where confidence may be encoded by dopaminergic firing.

Precision and neuronal dynamics in the human posterior parietal cortex during evidence accumulation.

Primate studies show slow ramping activity in posterior parietal cortex (PPC) neurons during perceptual decision-making. These findings have inspired a rich theoretical literature to account for this activity. These accounts are largely unrelated to Bayesian theories of perception and predictive coding, a related formulation of perceptual inference in the cortical hierarchy. Here, we tested a key prediction of such hierarchical inference, namely that the estimated precision (reliability) of information ascending the cortical hierarchy plays a key role in determining both the speed of decision-making and the rate of increase of PPC activity. Using dynamic causal modelling of magnetoencephalographic (MEG) evoked responses, recorded during a simple perceptual decision-making task, we recover ramping-activity from an anatomically and functionally plausible network of regions, including early visual cortex, the middle temporal area (MT) and PPC. Precision, as reflected by the gain on pyramidal cell activity, was strongly correlated with both the speed of decision making and the slope of PPC ramping activity. Our findings indicate that the dynamics of neuronal activity in the human PPC during perceptual decision-making recapitulate those observed in the macaque, and in so doing we link observations from primate electrophysiology and human choice behaviour. Moreover, the synaptic gain control modulating these dynamics is consistent with predictive coding formulations of evidence accumulation.

Active inference, evidence accumulation, and the urn task.

Deciding how much evidence to accumulate before making a decision is a problem we and other animals often face, but one that is not completely understood. This issue is particularly important because a tendency to sample less information (often known as reflection impulsivity) is a feature in several psychopathologies, such as psychosis. A formal understanding of information sampling may therefore clarify the computational anatomy of psychopathology. In this theoretical letter, we consider evidence accumulation in terms of active (Bayesian) inference using a generic model of Markov decision processes. Here, agents are equipped with beliefs about their own behavior--in this case, that they will make informed decisions. Normative decision making is then modeled using variational Bayes to minimize surprise about choice outcomes. Under this scheme, different facets of belief updating map naturally onto the functional anatomy of the brain (at least at a heuristic level). Of particular interest is the key role played by the expected precision of beliefs about control, which we have previously suggested may be encoded by dopaminergic neurons in the midbrain. We show that manipulating expected precision strongly affects how much information an agent characteristically samples, and thus provides a possible link between impulsivity and dopaminergic dysfunction. Our study therefore represents a step toward understanding evidence accumulation in terms of neurobiologically plausible Bayesian inference and may cast light on why this process is disordered in psychopathology.

Revealing a brain network endophenotype in families with idiopathic generalised epilepsy.

Idiopathic generalised epilepsy (IGE) has a genetic basis. The mechanism of seizure expression is not fully known, but is assumed to involve large-scale brain networks. We hypothesised that abnormal brain network properties would be detected using EEG in patients with IGE, and would be manifest as a familial endophenotype in their unaffected first-degree relatives. We studied 117 participants: 35 patients with IGE, 42 unaffected first-degree relatives, and 40 normal controls, using scalp EEG. Graph theory was used to describe brain network topology in five frequency bands for each subject. Frequency bands were chosen based on a published Spectral Factor Analysis study which demonstrated these bands to be optimally robust and independent. Groups were compared, using Bonferroni correction to account for nonindependent measures and multiple groups. Degree distribution variance was greater in patients and relatives than controls in the 6-9 Hz band (p = 0.0005, p = 0.0009 respectively). Mean degree was greater in patients than healthy controls in the 6-9 Hz band (p = 0.0064). Clustering coefficient was higher in patients and relatives than controls in the 6-9 Hz band (p = 0.0025, p = 0.0013). Characteristic path length did not differ between groups. No differences were found between patients and unaffected relatives. These findings suggest brain network topology differs between patients with IGE and normal controls, and that some of these network measures show similar deviations in patients and in unaffected relatives who do not have epilepsy. This suggests brain network topology may be an inherited endophenotype of IGE, present in unaffected relatives who do not have epilepsy, as well as in affected patients. We propose that abnormal brain network topology may be an endophenotype of IGE, though not in itself sufficient to cause epilepsy.

Cross-modal effects of value on perceptual acuity and stimulus encoding.

Cross-modal interactions are very common in perception. An important feature of many perceptual stimuli is their reward-predicting properties, the utilization of which is essential for adaptive behavior. What is unknown is whether reward associations in one sensory modality influence perception of stimuli in another modality. Here we show that auditory stimuli with high-reward associations increase the sensitivity of visual perception, even when sounds and reward associations are both irrelevant for the visual task. This increased sensitivity correlates with a change in stimulus representation in the visual cortex, indexed by increased multivariate decoding accuracy in simultaneously acquired functional MRI data. Univariate analysis showed that reward associations modulated responses in regions associated with multisensory processing in which the strength of modulation was a better predictor of the magnitude of the behavioral effect than the modulation in classical reward regions. Our findings demonstrate a value-driven cross-modal interaction that affects perception and stimulus encoding, with a resemblance to well-described modulatory effects of attention. We suggest that multisensory processing areas may mediate the transfer of value signals across senses.

Model averaging, optimal inference, and habit formation.

Postulating that the brain performs approximate Bayesian inference generates principled and empirically testable models of neuronal function-the subject of much current interest in neuroscience and related disciplines. Current formulations address inference and learning under some assumed and particular model. In reality, organisms are often faced with an additional challenge-that of determining which model or models of their environment are the best for guiding behavior. Bayesian model averaging-which says that an agent should weight the predictions of different models according to their evidence-provides a principled way to solve this problem. Importantly, because model evidence is determined by both the accuracy and complexity of the model, optimal inference requires that these be traded off against one another. This means an agent's behavior should show an equivalent balance. We hypothesize that Bayesian model averaging plays an important role in cognition, given that it is both optimal and realizable within a plausible neuronal architecture. We outline model averaging and how it might be implemented, and then explore a number of implications for brain and behavior. In particular, we propose that model averaging can explain a number of apparently suboptimal phenomena within the framework of approximate (bounded) Bayesian inference, focusing particularly upon the relationship between goal-directed and habitual behavior.

Widespread age-related differences in the human brain microstructure revealed by quantitative magnetic resonance imaging.

A pressing need exists to disentangle age-related changes from pathologic neurodegeneration. This study aims to characterize the spatial pattern and age-related differences of biologically relevant measures in vivo over the course of normal aging. Quantitative multiparameter maps that provide neuroimaging biomarkers for myelination and iron levels, parameters sensitive to aging, were acquired from 138 healthy volunteers (age range: 19-75 years). Whole-brain voxel-wise analysis revealed a global pattern of age-related degeneration. Significant demyelination occurred principally in the white matter. The observed age-related differences in myelination were anatomically specific. In line with invasive histologic reports, higher age-related differences were seen in the genu of the corpus callosum than the splenium. Iron levels were significantly increased in the basal ganglia, red nucleus, and extensive cortical regions but decreased along the superior occipitofrontal fascicle and optic radiation. This whole-brain pattern of age-associated microstructural differences in the asymptomatic population provides insight into the neurobiology of aging. The results help build a quantitative baseline from which to examine and draw a dividing line between healthy aging and pathologic neurodegeneration.

Transcranial direct current stimulation of right dorsolateral prefrontal cortex does not affect model-based or model-free reinforcement learning in humans.

There is broad consensus that the prefrontal cortex supports goal-directed, model-based decision-making. Consistent with this, we have recently shown that model-based control can be impaired through transcranial magnetic stimulation of right dorsolateral prefrontal cortex in humans. We hypothesized that an enhancement of model-based control might be achieved by anodal transcranial direct current stimulation of the same region. We tested 22 healthy adult human participants in a within-subject, double-blind design in which participants were given Active or Sham stimulation over two sessions. We show Active stimulation had no effect on model-based control or on model-free ('habitual') control compared to Sham stimulation. These null effects are substantiated by a power analysis, which suggests that our study had at least 60% power to detect a true effect, and by a Bayesian model comparison, which favors a model of the data that assumes stimulation had no effect over models that assume stimulation had an effect on behavioral control. Although we cannot entirely exclude more trivial explanations for our null effect, for example related to (faults in) our experimental setup, these data suggest that anodal transcranial direct current stimulation over right dorsolateral prefrontal cortex does not improve model-based control, despite existing evidence that transcranial magnetic stimulation can disrupt such control in the same brain region.