Belief inference for hierarchical hidden states in spatial navigation.

Uncertainty abounds in the real world, and in environments with multiple layers of unobservable hidden states, decision-making requires resolving uncertainties based on mutual inference. Focusing on a spatial navigation problem, we develop a Tiger maze task that involved simultaneously inferring the local hidden state and the global hidden state from probabilistically uncertain observation. We adopt a Bayesian computational approach by proposing a hierarchical inference model. Applying this to human task behaviour, alongside functional magnetic resonance brain imaging, allows us to separate the neural correlates associated with reinforcement and reassessment of belief in hidden states. The imaging results also suggest that different layers of uncertainty differentially involve the basal ganglia and dorsomedial prefrontal cortex, and that the regions responsible are organised along the rostral axis of these areas according to the type of inference and the level of abstraction of the hidden state, i.e. higher-order state inference involves more anterior parts.

Deep learning-based image deconstruction method with maintained saliency.

Visual properties that primarily attract bottom-up attention are collectively referred to as saliency. In this study, to understand the neural activity involved in top-down and bottom-up visual attention, we aim to prepare pairs of natural and unnatural images with common saliency. For this purpose, we propose an image transformation method based on deep neural networks that can generate new images while maintaining the consistent feature map, in particular the saliency map. This is an ill-posed problem because the transformation from an image to its corresponding feature map could be many-to-one, and in our particular case, the various images would share the same saliency map. Although stochastic image generation has the potential to solve such ill-posed problems, the most existing methods focus on adding diversity of the overall style/touch information while maintaining the naturalness of the generated images. To this end, we developed a new image transformation method that incorporates higher-dimensional latent variables so that the generated images appear unnatural with less context information but retain a high diversity of local image structures. Although such high-dimensional latent spaces are prone to collapse, we proposed a new regularization based on Kullback-Leibler divergence to avoid collapsing the latent distribution. We also conducted human experiments using our newly prepared natural and corresponding unnatural images to measure overt eye movements and functional magnetic resonance imaging, and found that those images induced distinctive neural activities related to top-down and bottom-up attentional processing.

Confidence modulates the decodability of scene prediction during partially-observable maze exploration in humans.

Prediction ability often involves some degree of uncertainty-a key determinant of confidence. Here, we sought to assess whether predictions are decodable in partially-observable environments where one's state is uncertain, and whether this information is sensitive to confidence produced by such uncertainty. We used functional magnetic resonance imaging-based, partially-observable maze navigation tasks in which subjects predicted upcoming scenes and reported their confidence regarding these predictions. Using a multi-voxel pattern analysis, we successfully decoded both scene predictions and subjective confidence from activities in the localized parietal and prefrontal regions. We also assessed confidence in their beliefs about where they were in the maze. Importantly, prediction decodability varied according to subjective scene confidence in the superior parietal lobule and state confidence estimated by the behavioral model in the inferior parietal lobule. These results demonstrate that prediction in uncertain environments depends on the prefrontal-parietal network within which prediction and confidence interact.

A multi-site, multi-disorder resting-state magnetic resonance image database.

Machine learning classifiers for psychiatric disorders using resting-state functional magnetic resonance imaging (rs-fMRI) have recently attracted attention as a method for directly examining relationships between neural circuits and psychiatric disorders. To develop accurate and generalizable classifiers, we compiled a large-scale, multi-site, multi-disorder neuroimaging database. The database comprises resting-state fMRI and structural images of the brain from 993 patients and 1,421 healthy individuals, as well as demographic information such as age, sex, and clinical rating scales. To harmonize the multi-site data, nine healthy participants ("traveling subjects") visited the sites from which the above datasets were obtained and underwent neuroimaging with 12 scanners. All participants consented to having their data shared and analyzed at multiple medical and research institutions participating in the project, and 706 patients and 1,122 healthy individuals consented to having their data disclosed. Finally, we have published four datasets: 1) the SRPBS Multi-disorder Connectivity Dataset 2), the SRPBS Multi-disorder MRI Dataset (restricted), 3) the SRPBS Multi-disorder MRI Dataset (unrestricted), and 4) the SRPBS Traveling Subject MRI Dataset.

Pain Control by Co-adaptive Learning in a Brain-Machine Interface.

Innovation in the field of brain-machine interfacing offers a new approach to managing human pain. In principle, it should be possible to use brain activity to directly control a therapeutic intervention in an interactive, closed-loop manner. But this raises the question as to whether the brain activity changes as a function of this interaction. Here, we used real-time decoded functional MRI responses from the insula cortex as input into a closed-loop control system aimed at reducing pain and looked for co-adaptive neural and behavioral changes. As subjects engaged in active cognitive strategies orientated toward the control system, such as trying to enhance their brain activity, pain encoding in the insula was paradoxically degraded. From a mechanistic perspective, we found that cognitive engagement was accompanied by activation of the endogenous pain modulation system, manifested by the attentional modulation of pain ratings and enhanced pain responses in pregenual anterior cingulate cortex and periaqueductal gray. Further behavioral evidence of endogenous modulation was confirmed in a second experiment using an EEG-based closed-loop system. Overall, the results show that implementing brain-machine control systems for pain induces a parallel set of co-adaptive changes in the brain, and this can interfere with the brain signals and behavior under control. More generally, this illustrates a fundamental challenge of brain decoding applications-that the brain inherently adapts to being decoded, especially as a result of cognitive processes related to learning and cooperation. Understanding the nature of these co-adaptive processes informs strategies to mitigate or exploit them.

Model-based and model-free pain avoidance learning.

Background: While there is good evidence that reward learning is underpinned by two distinct decision control systems - a cognitive 'model-based' and a habitbased 'model-free' system, a comparable distinction for punishment avoidance has been much less clear. Methods: We implemented a pain avoidance task that placed differential emphasis on putative model-based and model-free processing, mirroring a paradigm and modelling approach recently developed for reward-based decision-making. Subjects performed a two-step decision-making task with probabilistic pain outcomes of different quantities. The delivery of outcomes was sometimes contingent on a rule signalled at the beginning of each trial, emulating a form of outcome devaluation. Results: The behavioural data showed that subjects tended to use a mixed strategy - favouring the simpler model-free learning strategy when outcomes did not depend on the rule, and favouring a model-based when they did. Furthermore, the data were well described by a dynamic transition model between the two controllers. When compared with data from a reward-based task (albeit tested in the context of the scanner), we observed that avoidance involved a significantly greater tendency for subjects to switch between model-free and model-based systems in the face of changes in uncertainty. Conclusion: Our study suggests a dual-system model of pain avoidance, similar to but possibly more dynamically flexible than reward-based decision-making.

Classification and characterisation of brain network changes in chronic back pain: A multicenter study.

Background. Chronic pain is a common, often disabling condition thought to involve a combination of peripheral and central neurobiological factors. However, the extent and nature of changes in the brain is poorly understood. Methods. We investigated brain network architecture using resting-state fMRI data in chronic back pain patients in the UK and Japan (41 patients, 56 controls), as well as open data from USA. We applied machine learning and deep learning (conditional variational autoencoder architecture) methods to explore classification of patients/controls based on network connectivity. We then studied the network topology of the data, and developed a multislice modularity method to look for consensus evidence of modular reorganisation in chronic back pain. Results. Machine learning and deep learning allowed reliable classification of patients in a third, independent open data set with an accuracy of 63%, with 68% in cross validation of all data. We identified robust evidence of network hub disruption in chronic pain, most consistently with respect to clustering coefficient and betweenness centrality. We found a consensus pattern of modular reorganisation involving extensive, bilateral regions of sensorimotor cortex, and characterised primarily by negative reorganisation - a tendency for sensorimotor cortex nodes to be less inclined to form pairwise modular links with other brain nodes. Furthermore, these regions were found to display increased connectivity with the pregenual anterior cingulate cortex, a region known to be involved in endogenous pain control. In contrast, intraparietal sulcus displayed a propensity towards positive modular reorganisation, suggesting that it might have a role in forming modules associated with the chronic pain state. Conclusion. The results provide evidence of consistent and characteristic brain network changes in chronic pain, characterised primarily by extensive reorganisation of the network architecture of the sensorimotor cortex.

The control of tonic pain by active relief learning.

Tonic pain after injury characterises a behavioural state that prioritises recovery. Although generally suppressing cognition and attention, tonic pain needs to allow effective relief learning to reduce the cause of the pain. Here, we describe a central learning circuit that supports learning of relief and concurrently suppresses the level of ongoing pain. We used computational modelling of behavioural, physiological and neuroimaging data in two experiments in which subjects learned to terminate tonic pain in static and dynamic escape-learning paradigms. In both studies, we show that active relief-seeking involves a reinforcement learning process manifest by error signals observed in the dorsal putamen. Critically, this system uses an uncertainty ('associability') signal detected in pregenual anterior cingulate cortex that both controls the relief learning rate, and endogenously and parametrically modulates the level of tonic pain. The results define a self-organising learning circuit that reduces ongoing pain when learning about potential relief.

Thermosensory Perceptual Learning Is Associated with Structural Brain Changes in Parietal-Opercular (SII) Cortex.

The location of a sensory cortex for temperature perception remains a topic of substantial debate. Both the parietal-opercular (SII) and posterior insula have been consistently implicated in thermosensory processing, but neither region has yet been identified as the locus of fine temperature discrimination. Using a perceptual learning paradigm in male and female humans, we show improvement in discrimination accuracy for subdegree changes in both warmth and cool detection over 5 d of repetitive training. We found that increases in discriminative accuracy were specific to the temperature (cold or warm) being trained. Using structural imaging to look for plastic changes associated with perceptual learning, we identified symmetrical increases in gray matter volume in the SII cortex. Furthermore, we observed distinct, adjacent regions for cold and warm discrimination, with cold discrimination having a more anterior locus than warm. The results suggest that thermosensory discrimination is supported by functionally and anatomically distinct temperature-specific modules in the SII cortex.SIGNIFICANCE STATEMENT We provide behavioral and neuroanatomical evidence that perceptual learning is possible within the temperature system. We show that structural plasticity localizes to parietal-opercular (SII), and not posterior insula, providing the best evidence to date resolving a longstanding debate about the location of putative "temperature cortex." Furthermore, we show that cold and warm pathways are behaviorally and anatomically dissociable, suggesting that the temperature system has distinct temperature-dependent processing modules.

Decisions about decisions.

A major puzzle of decision making is how the brain decides which decision system to use at any one time. In this issue of Neuron, Lee et al. (2014) provide a theoretical, behavioral, and neurobiological account of a prefrontal reliability-based arbitration system.

Prices need no preferences: social trends determine decisions in experimental markets for pain relief.

OBJECTIVE: A standard view in health economics is that, although there is no market that determines the "prices" for health states, people can nonetheless associate health states with monetary values (or other scales, such as quality adjusted life year [QALYs] and disability adjusted life year [DALYs]). Such valuations can be used to shape health policy, and a major research challenge is to elicit such values from people; creating experimental "markets" for health states is a theoretically attractive way to address this. We explore the possibility that this framework may be fundamentally flawed-because there may not be any stable values to be revealed. Instead, perhaps people construct ad hoc values, influenced by contextual factors, such as the observed decisions of others. METHOD: The participants bid to buy relief from equally painful electrical shocks to the leg and arm in an experimental health market based on an interactive second-price auction. Thirty subjects were randomly assigned to two experimental conditions where the bids by "others" were manipulated to follow increasing or decreasing price trends for one, but not the other, pain. After the auction, a preference test asked the participants to choose which pain they prefer to experience for a longer duration. RESULTS: Players remained indifferent between the two pain-types throughout the auction. However, their bids were differentially attracted toward what others bid for each pain, with overbidding during decreasing prices and underbidding during increasing prices. CONCLUSION: Health preferences are dissociated from market prices, which are strongly referenced to others' choices. This suggests that the price of health care in a free-market has the capacity to become critically detached from people's underlying preferences.

Uncertainty increases pain: evidence for a novel mechanism of pain modulation involving the periaqueductal gray.

Predictions about sensory input exert a dominant effect on what we perceive, and this is particularly true for the experience of pain. However, it remains unclear what component of prediction, from an information-theoretic perspective, controls this effect. We used a vicarious pain observation paradigm to study how the underlying statistics of predictive information modulate experience. Subjects observed judgments that a group of people made to a painful thermal stimulus, before receiving the same stimulus themselves. We show that the mean observed rating exerted a strong assimilative effect on subjective pain. In addition, we show that observed uncertainty had a specific and potent hyperalgesic effect. Using computational functional magnetic resonance imaging, we found that this effect correlated with activity in the periaqueductal gray. Our results provide evidence for a novel form of cognitive hyperalgesia relating to perceptual uncertainty, induced here by vicarious observation, with control mediated by the brainstem pain modulatory system.

Neural mechanisms of belief inference during cooperative games.

Humans have the arguably unique ability to understand the mental representations of others. For success in both competitive and cooperative interactions, however, this ability must be extended to include representations of others' belief about our intentions, their model about our belief about their intentions, and so on. We developed a "stag hunt" game in which human subjects interacted with a computerized agent using different degrees of sophistication (recursive inferences) and applied an ecologically valid computational model of dynamic belief inference. We show that rostral medial prefrontal (paracingulate) cortex, a brain region consistently identified in psychological tasks requiring mentalizing, has a specific role in encoding the uncertainty of inference about the other's strategy. In contrast, dorsolateral prefrontal cortex encodes the depth of recursion of the strategy being used, an index of executive sophistication. These findings reveal putative computational representations within prefrontal cortex regions, supporting the maintenance of cooperation in complex social decision making.

Cooperation and heterogeneity of the autistic mind.

Individuals with autism spectrum conditions (ASCs) have a core difficulty in recursively inferring the intentions of others. The precise cognitive dysfunctions that determine the heterogeneity at the heart of this spectrum, however, remains unclear. Furthermore, it remains possible that impairment in social interaction is not a fundamental deficit but a reflection of deficits in distinct cognitive processes. To better understand heterogeneity within ASCs, we employed a game-theoretic approach to characterize unobservable computational processes implicit in social interactions. Using a social hunting game with autistic adults, we found that a selective difficulty representing the level of strategic sophistication of others, namely inferring others' mindreading strategy, specifically predicts symptom severity. In contrast, a reduced ability in iterative planning was predicted by overall intellectual level. Our findings provide the first quantitative approach that can reveal the underlying computational dysfunctions that generate the autistic "spectrum."

Hierarchical rule switching in prefrontal cortex.

Most real-world decision-making problems involve consideration of numerous possible actions, and it is often impossible to evaluate all of them before settling on preferred strategy. In such situations, humans might explore actions more efficiently by searching only the most likely subspace of the whole action space. To study how the brain solves such action selection problems, we designed a Multi Feature Sorting Task in which the task rules defining an optimal action have a hierarchical structure and studied concurrent brain activity using it. The task consisted of two kinds of rule switches: a higher-order switch to search for a rule across different subspaces and a lower-order switch to change a rule within the same subspace. The results revealed that the left dorsolateral prefrontal cortex (DLPFC) was more active in the higher-order switching, and the right fronto-polar cortex (FPC) was significantly activated with the lower-order switching. We discuss a possible functional model in the prefrontal cortex where the left DLPFC encodes the hierarchical organization of behaviours and the right FPC maintains and updates multiple behavioural. This interpretation is highly consistent with the previous findings and current theories of hierarchical organization in the prefrontal functional network.

Altruistic learning.

The origin of altruism remains one of the most enduring puzzles of human behaviour. Indeed, true altruism is often thought either not to exist, or to arise merely as a miscalculation of otherwise selfish behaviour. In this paper, we argue that altruism emerges directly from the way in which distinct human decision-making systems learn about rewards. Using insights provided by neurobiological accounts of human decision-making, we suggest that reinforcement learning in game-theoretic social interactions (habitisation over either individuals or games) and observational learning (either imitative of inference based) lead to altruistic behaviour. This arises not only as a result of computational efficiency in the face of processing complexity, but as a direct consequence of optimal inference in the face of uncertainty. Critically, we argue that the fact that evolutionary pressure acts not over the object of learning ('what' is learned), but over the learning systems themselves ('how' things are learned), enables the evolution of altruism despite the direct threat posed by free-riders.

Game theory of mind.

This paper introduces a model of 'theory of mind', namely, how we represent the intentions and goals of others to optimise our mutual interactions. We draw on ideas from optimum control and game theory to provide a 'game theory of mind'. First, we consider the representations of goals in terms of value functions that are prescribed by utility or rewards. Critically, the joint value functions and ensuing behaviour are optimised recursively, under the assumption that I represent your value function, your representation of mine, your representation of my representation of yours, and so on ad infinitum. However, if we assume that the degree of recursion is bounded, then players need to estimate the opponent's degree of recursion (i.e., sophistication) to respond optimally. This induces a problem of inferring the opponent's sophistication, given behavioural exchanges. We show it is possible to deduce whether players make inferences about each other and quantify their sophistication on the basis of choices in sequential games. This rests on comparing generative models of choices with, and without, inference. Model comparison is demonstrated using simulated and real data from a 'stag-hunt'. Finally, we note that exactly the same sophisticated behaviour can be achieved by optimising the utility function itself (through prosocial utility), producing unsophisticated but apparently altruistic agents. This may be relevant ethologically in hierarchal game theory and coevolution.

Resolution of uncertainty in prefrontal cortex.

Making optimal decisions in the face of uncertain or incomplete information arises as a common problem in everyday behavior, but the neural processes underlying this ability remain poorly understood. A typical case is navigation, in which a subject has to search for a known goal from an unknown location. Navigating under uncertain conditions requires making decisions on the basis of the current belief about location and updating that belief based on incoming information. Here, we use functional magnetic resonance imaging during a maze navigation task to study neural activity relating to the resolution of uncertainty as subjects make sequential decisions to reach a goal. We show that distinct regions of prefrontal cortex are engaged in specific computational functions that are well described by a Bayesian model of decision making. This permits efficient goal-oriented navigation and provides new insights into decision making by humans.

Control of exploitation-exploration meta-parameter in reinforcement learning.

In reinforcement learning (RL), the duality between exploitation and exploration has long been an important issue. This paper presents a new method that controls the balance between exploitation and exploration. Our learning scheme is based on model-based RL, in which the Bayes inference with forgetting effect estimates the state-transition probability of the environment. The balance parameter, which corresponds to the randomness in action selection, is controlled based on variation of action results and perception of environmental change. When applied to maze tasks, our method successfully obtains good controls by adapting to environmental changes. Recently, Usher et al. [Science 283 (1999) 549] has suggested that noradrenergic neurons in the locus coeruleus may control the exploitation-exploration balance in a real brain and that the balance may correspond to the level of animal's selective attention. According to this scenario, we also discuss a possible implementation in the brain.