Revisiting the importance of model fitting for model-based fMRI: It does matter in computational psychiatry.

Computational modeling has been applied for data analysis in psychology, neuroscience, and psychiatry. One of its important uses is to infer the latent variables underlying behavior by which researchers can evaluate corresponding neural, physiological, or behavioral measures. This feature is especially crucial for computational psychiatry, in which altered computational processes underlying mental disorders are of interest. For instance, several studies employing model-based fMRI-a method for identifying brain regions correlated with latent variables-have shown that patients with mental disorders (e.g., depression) exhibit diminished neural responses to reward prediction errors (RPEs), which are the differences between experienced and predicted rewards. Such model-based analysis has the drawback that the parameter estimates and inference of latent variables are not necessarily correct-rather, they usually contain some errors. A previous study theoretically and empirically showed that the error in model-fitting does not necessarily cause a serious error in model-based fMRI. However, the study did not deal with certain situations relevant to psychiatry, such as group comparisons between patients and healthy controls. We developed a theoretical framework to explore such situations. We demonstrate that the parameter-misspecification can critically affect the results of group comparison. We demonstrate that even if the RPE response in patients is completely intact, a spurious difference to healthy controls is observable. Such a situation occurs when the ground-truth learning rate differs between groups but a common learning rate is used, as per previous studies. Furthermore, even if the parameters are appropriately fitted to individual participants, spurious group differences in RPE responses are observable when the model lacks a component that differs between groups. These results highlight the importance of appropriate model-fitting and the need for caution when interpreting the results of model-based fMRI.

A simple computational algorithm of model-based choice preference.

A broadly used computational framework posits that two learning systems operate in parallel during the learning of choice preferences-namely, the model-free and model-based reinforcement-learning systems. In this study, we examined another possibility, through which model-free learning is the basic system and model-based information is its modulator. Accordingly, we proposed several modified versions of a temporal-difference learning model to explain the choice-learning process. Using the two-stage decision task developed by Daw, Gershman, Seymour, Dayan, and Dolan (2011), we compared their original computational model, which assumes a parallel learning process, and our proposed models, which assume a sequential learning process. Choice data from 23 participants showed a better fit with the proposed models. More specifically, the proposed eligibility adjustment model, which assumes that the environmental model can weight the degree of the eligibility trace, can explain choices better under both model-free and model-based controls and has a simpler computational algorithm than the original model. In addition, the forgetting learning model and its variation, which assume changes in the values of unchosen actions, substantially improved the fits to the data. Overall, we show that a hybrid computational model best fits the data. The parameters used in this model succeed in capturing individual tendencies with respect to both model use in learning and exploration behavior. This computational model provides novel insights into learning with interacting model-free and model-based components.

Does the reliability of computational models truly improve with hierarchical modeling? Some recommendations and considerations for the assessment of model parameter reliability : Reliability of computational model parameters.

Computational modeling of behavior is increasingly being adopted as a standard methodology in psychology, cognitive neuroscience, and computational psychiatry. This approach involves estimating parameters in a computational (or cognitive) model that represents the computational processes of the underlying behavior. In this approach, the reliability of the parameter estimates is an important issue. The use of hierarchical (Bayesian) approaches, which place a prior on each model parameter of the individual participants, is thought to improve the reliability of the parameters. However, the characteristics of reliability in parameter estimates, especially when individual-level priors are assumed, as in hierarchical models, have not yet been fully discussed. Furthermore, the suitability of different reliability measures for assessing parameter reliability is not thoroughly understood. In this study, we conduct a systematic examination of these issues through theoretical analysis and numerical simulations, focusing specifically on reinforcement learning models. We note that the heterogeneity in the estimation precision of individual parameters, particularly with priors, can skew reliability measures toward individuals with higher precision. We further note that there are two factors that reduce reliability, namely estimation error and intersession variation in the true parameters, and we discuss how to evaluate these factors separately. Based on the considerations of this study, we present several recommendations and cautions for assessing the reliability of the model parameters.

Cardiac Cycle Affects the Asymmetric Value Updating in Instrumental Reward Learning.

This study aimed to investigate whether instrumental reward learning is affected by the cardiac cycle. To this end, we examined the effects of the cardiac cycle (systole or diastole) on the computational processes underlying the participants' choices in the instrumental learning task. In the instrumental learning task, participants were required to select one of two discriminative stimuli (neutral visual stimuli) and immediately receive reward/punishment feedback depending on the probability assigned to the chosen stimuli. To manipulate the cardiac cycle, the presentation of discriminative stimuli was timed to coincide with either cardiac systole or diastole. We fitted the participants' choices in the task with reinforcement learning (RL) models and estimated parameters involving instrumental learning (i.e., learning rate and inverse temperature) separately in the systole and diastole trials. Model-based analysis revealed that the learning rate for positive prediction errors was higher than that for negative prediction errors in the systole trials; however, learning rates did not differ between positive and negative prediction errors in the diastole trials. These results demonstrate that the natural fluctuation of cardiac afferent signals can affect asymmetric value updating in instrumental reward learning.

Reinforcement Learning With Parsimonious Computation and a Forgetting Process.

Decision-making is assumed to be supported by model-free and model-based systems: the model-free system is based purely on experience, while the model-based system uses a cognitive map of the environment and is more accurate. The recently developed multistep decision-making task and its computational model can dissociate the contributions of the two systems and have been used widely. This study used this task and model to understand our value-based learning process and tested alternative algorithms for the model-free and model-based learning systems. The task used in this study had a deterministic transition structure, and the degree of use of this structure in learning is estimated as the relative contribution of the model-based system to choices. We obtained data from 29 participants and fitted them with various computational models that differ in the model-free and model-based assumptions. The results of model comparison and parameter estimation showed that the participants update the value of action sequences and not each action. Additionally, the model fit was improved substantially by assuming that the learning mechanism includes a forgetting process, where the values of unselected options change to a certain default value over time. We also examined the relationships between the estimated parameters and psychopathology and other traits measured by self-reported questionnaires, and the results suggested that the difference in model assumptions can change the conclusion. In particular, inclusion of the forgetting process in the computational models had a strong impact on estimation of the weighting parameter of the model-free and model-based systems.

Maternal separation as a risk factor for aggravation of neuropathic pain in later life in mice.

Psychological stresses such as social loss and separation during childhood induce hardship, referred to as emotional pain. These experiences are well-documented risk factors for the development of physical pain in adulthood. However, the underlying neuronal mechanisms of this exacerbation of pain are largely unknown, and consequently there is no effective pharmacotherapy. In this study, we sought to determine whether infant maternal separation (MS) contributes to aggravation of neuropathic pain in adult mice. MS increased anxiety- and depression-like behavioral responses to adult stress. In MS animals, chronic constriction injury (CCI) heightened the sensory dimension of chronic pain relative to that of control mice. However, MS mice treated with fluoxetine for 4 weeks after MS did not exhibit augmentation of allodynia, and their emotional response was attenuated. Microglia were more abundant in the spinal cord in MS/CCI mice than in control/CCI mice. These results suggest that emotional impairment is related to augmentation of neuropathic pain, and that dysfunction of microglial activation contributes to heightened pain sensitivity.

Brain-immune interaction accompanying odor-evoked autobiographic memory.

The phenomenon in which a certain smell evokes a specific memory is known as the Proust phenomenon. Odor-evoked autobiographic memories are more emotional than those elicited by other sensory stimuli. The results of our previous study indicated that odor-evoked autobiographic memory accompanied by positive emotions has remarkable effects on various psychological and physiological activities, including the secretion of cytokines, which are immune-signaling molecules that modulate systemic inflammation. In this study, we aimed to clarify the neural substrates associated with the interaction between odor-evoked autobiographic memory and peripheral circulating cytokines. We recruited healthy male and female volunteers and investigated the association between brain responses and the concentration of several cytokines in the plasma by using positron emission tomography (PET) recordings when an autographic memory was evoked in participants by asking them to smell an odor that was nostalgic to them. Participants experienced positive emotions and autobiographic memories when nostalgic odors were presented to them. The levels of peripheral proinflammatory cytokines, such as the tumor necrosis factor-α (TNF-α) and interferon-γ (IFN-γ), were significantly reduced after experiencing odor-evoked autobiographic memory. Subtraction analysis of PET images indicated that the medial orbitofrontal cortex (mOFC) and precuneus/posterior cingulate cortex (PCC) were significantly activated during experiences of odor-evoked autobiographic memory. Furthermore, a correlation analysis indicated that activities of the mOFC and precuneus/PCC were negatively correlated with IFN-γ concentration. These results indicate that the neural networks including the precuneus/PCC and mOFC might regulate the secretion of peripheral proinflammatory cytokines during the experience of odor-evoked autobiographic memories accompanied with positive emotions.