Neurocomputational model of compulsivity: Deviating from an uncertain goal-directed system.

Despite a theory that an imbalance in goal-directed versus habitual systems serve as building blocks of compulsions, research has yet to delineate how it occurs during an arbitration process between the two systems in obsessive-compulsive disorder. Inspired by a brain model that the inferior frontal cortex selectively gates the putamen to guide goal-directed or habitual actions, this study aimed to examine whether disruptions in the arbitration process via the fronto-striatal circuit would underlie the imbalanced decision-making and compulsions in patients. Thirty patients with obsessive-compulsive disorder (mean [SD] age = 26.93 [6.23] years, 12 females [40%]) and thirty healthy controls (mean [SD] age = 24.97 [4.72] years, 17 females [57%]) underwent functional MRI scans while performing the two-step Markov decision task, which was designed to dissociate goal-directed behavior from habitual behavior. We employed a neurocomputational model to account for an uncertainty-based arbitration process, in which a prefrontal arbitrator (i.e., inferior frontal gyrus) allocates behavioral control to a more reliable strategy by selectively gating the putamen. We analyzed group differences in the neural estimates of uncertainty of each strategy. We also compared the psychophysiological interaction effects of system preference (goal-directed vs. habitual) on fronto-striatal coupling between groups. We examined the correlation between compulsivity score and the neural activity and connectivity involved in the arbitration process. The computational model captured subjects' preference between the strategies. Compared to healthy controls, patients had a stronger preference for the habitual system (t = -2.88, P = 0.006), which was attributed to a more uncertain goal-directed system (t = 2.72, P = 0.009). Before the allocation of controls, patients exhibited hypoactivity in the inferior frontal gyrus compared to healthy controls when this region tracked the inverse of uncertainty (i.e., reliability) of goal-directed behavior (P = 0.001, family-wise error rate corrected). When reorienting behaviors to reach specific goals, patients exhibited weaker right ipsilateral ventrolateral prefronto-putamen coupling than healthy controls (P = 0.001, family-wise error rate corrected). This hypoconnectivity was correlated with more severe compulsivity (r = -0.57, P = 0.002). Our findings suggest that the attenuated top-down control of the putamen by the prefrontal arbitrator underlies compulsivity in obsessive-compulsive disorder. Enhancing fronto-striatal connectivity may be a potential neurotherapeutic approach for compulsivity and adaptive decision-making.

Controlling human causal inference through in silico task design.

Learning causal relationships is crucial for survival. The human brain's functional flexibility allows for effective causal inference, underlying various learning processes. While past studies focused on environmental factors influencing causal inference, a fundamental question remains: can these factors be manipulated for strategic causal inference control? This paper presents a task control framework for orchestrating causal learning task design. It utilizes a two-player game setting where a neural network learns to manipulate task variables by interacting with a human causal inference model. Training the task controller to generate experimental designs, we confirm its ability to accommodate complexities of environmental causal structure. Experiments involving 126 human subjects successfully validate the impact of task control on performance and learning efficiency. Additionally, we find that task control policy reflects the intrinsic nature of human causal inference: one-shot learning. This framework holds promising potential for applications paving the way for targeted behavioral outcomes in humans.

Pneumatic tactile stimulus delivery system for studying brain responses evoked by active finger touch with fMRI.

BACKGROUND: Primates use their hands to actively touch objects and collect information. To study tactile information processing, it is important for participants to experience tactile stimuli through active touch while monitoring brain activities. NEW METHOD: Here, we developed a pneumatic tactile stimulus delivery system (pTDS) that delivers various tactile stimuli on a programmed schedule and allows voluntary finger touches during MRI scanning. The pTDS uses a pneumatic actuator to move tactile stimuli and place them in a finger hole. A photosensor detects the time when an index finger touches a tactile stimulus, enabling the analysis of the touch-elicited brain responses. RESULTS: We examined brain responses while the participants actively touched braille objects presented by the pTDS. BOLD responses during tactile perception were significantly stronger in a finger touch area of the contralateral somatosensory cortex compared with that of visual perception. CONCLUSION: The pTDS enables MR studies of brain mechanisms for tactile processes through natural finger touch.

External globus pallidus input to the dorsal striatum regulates habitual seeking behavior in male mice.

The external globus pallidus (GPe) coordinates action-selection through GABAergic projections throughout the basal ganglia. GPe arkypallidal (arky) neurons project exclusively to the dorsal striatum, which regulates goal-directed and habitual seeking. However, the role of GPe arky neurons in reward-seeking remains unknown. Here, we identified that a majority of arky neurons target the dorsolateral striatum (DLS). Using fiber photometry, we found that arky activities were higher during random interval (RI; habit) compared to random ratio (RR; goal) operant conditioning. Support vector machine analysis demonstrated that arky neuron activities have sufficient information to distinguish between RR and RI behavior. Genetic ablation of this arkyGPe→DLS circuit facilitated a shift from goal-directed to habitual behavior. Conversely, chemogenetic activation globally reduced seeking behaviors, which was blocked by systemic D1R agonism. Our findings reveal a role of this arkyGPe→DLS circuit in constraining habitual seeking in male mice, which is relevant to addictive behaviors and other compulsive disorders.

Astrocyte activities in the external globus pallidus regulate action-selection strategies in reward-seeking behaviors.

An imbalance in goal-directed and habitual behavioral control is a hallmark of decision-making-related disorders, including addiction. Although external globus pallidus (GPe) is critical for action selection, which harbors enriched astrocytes, the role of GPe astrocytes involved in action-selection strategies remained unknown. Using in vivo calcium signaling with fiber photometry, we found substantially attenuated GPe astrocytic activity during habitual learning compared to goal-directed learning. The support vector machine analysis predicted the behavioral outcomes. Chemogenetic activation of the astrocytes or inhibition of GPe pan-neuronal activities facilitates the transition from habit to goal-directed reward-seeking behavior. Next, we found increased astrocyte-specific GABA (γ-aminobutyric acid) transporter type 3 (GAT3) messenger RNA expression during habit learning. Notably, the pharmacological inhibition of GAT3 occluded astrocyte activation-induced transition from habitual to goal-directed behavior. On the other hand, attentional stimuli shifted the habit to goal-directed behaviors. Our findings suggest that the GPe astrocytes regulate the action selection strategy and behavioral flexibility.

Impairment of arbitration between model-based and model-free reinforcement learning in obsessive-compulsive disorder.

Introduction: Obsessive-compulsive disorder (OCD) is characterized by an imbalance between goal-directed and habitual learning systems in behavioral control, but it is unclear whether these impairments are due to a single system abnormality of the goal-directed system or due to an impairment in a separate arbitration mechanism that selects which system controls behavior at each point in time. Methods: A total of 30 OCD patients and 120 healthy controls performed a 2-choice, 3-stage Markov decision-making paradigm. Reinforcement learning models were used to estimate goal-directed learning (as model-based reinforcement learning) and habitual learning (as model-free reinforcement learning). In general, 29 high Obsessive-Compulsive Inventory-Revised (OCI-R) score controls, 31 low OCI-R score controls, and all 30 OCD patients were selected for the analysis. Results: Obsessive-compulsive disorder (OCD) patients showed less appropriate strategy choices than controls regardless of whether the OCI-R scores in the control subjects were high (p = 0.012) or low (p < 0.001), specifically showing a greater model-free strategy use in task conditions where the model-based strategy was optimal. Furthermore, OCD patients (p = 0.001) and control subjects with high OCI-R scores (H-OCI-R; p = 0.009) both showed greater system switching rather than consistent strategy use in task conditions where model-free use was optimal. Conclusion: These findings indicated an impaired arbitration mechanism for flexible adaptation to environmental demands in both OCD patients and healthy individuals reporting high OCI-R scores.

Hippocampal integration and separation processes with different temporal and spatial dynamics during learning for associative memory.

The hippocampus is known to be critically involved in associative memory formation. However, the role of the hippocampus during the learning of associative memory is still controversial; while the hippocampus is considered to play a critical role in the integration of related stimuli, numerous studies also suggest a role of the hippocampus in the separation of different memory traces for rapid learning. Here, we employed an associative learning paradigm consisting of repeated learning cycles. By tracking the changes in the hippocampal representations of associated stimuli on a cycle-by-cycle basis as learning progressed, we show that both integration and separation processes occur in the hippocampus with different temporal dynamics. We found that the degree of shared representations for associated stimuli decreased significantly during the early phase of learning, whereas it increased during the later phase of learning. Remarkably, these dynamic temporal changes were observed only for stimulus pairs remembered 1 day or 4 weeks after learning, but not for forgotten pairs. Further, the integration process during learning was prominent in the anterior hippocampus, while the separation process was obvious in the posterior hippocampus. These results demonstrate temporally and spatially dynamic hippocampal processing during learning that can lead to the maintenance of associative memory.

Concordance between the international standards for neurological classification of spinal cord injury motor examination and needle electromyography findings in muscles with a motor power grade of zero or trace.

Context/Objective: To evaluate the accuracy of the International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI) motor examination in individuals with spinal cord injury (SCI) with motor grade 0 or 1 and analyze its degree of concordance with needle electromyography (EMG) findings for each key muscle.Design: Retrospective study.Setting: University hospital in Goyang, Korea.Participants: Individuals with SCI admitted to the Department of Rehabilitation from January 2013 to June 2019.Interventions: In the enrolled persons, needle EMG was performed on muscles with motor grade 0 or 1 on ISNCSCI examination, and muscle contraction was confirmed through the detection of motor unit action potential.Outcome measures: The agreement between motor examination and needle EMG findings was analyzed.Results: In 175 key muscles, needle EMG findings in 115 and 60 muscles evaluated as grades 0 and 1 on ISNCSCI examination showed 80% and 50% agreements, respectively. We found a fair agreement between motor examination and needle EMG findings (κ = 0.309, P < 0.0001). Moreover, statistically significant agreement was seen only in T1, L2, and S1 key muscles (κ = 1, P < 0.0001; κ = 0.359, P = 0.019; and κ = 0.521, P = 0.004, respectively).Conclusions: It is important to accurately distinguish between grade 0 and 1 motor power to maximize the positive outcomes from rehabilitation treatment and predict the possibility of recovery in individuals with SCI. Therefore, to improve the accuracy of motor examination and the American Spinal Injury Association Impairment Scale, needle EMG confirmation could be considered for muscles with motor grade 0 or 1 in individuals with SCI.

Meta-control of social learning strategies.

Social learning, copying other's behavior without actual experience, offers a cost-effective means of knowledge acquisition. However, it raises the fundamental question of which individuals have reliable information: successful individuals versus the majority. The former and the latter are known respectively as success-based and conformist social learning strategies. We show here that while the success-based strategy fully exploits the benign environment of low uncertainly, it fails in uncertain environments. On the other hand, the conformist strategy can effectively mitigate this adverse effect. Based on these findings, we hypothesized that meta-control of individual and social learning strategies provides effective and sample-efficient learning in volatile and uncertain environments. Simulations on a set of environments with various levels of volatility and uncertainty confirmed our hypothesis. The results imply that meta-control of social learning affords agents the leverage to resolve environmental uncertainty with minimal exploration cost, by exploiting others' learning as an external knowledge base.

Importance of prefrontal meta control in human-like reinforcement learning.

Recent investigation on reinforcement learning (RL) has demonstrated considerable flexibility in dealing with various problems. However, such models often experience difficulty learning seemingly easy tasks for humans. To reconcile the discrepancy, our paper is focused on the computational benefits of the brain's RL. We examine the brain's ability to combine complementary learning strategies to resolve the trade-off between prediction performance, computational costs, and time constraints. The complex need for task performance created by a volatile and/or multi-agent environment motivates the brain to continually explore an ideal combination of multiple strategies, called meta-control. Understanding these functions would allow us to build human-aligned RL models.

Prefrontal solution to the bias-variance tradeoff during reinforcement learning.

Evidence that the brain combines different value learning strategies to minimize prediction error is accumulating. However, the tradeoff between bias and variance error, which imposes different constraints on each learning strategy's performance, poses a challenge for value learning. While this tradeoff specifies the requirements for optimal learning, little has been known about how the brain deals with this issue. Here, we hypothesize that the brain adaptively resolves the bias-variance tradeoff during reinforcement learning. Our theory suggests that the solution necessitates baseline correction for prediction error, which offsets the adverse effects of irreducible error on value learning. We show behavioral evidence of adaptive control using a Markov decision task with context changes. The prediction error baseline seemingly signals context changes to improve adaptability. Critically, we identify multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide model-based and model-free reinforcement learning.

Neurocomputational mechanism of controllability inference under a multi-agent setting.

Controllability perception significantly influences motivated behavior and emotion and requires an estimation of one's influence on an environment. Previous studies have shown that an agent can infer controllability by observing contingency between one's own action and outcome if there are no other outcome-relevant agents in an environment. However, if there are multiple agents who can influence the outcome, estimation of one's genuine controllability requires exclusion of other agents' possible influence. Here, we first investigated a computational and neural mechanism of controllability inference in a multi-agent setting. Our novel multi-agent Bayesian controllability inference model showed that other people's action-outcome contingency information is integrated with one's own action-outcome contingency to infer controllability, which can be explained as a Bayesian inference. Model-based functional MRI analyses showed that multi-agent Bayesian controllability inference recruits the temporoparietal junction (TPJ) and striatum. Then, this inferred controllability information was leveraged to increase motivated behavior in the vmPFC. These results generalize the previously known role of the striatum and vmPFC in single-agent controllability to multi-agent controllability, and this generalized role requires the TPJ in addition to the striatum of single-agent controllability to integrate both self- and other-related information. Finally, we identified an innate positive bias toward the self during the multi-agent controllability inference, which facilitated behavioral adaptation under volatile controllability. Furthermore, low positive bias and high negative bias were associated with increased daily feelings of guilt. Our results provide a mechanism of how our sense of controllability fluctuates due to other people in our lives, which might be related to social learned helplessness and depression.

Effects of subclinical depression on prefrontal-striatal model-based and model-free learning.

Depression is characterized by deficits in the reinforcement learning (RL) process. Although many computational and neural studies have extended our knowledge of the impact of depression on RL, most focus on habitual control (model-free RL), yielding a relatively poor understanding of goal-directed control (model-based RL) and arbitration control to find a balance between the two. We investigated the effects of subclinical depression on model-based and model-free learning in the prefrontal-striatal circuitry. First, we found that subclinical depression is associated with the attenuated state and reward prediction error representation in the insula and caudate. Critically, we found that it accompanies the disrupted arbitration control between model-based and model-free learning in the predominantly inferior lateral prefrontal cortex and frontopolar cortex. We also found that depression undermines the ability to exploit viable options, called exploitation sensitivity. These findings characterize how subclinical depression influences different levels of the decision-making hierarchy, advancing previous conflicting views that depression simply influences either habitual or goal-directed control. Our study creates possibilities for various clinical applications, such as early diagnosis and behavioral therapy design.

Epidemiology of Spinal Cord Injury: Changes to Its Cause Amid Aging Population, a Single Center Study.

OBJECTIVE: To investigate the epidemiologic and demographic characteristics of patients with spinal cord injury (SCI) who were admitted to a department of rehabilitation of a university hospital. METHODS: This was a descriptive cross-sectional study. Medical records including sex, age at injury, type of disability, traumatic or non-traumatic etiology and presence of ossification of posterior longitudinal ligament (OPLL) of patients with SCI who were admitted to the department of rehabilitation between 2012 and 2018 were reviewed. RESULTS: Of the 221 cases of SCI, 161 were traumatic and 60 were non-traumatic. The mean age at injury was 52.8 years. People aged 40-49 years showed highest proportion among overall SCI patients (19.0%). The proportion of male patients was higher in traumatic SCI at 4.96:1 than in non-traumatic SCI at 1.30:1. The most common cause of traumatic SCI was falling off (37.3%), followed by motor vehicle crash (35.4%) and tripping over (19.3%). Meanwhile, the most common cause of non-traumatic SCI was neoplasm (35.0%). Tripping over was the leading cause of traumatic SCI in patients aged ≥60 years (42.6%). A high proportion of traumatic SCI patients were found to have underlying OPLL (26.1%), particularly those who were injured by tripping over (64.5%). CONCLUSION: The mean age of SCI patients was higher than that of previous studies. Falls was the single most common cause of traumatic SCI, and tripping over was the most common cause of injury in the elderly patients. OPLL was prevalent in patients who were injured from tripping over.

The efficacy of computerized cognitive rehabilitation in improving attention and executive functions in acquired brain injury patients, in acute and postacute phase.

BACKGROUND: Cognitive deficits, particularly executive dysfunction is common following acquired brain injury (ABI) and has detrimental effect on functional status and autonomy in daily life. Among various cognitive training methods, computerized cognitive rehabilitation (CCR) has been investigated as an alternative method to therapist-driven cognitive rehabilitation (TCR). However, previous studies have shown conflicting results on the superiority or inferiority of CCR and TCR. AIM: To investigate the efficacy of TCR and CCR in improving executive function in patients with acute-to-subacute ABI. DESIGN: A prospective, assessor-blinded randomized controlled trial. SETTING: Hospitalized care setting in the department of rehabilitation in a university hospital. POPULATION: Thirty-two acute-to-subacute (less than 3 months after onset) ABI patients with executive dysfunctions were included in this study. The mean time after injury was 25.1±18.1 days. METHODS: Participants were assigned to the TCR group (N.=14) or the CCR group (N.=18). Each group performed TCR or CCR for 30 minutes each day for two weeks in addition to routine rehabilitation. Neurocognitive function tests to assess complex attention, executive function, general cognitive function (mini-mental status examination [MMSE] and Montreal Cognitive Assessment [MoCA]), and functional evaluations [modified Barthel Index, MBI]) were performed at baseline (T0) and at the end of treatment (T1). RESULTS: The TCR and CCR groups showed significant improvements in the MMSE (P=0.004, 0.000), MoCA (P=0.003, 0.006), and MBI (P=0.000, 0.000) scores. TCR and CCR groups both showed significant improvements in some of the complex attention tests (trail-making test A, P=0.002, 0.005) and executive function tests (trail-making test B, P=0.016, 0.016). The TCR group showed significant improvements in the additional executive function tests (phonemic fluency test, P=0.004, semantic fluency test, P=0.001), while the CCR group showed significant improvements in the additional complex attention tests (symbol search, P=0.02, digit symbol coding, P=0.002). In the intergroup comparison of the changes from pre- to postintervention, only the TCR group showed a significant improvement in the phonemic fluency test (P=0.013). CONCLUSIONS: TCR might be more effective than CCR in improving frontal lobe-related executive function in ABI patients. CCR might be beneficial for improving psychomotor speed and working memory. CLINICAL REHABILITATION IMPACT: TCR or CCR should be chosen according to the targeted domain of cognitive dysfunction in acute-to-subacute ABI patients.

Why and how the brain weights contributions from a mixture of experts.

It has long been suggested that human behavior reflects the contributions of multiple systems that cooperate or compete for behavioral control. Here we propose that the brain acts as a "Mixture of Experts" in which different expert systems propose strategies for action. It will be argued that the brain determines which experts should control behavior at any one moment in time by keeping track of the reliability of the predictions within each system, and by allocating control over behavior in a manner that depends on the relative reliabilities across experts. fMRI and neurostimulation studies suggest a specific contribution of the anterior prefrontal cortex in this process. Further, such a mechanism also takes into consideration the complexity of the expert, favoring simpler over more cognitively complex experts. Results from the study of different expert systems in both experiential and social learning domains hint at the possibility that this reliability-based control mechanism is domain general, exerting control over many different expert systems simultaneously in order to produce sophisticated behavior.

Sick Sinus Syndrome Combined with Wallenberg Syndrome: a Case Report.

Cardiac arrhythmia is a rare manifestation of the Wallenberg syndrome; lesions are located in the brainstem, especially the lower medulla, which regulates sympathetic and parasympathetic activity. A 55-year-old man was admitted to the university hospital with symptoms including ataxia, left ptosis, decreased sensation of pain and temperature on the right side, left facial numbness, and dizziness. Brain magnetic resonance imaging revealed an infarction in the left dorsolateral medulla. Therefore, he was diagnosed with Wallenberg syndrome. While he underwent conservative treatment for Wallenberg syndrome, he experienced several events of self-limiting heart pounding, which required an evaluation of cardiac function. The 24-hour Holter monitor showed an increased RR interval with bradycardia and prolonged sinus pause. As a result, the diagnosis of sick sinus syndrome combined with Wallenberg syndrome was made. Sick sinus syndrome is a rare cardiac complication of the Wallenberg syndrome, and clinicians could overlook it when the initial electrocardiography shows a normal sinus rhythm. Sick sinus syndrome can cause sudden death without appropriate medical intervention. Therefore, clinicians should consider further evaluation, including a 24-hour Holter monitor, to check for the potential presence of sick sinus syndrome in the acute phase of Wallenberg syndrome.

Behavioral evidence for memory replay of video episodes in the macaque.

Humans recall the past by replaying fragments of events temporally. Here, we demonstrate a similar effect in macaques. We trained six rhesus monkeys with a temporal-order judgement (TOJ) task and collected 5000 TOJ trials. In each trial, the monkeys watched a naturalistic video of about 10 s comprising two across-context clips, and after a 2 s delay, performed TOJ between two frames from the video. The data are suggestive of a non-linear, time-compressed forward memory replay mechanism in the macaque. In contrast with humans, such compression of replay is, however, not sophisticated enough to allow these monkeys to skip over irrelevant information by compressing the encoded video globally. We also reveal that the monkeys detect event contextual boundaries, and that such detection facilitates recall by increasing the rate of information accumulation. Demonstration of a time-compressed, forward replay-like pattern in the macaque provides insights into the evolution of episodic memory in our lineage.

Dynamic resource allocation during reinforcement learning accounts for ramping and phasic dopamine activity.

For an animal to learn about its environment with limited motor and cognitive resources, it should focus its resources on potentially important stimuli. However, too narrow focus is disadvantageous for adaptation to environmental changes. Midbrain dopamine neurons are excited by potentially important stimuli, such as reward-predicting or novel stimuli, and allocate resources to these stimuli by modulating how an animal approaches, exploits, explores, and attends. The current study examined the theoretical possibility that dopamine activity reflects the dynamic allocation of resources for learning. Dopamine activity may transition between two patterns: (1) phasic responses to cues and rewards, and (2) ramping activity arising as the agent approaches the reward. Phasic excitation has been explained by prediction errors generated by experimentally inserted cues. However, when and why dopamine activity transitions between the two patterns remain unknown. By parsimoniously modifying a standard temporal difference (TD) learning model to accommodate a mixed presentation of both experimental and environmental stimuli, we simulated dopamine transitions and compared them with experimental data from four different studies. The results suggested that dopamine transitions from ramping to phasic patterns as the agent focuses its resources on a small number of reward-predicting stimuli, thus leading to task dimensionality reduction. The opposite occurs when the agent re-distributes its resources to adapt to environmental changes, resulting in task dimensionality expansion. This research elucidates the role of dopamine in a broader context, providing a potential explanation for the diverse repertoire of dopamine activity that cannot be explained solely by prediction error.