Credit assignment in movement-dependent reinforcement learning.

When a person fails to obtain an expected reward from an object in the environment, they face a credit assignment problem: Did the absence of reward reflect an extrinsic property of the environment or an intrinsic error in motor execution? To explore this problem, we modified a popular decision-making task used in studies of reinforcement learning, the two-armed bandit task. We compared a version in which choices were indicated by key presses, the standard response in such tasks, to a version in which the choices were indicated by reaching movements, which affords execution failures. In the key press condition, participants exhibited a strong risk aversion bias; strikingly, this bias reversed in the reaching condition. This result can be explained by a reinforcement model wherein movement errors influence decision-making, either by gating reward prediction errors or by modifying an implicit representation of motor competence. Two further experiments support the gating hypothesis. First, we used a condition in which we provided visual cues indicative of movement errors but informed the participants that trial outcomes were independent of their actual movements. The main result was replicated, indicating that the gating process is independent of participants' explicit sense of control. Second, individuals with cerebellar degeneration failed to modulate their behavior between the key press and reach conditions, providing converging evidence of an implicit influence of movement error signals on reinforcement learning. These results provide a mechanistically tractable solution to the credit assignment problem.

Case Report of Dual-Site Neurostimulation and Chronic Recording of Cortico-Striatal Circuitry in a Patient With Treatment Refractory Obsessive Compulsive Disorder.

Psychiatric disorders are increasingly understood as dysfunctions of hyper- or hypoconnectivity in distributed brain circuits. A prototypical example is obsessive compulsive disorder (OCD), which has been repeatedly linked to hyper-connectivity of cortico-striatal-thalamo-cortical (CSTC) loops. Deep brain stimulation (DBS) and lesions of CSTC structures have shown promise for treating both OCD and related disorders involving over-expression of automatic/habitual behaviors. Physiologically, we propose that this CSTC hyper-connectivity may be reflected in high synchrony of neural firing between loop structures, which could be measured as coherent oscillations in the local field potential (LFP). Here we report the results from the pilot patient in an Early Feasibility study (https://clinicaltrials.gov/ct2/show/NCT03184454) in which we use the Medtronic Activa PC+ S device to simultaneously record and stimulate in the supplementary motor area (SMA) and ventral capsule/ventral striatum (VC/VS). We hypothesized that frequency-mismatched stimulation should disrupt coherence and reduce compulsive symptoms. The patient reported subjective improvement in OCD symptoms and showed evidence of improved cognitive control with the addition of cortical stimulation, but these changes were not reflected in primary rating scales specific to OCD and depression, or during blinded cortical stimulation. This subjective improvement was correlated with increased SMA and VC/VS coherence in the alpha, beta, and gamma bands, signals which persisted after correcting for stimulation artifacts. We discuss the implications of this research, and propose future directions for research in network modulation in OCD and more broadly across psychiatric disorders.