Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T.

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Dynamic Flexibility in Striatal-Cortical Circuits Supports Reinforcement Learning.

Complex learned behaviors must involve the integrated action of distributed brain circuits. Although the contributions of individual regions to learning have been extensively investigated, much less is known about how distributed brain networks orchestrate their activity over the course of learning. To address this gap, we used fMRI combined with tools from dynamic network neuroscience to obtain time-resolved descriptions of network coordination during reinforcement learning in humans. We found that learning to associate visual cues with reward involves dynamic changes in network coupling between the striatum and distributed brain regions, including visual, orbitofrontal, and ventromedial prefrontal cortex (n = 22; 13 females). Moreover, we found that this flexibility in striatal network coupling correlates with participants' learning rate and inverse temperature, two parameters derived from reinforcement learning models. Finally, we found that episodic learning, measured separately in the same participants at the same time, was related to dynamic connectivity in distinct brain networks. These results suggest that dynamic changes in striatal-centered networks provide a mechanism for information integration during reinforcement learning.SIGNIFICANCE STATEMENT Learning from the outcomes of actions, referred to as reinforcement learning, is an essential part of life. The roles of individual brain regions in reinforcement learning have been well characterized in terms of updating values for actions or cues. Missing from this account, however, is an understanding of how different brain areas interact during learning to integrate sensory and value information. Here we characterize flexible striatal-cortical network dynamics that relate to reinforcement learning behavior.

Brain atrophy can introduce age-related differences in BOLD response.

Use of functional magnetic resonance imaging (fMRI) in studies of aging is often hampered by uncertainty about age-related differences in the amplitude and timing of the blood oxygenation level dependent (BOLD) response (i.e., hemodynamic impulse response function (HRF)). Such uncertainty introduces a significant challenge in the interpretation of the fMRI results. Even though this issue has been extensively investigated in the field of neuroimaging, there is currently no consensus about the existence and potential sources of age-related hemodynamic alterations. Using an event-related fMRI experiment with two robust and well-studied stimuli (visual and auditory), we detected a significant age-related difference in the amplitude of response to auditory stimulus. Accounting for brain atrophy by circumventing spatial normalization and processing the data in subjects' native space eliminated these observed differences. In addition, we simulated fMRI data using age differences in brain morphology while controlling HRF shape. Analyzing these simulated fMRI data using standard image processing resulted in differences in HRF amplitude, which were eliminated when the data were analyzed in subjects' native space. Our results indicate that age-related atrophy introduces inaccuracy in co-registration to standard space, which subsequently appears as attenuation in BOLD response amplitude. Our finding could explain some of the existing contradictory reports regarding age-related differences in the fMRI BOLD responses. Hum Brain Mapp 38:3402-3414, 2017. © 2017 Wiley Periodicals, Inc.

Transfer of learning relates to intrinsic connectivity between hippocampus, ventromedial prefrontal cortex, and large-scale networks.

An important aspect of adaptive learning is the ability to flexibly use past experiences to guide new decisions. When facing a new decision, some people automatically leverage previously learned associations, while others do not. This variability in transfer of learning across individuals has been demonstrated repeatedly and has important implications for understanding adaptive behavior, yet the source of these individual differences remains poorly understood. In particular, it is unknown why such variability in transfer emerges even among homogeneous groups of young healthy participants who do not vary on other learning-related measures. Here we hypothesized that individual differences in the transfer of learning could be related to relatively stable differences in intrinsic brain connectivity, which could constrain how individuals learn. To test this, we obtained a behavioral measure of memory-based transfer outside of the scanner and on a separate day acquired resting-state functional MRI images in 42 participants. We then analyzed connectivity across independent component analysis-derived brain networks during rest, and tested whether intrinsic connectivity in learning-related networks was associated with transfer. We found that individual differences in transfer were related to intrinsic connectivity between the hippocampus and the ventromedial prefrontal cortex, and between these regions and large-scale functional brain networks. Together, the findings demonstrate a novel role for intrinsic brain dynamics in flexible learning-guided behavior, both within a set of functionally specific regions known to be important for learning, as well as between these regions and the default and frontoparietal networks, which are thought to serve more general cognitive functions.

Age-related differences in working memory deficits during nicotine withdrawal.

Nicotine withdrawal is associated with subtle working memory deficits that predict subsequent relapse. We examined the neural substrates underlying these processes in treatment-seeking smokers, and explored the moderating influence of age on abstinence-induced alterations in brain activity and performance. Sixty-three smokers participated in two blood oxygen level-dependent (BOLD) functional magnetic resonance imaging scans while performing a visual N-back task on two separate occasions: smoking as usual and after 24 hours of biochemically confirmed abstinence (order counterbalanced). Abstinence (versus smoking) led to reduced accuracy, slower median correct response time and reduced BOLD signal change in the three a priori regions of interest: medial frontal/cingulate gyrus and right and left dorsolateral prefrontal cortex. Significant age × session effects were found for BOLD signal change in all three regions, as well as for withdrawal and craving; for all measures, abstinence effects were attenuated in smokers aged ≥50 years compared with those <50 years old. These results suggest that abstinence effects on neurocognitive function may be more pronounced for younger smokers, and may indicate a new avenue for research exploring mechanisms underlying age differences in smoking cessation success.

Two routes to value-based decisions in Parkinson's disease: differentiating incremental reinforcement learning from episodic memory.

Patients with Parkinson's disease are impaired at incremental reward-based learning. It is typically assumed that this impairment reflects a loss of striatal dopamine. However, many open questions remain about the nature of reward-based learning deficits in Parkinson's. Recent studies have found that a combination of different cognitive and computational strategies contribute even to simple reward-based learning tasks, suggesting a possible role for episodic memory. These findings raise critical questions about how incremental learning and episodic memory interact to support learning from past experience and what their relative contributions are to impaired decision-making in Parkinson's disease. Here we addressed these questions by asking patients with Parkinson's disease (n=26) both on and off their dopamine replacement medication and age- and education-matched healthy controls (n=26) to complete a task designed to isolate the contributions of incremental learning and episodic memory to reward-based learning and decision-making. We found that Parkinson's patients performed as well as healthy controls when using episodic memory, but were impaired at incremental reward-based learning. Dopamine replacement medication remediated this deficit while enhancing subsequent episodic memory for the value of motivationally relevant stimuli. These results demonstrate that Parkinson's patients are impaired at learning about reward from trial-and-error when episodic memory is properly controlled for, and that learning based on the value of single experiences remains intact in patients with Parkinson's disease.

An improved framework for confound regression and filtering for control of motion artifact in the preprocessing of resting-state functional connectivity data.

Several recent reports in large, independent samples have demonstrated the influence of motion artifact on resting-state functional connectivity MRI (rsfc-MRI). Standard rsfc-MRI preprocessing typically includes regression of confounding signals and band-pass filtering. However, substantial heterogeneity exists in how these techniques are implemented across studies, and no prior study has examined the effect of differing approaches for the control of motion-induced artifacts. To better understand how in-scanner head motion affects rsfc-MRI data, we describe the spatial, temporal, and spectral characteristics of motion artifacts in a sample of 348 adolescents. Analyses utilize a novel approach for describing head motion on a voxelwise basis. Next, we systematically evaluate the efficacy of a range of confound regression and filtering techniques for the control of motion-induced artifacts. Results reveal that the effectiveness of preprocessing procedures on the control of motion is heterogeneous, and that improved preprocessing provides a substantial benefit beyond typical procedures. These results demonstrate that the effect of motion on rsfc-MRI can be substantially attenuated through improved preprocessing procedures, but not completely removed.

Machine learning within the Parkinson's progression markers initiative: Review of the current state of affairs.

The Parkinson's Progression Markers Initiative (PPMI) has collected more than a decade's worth of longitudinal and multi-modal data from patients, healthy controls, and at-risk individuals, including imaging, clinical, cognitive, and 'omics' biospecimens. Such a rich dataset presents unprecedented opportunities for biomarker discovery, patient subtyping, and prognostic prediction, but it also poses challenges that may require the development of novel methodological approaches to solve. In this review, we provide an overview of the application of machine learning methods to analyzing data from the PPMI cohort. We find that there is significant variability in the types of data, models, and validation procedures used across studies, and that much of what makes the PPMI data set unique (multi-modal and longitudinal observations) remains underutilized in most machine learning studies. We review each of these dimensions in detail and provide recommendations for future machine learning work using data from the PPMI cohort.

Being right is its own reward: load and performance related ventral striatum activation to correct responses during a working memory task in youth.

The ventral striatum (VS) is a critical brain region for reinforcement learning and motivation. Intrinsically motivated subjects performing challenging cognitive tasks engage reinforcement circuitry including VS even in the absence of external feedback or incentives. However, little is known about how such VS responses develop with age, relate to task performance, and are influenced by task difficulty. Here we used fMRI to examine VS activation to correct and incorrect responses during a standard n-back working memory task in a large sample (n=304) of healthy children, adolescents and young adults aged 8-22. We found that bilateral VS activates more strongly to correct than incorrect responses, and that the VS response scales with the difficulty of the working memory task. Furthermore, VS response was correlated with discrimination performance during the task, and the magnitude of VS response peaked in mid-adolescence. These findings provide evidence for scalable intrinsic reinforcement signals during standard cognitive tasks, and suggest a novel link between motivation and cognition during adolescent development.

A meta-analytic investigation of neurocognitive deficits in bipolar illness: profile and effects of clinical state.

A meta-analysis of neuropsychological studies of patients with bipolar disorder comprised of 42 studies of 1,197 patients in euthymia, 13 studies consisting of 314 patients in a manic/mixed phase of illness, and 5 studies of 96 patients in a depressed state. Cohen d values were calculated for each study as the mean difference between patient and control group score on each neuropsychological measure, expressed in pooled SD units. For euthymia, results revealed impairment across all neuropsychological domains, with d values in the moderate-to-large range (d = .5 - .8) for the vast majority of measures. There was evidence of large effect-size impairment on measures of verbal learning (d = .81), and delayed verbal and nonverbal memory (d = .80 - .92), while effect-size impairment on measures of visuospatial function was small-to-moderate (d

OPTIMAL SIGNAL RECOVERY FROM INTERLEAVED FMRI DATA.

Due to the nature of fMRI acquisition protocols, slices in the plane of acquisition are not acquired simultaneously or sequentially, and therefore are temporally misaligned with each other. Slice timing correction (STC) is a critical preprocessing step that corrects for this misalignment. STC is applied in all major software packages. To date, little effort has gone towards assessing the optimal method of STC. In this study, we examine the most popular methods of STC, and propose a new optimal method based on the fundamental properties of sampling theory. We evaluate this method using 20 simulated fMRI data and demonstrate the utility of STC in general as well as the superiority of the proposed method in comparison to existing ones.