Vicarious reinforcement learning signals when instructing others.

Reinforcement learning (RL) theory posits that learning is driven by discrepancies between the predicted and actual outcomes of actions (prediction errors [PEs]). In social environments, learning is often guided by similar RL mechanisms. For example, teachers monitor the actions of students and provide feedback to them. This feedback evokes PEs in students that guide their learning. We report the first study that investigates the neural mechanisms that underpin RL signals in the brain of a teacher. Neurons in the anterior cingulate cortex (ACC) signal PEs when learning from the outcomes of one's own actions but also signal information when outcomes are received by others. Does a teacher's ACC signal PEs when monitoring a student's learning? Using fMRI, we studied brain activity in human subjects (teachers) as they taught a confederate (student) action-outcome associations by providing positive or negative feedback. We examined activity time-locked to the students' responses, when teachers infer student predictions and know actual outcomes. We fitted a RL-based computational model to the behavior of the student to characterize their learning, and examined whether a teacher's ACC signals when a student's predictions are wrong. In line with our hypothesis, activity in the teacher's ACC covaried with the PE values in the model. Additionally, activity in the teacher's insula and ventromedial prefrontal cortex covaried with the predicted value according to the student. Our findings highlight that the ACC signals PEs vicariously for others' erroneous predictions, when monitoring and instructing their learning. These results suggest that RL mechanisms, processed vicariously, may underpin and facilitate teaching behaviors.

The anterior cingulate gyrus signals the net value of others' rewards.

Evaluating the costs and benefits of our own choices is central to most forms of decision-making and its mechanisms in the brain are becoming increasingly well understood. To interact successfully in social environments, it is also essential to monitor the rewards that others receive. Previous studies in nonhuman primates have found neurons in the anterior cingulate cortex (ACC) that signal the net value (benefit minus cost) of rewards that will be received oneself and also neurons that signal when a reward will be received by someone else. However, little is understood about the way in which the human brain engages in cost-benefit analyses during social interactions. Does the ACC signal the net value (the benefits minus the costs) of rewards that others will receive? Here, using fMRI, we examined activity time locked to cues that signaled the anticipated reward magnitude (benefit) to be gained and the level of effort (cost) to be incurred either by a subject themselves or by a social confederate. We investigated whether activity in the ACC covaries with the net value of rewards that someone else will receive when that person is required to exert effort for the reward. We show that, although activation in the sulcus of the ACC signaled the costs on all trials, gyral ACC (ACC(g)) activity varied parametrically only with the net value of rewards gained by others. These results suggest that the ACC(g) plays an important role in signaling cost-benefit information by signaling the value of others' rewards during social interactions.

Consensus paper: the cerebellum's role in movement and cognition.

While the cerebellum's role in motor function is well recognized, the nature of its concurrent role in cognitive function remains considerably less clear. The current consensus paper gathers diverse views on a variety of important roles played by the cerebellum across a range of cognitive and emotional functions. This paper considers the cerebellum in relation to neurocognitive development, language function, working memory, executive function, and the development of cerebellar internal control models and reflects upon some of the ways in which better understanding the cerebellum's status as a "supervised learning machine" can enrich our ability to understand human function and adaptation. As all contributors agree that the cerebellum plays a role in cognition, there is also an agreement that this conclusion remains highly inferential. Many conclusions about the role of the cerebellum in cognition originate from applying known information about cerebellar contributions to the coordination and quality of movement. These inferences are based on the uniformity of the cerebellum's compositional infrastructure and its apparent modular organization. There is considerable support for this view, based upon observations of patients with pathology within the cerebellum.

Cerebellum and cognition: evidence for the encoding of higher order rules.

Converging anatomical and functional evidence suggests that the cerebellum processes both motor and nonmotor information originating from the primary motor cortex and prefrontal cortex, respectively. However, it has not been established whether the cerebellum only processes prefrontal information where rules specify actions or whether the cerebellum processes any form of prefrontal information no matter how abstract. Using functional magnetic resonance imaging, we distinguish between two competing hypotheses: (1) activity within prefrontal-projecting cerebellar lobules (Crus I and II) will only be evoked by rules that specify action (i.e. first-order rules; arbitrary S-R mappings) and (2) activity will be evoked in these lobules by both first-order rules and second-order rules that govern the application of lower order rules. The results showed that prefrontal-projecting cerebellar lobules Crus I and II were commonly activated by processing both first- and second-order rules. We demonstrate for the first time that cerebellar circuits engage both first- and second-order rules and in doing so show that the cerebellum can contribute to cognitive control independent of motor control.

Cerebellar plasticity and the automation of first-order rules.

Theories of corticocerebellar function propose roles for the cerebellum in automating motor control, a process thought to depend on plasticity in cerebellar circuits that exchange information with the motor cortex. Little is known, however, about automating behaviors beyond the motor domain. The present study tested the hypothesis that cerebellar plasticity also subserves the development of automaticity in behavior based on low-order rules. Human subjects were required to learn two sets of first-order rules in which visual stimuli of different shapes each arbitrarily instructed a particular finger movement. We used event-related functional magnetic resonance imaging to scan subjects while these response rules became increasingly automatic with practice, as assessed with a dual-task procedure. We found that the amplitude of the blood oxygenation level-dependent signal gradually decreased as a function of practice, as responses became increasingly automatic, and that this effect was greater for a set of rules that became automatic rapidly compared with a second set, which became automatic more slowly. These trial-by-trial activity changes occurred in Crus I of cerebellar cortical lobule HVIIA, in which neurons exchange information with the prefrontal cortex rather than the motor cortex. Activity in Crus I was time locked specifically to the processing of these rules, rather than to subsequent actions. The results support the hypothesis that decreases in cerebellar cortical activity underlie the automation of behavior, whether related to motor control and motor cortex or to response rules and prefrontal cortex.

Frontal lobe and posterior parietal contributions to the cortico-cerebellar system.

Our growing understanding of how cerebral cortical areas communicate with the cerebellum in primates has enriched our understanding of the data that cerebellar circuits can access, and the neocortical areas that cerebellar activity can influence. The cerebellum is part of some large-scale networks involving several parts of the neocortex including association areas in the frontal lobe and the posterior parietal cortex that are known for their contributions to higher cognitive function. Understanding their connections with the cerebellum informs the debates around the role of the cerebellum in higher cognitive functions because they provide mechanisms through which association areas and the cerebellum can influence each others' operations. In recent years, evidence from connectional anatomy and human neuroimaging have comprehensively overturned the view that the cerebellum contributes only to motor control. The aim of this review is to examine our changing perspectives on the nature of cortico-cerebellar anatomy and the ways in which it continues to shape our views on its contributions to function. The review considers the anatomical connectivity of the cerebellar cortex with frontal lobe areas and the posterior parietal cortex. It will first focus on the anatomical organisation of these circuits in non-human primates before discussing new findings about this system in the human brain. It has been suggested that in non-human primates "although there is a modest input from medial prefrontal cortex, there is very little or none from the more lateral prefrontal areas" [33]. This review discusses anatomical investigations that challenge this claim. It also attempts to dispel the misconception that prefrontal projections to the cerebellum are from areas concerned only with the kinematic control of eye movements. Finally, I argue that our revised understanding of anatomy compels us to reconsider conventional views of how these systems operate in the human brain.

Distinct and overlapping functional zones in the cerebellum defined by resting state functional connectivity.

The cerebellum processes information from functionally diverse regions of the cerebral cortex. Cerebellar input and output nuclei have connections with prefrontal, parietal, and sensory cortex as well as motor and premotor cortex. However, the topography of the connections between the cerebellar and cerebral cortices remains largely unmapped, as it is relatively unamenable to anatomical methods. We used resting-state functional magnetic resonance imaging to define subregions within the cerebellar cortex based on their functional connectivity with the cerebral cortex. We mapped resting-state functional connectivity voxel-wise across the cerebellar cortex, for cerebral-cortical masks covering prefrontal, motor, somatosensory, posterior parietal, visual, and auditory cortices. We found that the cerebellum can be divided into at least 2 zones: 1) a primary sensorimotor zone (Lobules V, VI, and VIII), which contains overlapping functional connectivity maps for domain-specific motor, somatosensory, visual, and auditory cortices; and 2) a supramodal zone (Lobules VIIa, Crus I, and II), which contains overlapping functional connectivity maps for prefrontal and posterior-parietal cortex. The cortical connectivity of the supramodal zone was driven by regions of frontal and parietal cortex which are not directly involved in sensory or motor processing, including dorsolateral prefrontal cortex and the frontal pole, and the inferior parietal lobule.

A probabilistic MR atlas of the human cerebellum.

The functional organization of the cerebellum is reflected in large part by the unique afferent and efferent connectivity of the individual cerebellar lobules. This functional diversity on a relatively small spatial scale makes accurate localization methods for human functional imaging and anatomical patient-based research indispensable. Here we present a probabilistic atlas of the cerebellar lobules in the anatomical space defined by the MNI152 template. We separately masked the lobules on T1-weighted MRI scans (1 mm isotropic resolution) of 20 healthy young participants (10 male, 10 female, average age 23.7 yrs). These cerebella were then aligned to the standard or non-linear version of the whole-brain MNI152 template using a number of commonly used normalization algorithms, or to a previously published cerebellum-only template (Diedrichsen, J., 2006. A spatially unbiased atlas template of the human cerebellum. NeuroImage 33, 127-138.). The resulting average overlap was higher for the cerebellum-only template than for any of the whole-brain normalization methods. The probabilistic maps allow for the valid assignment of functional activations to specific cerebellar lobules, while providing a quantitative measure of the uncertainty of such assignments. Furthermore, maximum probability maps derived from these atlases can be used to define regions of interest (ROIs) in functional neuroimaging and neuroanatomical research. The atlas, made freely available online, is compatible with a number of widely used analysis packages.

A system in the human brain for predicting the actions of others.

The ability to attribute mental states to others, and therefore to predict others' behavior, is particularly advanced in humans. A controversial but untested idea is that this is achieved by simulating the other person's mental processes in one's own mind. If this is the case, then the same neural systems activated by a mental function should re-activate when one thinks about that function performed by another. Here, using functional magnetic resonance imaging (fMRI), we tested whether the neural processes involved in preparing one's own actions are also used for predicting the future actions of others. We provide compelling evidence that areas within the action control system of the human brain are indeed activated when predicting others' actions, but a different action sub-system is activated when preparing one's own actions.

Exploring brain connectivity: a new frontier in systems neuroscience. Functional Brain Connectivity, 4-6 April 2002, Dusseldorf, Germany.

Functional Brain Connectivity, held on 4-6 April 2002, Dusseldorf, Germany.

New approaches for exploring anatomical and functional connectivity in the human brain.

Information processing in the primate brain is based on the complementary principles of modular and distributed information processing. The former emphasizes the specialization of functions within different brain areas. The latter emphasizes the massively parallel nature of brain networks and the fact that function also emerges from the flow of information between brain areas. The localization of function to specific brain areas ("functional segregation") is the commonest approach to investigating function; however, an emerging, complementary approach ("functional integration") describes function in terms of the information flow across networks of areas. Here, we highlight recent advances in neuroimaging methodology that have made it possible to investigate the anatomical architecture of networks in the living human brain with diffusion tensor imaging (DTI). We also highlight recent thinking on the ways in which functional imaging can be used to characterize information transmission across networks in the human brain (functional and effective connectivity).

Distinct portions of anterior cingulate cortex and medial prefrontal cortex are activated by reward processing in separable phases of decision-making cognition.

BACKGROUND: Choosing between actions associated with uncertain rewards and punishments is mediated by neural circuitry encompassing the orbitofrontal cortex, anterior cingulate cortex (ACC), and striatum; however, the precise conditions under which these different components are activated during decision-making cognition remain uncertain. METHODS: Fourteen healthy volunteers completed an event-based functional magnetic resonance imaging protocol to investigate blood-oxygenation-level-dependent (BOLD) responses during independently modeled phases of choice cognition. In the "decision phase," participants decided which of two simultaneous visually presented gambles they wished to play for monetary reward. The gambles differed in their magnitude of gains, magnitude of losses, and the probabilities with which these outcomes were delivered. In the "outcome phase," the result of each choice was indicated on the visual display. RESULTS: In the decision phase, choices involving large gains were associated with increased BOLD responses in the pregenual ACC, paracingulate, and right posterior orbitolateral cortex compared with choices involving small gains. In the outcome phase, good outcomes were associated with increased BOLD responses in the posterior orbitomedial cortex, subcallosal ACC, and ventral striatum compared with negative outcomes. There was only limited overlap between reward-related activity in ACC and orbitofrontal cortex during the decision and outcome phases. CONCLUSIONS: Neural activity within the medial and lateral orbitofrontal cortex, pregenual ACC, and striatum mediate distinct representations of reward-related information that are deployed at different stages during a decision-making episode.

Automatic and controlled processing in the corticocerebellar system.

During learning, performance changes often involve a transition from controlled processing in which performance is flexible and responsive to ongoing error feedback, but effortful and slow, to a state in which processing becomes swift and automatic. In this state, performance is unencumbered by the requirement to process feedback, but its insensitivity to feedback reduces its flexibility. Many properties of automatic processing are similar to those that one would expect of forward models, and many have suggested that these may be instantiated in cerebellar circuitry. Since hierarchically organized frontal lobe areas can both send and receive commands, I discuss the possibility that they can act both as controllers and controlled objects and that their behaviors can be independently modeled by forward models in cerebellar circuits. Since areas of the prefrontal cortex contribute to this hierarchically organized system and send outputs to the cerebellar cortex, I suggest that the cerebellum is likely to contribute to the automation of cognitive skills, and to the formation of habitual behavior which is resistant to error feedback. An important prerequisite to these ideas is that cerebellar circuitry should have access to higher order error feedback that signals the success or failure of cognitive processing. I have discussed the pathways through which such feedback could arrive via the inferior olive and the dopamine system. Cerebellar outputs inhibit both the inferior olive and the dopamine system. It is possible that learned representations in the cerebellum use this as a mechanism to suppress the processing of feedback in other parts of the nervous system. Thus, cerebellar processes that control automatic performance may be completed without triggering the engagement of controlled processes by prefrontal mechanisms.

Cognitive deficits from a cerebellar tumour: a historical case report from Luria's Laboratory.

In 1964 an original case report from A.R. Luria's Laboratory of Neuropsychology was published in Cortex, being one of the first to draw a link between cerebellum and cognition, by highlighting the manifestation of 'pseudo-frontal' symptoms resulting from a cerebellar tumour. The findings of Luria and his team seem more consistent with modern views about cerebellar interactions with the frontal lobe and its contributions to behaviour than the views prevalent at the time of publication. The paper was originally submitted in Russian, and translated into Italian for its publication by Cortex. However, Cortex did not preserve the original manuscript in Russian. With the passage of time, and available only to the Italian readership, this case report inevitably fell into obscurity. Hence, we present a translation in English based on the published Italian version of the manuscript and discuss it in the context of Luria's general thinking about information processing in the brain and our current understanding of cortico-cerebellar system. The publication of this article gives readers an opportunity to consider the substantial influence of Soviet neuropsychology on the field internationally under Luria's leadership in the 1960s. It also shows that time is the best judge of ones scientific endeavours, and what may seem implausible today might prove to be valid and worthy of exploration tomorrow.

The primate cortico-cerebellar system: anatomy and function.

Evidence has been accumulating that the primate cerebellum contributes not only to motor control, but also to higher 'cognitive' function. However, there is no consensus about how the cerebellum processes such information. The answer to this puzzle can be found in the nature of cerebellar connections to areas of the cerebral cortex (particularly the prefrontal cortex) and in the uniformity of its intrinsic cellular organization, which implies uniformity in information processing regardless of the area of origin in the cerebral cortex. With this in mind, the relatively well-developed models of how the cerebellum processes information from the motor cortex might be extended to explain how it could also process information from the prefrontal cortex.

The evolution of prefrontal inputs to the cortico-pontine system: diffusion imaging evidence from Macaque monkeys and humans.

The cortico-ponto-cerebellar system is one of the largest projection systems in the primate brain, but in the human brain the nature of the information processing in this system remains elusive. Determining the areas of the cerebral cortex which contribute projections to this system will allow us to better understand information processing within it. Information from the cerebral cortex is conveyed to the cerebellum by topographically arranged fibres in the cerebral peduncle - an important fibre system in which all cortical outputs spatially converge on their way to the cerebellum via the pontine nuclei. Little is known of their anatomical organization in the human brain. New in vivo diffusion imaging and probabilistic tractography methods now offer a way in which input tracts in the cerebral peduncle can be characterized in detail. Here we use these methods to contrast their organization in humans and macaque monkeys. We confirm the dominant contribution of the cortical motor areas to the macaque monkey cerebral peduncle. However, we also present novel anatomical evidence for a relatively large prefrontal contribution to the human cortico-ponto-cerebellar system in the cerebral peduncle. These findings suggest the selective evolution of prefrontal inputs to the human cortico-ponto-cerebellar system.

Variability in fMRI: a re-examination of inter-session differences.

We revisit a previous study on inter-session variability (McGonigle et al. [2000]: Neuroimage 11:708-734), showing that contrary to one popular interpretation of the original article, inter-session variability is not necessarily high. We also highlight how evaluating variability based on thresholded single-session images alone can be misleading. Finally, we show that the use of different first-level preprocessing, time-series statistics, and registration analysis methodologies can give significantly different inter-session analysis results.

Learning of sequences of finger movements and timing: frontal lobe and action-oriented representation.

Motor sequence learning involves learning of a sequence of effectors with which to execute a series of movements and learning of a sequence of timings at which to execute the movements. In this study, we have segregated the neural correlates of the two learning mechanisms. Moreover, we have found an interaction between the two learning mechanisms in the frontal areas, which we claim as suggesting action-oriented coding in the frontal lobe. We used positron emission tomography and compared three learning conditions with a visuo-motor control condition. In two learning conditions, the subjects learned either a sequence of finger movements with random timing or a sequence of timing with random use of fingers. In the third condition the subjects learned to execute a sequence of specific finger movements at specific timing; we argue that it was only in this condition that the motor sequence was coded as an action-oriented representation. By looking for condition by session interactions (learning vs. control conditions over sessions), we have removed nonspecific time effects and identified areas that showed a learning-related increment of activation during learning. Learning of a finger sequence was associated with an increment of activation in the right intraparietal sulcus region and medial parietal cortex, whereas learning of a timing sequence was associated with an increment of activation in the lateral cerebellum, suggesting separate mechanisms for learning effector and temporal sequences. The left intraparietal sulcus region showed an increment of activation in learning of both finger and timing sequences, suggesting an overlap between the two learning mechanisms. We also found that the mid-dorsolateral prefrontal cortex, together with the medial and lateral premotor areas, became increasingly active when subjects learned a sequence that specified both fingers and timing, that is, when subjects were able to prepare specific motor action. These areas were not active when subjects learned a sequence that specified fingers or timing alone, that is, when subjects were still dependent on external stimuli as to the timing or fingers with which to execute the movements. Frontal areas may integrate the effector and temporal information of a motor sequence and implement an action-oriented representation so as to perform a motor sequence accurately and quickly. We also found that the mid-dorsolateral prefrontal cortex was distinguished from the ventrolateral prefrontal cortex and anterior fronto-polar cortex, which showed sustained activity throughout learning sessions and did not show either an increment or decrement of activation.