Reinforcement Motor Learning After Cerebellar Damage Is Related to State Estimation.

Recent work showed that individuals with cerebellar degeneration could leverage intact reinforcement learning (RL) to alter their movement. However, there was marked inter-individual variability in learning, and the factors underlying it were unclear. Cerebellum-dependent sensory prediction may contribute to RL in motor contexts by enhancing body state estimates, which are necessary to solve the credit-assignment problem. The objective of this study was to test the relationship between the predictive component of state estimation and RL in individuals with cerebellar degeneration. Individuals with cerebellar degeneration and neurotypical control participants completed two tasks: an RL task that required them to alter the angle of reaching movements and a state estimation task that tested the somatosensory perception of active and passive movement. The state estimation task permitted the calculation of the active benefit shown by each participant, which is thought to reflect the cerebellum-dependent predictive component of state estimation. We found that the cerebellar and control groups showed similar magnitudes of learning with reinforcement and active benefit on average, but there was substantial variability across individuals. Using multiple regression, we assessed potential predictors of RL. Our analysis included active benefit, somatosensory acuity, clinical ataxia severity, movement variability, movement speed, and age. We found a significant relationship in which greater active benefit predicted better learning with reinforcement in the cerebellar, but not the control group. No other variables showed significant relationships with learning. Overall, our results support the hypothesis that the integrity of sensory prediction is a strong predictor of RL after cerebellar damage.

Different sensory information is used for state estimation when stationary or moving.

Accurate estimation of limb state is necessary for movement planning and execution. State estimation requires both feedforward and feedback information; here we focus on the latter. Prior literature has shown that integrating visual and proprioceptive feedback improve estimates of static limb position. However, differences in visual and proprioceptive feedback delays suggest that multisensory integration could be disadvantageous when the limb is moving. To investigate multisensory integration in different passive movement contexts, we compared the degree of interference created by discrepant visual or proprioceptive feedback when estimating the position of the limb either statically at the end of the movement or dynamically at movement midpoint. In the static context, we observed idiosyncratic interference: discrepant proprioceptive feedback significantly interfered with reports of the visual target location, leading to a bias of the reported position toward the proprioceptive cue. In the dynamic context, no interference was seen: participants could ignore sensory feedback from one modality and accurately reproduce the motion indicated by the other modality. We modeled feedback-based state estimation by updating the longstanding maximum likelihood estimation model of multisensory integration to account for sensory delays. Consistent with our behavioral results, the model showed that the benefit of multisensory integration was largely lost when the limb was passively moving. Together, these findings suggest that the sensory feedback used to compute a state estimate differs depending on whether the limb is stationary or moving. While the former may tend toward multimodal integration, the latter is more likely to be based on feedback from a single sensory modality.

Cerebellar Dysfunction in Multiple Sclerosis: Considerations for Research and Rehabilitation Therapy.

Introduction. Cerebellar pathology is common among persons with multiple sclerosis (PwMS). The cerebellum is well recognized for its role in motor control and motor learning and cerebellar pathology in multiple sclerosis is associated with enhanced motor impairment and disability progression. The Problem. To mitigate motor disability progression, PwMS are commonly prescribed exercise and task-specific rehabilitation training. Yet, whether cerebellar dysfunction differentially affects rehabilitation outcomes in this population remains unknown. Furthermore, we lack rehabilitation interventions targeting cerebellar dysfunction. The Solution. Here, we summarize the current understanding of the impact of cerebellar dysfunction on motor control, motor training, and rehabilitation in persons with multiple sclerosis. Recommendations. Additionally, we highlight critical knowledge gaps and propose that these guide future research studying cerebellar dysfunction in persons with multiple sclerosis.

Aberrant activity in an intact residual muscle is associated with phantom limb pain in above-knee amputees.

Many individuals who undergo limb amputation experience persistent phantom limb pain (PLP), but the underlying mechanisms of PLP are unknown. The traditional hypothesis was that PLP resulted from maladaptive plasticity in sensorimotor cortex that degrades the neural representation of the missing limb. However, a recent study of individuals with upper limb amputations has shown that PLP is correlated with aberrant electromyographic (EMG) activity in residual muscles, posited to reflect a retargeting of efferent projections from a preserved representation of a missing limb. Here, we assessed EMG activity in a residual thigh muscle (vastus lateralis, VL) in patients with transfemoral amputations during cyclical movements of a phantom foot. VL activity on the amputated side was compared to that recorded on patients' intact side while they moved both the phantom and intact feet synchronously. VL activity in the patient group was also compared to a sample of control participants with no amputation. We show that phantom foot movement is associated with greater VL activity in the amputated leg than that seen in the intact leg as well as that exhibited by controls. The magnitude of residual VL activity was also positively related to ratings of PLP. These results show that phantom limb movement is associated with aberrant activity in a residual muscle after lower-limb amputation and provide evidence of a positive relationship between this activity and phantom limb pain.NEW & NOTEWORTHY This study is the first to assess residual muscle activity during movement of a phantom limb in individuals with lower limb amputations. We find that phantom foot movement is associated with aberrant recruitment of a residual thigh muscle and that this aberrant activity is related to phantom limb pain.

Mechanisms of Human Motor Learning Do Not Function Independently.

Human motor learning is governed by a suite of interacting mechanisms each one of which modifies behavior in distinct ways and rely on different neural circuits. In recent years, much attention has been given to one type of motor learning, called motor adaptation. Here, the field has generally focused on the interactions of three mechanisms: sensory prediction error SPE-driven, explicit (strategy-based), and reinforcement learning. Studies of these mechanisms have largely treated them as modular, aiming to model how the outputs of each are combined in the production of overt behavior. However, when examined closely the results of some studies also suggest the existence of additional interactions between the sub-components of each learning mechanism. In this perspective, we propose that these sub-component interactions represent a critical means through which different motor learning mechanisms are combined to produce movement; understanding such interactions is critical to advancing our knowledge of how humans learn new behaviors. We review current literature studying interactions between SPE-driven, explicit, and reinforcement mechanisms of motor learning. We then present evidence of sub-component interactions between SPE-driven and reinforcement learning as well as between SPE-driven and explicit learning from studies of people with cerebellar degeneration. Finally, we discuss the implications of interactions between learning mechanism sub-components for future research in human motor learning.

Reinforcement Signaling Can Be Used to Reduce Elements of Cerebellar Reaching Ataxia.

Damage to the cerebellum causes a disabling movement disorder called ataxia, which is characterized by poorly coordinated movement. Arm ataxia causes dysmetria (over- or under-shooting of targets) with many corrective movements. As a result, people with cerebellar damage exhibit reaching movements with highly irregular and prolonged movement paths. Cerebellar patients are also impaired in error-based motor learning, which may impede rehabilitation interventions. However, we have recently shown that cerebellar patients can learn a simple reaching task using a binary reinforcement paradigm, in which feedback is based on participants' mean performance. Here, we present a pilot study that examined whether patients with cerebellar damage can use this reinforcement training to learn a more complex motor task-to decrease the path length of their reaches. We compared binary reinforcement training to a control condition of massed practice without reinforcement feedback. In both conditions, participants made target-directed reaches in 3-dimensional space while vision of their movement was occluded. In the reinforcement training condition, reaches with a path length below participants' mean were reinforced with an auditory stimulus at reach endpoint. We found that patients were able to use reinforcement signaling to significantly reduce their reach paths. Massed practice produced no systematic change in patients' reach performance. Overall, our results suggest that binary reinforcement training can improve reaching movements in patients with cerebellar damage and the benefit cannot be attributed solely to repetition or reduced visual control.

Interactions between motor exploration and reinforcement learning.

Motor exploration, a trial-and-error process in search for better motor outcomes, is known to serve a critical role in motor learning. This is particularly relevant during reinforcement learning, where actions leading to a successful outcome are reinforced while unsuccessful actions are avoided. Although early on motor exploration is beneficial to finding the correct solution, maintaining high levels of exploration later in the learning process might be deleterious. Whether and how the level of exploration changes over the course of reinforcement learning, however, remains poorly understood. Here we evaluated temporal changes in motor exploration while healthy participants learned a reinforcement-based motor task. We defined exploration as the magnitude of trial-to-trial change in movements as a function of whether the preceding trial resulted in success or failure. Participants were required to find the optimal finger-pointing direction using binary feedback of success or failure. We found that the magnitude of exploration gradually increased over time when participants were learning the task. Conversely, exploration remained low in participants who were unable to correctly adjust their pointing direction. Interestingly, exploration remained elevated when participants underwent a second training session, which was associated with faster relearning. These results indicate that the motor system may flexibly upregulate the extent of exploration during reinforcement learning as if acquiring a specific strategy to facilitate subsequent learning. Also, our findings showed that exploration affects reinforcement learning and vice versa, indicating an interactive relationship between them. Reinforcement-based tasks could be used as primers to increase exploratory behavior leading to more efficient subsequent learning.NEW & NOTEWORTHY Motor exploration, the ability to search for the correct actions, is critical to learning motor skills. Despite this, whether and how the level of exploration changes over the course of training remains poorly understood. We showed that exploration increased and remained high throughout training of a reinforcement-based motor task. Interestingly, elevated exploration persisted and facilitated subsequent learning. These results suggest that the motor system upregulates exploration as if learning a strategy to facilitate subsequent learning.

The cerebellum as a movement sensor.

In this article, we review a broad range of studies of cerebellar function and dysfunction and interpret them within the framework that the cerebellum acts as part of a mechanism of predictive control. We describe studies that span human behaviour and consider the motor and sensory impairments that result from cerebellar damage. We conclude that a parsimonious explanation of cerebellar function is as a predictor of the sensory outcomes of movement. However, future studies are needed to more rigorously test this hypothesis and determine how the cerebellar circuit might perform this type of computation.

Increasing Motor Noise Impairs Reinforcement Learning in Healthy Individuals.

Motor variability from exploration is crucial for reinforcement learning as it allows the nervous system to find new task solutions. However, motor variability from noise can be detrimental to learning and may underlie slowed reinforcement learning performance observed in individuals with cerebellar damage. Here we examine whether artificially increasing noise in healthy individuals slows reinforcement learning in a manner similar to that seen in patients with cerebellar damage. Participants used binary reinforcement to learn to rotate their reach angle in a series of directions. By comparing task performance between conditions with different levels of added noise, we show that adding a high level of noise-matched to a group of patients with cerebellar damage-slows learning. In additional experiments, we show that the detrimental effect of noise may lie in reinforcing incorrect behavior, rather than not reinforcing correct behavior. By comparing performance between healthy participants with added noise and a group of patients with cerebellar damage, we found that added noise does not slow the learning of the control group to the same degree observed in the patient group. Using a mechanistic model, we show that added noise in the present study matched patients' motor noise and total learning. However, increased exploration in the control group relative to the group with cerebellar damage supports faster learning. Our results suggest that motor noise slows reinforcement learning by impairing the mapping of reward to the correct action and that this may underlie deficits induced by cerebellar damage.

The cerebellum contributes to proprioception during motion.

Proprioception, the sense of limb position and motion, is essential for generating accurate movements. Limb position sense has typically been studied under static conditions (i.e., the fixed position of a limb in space), with less known about dynamic position sense (i.e., limb position during movement). Here we investigated how a person's estimate of hand position varies when using spatial or temporal information to judge the unseen hand's location during reaching. We assessed the acuity of dynamic position sense in two directions, orthogonal to hand movement, which only requires spatial information, and in line with hand movement, which has both spatial and temporal components. Our results showed that people have better proprioceptive acuity in the orthogonal condition where only spatial information is used. We then assessed whether cerebellar damage impairs proprioceptive acuity in both tasks during passive and active movement. Cerebellar patients showed reduced acuity in both tasks and in both movement conditions relative to age-matched controls. However, patients' deficits were most apparent when judgments of active movement relied on temporal information. Furthermore, both cerebellar patient and control performance correlated with the trial-to-trial variability of their active movements: subjects are worse at the proprioceptive tasks when movements are variable. Our results suggest that, during active movements, proprioceptive acuity may be reliant on the motor system's ability to predict motor output. Therefore, the resultant proprioceptive deficits occurring after cerebellar damage may be related to a more general impairment in movement prediction.NEW & NOTEWORTHY We assessed limb position sense during movement in patients with cerebellar damage and found deficits in proprioceptive acuity during both passive and active movement. The effect of cerebellar damage was most apparent when individuals relied on both timing and spatial information during active movement. Thus proprioceptive acuity during active movements may be reliant on the motor system's ability to predict motor output.

Proprioceptive Localization Deficits in People With Cerebellar Damage.

It has been hypothesized that an important function of the cerebellum is predicting the state of the body during movement. Yet, the extent of cerebellar involvement in perception of limb state (i.e., proprioception, specifically limb position sense) has yet to be determined. Here, we investigated whether patients with cerebellar damage have deficits when trying to locate their hand in space (i.e., proprioceptive localization), which is highly important for everyday movements. By comparing performance during passive robot-controlled and active self-made multi-joint movements, we were able to determine that some cerebellar patients show improved precision during active movement (i.e., active benefit), comparable to controls, whereas other patients have reduced active benefit. Importantly, the differences in patient performance are not explained by patient diagnosis or clinical ratings of impairment. Furthermore, a subsequent experiment confirmed that active deficits in proprioceptive localization occur during both single-joint and multi-joint movements. As such, it is unlikely that localization deficits can be explained by the multi-joint coordination deficits occurring after cerebellar damage. Our results suggest that cerebellar damage may cause varied impairments to different elements of proprioceptive sense. It follows that proprioceptive localization should be adequately accounted for in clinical testing and rehabilitation of people with cerebellar damage.

Effective reinforcement learning following cerebellar damage requires a balance between exploration and motor noise.

Reinforcement and error-based processes are essential for motor learning, with the cerebellum thought to be required only for the error-based mechanism. Here we examined learning and retention of a reaching skill under both processes. Control subjects learned similarly from reinforcement and error-based feedback, but showed much better retention under reinforcement. To apply reinforcement to cerebellar patients, we developed a closed-loop reinforcement schedule in which task difficulty was controlled based on recent performance. This schedule produced substantial learning in cerebellar patients and controls. Cerebellar patients varied in their learning under reinforcement but fully retained what was learned. In contrast, they showed complete lack of retention in error-based learning. We developed a mechanistic model of the reinforcement task and found that learning depended on a balance between exploration variability and motor noise. While the cerebellar and control groups had similar exploration variability, the patients had greater motor noise and hence learned less. Our results suggest that cerebellar damage indirectly impairs reinforcement learning by increasing motor noise, but does not interfere with the reinforcement mechanism itself. Therefore, reinforcement can be used to learn and retain novel skills, but optimal reinforcement learning requires a balance between exploration variability and motor noise.

Cerebellar damage impairs internal predictions for sensory and motor function.

The cerebellum is connected to cerebral areas that subserve a range of sensory and motor functions. In this review, we summarize new literature demonstrating deficits in visual perception, proprioception, motor control, and motor learning performance following cerebellar damage. In particular, we highlight novel results that together suggest a general role of the cerebellum in estimating and predicting movement dynamics of the body and environmental stimuli. These findings agree with the hypothesized role of the cerebellum in the generation and calibration of predictive models for a variety of functions.

Continuous theta-burst stimulation to primary motor cortex reveals asymmetric compensation for sensory attenuation in bimanual repetitive force production.

Studies of fingertip force production have shown that self-produced forces are perceived as weaker than externally generated forces. This is due to mechanisms of sensory reafference where the comparison between predicted and actual sensory feedback results in attenuated perceptions of self-generated forces. Without an external reference to calibrate attenuated performance judgments, a compensatory overproduction of force is exhibited. It remains unclear whether the force overproduction seen in the absence of visual reference stimuli differs when forces are produced bimanually. We studied performance of two versions of a bimanual sequential force production task compared with each hand performing the task unimanually. When the task goal was shared, force series produced by each hand in bimanual conditions were found to be uncorrelated. When the bimanual task required each hand to reach a target force level, we found asymmetries in the degree of force overproduction between the hands following visual feedback removal. Unilateral continuous theta-burst stimulation of the left primary motor cortex yielded a selective reduction of force overproduction in the hand contralateral to stimulation by disrupting sensory reafference processes. While variability was lower in bimanual trials when the task goal was shared, this influence of hand condition disappeared when the target force level was to be reached by each hand simultaneously. Our findings strengthen the notion that force control in bimanual action is less tightly coupled than other mechanisms of bimanual motor control and show that this effector specificity may be extended to the processing and compensation for mechanisms of sensory reafference.

Sensory attenuation of self-produced feedback: the Lombard effect revisited.

The Lombard effect describes the automatic and involuntary increase in vocal intensity that speakers exhibit in a noisy environment. Previous studies of the Lombard effect have typically focused on the relationship between speaking and hearing. Automatic and involuntary increases in motor output have also been noted in studies of finger force production, an effect attributed to mechanisms of sensory attenuation. The present study tested the hypothesis that sensory attenuation mechanisms also underlie expression of the Lombard effect. Participants vocalized phonemes in time with a metronome, while auditory and visual feedback of their performance were manipulated or removed during the course of the trial. We demonstrate that providing a visual reference to calibrate somatosensory-based judgments of current vocal intensity resulted in reduced expression of the Lombard effect. Our results suggest that sensory attenuation effects typically seen in fingertip force production play an important role in the control of speech volume.

Continuous theta-burst stimulation to primary motor cortex reduces the overproduction of forces following removal of visual feedback.

Forward models, generated from the efference copies of motor commands, are thought to monitor the accuracy of ongoing movement. By comparing predicted with actual afferent information, forward models also aid in the differentiation of self-produced movements from externally generated ones. Many have proposed that a consequence of this comparison is attenuation of the predicted component of incoming sensory signals. Previous work from our laboratory has shown that following the removal of an external visual reference, discrete sequential forces exceed target values. Forces produced at the fingertip were perceived as weaker, which lead to a systematic, compensatory over-production of the magnitudes required. The relatively new repetitive TMS protocol of continuous theta-burst stimulation (cTBS) has been shown to reliably depress cortical excitability for a period following stimulation. If sensory attenuation mechanisms were responsible for the overproduction of forces found in our previous results, we hypothesized that reducing cortical excitability of M1 through application of cTBS would induce discrepancy between the efference copy generated and motor output produced. As a result, we expected the overproduction of forces following visual feedback removal would be reduced after receiving cTBS. Participants produced series of pinch grip forces in time to a metronome and to visually specified force magnitudes. Visual feedback of force output was extinguished 10s into experimental trials and participants performed continued responses for the remaining 10s. Results confirmed our hypothesis. Mean peak force and constant error were greater and more positive in the absence of visual feedback regardless of stimulation condition; however, the magnitude of increase was significantly reduced following cTBS compared with baseline and sham conditions. Variability was not differentially affected by stimulation condition, increasing only with removal of visual feedback contingent upon the larger forces produced in these trials. Our findings provide further evidence to support the idea that TBS may differentially affect motor output and efference copy generation.

Repetitive finger force production in predictable environments.

Previous research suggests that removal of visual feedback of force output in a sequential force production task results in a continuous escalation of the force magnitudes produced. Central predictive mechanisms involving reafference result in self-generated forces being perceived as weaker, thus leading to a systematic over-production of force. While this effect has been widely examined with respect to sensation of self-produced stimuli, its role in the sequential production of discrete forces has not been extensively studied. The objective of the present experiment was to further investigate the force escalation effect by examining the sequential force production task in two predictable environments where force targets progressively changed between 8 and 16 N. In one environment target force magnitudes continually increased at a predictable rate, while in the second target force magnitudes decreased at the same rate. Eight healthy participants produced a series of repetitive pinch grip forces in time to a metronome at a frequency of 2 Hz. Visual feedback of force output was removed 10 s in to the trial, with participants performing continued responses for the remaining 20 s. Following removal of visual feedback participants rapidly accrued errors in force magnitude, but slopes of the peak force series did not differ from those of the target series. Errors in magnitude were bi-directional, showing dependence on the increasing or decreasing environment in which the task was performed. The observed escalation and de-escalation effect can both be attributed to central predictive mechanisms and the force salience effect in reafference.

Timing and visual feedback constraints on repetitive finger force production.

While much is known about sequential effects in motor timing, less is understood about whether movement parameters such as force show sequential dependencies. In this study, we examined the effect of timing constraints on repetitive unimanual force production sequences. Ten healthy participants produced a series of pinch grip forces in time to a metronome and to visually specified force amplitudes. Either visual feedback of force produced or the auditory metronome removed 10 s into the experimental trial, with participants performing continued responses for the remaining 20 s. In the continuation trials, a negative lag-1 autocorrelation in the inter-response intervals (IRIs) was observed as is commonly seen in motor timing tasks. However, removal of visual feedback resulted in a systematic increase in mean force output through the course of the trial, resulting in positive lag-1 autocorrelation values. An interaction was found between mean IRI and peak force (PF) magnitude, with greater force variability seen for the larger intervals. However, the imposition of dual force and timing constraints had no effect either on the underlying variability of the PF or on the IRIs. The results are discussed in the context of force and time being independently specified components of a generalized motor program.