Reinforcement learning establishes a minimal metacognitive process to monitor and control motor learning performance.

Humans and animals develop learning-to-learn strategies throughout their lives to accelerate learning. One theory suggests that this is achieved by a metacognitive process of controlling and monitoring learning. Although such learning-to-learn is also observed in motor learning, the metacognitive aspect of learning regulation has not been considered in classical theories of motor learning. Here, we formulated a minimal mechanism of this process as reinforcement learning of motor learning properties, which regulates a policy for memory update in response to sensory prediction error while monitoring its performance. This theory was confirmed in human motor learning experiments, in which the subjective sense of learning-outcome association determined the direction of up- and down-regulation of both learning speed and memory retention. Thus, it provides a simple, unifying account for variations in learning speeds, where the reinforcement learning mechanism monitors and controls the motor learning process.

Transcranial magnetic stimulation on the dorsal premotor cortex facilitates human visuomotor adaptation.

The premotor cortex is traditionally known to be involved in motor preparation and execution. More recently, evidence from neuroscience research shows that the dorsal premotor cortex (PMd) is also involved in sensory error-based motor adaptation and that invasive brain stimulation on PMd can attenuate adaptation in monkeys. The present study examines if adaptation can be modulated noninvasively in humans. Twenty-five healthy volunteers participated in a motor task in which rapid arm-reaching movements were made to hit a target, whereas the online cursor feedback about the hand position was visually rotated, inducing sensory error that drove motor adaptation. Transcranial magnetic stimulation (TMS) was delivered to PMd just before experiencing a sensory error, as in the previous study on monkeys. The degree of motor adaptation was measured as the change in the hand direction in response to the experienced error. TMS was found to increase adaptation compared with control conditions. Interestingly, the direction of modulation was opposite to the previous study on monkeys, which might originate from different methods and parameters of stimulation. The effect was also location-specific and was not a mere artifact of applying TMS because the facilitatory modulation occurred when stimulating PMd but not when stimulating the ventral premotor cortex, which was known for different roles and networks from PMd. Since noninvasive neuromodulation is a promising tool for research and clinical practice, the present study demonstrates that PMd is a feasible target region of neuromodulation to understand human motor adaptation and improve motor rehabilitation.

Reward prediction errors, not sensory prediction errors, play a major role in model selection in human reinforcement learning.

Model-based reinforcement learning enables an agent to learn in variable environments and tasks by optimizing its actions based on the predicted states and outcomes. This mechanism has also been considered in the brain. However, exactly how the brain selects an appropriate model for confronting environments has remained unclear. Here, we investigated the model selection algorithm in the human brain during a reinforcement learning task. One primary theory of model selection in the brain is based on sensory prediction errors. Here, we compared this theory with an alternative possibility of internal model selection with reward prediction errors. To compare these two theories, we devised a switching experiment from a first-order Markov decision process to a second-order Markov decision process that provides either reward- or sensory prediction error regarding environmental change. We tested two representative computational models driven by different prediction errors. One is the sensory prediction-error-driven Bayesian algorithm, which has been discussed as a representative internal model selection algorithm in the animal reinforcement learning task. The other is the reward-prediction-error-driven policy gradient algorithm. We compared the simulation results of these two computational models with human reinforcement learning behaviors. The model fitting result supports that the policy gradient algorithm is preferable to the Bayesian algorithm. This suggests that the human brain employs the reward prediction error to select an appropriate internal model in the reinforcement learning task.

Computational role of exploration noise in error-based de novo motor learning.

The redundancy inherent to the human body is a central problem that must be solved by the brain when acquiring new motor skills. The problem of redundancy becomes particularly critical when learning a new motor policy from scratch in a novel environment and task (i.e., de novo learning). It has been proposed that motor variability could be leveraged to explore and identify task-potent motor commands, and recent results indicated a possible role of motor exploration in error-based motor learning, including in de novo learning tasks. However, the precise computational mechanisms underlying this role remain poorly understood. A new controller in a de novo motor task can potentially be learned by first using motor exploration to learn a sensitivity derivative, which can transform observed task errors into motor corrections, enabling the error-based learning of the controller. Although this approach has been discussed, the computational properties of exploration and how this mechanism can explain recent reports of motor exploration in error-based de-novo learning have not been thoroughly examined. Here, we used this approach to simulate the tasks used in several recent studies of human motor learning tasks in which motor exploration was observed, and replicating their main results. Analyses of the proposed learning mechanism using equations and simulations suggested that exploring the entire motor command space leads to the training of an efficient sensitivity derivative, enabling rapid learning of the controller, in visuomotor adaptation and de novo tasks. The successful replication of previous experimental results elucidated the role of motor exploration in motor learning.

Accounting for the valley of recovery during post-stroke rehabilitation training via a model-based analysis of macaque manual dexterity.

Background: True recovery, in which a stroke patient regains the same precise motor skills observed in prestroke conditions, is the fundamental goal of rehabilitation training. However, a transient drop in task performance during rehabilitation training after stroke, observed in human clinical outcome as well as in both macaque and squirrel monkey retrieval data, might prevent smooth transitions during recovery. This drop, i.e., recovery valley, often occurs during the transition from compensatory skill to precision skill. Here, we sought computational mechanisms behind such transitions and recovery. Analogous to motor skill learning, we considered that the motor recovery process is composed of spontaneous recovery and training-induced recovery. Specifically, we hypothesized that the interaction of these multiple skill update processes might determine profiles of the recovery valley. Methods: A computational model of motor recovery was developed based on a state-space model of motor learning that incorporates a retention factor and interaction terms for training-induced recovery and spontaneous recovery. The model was fit to previously reported macaque motor recovery data where the monkey practiced precision grip skills after a lesion in the sensorimotor area in the cortex. Multiple computational models and the effects of each parameter were examined by model comparisons based on information criteria and sensitivity analyses of each parameter. Result: Both training-induced and spontaneous recoveries were necessary to explain the behavioral data. Since these two factors contributed following logarithmic function, the training-induced recovery were effective only after spontaneous biological recovery had developed. In the training-induced recovery component, the practice of the compensation also contributed to recovery of the precision grip skill as if there is a significant generalization effect of learning between these two skills. In addition, a retention factor was critical to explain the recovery profiles. Conclusions: We found that spontaneous recovery, training-induced recovery, retention factors, and interaction terms are crucial to explain recovery and recovery valley profiles. This simulation-based examination of the model parameters provides suggestions for effective rehabilitation methods to prevent the recovery valley, such as plasticity-promoting medications, brain stimulation, and robotic rehabilitation technologies.

The role of motor memory dynamics in structuring bodily self-consciousness.

Bodily self-consciousness has been considered a sensorimotor root of self-consciousness. If this is the case, how does sensorimotor memory, which is important for the prediction of sensory consequences of volitional actions, influence awareness of bodily self-consciousness? This question is essential for understanding the effective acquisition and recovery of self-consciousness following its impairment, but it has remained unexamined. Here, we investigated how body ownership and agency recovered following body schema distortion in a virtual reality environment along with two kinds of motor memories: memories that were rapidly updated and memories that were gradually updated. We found that, although agency and body ownership recovered in parallel, the recovery of body ownership was predicted by fast memories and that of agency was predicted by slow memories. Thus, the bodily self was represented in multiple motor memories with different dynamics. This finding demystifies the controversy about the causal relationship between body ownership and agency.

Raman lidar for remote sensing of oil in water.

We describe a portable Raman lidar system that can remotely detect oil leakages in water. The system has been developed based on a frequency-doubled, Q-switched Nd:YAG laser, operated at 532 nm with a receiver telescope equipped with some filters and photomultipliers. Stand-off detection of oil is achieved in a 6-m-long water tank, which allowed us to considerably increase the survey capability of subsea infrastructures, including both the range observation and target identification.

Greater reliance on proprioceptive information during a reaching task with perspective manipulation among children with autism spectrum disorders.

Difficulties with visual perspective-taking among individuals with autism spectrum disorders remain poorly understood. Many studies have presumed that first-person visual input can be mentally transformed to a third-person perspective during visual perspective-taking tasks; however, existing research has not fully revealed the computational strategy used by those with autism spectrum disorders for taking another person's perspective. In this study, we designed a novel approach to test a strategy using the opposite-directional effect among children with autism spectrum disorders. This effect refers to how a third-person perspective as a visual input alters a cognitive process. We directly manipulated participants' visual perspective by placing a camera at different positions; participants could watch themselves from a third-person perspective during a reaching task with no endpoint feedback. During a baseline task, endpoint bias (with endpoint feedback but no visual transformation) did not differ significantly between groups. However, the endpoint was affected by extrinsic coordinate information in the control group relative to the autism spectrum disorders group when the visual perspective was transformed. These results indicate an increased reliance on proprioception during the reaching task with perspective manipulation in the autism spectrum disorders group.

Task-relevant and task-irrelevant variability causally shape error-based motor learning.

Recent studies of motor learning show dissociable roles of reward- and sensory-prediction errors in updating motor commands by using typical adaptation paradigms where force or visual perturbations are imposed on hand movements. Such classic adaptation paradigms ignore a problem of redundancy inherently embedded in the motor pathways where the central nervous system has to find a unique solution in the high-dimensional motor command space. Computationally, a possible way of solving such a redundancy problem is exploring and updating motor commands based on the learned knowledge of the structures of both the motor pathways and the tasks. However, the effects of task-irrelevant motor command exploration in structure learning and its effects on reward-based and error-based learning have yet to be examined. Here, we used a redundant motor task where participants manipulated a cursor on a monitor screen with their hand gesture movements and then analyzed single-trial motor learning by fitting models consisting of reward-based and error-based learning contributions. We found that the error-based learning rate positively correlated with both task-relevant and task-irrelevant variability, likely reflecting the effect of motor exploration in structure learning. Further modeling results show that the effects of both task-relevant and task-irrelevant variability are simultaneous, and not mediated by one another. In contrast, the reward-based learning rate correlated with neither task-relevant nor task-irrelevant variability. Thus, although not having a direct influence on the task outcome, exploration in the task-irrelevant space late in training has a significant effect on the learning of a task structure used for error-based learning. This suggests that motor exploration, in both task-relevant and task-irrelevant spaces, has an essential role in error-based motor learning in a redundant motor mechanism.

Experience of After-Effect of Memory Update Reduces Sensitivity to Errors During Sensory-Motor Adaptation Task.

Motor learning is the process of updating motor commands in response to a trajectory error induced by a perturbation to the body or vision. The brain has a great capability to accelerate learning by increasing the sensitivity of the memory update to the perceived trajectory errors. Conventional theory suggests that the statistics of perturbations or the statistics of the experienced errors induced by the external perturbations determine the learning speeds. However, the potential effect of another type of error perception, a self-generated error as a result of motor command updates (i.e., an aftereffect), on the learning speeds has not been examined yet. In this study, we dissociated the two kinds of errors by controlling the perception of the aftereffect using a channel-force environment. One group experienced errors due to the aftereffect of the learning process, while the other did not. We found that the participants who perceived the aftereffect of the memory updates exhibited a significant decrease in error-sensitivity, whereas the participants who did not perceive the aftereffect did not show an increase or decrease in error-sensitivity. This suggests that the perception of the aftereffect of learning attenuated updating the motor commands from the perceived errors. Thus, both self-generated and externally induced errors may modulate learning speeds.

Neuroanatomical Basis of Individuality in Muscle Tuning Function: Neural Correlates of Muscle Tuning.

In a conventional view of motor control, the human brain might employ an optimization principle that leads a stereotypical motor behavior which we observe as an averaged behavioral data over subjects. In this scenario, the inter-individual motor variability is considered as an observation noise. Here, we challenged this view. We considered a motor control task where the human participants manipulated arm force by coordinating shoulder and elbow torques and investigated the muscle-tuning function that represents how the brain distributed the ideal joint torques to multiple muscles. In the experimental data, we observed large inter-individual variability in the profile of a muscle-tuning function. This contradicts with a well-established optimization theory that is based on minimization of muscle energy consumption and minimization of motor variability. We then hypothesized the inter-subject differences in the structure of the motor cortical areas might be the source of the across-subjects variability of the motor behavior. This was supported by a voxel-based morphometry analysis of magnetic resonance imaging; The inter-individual variability of the muscle tuning profile was correlated with that of the gray matter volume in the premotor cortex which is ipsilateral to the used arm (i.e., right hemisphere for the right arm). This study suggests that motor individuality may originate from inter-individual variation in the cortical structure.

Sensory prediction errors, not performance errors, update memories in visuomotor adaptation.

Sensory prediction errors are thought to update memories in motor adaptation, but the role of performance errors is largely unknown. To dissociate these errors, we manipulated visual feedback during fast shooting movements under visuomotor rotation. Participants were instructed to strategically correct for performance errors by shooting to a neighboring target in one of four conditions: following the movement onset, the main target, the neighboring target, both targets, or none of the targets disappeared. Participants in all conditions experienced a drift away from the main target following the strategy. In conditions where the main target was shown, participants often tried to minimize performance errors caused by the drift by generating corrective movements. However, despite differences in performance during adaptation between conditions, memory decay in a delayed washout block was indistinguishable between conditions. Our results thus suggest that, in visuomotor adaptation, sensory predictions errors, but not performance errors, update the slow, temporally stable, component of motor memory.

Alpha Phase Synchronization of Parietal Areas Reflects Switch-Specific Activity During Mental Rotation: An EEG Study.

Action selection is typically influenced by the history of previously selected actions (the immediate motor history), which is apparent when a selected action is switched from a previously selected one to a new one. This history dependency of the action selection is even observable during a mental hand rotation task. Thus, we hypothesized that the history-dependent interaction of actions might share the same neural mechanisms among different types of action switching tasks. An alternative hypothesis is that the history dependency of the mental hand rotation task might involve a distinctive neural mechanism from the general action selection tasks so that the reported observation with the mental hand rotation task in the previously published literature might lack generality. To refute this possibility, we compared neural activity during action switching in the mental hand rotation with the general action switching task which is triggered by a simple visual stimulus. In the experiment, to focus on temporal changes in whole brain oscillatory activity, we recorded electroencephalographic (EEG) signals while 25 healthy subjects performed the two tasks. For analysis, we examined functional connectivity reflected in EEG phase synchronization and analyzed temporal changes in brain activity when subjects switched from a previously selected action to a new action. Using a clustering-based method to identify functional connectivity reflected in time-varying phase synchronization, we identified alpha-power inter-parietal synchronization that appears only during switching of the selected action, regardless of the hand laterality in the presented image. Moreover, the current study revealed that for both tasks the extent of this alpha-power inter-parietal synchronization was altered by the history of the selected actions. These findings suggest that alpha-power inter-parietal synchronization is engaged as a form of switching-specific functional connectivity, and that switching-related activity is independent of the task paradigm.

Diverse coordinate frames on sensorimotor areas in visuomotor transformation.

The visuomotor transformation during a goal-directed movement may involve a coordinate transformation from visual 'extrinsic' to muscle-like 'intrinsic' coordinate frames, which might be processed via a multilayer network architecture composed of neural basis functions. This theory suggests that the postural change during a goal-directed movement task alters activity patterns of the neurons in the intermediate layer of the visuomotor transformation that recieves both visual and proprioceptive inputs, and thus influence the multi-voxel pattern of the blood oxygenation level dependent signal. Using a recently developed multi-voxel pattern decoding method, we found extrinsic, intrinsic and intermediate coordinate frames along the visuomotor cortical pathways during a visuomotor control task. The presented results support the hypothesis that, in human, the extrinsic coordinate frame was transformed to the muscle-like frame over the dorsal pathway from the posterior parietal cortex and the dorsal premotor cortex to the primary motor cortex.

Transient increase in systemic interferences in the superficial layer and its influence on event-related motor tasks: a functional near-infrared spectroscopy study.

Functional near-infrared spectroscopy (fNIRS) is a widely utilized neuroimaging tool in fundamental neuroscience research and clinical investigation. Previous research has revealed that task-evoked systemic artifacts mainly originating from the superficial-tissue may preclude the identification of cerebral activation during a given task. We examined the influence of such artifacts on event-related brain activity during a brisk squeezing movement. We estimated task-evoked superficial-tissue hemodynamics from short source­detector distance channels (15 mm) by applying principal component analysis. The estimated superficial-tissue hemodynamics exhibited temporal profiles similar to the canonical cerebral hemodynamic model. Importantly, this task-evoked profile was also observed in data from a block design motor experiment, suggesting a transient increase in superficial-tissue hemodynamics occurs following motor behavior, irrespective of task design. We also confirmed that estimation of event-related cerebral hemodynamics was improved by a simple superficial-tissue hemodynamic artifact removal process using 15-mm short distance channels, compared to the results when no artifact removal was applied. Thus, our results elucidate task design-independent characteristics of superficial-tissue hemodynamics and highlight the need for the application of superficial-tissue hemodynamic artifact removal methods when analyzing fNIRS data obtained during event-related motor tasks.

Computational motor control as a window to understanding schizophrenia.

In addition to mental disorders such as attention, emotion, delusions, hallucinations, and difficulties in social skills, the patients with schizophrenia exhibits significant abnormality in sensorimotor perception and control. To seek a neurobiological cause of the heterogeneous symptoms in schizophrenia, we focused on the impaired inference mechanism of the self-agency of the schizophrenia's brain where the sensory outcome generated by the self-initiated action was misattributed to the other agent's action. By developing a novel computational model of agency experience using a Bayesian decision making framework, we united the computational mechanisms of agency and motor control via internal model: a model for one to predict the sensory consequence of action. Our theory based on optimal feedback control with Kalman filtering successfully predicted a variety of schizophrenia's motor abnormalities assuming a deformed internal model. To discuss the plausibility of these model predictions, we reviewed literature that might support these predictions. We further proposed some experiments that potentially examine the proposed model of schizophrenia. Our approach in investigating a problem of mind by projecting it on the coordinates system of the embodiment effectively shed light on a central neuropathology of this disease that had been latent behind the observed behaviors.

The cerebro-cerebellum: Could it be loci of forward models?

It is widely accepted that the cerebellum acquires and maintain internal models for motor control. An internal model simulates mapping between a set of causes and effects. There are two candidates of cerebellar internal models, forward models and inverse models. A forward model transforms a motor command into a prediction of the sensory consequences of a movement. In contrast, an inverse model inverts the information flow of the forward model. Despite the clearly different formulations of the two internal models, it is still controversial whether the cerebro-cerebellum, the phylogenetically newer part of the cerebellum, provides inverse models or forward models for voluntary limb movements or other higher brain functions. In this article, we review physiological and morphological evidence that suggests the existence in the cerebro-cerebellum of a forward model for limb movement. We will also discuss how the characteristic input-output organization of the cerebro-cerebellum may contribute to forward models for non-motor higher brain functions.

Reward-dependent modulation of movement variability.

Movement variability is often considered an unwanted byproduct of a noisy nervous system. However, variability can signal a form of implicit exploration, indicating that the nervous system is intentionally varying the motor commands in search of actions that yield the greatest success. Here, we investigated the role of the human basal ganglia in controlling reward-dependent motor variability as measured by trial-to-trial changes in performance during a reaching task. We designed an experiment in which the only performance feedback was success or failure and quantified how reach variability was modulated as a function of the probability of reward. In healthy controls, reach variability increased as the probability of reward decreased. Control of variability depended on the history of past rewards, with the largest trial-to-trial changes occurring immediately after an unrewarded trial. In contrast, in participants with Parkinson's disease, a known example of basal ganglia dysfunction, reward was a poor modulator of variability; that is, the patients showed an impaired ability to increase variability in response to decreases in the probability of reward. This was despite the fact that, after rewarded trials, reach variability in the patients was comparable to healthy controls. In summary, we found that movement variability is partially a form of exploration driven by the recent history of rewards. When the function of the human basal ganglia is compromised, the reward-dependent control of movement variability is impaired, particularly affecting the ability to increase variability after unsuccessful outcomes.

The influence of an uncertain force environment on reshaping trial-to-trial motor variability.

Motor memory is updated to generate ideal movements in a novel environment. When the environment changes every trial randomly, how does the brain incorporate this uncertainty into motor memory? To investigate how the brain adapts to an uncertain environment, we considered a reach adaptation protocol where individuals practiced moving in a force field where a noise was injected. After they had adapted, we measured the trial-to-trial variability in the temporal profiles of the produced hand force. We found that the motor variability was significantly magnified by the adaptation to the random force field. Temporal profiles of the motor variance were significantly dissociable between two different types of random force fields experienced. A model-based analysis suggests that the variability is generated by noise in the gains of the internal model. It further suggests that the trial-to-trial motor variability magnified by the adaptation in a random force field is generated by the uncertainty of the internal model formed in the brain as a result of the adaptation.

Temporal changes of beta rhythms and rotation-related negativity reflect switches in motor imagery.

While motor imagery has been known as a powerful tool for neuro-rehabilitation in stroke patients, whether this technique is also effective for other brain disorders is unclear. For instance, patients with Parkinson's disease or attention-deficit hyperactivity disorder who are impaired at real motor switching may benefit therapeutically from training that consists of switching their imagined motor movements, and eventually recover from the dysfunction. However, despite its importance little is known about exactly how switching mental images of one's actions is processed in the brain. Therefore, we set out to clarify this issue by measuring brain activity reflected in electroencephalograms as subjects switched an imagined hand rotation from one hand to the other during a motor-imagery task. By comparing electroencephalogram signals from repeated mental imaging of hand movements, we found a switch-specific decrease in the beta-band activity in parietal and frontal regions around 0.6 s after stimulus presentation. Further, we found rotation-related negativity in the parietal cortex at the same time as the decreased beta-band power. These results suggest that the parietal area is dynamically involved in the switching of imagined hand motion, and that frontal areas may have an important role in inhibiting mental imagery of the deselected hand's motion.