Reinforcement-based processes actively regulate motor exploration along redundant solution manifolds.

From a baby's babbling to a songbird practising a new tune, exploration is critical to motor learning. A hallmark of exploration is the emergence of random walk behaviour along solution manifolds, where successive motor actions are not independent but rather become serially dependent. Such exploratory random walk behaviour is ubiquitous across species' neural firing, gait patterns and reaching behaviour. The past work has suggested that exploratory random walk behaviour arises from an accumulation of movement variability and a lack of error-based corrections. Here, we test a fundamentally different idea-that reinforcement-based processes regulate random walk behaviour to promote continual motor exploration to maximize success. Across three human reaching experiments, we manipulated the size of both the visually displayed target and an unseen reward zone, as well as the probability of reinforcement feedback. Our empirical and modelling results parsimoniously support the notion that exploratory random walk behaviour emerges by utilizing knowledge of movement variability to update intended reach aim towards recently reinforced motor actions. This mechanism leads to active and continuous exploration of the solution manifold, currently thought by prominent theories to arise passively. The ability to continually explore muscle, joint and task redundant solution manifolds is beneficial while acting in uncertain environments, during motor development or when recovering from a neurological disorder to discover and learn new motor actions.

Implicit reward-based motor learning.

Binary feedback, providing information solely about task success or failure, can be sufficient to drive motor learning. While binary feedback can induce explicit adjustments in movement strategy, it remains unclear if this type of feedback also induces implicit learning. We examined this question in a center-out reaching task by gradually moving an invisible reward zone away from a visual target to a final rotation of 7.5° or 25° in a between-group design. Participants received binary feedback, indicating if the movement intersected the reward zone. By the end of the training, both groups modified their reach angle by about 95% of the rotation. We quantified implicit learning by measuring performance in a subsequent no-feedback aftereffect phase, in which participants were told to forgo any adopted movement strategies and reach directly to the visual target. The results showed a small, but robust (2-3°) aftereffect in both groups, highlighting that binary feedback elicits implicit learning. Notably, for both groups, reaches to two flanking generalization targets were biased in the same direction as the aftereffect. This pattern is at odds with the hypothesis that implicit learning is a form of use-dependent learning. Rather, the results suggest that binary feedback can be sufficient to recalibrate a sensorimotor map.

Implicit reward-based motor learning.

Binary feedback, providing information solely about task success or failure, can be sufficient to drive motor learning. While binary feedback can induce explicit adjustments in movement strategy, it remains unclear if this type of feedback also induce implicit learning. We examined this question in a center-out reaching task by gradually moving an invisible reward zone away from a visual target to a final rotation of 7.5° or 25° in a between-group design. Participants received binary feedback, indicating if the movement intersected the reward zone. By the end of the training, both groups modified their reach angle by about 95% of the rotation. We quantified implicit learning by measuring performance in a subsequent no-feedback aftereffect phase, in which participants were told to forgo any adopted movement strategies and reach directly to the visual target. The results showed a small, but robust (2-3°) aftereffect in both groups, highlighting that binary feedback elicits implicit learning. Notably, for both groups, reaches to two flanking generalization targets were biased in the same direction as the aftereffect. This pattern is at odds with the hypothesis that implicit learning is a form of use-dependent learning. Rather, the results suggest that binary feedback can be sufficient to recalibrate a sensorimotor map.

Failure induces task-irrelevant exploration during a stencil task.

During reward-based motor tasks, performance failure leads to an increase in movement variability along task-relevant dimensions. These increases in movement variability are indicative of exploratory behaviour in search of a better, more successful motor action. It is unclear whether failure also induces exploration along task-irrelevant dimensions that do not influence performance. In this study, we ask whether participants would explore the task-irrelevant dimension while they performed a stencil task. With a stylus, participants applied downward, normal force that influenced whether they received reward (task-relevant) as they simultaneously made erasing-like movement patterns along the tablet that did not influence performance (task-irrelevant). In this task, the movement pattern was analyzed as the distribution of movement directions within a movement. The results showed significant exploration of task-relevant force and task-irrelevant movement patterns. We conclude that failure can induce additional movement variability along a task-irrelevant dimension.

Parental praise and children's exploration: a virtual reality experiment.

When children practice a new skill and fail, it is critical for them to explore new strategies to succeed. How can parents encourage children's exploration? Bridging insights from developmental psychology and the neuroscience of motor control, we examined the effects of parental praise on children's motor exploration. We theorize that modest praise can spark exploration. Unlike inflated praise, modest praise acknowledges children's performance, without setting a high standard for future performance. This may be reassuring to children with lower levels of self-esteem, who often doubt their ability. We conducted a novel virtual-reality experiment. Children (N = 202, ages 8-12) reported self-esteem and performed a virtual-reality 3D trajectory-matching task, with success/failure feedback after each trial. Children received modest praise ("You did well!"), inflated praise ("You did incredibly well!"), or no praise from their parent. We measured motor exploration as children's tendency to vary their movements following failure. Relative to no praise, modest praise-unlike inflated praise-encouraged exploration in children with lower levels of self-esteem. By contrast, modest praise discouraged exploration in children with higher levels of self-esteem. Effects were small yet robust. This experiment demonstrates that modest praise can spark exploration in children with lower levels of self-esteem.

Motivation as a function of success frequency.

It is well-established that intermediate challenge is optimally motivating. We tested whether this can be quantified into an inverted-U relationship between motivation and success frequency. Participants played a game in which they navigated a scene to catch targets. In Experiment 1 (N = 101), play duration was free and the motivating value of success frequency was measured from the probability that a player would continue at that frequency. In Experiment 2 (N = 70), play duration was fixed, and motivation was measured using repeated self-reports. In Experiment 1, the probability to continue increased linearly with the success frequency whereas play duration did show the inverted-U relationship with success frequency. In Experiment 2, self-reported motivation showed the inverted-U relationship with success frequency. Together, this shows that motivation depends on success frequency. In addition, we provide tentative evidence that the concept of intermediate challenge being most motivating can be quantified into an inverted-U relationship between motivation and success frequency.

Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them.

When learning a movement based on binary success information, one is more variable following failure than following success. Theoretically, the additional variability post-failure might reflect exploration of possibilities to obtain success. When average behavior is changing (as in learning), variability can be estimated from differences between subsequent movements. Can one estimate exploration reliably from such trial-to-trial changes when studying reward-based motor learning? To answer this question, we tried to reconstruct the exploration underlying learning as described by four existing reward-based motor learning models. We simulated learning for various learner and task characteristics. If we simply determined the additional change post-failure, estimates of exploration were sensitive to learner and task characteristics. We identified two pitfalls in quantifying exploration based on trial-to-trial changes. Firstly, performance-dependent feedback can cause correlated samples of motor noise and exploration on successful trials, which biases exploration estimates. Secondly, the trial relative to which trial-to-trial change is calculated may also contain exploration, which causes underestimation. As a solution, we developed the additional trial-to-trial change (ATTC) method. By moving the reference trial one trial back and subtracting trial-to-trial changes following specific sequences of trial outcomes, exploration can be estimated reliably for the three models that explore based on the outcome of only the previous trial. Since ATTC estimates are based on a selection of trial sequences, this method requires many trials. In conclusion, if exploration is a binary function of previous trial outcome, the ATTC method allows for a model-free quantification of exploration.

Learning a reach trajectory based on binary reward feedback.

Binary reward feedback on movement success is sufficient for learning some simple sensorimotor mappings in a reaching task, but not for some other tasks in which multiple kinematic factors contribute to performance. The critical condition for learning in more complex tasks remains unclear. Here, we investigate whether reward-based motor learning is possible in a multi-dimensional trajectory matching task and whether simplifying the task by providing feedback on one factor at a time ('factorized feedback') can improve learning. In two experiments, participants performed a trajectory matching task in which learning was measured as a reduction in the error. In Experiment 1, participants matched a straight trajectory slanted in depth. We factorized the task by providing feedback on the slant error, the length error, or on their composite. In Experiment 2, participants matched a curved trajectory, also slanted in depth. In this experiment, we factorized the feedback by providing feedback on the slant error, the curvature error, or on the integral difference between the matched and target trajectory. In Experiment 1, there was anecdotal evidence that participants learnt the multidimensional task. Factorization did not improve learning. In Experiment 2, there was anecdotal evidence the multidimensional task could not be learnt. We conclude that, within a complexity range, multiple kinematic factors can be learnt in parallel.

Quantifying exploration in reward-based motor learning.

Exploration in reward-based motor learning is observable in experimental data as increased variability. In order to quantify exploration, we compare three methods for estimating other sources of variability: sensorimotor noise. We use a task in which participants could receive stochastic binary reward feedback following a target-directed weight shift. Participants first performed six baseline blocks without feedback, and next twenty blocks alternating with and without feedback. Variability was assessed based on trial-to-trial changes in movement endpoint. We estimated sensorimotor noise by the median squared trial-to-trial change in movement endpoint for trials in which no exploration is expected. We identified three types of such trials: trials in baseline blocks, trials in the blocks without feedback, and rewarded trials in the blocks with feedback. We estimated exploration by the median squared trial-to-trial change following non-rewarded trials minus sensorimotor noise. As expected, variability was larger following non-rewarded trials than following rewarded trials. This indicates that our reward-based weight-shifting task successfully induced exploration. Most importantly, our three estimates of sensorimotor noise differed: the estimate based on rewarded trials was significantly lower than the estimates based on the two types of trials without feedback. Consequently, the estimates of exploration also differed. We conclude that the quantification of exploration depends critically on the type of trials used to estimate sensorimotor noise. We recommend the use of variability following rewarded trials.

Correction: Reward abundance interferes with error-based learning in a visuomotor adaptation task.

[This corrects the article DOI: 10.1371/journal.pone.0193002.].

A review of grasping as the movements of digits in space.

It is tempting to describe human reach-to-grasp movements in terms of two, more or less independent visuomotor channels, one relating hand transport to the object's location and the other relating grip aperture to the object's size. Our review of experimental work questions this framework for reasons that go beyond noting the dependence between the two channels. Both the lack of effect of size illusions on grip aperture and the finding that the variability in grip aperture does not depend on the object's size indicate that size information is not used to control grip aperture. An alternative is to describe grip formation as emerging from controlling the movements of the digits in space. Each digit's trajectory when grasping an object is remarkably similar to its trajectory when moving to tap the same position on its own. The similarity is also evident in the fast responses when the object is displaced. This review develops a new description of the speed-accuracy trade-off for multiple effectors that is applied to grasping. The most direct support for the digit-in-space framework is that prism-induced adaptation of each digit's tapping movements transfers to that digit's movements when grasping, leading to changes in grip aperture for adaptation in opposite directions for the two digits. We conclude that although grip aperture and hand transport are convenient variables to describe grasping, treating grasping as movements of the digits in space is a more suitable basis for understanding the neural control of grasping.

Gamification as a Sustainable Source of Enjoyment During Balance and Gait Exercises.

We may be motivated to engage in a certain motor activity because it is instrumental to obtaining reward (e.g., money) or because we enjoy the activity, making it intrinsically rewarding. Enjoyment is related to intrinsic motivation which is considered to be a durable form of motivation. Therefore, many rehabilitation programs aim to increase task enjoyment by adding game elements ("gamification"). Here we ask how the influence of game elements on motivation develops over time and additionally explore whether enjoyment influences motor performance. We describe two different studies that varied game elements in different exercises. Experiment 1 compared the durability of enjoyment for a gamified and a conventional balance exercise in elderly. Experiment 2 addressed the question whether adding game elements to a gait adaptability exercise enhances the durability of enjoyment and additionally tested whether the game elements influenced movement vigor and accuracy (motor performance). The results show that the game elements enhanced enjoyment. Enjoyment faded over time, but this decrease tended to be less pronounced in gamified exercises. There was no evidence that the game elements affected movement vigor or accuracy.

Reward-based motor adaptation can generalize across actions.

Recently it has been shown that rewarded variability can be used to adapt visuomotor behavior. However, its relevance seems limited because adaptation to binary rewards has been demonstrated only when the same movement is repeated throughout the experiment. We therefore investigated whether the adaptation is action-specific and whether the amount of exploration depends on spatial complexity. Participants pointed to 3-D visual targets without seeing their hand and could use only binary reward feedback to adapt their movements. We varied the number of target positions and the number of dimensions the feedback was based on. Because the feedback was based on a 5-cm rightward shifted hand position, adaptation was needed for good performance. The participants started naïve to the perturbation. If actions were made toward a single target position and the feedback was based on the lateral component of their response only, participants adapted completely within 200 trials. Having more than 1 target position or more than 1 dimension of performance resulted in considerably less adaptation but did not affect the exploration. Thus, reward-based adaptation can generalize across actions but is reduced by spatial complexity, whereas exploration is not affected by spatial complexity. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Reward abundance interferes with error-based learning in a visuomotor adaptation task.

The brain rapidly adapts reaching movements to changing circumstances by using visual feedback about errors. Providing reward in addition to error feedback facilitates the adaptation but the underlying mechanism is unknown. Here, we investigate whether the proportion of trials rewarded (the 'reward abundance') influences how much participants adapt to their errors. We used a 3D multi-target pointing task in which reward alone is insufficient for motor adaptation. Participants (N = 423) performed the pointing task with feedback based on a shifted hand-position. On a proportion of trials we gave them rewarding feedback that their hand hit the target. Half of the participants only received this reward feedback. The other half also received feedback about endpoint errors. In different groups, we varied the proportion of trials that was rewarded. As expected, participants who received feedback about their errors did adapt, but participants who only received reward-feedback did not. Critically, participants who received abundant rewards adapted less to their errors than participants who received less reward. Thus, reward abundance negatively influences how much participants learn from their errors. Probably participants used a mechanism that relied more on the reward feedback when the reward was abundant. Because participants could not adapt to the reward, this interfered with adaptation to errors.

Temporally stable adaptation is robust, incomplete and specific.

Sensorimotor adaptation, the process that reduces movement errors by learning from sensory feedback, is often studied within a session of about half an hour. Within such a single session, adaptation generally reaches plateau before errors are completely removed. However, adaptation may complete on longer timescales: the slow components of error-based adaptation are associated with good retention. In this study, we tested how adaptation evolves over time by asking participants to perform six adaptation sessions on different days. In these sessions, participants performed a three-dimensional reaching task while visual feedback about endpoint errors was rotated around the cyclopean eye. In addition, context specificity of the adaptation was addressed by measuring inter-limb transfer and transfer to visual and proprioceptive perceptual tasks. We show that from the second session on, the adaptation was retained almost completely across sessions. However, after six learning sessions, adaptation still reached plateau before errors were completely removed. The adaptation was specific: the adaptation did neither transfer to the other hand, nor to the visual, and only marginally to the proprioceptive perceptual estimates. We conclude that motor adaptation is robust, specific and incomplete.

Visuomotor adaptation: how forgetting keeps us conservative.

Even when provided with feedback after every movement, adaptation levels off before biases are completely removed. Incomplete adaptation has recently been attributed to forgetting: the adaptation is already partially forgotten by the time the next movement is made. Here we test whether this idea is correct. If so, the final level of adaptation is determined by a balance between learning and forgetting. Because we learn from perceived errors, scaling these errors by a magnification factor has the same effect as subjects increasing the amount by which they learn from each error. In contrast, there is no reason to expect scaling the errors to affect forgetting. The magnification factor should therefore influence the balance between learning and forgetting, and thereby the final level of adaptation. We found that adaptation was indeed more complete for larger magnification factors. This supports the idea that incomplete adaptation is caused by part of what has been learnt quickly being forgotten.

Perception of 3D Slant Out of the Box.

Evidence for contextual effects is widespread in visual perception. Although this suggests that contextual effects are the result of a generic property of the visual system, current explanations are limited to the domain in which they occur. In this paper we propose a more general mechanism of global influences on the perception of slant. We review empirical data and evaluate proposed explanations of contextual biases. By assessing not only a model about three-dimensional slant perception but also evaluating more generic mechanisms of contextual modulation, we show that surround suppression of neural responses explains the major phenomena in the empirical data on contextual biases. Moreover, contextual biases may be part of a mechanism of grouping and segmentation.

Shape contrast: a global mechanism?

We investigated whether a shape contrast bias is caused by local contrast enhancement or by a global mechanism. In a baseline condition, observers performed a shape discrimination task on an isolated hinged plane. But in the experimental conditions, five dihedral surfaces, of which we varied the dihedral angle distribution, were added on each side. Shape perception was influenced not only by the adjacent surface but also by the mean of the shape distribution in the extended surround. Thus, shape contrast is not locally determined and has to be understood from a global mechanism. We propose divisive normalization of shape signals as such a mechanism.

Uncertainty reveals surround modulation of shape.

Noisy estimations of shape can be partially resolved by incorporating relevant information from the context. The effect of surround stimuli on shape perception becomes clear in illusions of shape contrast and assimilation. In this study, we answer the question how a surround-induced bias depends on the reliability of shape signals. This way, we assess the processes by which an observer incorporates relevant data from the context into the shape estimate. We selectively added visual noise to the center and surround and compared a bias in shape perception with a control condition where no noise was added. In the conditions where shape and surround stimuli were well defined, we found a shape-contrast bias. When the surround stimuli were degraded, this contrast bias decreased. Most interestingly, when the central shape was degraded, an assimilation bias was observed. This bias was larger when the entire stimulus was degraded compared to when only the central shape was degraded. This suggests that shape contrast is the result of inference processes relying on local representations in early visual areas whereas assimilation is related to inference processes by global representations in higher visual areas.

Perception of 3D shape in context: contrast and assimilation.

Whereas integration of shape and surround is held to occur through cue-dependent representations, we show that both cue-invariant and cue-dependent representations are involved. A central hinged plane and larger flanking plane were defined by either binocular disparity or motion. In a 'within-cue' condition, shape and surround were defined by the same cue and in a 'cross-cue' condition they were defined by a different cue. Observers compared the dihedral angle of the central shape with a constant reference. When the central shape was defined by disparity, the surround stimuli invoked a contrast bias in the within-cue condition, but shape assimilation occurred in the cross-cue condition. When the central shape was defined by motion there were overall no significant results, but if a contrast bias was observed, it was in the within-cue condition where integration could occur through cue-dependent representations.