Computational reproductions of external force field adaption without assuming desired trajectories.

Optimal feedback control is an established framework that is used to characterize human movement. However, it is not fully understood how the brain computes optimal gains through interactions with the environment. In the past study, we proposed a model of motor learning that identifies a set of feedback and feedforward controllers and a state predictor of the arm musculoskeletal system to control free reaching movements. In this study, we applied the model to force field adaptation tasks where normal reaching movements are disturbed by an external force imposed on the hand. Without a priori knowledge about the arm and environment, the model was able to adapt to the force field by generating counteracting forces to overcome it in a manner similar to what is reported in the behavioral literature. The kinematics of the movements generated by our model share characteristic features of human movements observed before and after force field adaptation. In addition, we demonstrate that the structure and learning algorithm introduced in our model induced a shift in the end-point's equilibrium position and a static force modulation, accompanied by a fast and a slow learning process. Importantly, our model does not require desired trajectories, yields movements without specifying movement duration, and predicts force generation patterns by exploring the environment. Our model demonstrates a possible mechanism through which the central nervous system may control and adapt a point-to-point reaching movement without specifying a desired trajectory by continuously updating the body's musculoskeletal model.

Analogous adaptations in speed, impulse and endpoint stiffness when learning a real and virtual insertion task with haptic feedback.

Humans have the ability to use a diverse range of handheld tools. Owing to its versatility, a virtual environment with haptic feedback of the force is ideally suited to investigating motor learning during tool use. However, few simulators exist to recreate the dynamic interactions during real tool use, and no study has compared the correlates of motor learning between a real and virtual tooling task. To this end, we compared two groups of participants who either learned to insert a real or virtual tool into a fixture. The trial duration, the movement speed, the force impulse after insertion and the endpoint stiffness magnitude decreased as a function of trials, but they changed at comparable rates in both environments. A ballistic insertion strategy observed in both environments suggests some interdependence when controlling motion and controlling interaction, contradicting a prominent theory of these two control modalities being independent of one another. Our results suggest that the brain learns real and virtual insertion in a comparable manner, thereby supporting the use of a virtual tooling task with haptic feedback to investigate motor learning during tool use.

Independent control of cocontraction and reciprocal activity during goal-directed reaching in muscle space.

The movement in a joint is facilitated by a pair of muscles that pull in opposite directions. The difference in the pair's muscle force or reciprocal activity results in joint torque, while the overlapping muscle force or the cocontraction is related to the joint's stiffness. Cocontraction knowingly adapts implicitly over a number of movements, but it is unclear whether the central nervous system can actively regulate cocontraction in a goal-directed manner in a short span of time. We developed a muscle interface where a cursor's horizontal position was determined by the reciprocal activity of the shoulder flexion-extension muscle pair, while the vertical position was controlled by its cocontraction. Participants made goal-directed movements to single and via-point targets in the two-dimensional muscle space, learning to move the cursor along the shortest path. Simulations using an optimal control framework suggest that the reciprocal activity and the cocontraction may be controlled independently by the CNS, albeit at a rate orders of magnitude slower than the muscle's maximal activation speed.

Muscle Synergy and Musculoskeletal Model-Based Continuous Multi-Dimensional Estimation of Wrist and Hand Motions.

In this study, seven-channel electromyography signal-based two-dimensional wrist joint movement estimation with and without handgrip motions was carried out. Electromyography signals were analyzed using the synergy-based linear regression model and musculoskeletal model; they were subsequently compared with respect to single and combined wrist joint movements and handgrip. Using each one of wrist motion and grip trial as a training set, the synergy-based linear regression model exhibited a statistically significant performance with 0.7891 ± 0.0844 Pearson correlation coefficient (r) value in two-dimensional wrist motion estimation compared with 0.7608 ± 0.1037 r value of the musculoskeletal model. Estimates on the grip force produced 0.8463 ± 0.0503 r value with 0.2559 ± 0.1397 normalized root-mean-square error of the wrist motion range. This continuous wrist and handgrip estimation can be considered when electromyography-based multi-dimensional input signals in the prosthesis, virtual interface, and rehabilitation are needed.

Control of a Robot Arm Using Decoded Joint Angles from Electrocorticograms in Primate.

Electrocorticogram (ECoG) is a well-known recording method for the less invasive brain machine interface (BMI). Our previous studies have succeeded in predicting muscle activities and arm trajectories from ECoG signals. Despite such successful studies, there still remain solving works for the purpose of realizing an ECoG-based prosthesis. We suggest a neuromuscular interface to control robot using decoded muscle activities and joint angles. We used sparse linear regression to find the best fit between band-passed ECoGs and electromyograms (EMG) or joint angles. The best coefficient of determination for 100 s continuous prediction was 0.6333 ± 0.0033 (muscle activations) and 0.6359 ± 0.0929 (joint angles), respectively. We also controlled a 4 degree of freedom (DOF) robot arm using only decoded 4 DOF angles from the ECoGs in this study. Consequently, this study shows the possibility of contributing to future advancements in neuroprosthesis and neurorehabilitation technology.

Reduced Effort Does Not Imply Slacking: Responsiveness to Error Increases With Robotic Assistance.

In both neurorehabilitation and functional augmentation, the patient or the user's muscular effort diminishes when the movement of their limb is supported by a robot. Is this relaxation a result of "slacking" by letting the robot take-over the movement, resulting in less responsiveness in the task? To address this question, we tested subjects who controlled a virtual cursor isometrically to track a moving target without and with different assistants. We measured the force applied by the subject as a metric for effort and estimated their control gain as the metric for responsiveness in the task. Although subjects applied less force with position assistance, the norm of the control gain increased with all assistants, i.e., they applied proportionately larger forces for the same difference between the cursor and the target states. Furthermore, assisting velocity errors improved baseline performance without reducing effort. Though all assistants improved task performance, the control gain adapted differently to position and velocity assistance. Position assistance was exploited to accurately track the target, whereas velocity assistance was treated as a disturbance, and was effectively nullified as it prevented submovements that minimized positional error. Our results show that robotic assistance increases task responsiveness in healthy individuals and that assisting velocity errors could boost patient performance without reducing their motor effort.

Decoding finger movement in humans using synergy of EEG cortical current signals.

The synchronized activity of neuronal populations across multiple distant brain areas may reflect coordinated interactions of large-scale brain networks. Currently, there is no established method to investigate the temporal transitions between these large-scale networks that would allow, for example, to decode finger movements. Here we applied a matrix factorization method employing principal component and temporal independent component analyses to identify brain activity synchronizations. In accordance with previous studies investigating "muscle synergies", we refer to this activity as "brain activity synergy". Using electroencephalography (EEG), we first estimated cortical current sources (CSs) and then identified brain activity synergies within the estimated CS signals. A decoding analysis for finger movement in eight directions showed that such CS synergies provided more information for dissociating between movements than EEG sensor signals, EEG synergy, or CS signals, suggesting that temporal activation patterns of the synchronizing CSs may contain information related to motor control. A quantitative analysis of features selected by the decoders further revealed temporal transitions among the primary motor area, dorsal and ventral premotor areas, pre-supplementary motor area, and supplementary motor area, which may reflect transitions in motor planning and execution. These results provide a proof of concept for brain activity synergy estimation using CSs.

Mapping ECoG channel contributions to trajectory and muscle activity prediction in human sensorimotor cortex.

Studies on brain-machine interface techniques have shown that electrocorticography (ECoG) is an effective modality for predicting limb trajectories and muscle activity in humans. Motor control studies have also identified distributions of "extrinsic-like" and "intrinsic-like" neurons in the premotor (PM) and primary motor (M1) cortices. Here, we investigated whether trajectories and muscle activity predicted from ECoG were obtained based on signals derived from extrinsic-like or intrinsic-like neurons. Three participants carried objects of three different masses along the same counterclockwise path on a table. Trajectories of the object and upper arm muscle activity were predicted using a sparse linear regression. Weight matrices for the predictors were then compared to determine if the ECoG channels contributed more information about trajectory or muscle activity. We found that channels over both PM and M1 contributed highly to trajectory prediction, while a channel over M1 was the highest contributor for muscle activity prediction.

Decoding of Ankle Flexion and Extension from Cortical Current Sources Estimated from Non-invasive Brain Activity Recording Methods.

The classification of ankle movements from non-invasive brain recordings can be applied to a brain-computer interface (BCI) to control exoskeletons, prosthesis, and functional electrical stimulators for the benefit of patients with walking impairments. In this research, ankle flexion and extension tasks at two force levels in both legs, were classified from cortical current sources estimated by a hierarchical variational Bayesian method, using electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) recordings. The hierarchical prior for the current source estimation from EEG was obtained from activated brain areas and their intensities from an fMRI group (second-level) analysis. The fMRI group analysis was performed on regions of interest defined over the primary motor cortex, the supplementary motor area, and the somatosensory area, which are well-known to contribute to movement control. A sparse logistic regression method was applied for a nine-class classification (eight active tasks and a resting control task) obtaining a mean accuracy of 65.64% for time series of current sources, estimated from the EEG and the fMRI signals using a variational Bayesian method, and a mean accuracy of 22.19% for the classification of the pre-processed of EEG sensor signals, with a chance level of 11.11%. The higher classification accuracy of current sources, when compared to EEG classification accuracy, was attributed to the high number of sources and the different signal patterns obtained in the same vertex for different motor tasks. Since the inverse filter estimation for current sources can be done offline with the present method, the present method is applicable to real-time BCIs. Finally, due to the highly enhanced spatial distribution of current sources over the brain cortex, this method has the potential to identify activation patterns to design BCIs for the control of an affected limb in patients with stroke, or BCIs from motor imagery in patients with spinal cord injury.

Decoding of Covert Vowel Articulation Using Electroencephalography Cortical Currents.

With the goal of providing assistive technology for the communication impaired, we proposed electroencephalography (EEG) cortical currents as a new approach for EEG-based brain-computer interface spellers. EEG cortical currents were estimated with a variational Bayesian method that uses functional magnetic resonance imaging (fMRI) data as a hierarchical prior. EEG and fMRI data were recorded from ten healthy participants during covert articulation of Japanese vowels /a/ and /i/, as well as during a no-imagery control task. Applying a sparse logistic regression (SLR) method to classify the three tasks, mean classification accuracy using EEG cortical currents was significantly higher than that using EEG sensor signals and was also comparable to accuracies in previous studies using electrocorticography. SLR weight analysis revealed vertices of EEG cortical currents that were highly contributive to classification for each participant, and the vertices showed discriminative time series signals according to the three tasks. Furthermore, functional connectivity analysis focusing on the highly contributive vertices revealed positive and negative correlations among areas related to speech processing. As the same findings were not observed using EEG sensor signals, our results demonstrate the potential utility of EEG cortical currents not only for engineering purposes such as brain-computer interfaces but also for neuroscientific purposes such as the identification of neural signaling related to language processing.

Real-Time Control of a Video Game Using Eye Movements and Two Temporal EEG Sensors.

EEG-controlled gaming applications range widely from strictly medical to completely nonmedical applications. Games can provide not only entertainment but also strong motivation for practicing, thereby achieving better control with rehabilitation system. In this paper we present real-time control of video game with eye movements for asynchronous and noninvasive communication system using two temporal EEG sensors. We used wavelets to detect the instance of eye movement and time-series characteristics to distinguish between six classes of eye movement. A control interface was developed to test the proposed algorithm in real-time experiments with opened and closed eyes. Using visual feedback, a mean classification accuracy of 77.3% was obtained for control with six commands. And a mean classification accuracy of 80.2% was obtained using auditory feedback for control with five commands. The algorithm was then applied for controlling direction and speed of character movement in two-dimensional video game. Results showed that the proposed algorithm had an efficient response speed and timing with a bit rate of 30 bits/min, demonstrating its efficacy and robustness in real-time control.

Decoding fingertip trajectory from electrocorticographic signals in humans.

Seeking to apply brain-machine interface technology in neuroprosthetics, a number of methods for predicting trajectory of the elbow and wrist have been proposed and have shown remarkable results. Recently, the prediction of hand trajectory and classification of hand gestures or grasping types have attracted considerable attention. However, trajectory prediction for precise finger motion has remained a challenge. We proposed a method for the prediction of fingertip motions from electrocorticographic signals in human cortex. A patient performed extension/flexion tasks with three fingers. Average Pearson's correlation coefficients and normalized root-mean-square errors between decoded and actual trajectories were 0.83-0.90 and 0.24-0.48, respectively. To confirm generalizability to other users, we applied our method to the BCI Competition IV open data sets. Our method showed that the prediction accuracy of fingertip trajectory could be equivalent to that of other results in the competition.

Dissociable neural representations of wrist motor coordinate frames in human motor cortices.

There is a growing interest in how the brain transforms body part positioning in the extrinsic environment into an intrinsic coordinate frame during motor control. To explore the human brain areas representing intrinsic and extrinsic coordinate frames, this fMRI study examined neural representation of motor cortices while human participants performed isometric wrist flexions and extensions in different forearm postures, thereby applying the same wrist actions (representing the intrinsic coordinate frame) to different movement directions (representing the extrinsic coordinate frame). Using sparse logistic regression, critical voxels involving pattern information that specifically discriminates wrist action (flexion vs. extension) and movement direction (upward vs. downward) were identified within the primary motor and premotor cortices. Analyses of classifier weights further identified contributions of the primary motor cortex to the intrinsic coordinate frame and the ventral and dorsal premotor cortex and supplementary motor area proper to the extrinsic coordinate frame. These results are consistent with existing findings using non-human primates and demonstrate the distributed representations of independent coordinate frames in the human brain.

Decoding grasp force profile from electrocorticography signals in non-human primate sensorimotor cortex.

The relatively low invasiveness of electrocorticography (ECoG) has made it a promising candidate for the development of practical, high-performance neural prosthetics. Recent ECoG-based studies have shown success in decoding hand and finger movements and muscle activity in reaching and grasping tasks. However, decoding of force profiles is still lacking. Here, we demonstrate that lateral grasp force profile can be decoded using a sparse linear regression from 15 and 16 channel ECoG signals recorded from sensorimotor cortex in two non-human primates. The best average correlation coefficients of prediction after 10-fold cross validation were 0.82±0.09 and 0.79±0.15 for our monkeys A and B, respectively. These results show that grasp force profile was successfully decoded from ECoG signals in reaching and grasping tasks and may potentially contribute to the development of more natural control methods for grasping in neural prosthetics.

Prediction of three-dimensional arm trajectories based on ECoG signals recorded from human sensorimotor cortex.

Brain-machine interface techniques have been applied in a number of studies to control neuromotor prostheses and for neurorehabilitation in the hopes of providing a means to restore lost motor function. Electrocorticography (ECoG) has seen recent use in this regard because it offers a higher spatiotemporal resolution than non-invasive EEG and is less invasive than intracortical microelectrodes. Although several studies have already succeeded in the inference of computer cursor trajectories and finger flexions using human ECoG signals, precise three-dimensional (3D) trajectory reconstruction for a human limb from ECoG has not yet been achieved. In this study, we predicted 3D arm trajectories in time series from ECoG signals in humans using a novel preprocessing method and a sparse linear regression. Average Pearson's correlation coefficients and normalized root-mean-square errors between predicted and actual trajectories were 0.44~0.73 and 0.18~0.42, respectively, confirming the feasibility of predicting 3D arm trajectories from ECoG. We foresee this method contributing to future advancements in neuroprosthesis and neurorehabilitation technology.

The effect of temporal perception on weight perception.

A successful catch of a falling ball requires an accurate estimation of the timing for when the ball hits the hand. In a previous experiment in which participants performed ball-catching task in virtual reality environment, we accidentally found that the weight of a falling ball was perceived differently when the timing of ball load force to the hand was shifted from the timing expected from visual information. Although it is well known that spatial information of an object, such as size, can easily deceive our perception of its heaviness, the relationship between temporal information and perceived heaviness is still not clear. In this study, we investigated the effect of temporal factors on weight perception. We conducted ball-catching experiments in a virtual environment where the timing of load force exertion was shifted away from the visual contact timing (i.e., time when the ball hit the hand in the display). We found that the ball was perceived heavier when force was applied earlier than visual contact and lighter when force was applied after visual contact. We also conducted additional experiments in which participants were conditioned to one of two constant time offsets prior to testing weight perception. After performing ball-catching trials with 60 ms advanced or delayed load force exertion, participants' subjective judgment on the simultaneity of visual contact and force exertion changed, reflecting a shift in perception of time offset. In addition, timing of catching motion initiation relative to visual contact changed, reflecting a shift in estimation of force timing. We also found that participants began to perceive the ball as lighter after conditioning to 60 ms advanced offset and heavier after the 60 ms delayed offset. These results suggest that perceived heaviness depends not on the actual time offset between force exertion and visual contact but on the subjectively perceived time offset between them and/or estimation error in force timing.

A computational model for optimal muscle activity considering muscle viscoelasticity in wrist movements.

To understand the mechanism of neural motor control, it is important to clarify how the central nervous system organizes the coordination of redundant muscles. Previous studies suggested that muscle activity for step-tracking wrist movements are optimized so as to reduce total effort or end-point variance under neural noise. However, since the muscle dynamics were assumed as a simple linear system, some characteristic patterns of experimental EMG were not seen in the simulated muscle activity of the previous studies. The biological muscle is known to have dynamic properties in which its elasticity and viscosity depend on activation level. The motor control system is supposed to consider the viscoelasticity of the muscles when generating motor command signals. In this study, we present a computational motor control model that can control a musculoskeletal system with nonlinear dynamics. We applied the model to step-tracking wrist movements actuated by five muscles with dynamic viscoelastic properties. To solve the motor redundancy, we designed the control model to generate motor commands that maximize end-point accuracy under signal-dependent noise, while minimizing the squared sum of them. Here, we demonstrate that the muscle activity simulated by our model exhibits spatiotemporal features of experimentally observed muscle activity of human and nonhuman primates. In addition, we show that the movement trajectories resulting from the simulated muscle activity resemble experimentally observed trajectories. These results suggest that, by utilizing inherent viscoelastic properties of the muscles, the neural system may optimize muscle activity to improve motor performance.

Prediction of hand trajectory from electrocorticography signals in primary motor cortex.

Due to their potential as a control modality in brain-machine interfaces, electrocorticography (ECoG) has received much focus in recent years. Studies using ECoG have come out with success in such endeavors as classification of arm movements and natural grasp types, regression of arm trajectories in two and three dimensions, estimation of muscle activity time series and so on. However, there still remains considerable work to be done before a high performance ECoG-based neural prosthetic can be realized. In this study, we proposed an algorithm to decode hand trajectory from 15 and 32 channel ECoG signals recorded from primary motor cortex (M1) in two primates. To determine the most effective areas for prediction, we applied two electrode selection methods, one based on position relative to the central sulcus (CS) and another based on the electrodes' individual prediction performance. The best coefficients of determination for decoding hand trajectory in the two monkeys were 0.4815 ± 0.0167 and 0.7780 ± 0.0164. Performance results from individual ECoG electrodes showed that those with higher performance were concentrated at the lateral areas and areas close to the CS. The results of prediction according with different numbers of electrodes based on proposed methods were also shown and discussed. These results also suggest that superior decoding performance can be achieved from a group of effective ECoG signals rather than an entire ECoG array.

Prediction of muscle activities from electrocorticograms in primary motor cortex of primates.

Electrocorticography (ECoG) has drawn attention as an effective recording approach for brain-machine interfaces (BMI). Previous studies have succeeded in classifying movement intention and predicting hand trajectories from ECoG. Despite such successes, however, there still remains considerable work for the realization of ECoG-based BMIs as neuroprosthetics. We developed a method to predict multiple muscle activities from ECoG measurements. We also verified that ECoG signals are effective for predicting muscle activities in time varying series when performing sequential movements. ECoG signals were band-pass filtered into separate sensorimotor rhythm bands, z-score normalized, and smoothed with a Gaussian filter. We used sparse linear regression to find the best fit between frequency bands of ECoG and electromyographic activity. The best average correlation coefficient and the normalized root-mean-square error were 0.92±0.06 and 0.06±0.10, respectively, in the flexor digitorum profundus finger muscle. The δ (1.5∼4Hz) and γ2 (50∼90Hz) bands contributed significantly more strongly than other frequency bands (P<0.001). These results demonstrate the feasibility of predicting muscle activity from ECoG signals in an online fashion.

Dynamic optimization of the sit-to-stand movement.

The purpose of this study was to clarify criteria that can predict trajectories during the sit-to-stand movement. In particular, the minimum jerk and minimum torque-change models were examined. Three patterns of sit-to-stand movement from a chair, i.e., upright, natural, and leaning forward, were measured in five young participants using a 3-D motion analysis device (200 Hz). The trajectory of the center of mass and its smoothness were examined, and the optimal trajectories predicted by both models were evaluated. Trajectories of the center of mass predicted by the minimum torque-change model, rather than the minimum jerk model, resembled the measured movements in all rising movement patterns. The upright pattern required greater extension torque of the knee and ankle joints at the instant of seat-off. The leaning-forward pattern required greater extension hip torque and higher movement cost than the natural and upright patterns. These results indicate that the natural sit-to-stand movement might be a result of dynamic optimization.