Cortical and subcortical mechanisms of brain-machine interfaces.

Technical advances in the field of Brain-Machine Interfaces (BMIs) enable users to control a variety of external devices such as robotic arms, wheelchairs, virtual entities and communication systems through the decoding of brain signals in real time. Most BMI systems sample activity from restricted brain regions, typically the motor and premotor cortex, with limited spatial resolution. Despite the growing number of applications, the cortical and subcortical systems involved in BMI control are currently unknown at the whole-brain level. Here, we provide a comprehensive and detailed report of the areas active during on-line BMI control. We recorded functional magnetic resonance imaging (fMRI) data while participants controlled an EEG-based BMI inside the scanner. We identified the regions activated during BMI control and how they overlap with those involved in motor imagery (without any BMI control). In addition, we investigated which regions reflect the subjective sense of controlling a BMI, the sense of agency for BMI-actions. Our data revealed an extended cortical-subcortical network involved in operating a motor-imagery BMI. This includes not only sensorimotor regions but also the posterior parietal cortex, the insula and the lateral occipital cortex. Interestingly, the basal ganglia and the anterior cingulate cortex were involved in the subjective sense of controlling the BMI. These results inform basic neuroscience by showing that the mechanisms of BMI control extend beyond sensorimotor cortices. This knowledge may be useful for the development of BMIs that offer a more natural and embodied feeling of control for the user. Hum Brain Mapp 38:2971-2989, 2017. © 2017 Wiley Periodicals, Inc.

Active tactile exploration using a brain-machine-brain interface.

Brain-machine interfaces use neuronal activity recorded from the brain to establish direct communication with external actuators, such as prosthetic arms. It is hoped that brain-machine interfaces can be used to restore the normal sensorimotor functions of the limbs, but so far they have lacked tactile sensation. Here we report the operation of a brain-machine-brain interface (BMBI) that both controls the exploratory reaching movements of an actuator and allows signalling of artificial tactile feedback through intracortical microstimulation (ICMS) of the primary somatosensory cortex. Monkeys performed an active exploration task in which an actuator (a computer cursor or a virtual-reality arm) was moved using a BMBI that derived motor commands from neuronal ensemble activity recorded in the primary motor cortex. ICMS feedback occurred whenever the actuator touched virtual objects. Temporal patterns of ICMS encoded the artificial tactile properties of each object. Neuronal recordings and ICMS epochs were temporally multiplexed to avoid interference. Two monkeys operated this BMBI to search for and distinguish one of three visually identical objects, using the virtual-reality arm to identify the unique artificial texture associated with each. These results suggest that clinical motor neuroprostheses might benefit from the addition of ICMS feedback to generate artificial somatic perceptions associated with mechanical, robotic or even virtual prostheses.

Expanding the primate body schema in sensorimotor cortex by virtual touches of an avatar.

The brain representation of the body, called the body schema, is susceptible to plasticity. For instance, subjects experiencing a rubber hand illusion develop a sense of ownership of a mannequin hand when they view it being touched while tactile stimuli are simultaneously applied to their own hand. Here, the cortical basis of such an embodiment was investigated through concurrent recordings from primary somatosensory (i.e., S1) and motor (i.e., M1) cortical neuronal ensembles while two monkeys observed an avatar arm being touched by a virtual ball. Following a period when virtual touches occurred synchronously with physical brushes of the monkeys' arms, neurons in S1 and M1 started to respond to virtual touches applied alone. Responses to virtual touch occurred 50 to 70 ms later than to physical touch, consistent with the involvement of polysynaptic pathways linking the visual cortex to S1 and M1. We propose that S1 and M1 contribute to the rubber hand illusion and that, by taking advantage of plasticity in these areas, patients may assimilate neuroprosthetic limbs as parts of their body schema.

Control strategies for active lower extremity prosthetics and orthotics: a review.

: Technological advancements have led to the development of numerous wearable robotic devices for the physical assistance and restoration of human locomotion. While many challenges remain with respect to the mechanical design of such devices, it is at least equally challenging and important to develop strategies to control them in concert with the intentions of the user.This work reviews the state-of-the-art techniques for controlling portable active lower limb prosthetic and orthotic (P/O) devices in the context of locomotive activities of daily living (ADL), and considers how these can be interfaced with the user's sensory-motor control system. This review underscores the practical challenges and opportunities associated with P/O control, which can be used to accelerate future developments in this field. Furthermore, this work provides a classification scheme for the comparison of the various control strategies.As a novel contribution, a general framework for the control of portable gait-assistance devices is proposed. This framework accounts for the physical and informatic interactions between the controller, the user, the environment, and the mechanical device itself. Such a treatment of P/Os--not as independent devices, but as actors within an ecosystem--is suggested to be necessary to structure the next generation of intelligent and multifunctional controllers.Each element of the proposed framework is discussed with respect to the role that it plays in the assistance of locomotion, along with how its states can be sensed as inputs to the controller. The reviewed controllers are shown to fit within different levels of a hierarchical scheme, which loosely resembles the structure and functionality of the nominal human central nervous system (CNS). Active and passive safety mechanisms are considered to be central aspects underlying all of P/O design and control, and are shown to be critical for regulatory approval of such devices for real-world use.The works discussed herein provide evidence that, while we are getting ever closer, significant challenges still exist for the development of controllers for portable powered P/O devices that can seamlessly integrate with the user's neuromusculoskeletal system and are practical for use in locomotive ADL.

Force feedback facilitates multisensory integration during robotic tool use.

The present study investigated the effects of force feedback in relation to tool use on the multisensory integration of visuo-tactile information. Participants learned to control a robotic tool through a surgical robotic interface. Following tool-use training, participants performed a crossmodal congruency task, by responding to tactile vibrations applied to their hands, while ignoring visual distractors superimposed on the robotic tools. In the first experiment it was found that tool-use training with force feedback facilitates multisensory integration of signals from the tool, as reflected in a stronger crossmodal congruency effect with the force feedback training compared to training without force feedback and to no training. The second experiment extends these findings by showing that training with realistic online force feedback resulted in a stronger crossmodal congruency effect compared to training in which force feedback was delayed. The present study highlights the importance of haptic information for multisensory integration and extends findings from classical tool-use studies to the domain of robotic tools. We argue that such crossmodal congruency effects are an objective measure of robotic tool integration and propose some potential applications in surgical robotics, robotic tools, and human-tool interaction.

Survey on surgical instrument handle design: ergonomics and acceptance.

Minimally invasive surgical approaches have revolutionized surgical care and considerably improved surgical outcomes. The instrumentation has changed significantly from open to laparoscopic and robotic surgery with various usability and ergonomics qualities. To establish guidelines for future designing of surgical instruments, this study assesses the effects of current surgical approaches and instruments on the surgeon. Furthermore, an analysis of surgeons' preferences with respect to instrument handles was performed to identify the main acceptance criteria. In all, 49 surgeons (24 with robotic surgery experience, 25 without) completed the survey about physical discomfort and working conditions. The respondents evaluated comfort, intuitiveness, precision, and stability of 7 instrument handles. Robotic surgery procedures generally take a longer time than conventional procedures but result in less back, shoulder, and wrist pain; 28% of surgeons complained about finger and neck pain during robotic surgery. Three handles (conventional needle holder, da Vinci wrist, and joystick-like handle) received significantly higher scores for most of the proposed criteria. The handle preference is best explained by a regression model related only to comfort and precision (R(2) = 0.91) and is significantly affected by the surgeon's background (P < .001). Although robotic surgery seems to alleviate physical discomfort during and after surgery, the results of this study show that there is room for improvement in the sitting posture and in the ergonomics of the handles. Comfort and precision have been found to be the most important aspects for the surgeon's choice of an instrument handle. Furthermore, surgeons' professional background should be considered when designing novel surgical instruments.

Haptic handles for robotic surgery.

Robotic surgery, i.e. master-slave telemanipulators for surgery, is rapidly developing. One of the key components is the surgeon's console, and, within the console, especially the "handles" (the "joysticks") for manipulation control. Two examples of haptic (force-feedback) handle designs developed and realized at the LSRO lab are briefly presented along with design considerations (ergonomics, usability). Incidence on patient safety and patient benefit and future trends in this field are hinted at, especially the possibilities offered VR in this context.

Evaluation of a Portable fMRI Compatible Robotic Wrist Interface.

This paper presents evaluation of a portable fMRI compatible haptic interface to study the brain correlates of sensorimotor control during wrist motion. The interface is actuated by a shielded DC motor located more than 2 m away from the 3T MR scanner's bore. The achievable wrist torque of the interface is up to 2 Nm, and the interface provides sufficient bandwidth for human motor control experiments. Ergonomic and fMRI compatibility testing with a 3T MR scanner showed that the interface is MR safe, compatible with a strong static magnetic field and radio frequency emission, and its operation does not affect the quality of the acquired images. Clinical Relevance- We present and evaluate an fMRI compatible robotic interface to study human wrist joint motor function.

Neural signatures of visuo-motor integration during human-robot interactions.

Visuo-motor integration shapes our daily experience and underpins the sense of feeling in control over our actions. The last decade has seen a surge in robotically and virtually mediated interactions, whereby bodily actions ultimately result in an artificial movement. But despite the growing number of applications, the neurophysiological correlates of visuo-motor processing during human-machine interactions under dynamic conditions remain scarce. Here we address this issue by employing a bimanual robotic interface able to track voluntary hands movement, rendered in real-time into the motion of two virtual hands. We experimentally manipulated the visual feedback in the virtual reality with spatial and temporal conflicts and investigated their impact on (1) visuo-motor integration and (2) the subjective experience of being the author of one's action (i.e., sense of agency). Using somatosensory evoked responses measured with electroencephalography, we investigated neural differences occurring when the integration between motor commands and visual feedback is disrupted. Our results show that the right posterior parietal cortex encodes for differences between congruent and spatially-incongruent interactions. The experimental manipulations also induced a decrease in the sense of agency over the robotically-mediated actions. These findings offer solid neurophysiological grounds that can be used in the future to monitor integration mechanisms during movements and ultimately enhance subjective experience during human-machine interactions.

Thought consciousness and source monitoring depend on robotically controlled sensorimotor conflicts and illusory states.

Thought insertion (TI) is characterized by the experience that certain thoughts, occurring in one's mind, are not one's own, but the thoughts of somebody else and suggestive of a psychotic disorder. We report a robotics-based method able to investigate the behavioral and subjective mechanisms of TI in healthy participants. We used a robotic device to alter body perception by providing online sensorimotor stimulation, while participants performed cognitive tasks implying source monitoring of mental states attributed to either oneself or another person. Across several experiments, conflicting sensorimotor stimulation reduced the distinction between self- and other-generated thoughts and was, moreover, associated with the experimentally generated feeling of being in the presence of an alien agent and subjective aspects of TI. Introducing a new robotics-based approach that enables the experimental study of the brain mechanisms of TI, these results link TI to predictable self-other shifts in source monitoring and specific sensorimotor processes.

Balance Control Strategies during Standing in a Locked-Ankle Passive Exoskeleton.

This paper investigates sensorimotor adaptation strategies of sagittal postural control in healthy subjects under kinematic constraints. A passive exoskeleton named CAPTUR, with locked ankle joints and legs motion restrained to the sagittal plane is used to restrict and measure participant's movements. The aim is to assess the role of the orientation of the shank and the trunk segments in maintaining the body center of mass above its support base, while the ankle strategy is inhibited. Five young healthy participants were asked to keep standing, while their balance was challenged by five experimental conditions. Participants mainly regulated quiet standing balance by flexing/extending the knees, in order to affect the shank and feet angles, and move the contact patch along the sagittal axis. In this case, the orientation of the trunk segment changes synchronously with the shank angle to keep an upright posture. Responses to more dramatic excursions of the center of pressure are ensured by changing the trunk tilt angle in opposition of phase with the shank angle. These observations could be used to implement a bioinspired balance controller for such constrained lower-limb exoskeletons.

Error Augmentation Improves Visuomotor Adaptation during a Full-Body Balance Task.

Visual amplification of kinematic errors has successfully been applied to improve performance for upper limb movements. In this study, we investigated whether visual error augmentation can promote faster adaptation during a full-body balance task. Healthy volunteers controlled a cursor by shifting their weight on the THERA-Trainer coro platform. Two experimental groups and one control group were asked to reach visual targets. For the two experimental groups, the cursor's deviation from the ideal straight line trajectory was augmented by a gain of 1.5 and 2, respectively, while the control group did not experience visual error amplification (gain of 1). Error augmentation with a gain of 1.5 enhanced the speed and the amount of motor adaptation, while the highest gain might have decreased the stability of adaptation. As visual feedback is commonly used in balance training, our preliminary data suggest that integrating visual error augmentation in postural exercises may facilitate balance control.

A modular simulation framework for colonoscopy using a new haptic device.

We have developed a multi-threaded framework for colonoscopy simulation utilising OpenGL with an interface to a real-time prototype colonoscopy haptic device. A modular framework has enabled us to support multiple haptic devices and efficiently integrate new research into physically based modelling of the colonoscope, colon and surrounding organs. The framework supports GPU accelerated algorithms as runtime modules, allowing the real-time calculations required for haptic feedback.

Ligament balancing in TKA: evaluation of a force-sensing device and the influence of patellar eversion and ligament release.

Ligament balancing in total knee arthroplasty may have an important influence on joint stability and prosthesis lifetime. In order to provide quantitative information and assistance during ligament balancing, a device that intraoperatively measures knee joint forces and moments was developed. Its performance and surgical advantages were evaluated on six cadaver specimens mounted on a knee joint loading apparatus allowing unconstrained knee motion as well as compression and varus-valgus loading. Four different experiments were performed on each specimen. (1) Knee joints were axially loaded. Comparison between applied and measured compressive forces demonstrated the accuracy and reliability of in situ measurements (1.8N). (2) Assessment of knee stability based on condyle contact forces or varus-valgus moments were compared to the current surgical method (difference of varus-valgus loads causing condyle lift-off). The force-based approach was equivalent to the surgical method while the moment-based, which is considered optimal, showed a tendency of lateral imbalance. (3) To estimate the importance of keeping the patella in its anatomical position during imbalance assessment, the effect of patellar eversion on the mediolateral distribution of tibiofemoral contact forces was measured. One fourth of the contact force induced by the patellar load was shifted to the lateral compartment. (4) The effect of minor and major medial collateral ligament releases was biomechanically quantified. On average, the medial contact force was reduced by 20% and 46%, respectively. Large variation among specimens reflected the difficulty of ligament release and the need for intraoperative force monitoring. This series of experiments thus demonstrated the device's potential to improve ligament balancing and survivorship of total knee arthroplasty.

Haptic interface module for hysteroscopy simulator system.

A fundamental element of a surgical simulator system aiming at high fidelity is the surgical instrument to be used by the simulation. The input device, as well as haptic feedback from the system should resemble the real environment. In this paper we describe all elements of the haptic interface module developed for a hysteroscopy simulator.

TWIICE - A lightweight lower-limb exoskeleton for complete paraplegics.

This paper introduces TWIICE, a lower-limb exoskeleton that enables people suffering from complete paraplegia to stand up and walk again. TWIICE provides complete mobilization of the lower-limbs, which is a first step toward enabling the user to regain independence in activities of the daily living. The tasks it can perform include level and inclined walking (up to 20° slope), stairs ascent and descent, sitting on a seat, and standing up. Participation in the world's first Cybathlon (Zurich, 2016) demonstrated good performance at these demanding tasks. In this paper, we describe the implementation details of the device and comment on preliminary results from a single user case study.

Positioning the endoscope in laparoscopic surgery by foot: Influential factors on surgeons' performance in virtual trainer.

We have investigated how surgeons can use the foot to position a laparoscopic endoscope, a task that normally requires an extra assistant. Surgeons need to train in order to exploit the possibilities offered by this new technique and safely manipulate the endoscope together with the hands movements. A realistic abdominal cavity has been developed as training simulator to investigate this multi-arm manipulation. In this virtual environment, the surgeon's biological hands are modelled as laparoscopic graspers while the viewpoint is controlled by the dominant foot. 23 surgeons and medical students performed single-handed and bimanual manipulation in this environment. The results show that residents had superior performance compared to both medical students and more experienced surgeons, suggesting that residency is an ideal period for this training. Performing the single-handed task improves the performance in the bimanual task, whereas the converse was not true.

EXiO-A Brain-Controlled Lower Limb Exoskeleton for Rhesus Macaques.

Recent advances in the field of brain-machine interfaces (BMIs) have demonstrated enormous potential to shape the future of rehabilitation and prosthetic devices. Here, a lower-limb exoskeleton controlled by the intracortical activity of an awake behaving rhesus macaque is presented as a proof-of-concept for a locomotorBMI. A detailed description of the mechanical device, including its innovative features and first experimental results, is provided. During operation, BMI-decoded position and velocity are directly mapped onto the bipedal exoskeleton's motions, which then move the monkey's legs as the monkey remains physicallypassive. To meet the unique requirements of such an application, the exoskeleton's features include: high output torque with backdrivable actuation, size adjustability, and safe user-robot interface. In addition, a novel rope transmission is introduced and implemented. To test the performance of the exoskeleton, a mechanical assessment was conducted, which yielded quantifiable results for transparency, efficiency, stiffness, and tracking performance. Usage under both brain control and automated actuation demonstrates the device's capability to fulfill the demanding needs of this application. These results lay the groundwork for further advancement in BMI-controlled devices for primates including humans.

A haptic device for guide wire in interventional radiology procedures.

Interventional Radiology (IR) is a minimally invasive procedure where thin tubular instruments, guide wires and catheters, are steered through the patient's vascular system under X-ray imaging. In order to perform these procedures, a radiologist has to be trained to master hand-eye coordination, instrument manipulation and procedure protocols. The existing simulation systems all have major drawbacks: the use of modified instruments, unrealistic insertion lengths, high inertia of the haptic device that creates a noticeably degraded dynamic behavior or excessive friction that is not properly compensated for. In this paper we propose a quality training environment dedicated to IR. The system is composed of a virtual reality (VR) simulation of the patient's anatomy linked to a robotic interface providing haptic force feedback. This paper focuses on the requirements, design and prototyping of a specific haptic interface for guide wires.

Highly-realistic, immersive training environment for hysteroscopy.

The primary driving application of our current research is the development of a generic surgical training simulator for hysteroscopy. A key target is to go beyond rehearsal of basic manipulative skills, and enable training of procedural skills like decision making and problem solving. In this respect, the sense of presence plays an important role in the achievable training effect. To enable user immersion into the training environment, the surrounding and interaction metaphors should be the same as during the real intervention. To this end, we replicated an OR in our lab, provided standard hysteroscopic tools for interaction, and generate a new virtual scene for every session. In this setting, the training starts, as soon as the trainees enter the OR, and ends when they leave the room.