A Low-Complexity Brain-Computer Interface for High-Complexity Robot Swarm Control.

A brain-computer interface (BCI) is a system that allows a human operator to use only mental commands in controlling end effectors that interact with the world around them. Such a system consists of a measurement device to record the human user's brain activity, which is then processed into commands that drive a system end effector. BCIs involve either invasive measurements which allow for high-complexity control but are generally infeasible, or noninvasive measurements which offer lower quality signals but are more practical to use. In general, BCI systems have not been developed that efficiently, robustly, and scalably perform high-complexity control while retaining the practicality of noninvasive measurements. Here we leverage recent results from feedback information theory to fill this gap by modeling BCIs as a communications system and deploying a human-implementable interaction algorithm for noninvasive control of a high-complexity robot swarm. We construct a scalable dictionary of robotic behaviors that can be searched simply and efficiently by a BCI user, as we demonstrate through a large-scale user study testing the feasibility of our interaction algorithm, a user test of the full BCI system on (virtual and real) robot swarms, and simulations that verify our results against theoretical models. Our results provide a proof of concept for how a large class of high-complexity effectors (even beyond robotics) can be effectively controlled by a BCI system with low-complexity and noisy inputs.

Enhanced Accuracy in Magnetic Actuation: Closed-loop Control of a Magnetic Agent with Low-Error Numerical Magnetic Model Estimation.

Magnetic actuation holds promise for wirelessly controlling small, magnetic surgical tools and may enable the next generation of ultra minimally invasive surgical robotic systems. Precise torque and force exertion are required for safe surgical operations and accurate state control. Dipole field estimation models perform well far from electromagnets but yield large errors near coils. Thus, manipulations near coils suffer from severe (10×) field modeling errors. We experimentally quantify closed-loop magnetic agent control performance by using both a highly erroneous dipole model and a more accurate numerical magnetic model to estimate magnetic forces and torques for any given robot pose in 2D. We compare experimental measurements with estimation errors for the dipole model and our finite element analysis (FEA) based model of fields near coils. With five different paths designed for this study, we demonstrate that FEA-based magnetic field modeling reduces positioning root-mean-square (RMS) errors by 48% to 79% as compared with dipole models. Models demonstrate close agreement for magnetic field direction estimation, showing similar accuracy for orientation control. Such improved magnetic modelling is crucial for systems requiring robust estimates of magnetic forces for positioning agents, particularly in force-sensitive environments like surgical manipulation.

Overcoming the Force Limitations of Magnetic Robotic Surgery: Magnetic Pulse Actuated Collisions for Tissue-Penetrating-Needle for Tetherless Interventions.

The field of magnetic robotics aims to obviate physical connections between the actuators and end-effectors. Such tetherless control may enable new ultra-minimally invasive surgical manipulations in clinical settings. While wireless actuation offers advantages in medical applications, the challenge of providing sufficient force to magnetic needles for tissue penetration remains a barrier to practical application. Applying sufficient force for tissue penetration is required for tasks such as biopsy, suturing, cutting, drug delivery, and accessing deep seated regions of complex structures in organs such as the eye. To expand the force landscape for such magnetic surgical tools, an impact-force based suture needle capable of penetrating in vitro and ex vivo samples with 3-DOF planar motion is proposed. Using custom-built 14G and 25G needles, we demonstrate generation of 410 mN penetration force, a 22.7-fold force increase with more than 20 times smaller volume compared to similar magnetically guided needles. With the MPACT-Needle, in vitro suturing of a gauze mesh onto an agar gel is demonstrated. In addition, we have reduced the tip size to 25G, which is a typical needle size for interventions in the eye, to demonstrate ex vivo penetration in a rabbit eye, mimicking procedures such as corneal injections and transscleral drug delivery.

Control of Magnetic Surgical Robots With Model-Based Simulators and Reinforcement Learning.

Magnetically manipulated medical robots are a promising alternative to current robotic platforms, allowing for miniaturization and tetherless actuation. Controlling such systems autonomously may enable safe, accurate operation. However, classical control methods require rigorous models of magnetic fields, robot dynamics, and robot environments, which can be difficult to generate. Model-free reinforcement learning (RL) offers an alternative that can bypass these requirements. We apply RL to a robotic magnetic needle manipulation system. Reinforcement learning algorithms often require long runtimes, making them impractical for many surgical robotics applications, most of which require careful, constant monitoring. Our approach first constructs a model-based simulation (MBS) on guided real-world exploration, learning the dynamics of the environment. After intensive MBS environment training, we transfer the learned behavior from the MBS environment to the real-world. Our MBS method applies RL roughly 200 times faster than doing so in the real world, and achieves a 6 mm root-mean-square (RMS) error for a square reference trajectory. In comparison, pure simulation-based approaches fail to transfer, producing a 31 mm RMS error. These results demonstrate that MBS environments are a good solution for domains where running model-free RL is impractical, especially if an accurate simulation is not available.

Interpersonal motor synergy: coworking strategy depends on task constraints.

We investigated the role of task constraints on interpersonal interactions. Twenty-one pairs of coworkers performed a finger force production task on force sensors placed at two ends of a seesaw-like apparatus and matched a combined target force of 20 N for 23 s over 10 trials. There were two experimental conditions: 1) FIXED: the seesaw apparatus was mechanically held in place so that the only task constraint was to match the 20 N resultant force, and 2) MOVING: the lever in the apparatus was allowed to rotate freely around its fulcrum, acting like a seesaw, so an additional task constraint to (implicitly) balance the resultant moment was added. We hypothesized that the additional task constraint of moment stabilization imposed on the MOVING condition would deteriorate task performance compared with the FIXED condition; however, this was rejected, as the performance of the force matching task was similar between two conditions. We also hypothesized that the central nervous systems (CNSs) would employ distinct coworking strategies or interpersonal motor synergy (IPMS) between conditions to satisfy different task constraints, which was supported by our results. Negative covariance between coworker's forces in the FIXED condition suggested a force stabilization strategy, whereas positive covariance in the MOVING condition suggested a moment stabilization strategy, implying that independent CNSs adopt distinct IPMSs depending on task constraints. We speculate that in the absence of a central neural controller, shared visual and mechanical connections between coworkers may suffice to trigger modulations in the cerebellum of each CNS to satisfy competing task constraints.NEW & NOTEWORTHY To the best of our knowledge, this is the first study to investigate the coworking behavior or IPMS when an additional task constraint is imposed. Our proposed analytical framework quantifies IPMS and allows for investigating variability in offline (i.e., across multiple repetitions) and online (i.e., across time) control, which is novel in coworking research. Understanding variability while performing a task is essential, as repeating a task is not always possible, as in therapeutic contexts.

Localization and Control of Magnetic Suture Needles in Cluttered Surgical Site with Blood and Tissue.

Real-time visual localization of needles is necessary for various surgical applications, including surgical automation and visual feedback. In this study we investigate localization and autonomous robotic control of needles in the context of our magneto-suturing system. Our system holds the potential for surgical manipulation with the benefit of minimal invasiveness and reduced patient side effects. However, the nonlinear magnetic fields produce unintuitive forces and demand delicate position-based control that exceeds the capabilities of direct human manipulation. This makes automatic needle localization a necessity. Our localization method combines neural network-based segmentation and classical techniques, and we are able to consistently locate our needle with 0.73 mm RMS error in clean environments and 2.72 mm RMS error in challenging environments with blood and occlusion. The average localization RMS error is 2.16 mm for all environments we used in the experiments. We combine this localization method with our closed-loop feedback control system to demonstrate the further applicability of localization to autonomous control. Our needle is able to follow a running suture path in (1) no blood, no tissue; (2) heavy blood, no tissue; (3) no blood, with tissue; and (4) heavy blood, with tissue environments. The tip position tracking error ranges from 2.6 mm to 3.7 mm RMS, opening the door towards autonomous suturing tasks.

Towards Autonomous Control of Magnetic Suture Needles.

This paper proposes a magnetic needle steering controller to manipulate mesoscale magnetic suture needles for executing planned suturing motion. This is an initial step towards our research objective: enabling autonomous control of magnetic suture needles for suturing tasks in minimally invasive surgery. To demonstrate the feasibility of accurate motion control, we employ a cardinally-arranged four-coil electromagnetic system setup and control magnetic suture needles in a 2-dimensional environment, i.e., a Petri dish filled with viscous liquid. Different from only using magnetic field gradients to control small magnetic agents under high damping conditions, the dynamics of a magnetic suture needle are investigated and encoded in the controller. Based on mathematical formulations of magnetic force and torque applied on the needle, we develop a kinematically constrained dynamic model that controls the needle to rotate and only translate along its central axis for mimicking the behavior of surgical sutures. A current controller of the electromagnetic system combining with closed-loop control schemes is designed for commanding the magnetic suture needles to achieve desired linear and angular velocities. To evaluate control performance of magnetic suture needles, we conduct experiments including needle rotation control, needle position control by using discretized trajectories, and velocity control by using a time-varying circular trajectory. The experiment results demonstrate our proposed needle steering controller can perform accurate motion control of mesoscale magnetic suture needles.

MagnetoSuture: Tetherless Manipulation of Suture Needles.

This paper demonstrates the feasibility of ligation and tissue penetration for surgical suturing tasks using magnetically actuated suture needles. Manipulation of suture needles in minimally invasive surgery involves using articulated manual/robotic tools for needle steering and controlling needle-tissue or thread-tissue interactions. The large footprints of conventional articulated surgical tools significantly increase surgical invasiveness, potentially leading to longer recovery times, tissue damage, scarring, or associated infections. Aiming to address these issues, we investigate the feasibility of using magnetic fields to tetherlessly steer suture needles. The primary challenge of such a concept is to provide sufficient force for tissue penetration and ligation. In this work, we demonstrate proof-of-concept capabilities using the MagnetoSuture™ system, performing tissue penetration and ligation tasks using ex vivo tissues, customized NdFeB suture needles with attached threads, and remote-controlled magnetic fields. To evaluate the system performance, we conducted experiments demonstrating tetherless recreation of a purse string suture pattern, ligation of an excised segment of a rat intestine, and penetration of rat intestines.