LARLUS: laparoscopic augmented reality from laparoscopic ultrasound.

PURPOSE: This research endeavors to improve tumor localization in minimally invasive surgeries, a challenging task primarily attributable to the absence of tactile feedback and limited visibility. The conventional solution uses laparoscopic ultrasound (LUS) which has a long learning curve and is operator-dependent. METHODS: The proposed approach involves augmenting LUS images onto laparoscopic images to improve the surgeon's ability to estimate tumor and internal organ anatomy. This augmentation relies on LUS pose estimation and filtering. RESULTS: Experiments conducted with clinical data exhibit successful outcomes in both the registration and augmentation of LUS images onto laparoscopic images. Additionally, noteworthy results are observed in filtering, leading to reduced flickering in augmentations. CONCLUSION: The outcomes reveal promising results, suggesting the potential of LUS augmentation in surgical images to assist surgeons and serve as a training tool. We have used the LUS probe's shaft to disambiguate the rotational symmetry. However, in the long run, it would be desirable to find more convenient solutions.

Large-Area and Low-Cost Force/Tactile Capacitive Sensor for Soft Robotic Applications.

This paper presents a novel design and development of a low-cost and multi-touch sensor based on capacitive variations. This new sensor is very flexible and easy to fabricate, making it an appropriate choice for soft robot applications. Materials (conductive ink, silicone, and control boards) used in this sensor are inexpensive and easily found in the market. The proposed sensor is made of a wafer of different layers, silicone layers with electrically conductive ink, and a pressure-sensitive conductive paper sheet. Previous approaches like e-skin can measure the contact point or pressure of conductive objects like the human body or finger, while the proposed design enables the sensor to detect the object's contact point and the applied force without considering the material conductivity of the object. The sensor can detect five multi-touch points at the same time. A neural network architecture is used to calibrate the applied force with acceptable accuracy in the presence of noise, variation in gains, and non-linearity. The force measured in real time by a commercial precise force sensor (ATI) is mapped with the produced voltage obtained by changing the layers' capacitance between two electrode layers. Finally, the soft robot gripper embedding the suggested tactile sensor is utilized to grasp an object with position and force feedback signals.

Adaptive Deformation Control for Elastic Linear Objects.

This paper addresses the general problem of deformable linear object manipulation. The main application we consider is in the field of agriculture, for plant grasping, but may have interests in other tasks such as human daily activities and industrial production. We specifically consider an elastic linear object where one of its endpoints is fixed, and another point can be grasped by a robotic arm. To deal with the mentioned problem, we propose a model-free method to control the state of an arbitrary point that can be at any place along the object's length. Our approach allows the robot to manipulate the object without knowing any model parameters or offline information of the object's deformation. An adaptive control strategy is proposed for regulating the state of any point automatically deforming the object into the desired location. A control law is developed to regulate the object's shape thanks to the adaptive estimation of the system parameters and its states. This method can track a desired manipulation trajectory to reach the target point, which leads to a smooth deformation without drastic changes. A Lyapunov-based argument is presented for the asymptotic convergence of the system that shows the process's stability and convergence to desired state values. To validate the controller, numerical simulations involving two different deformation models are conducted, and performances of the proposed algorithm are investigated through full-scale experiments.

General Framework for the Optimization of the Human-Robot Collaboration Decision-Making Process Through the Ability to Change Performance Metrics.

This paper proposes a new decision-making framework in the context of Human-Robot Collaboration (HRC). State-of-the-art techniques consider the HRC as an optimization problem in which the utility function, also called reward function, is defined to accomplish the task regardless of how well the interaction is performed. When the performance metrics are considered, they cannot be easily changed within the same framework. In contrast, our decision-making framework can easily handle the change of the performance metrics from one case scenario to another. Our method treats HRC as a constrained optimization problem where the utility function is split into two main parts. Firstly, a constraint defines how to accomplish the task. Secondly, a reward evaluates the performance of the collaboration, which is the only part that is modified when changing the performance metrics. It gives control over the way the interaction unfolds, and it also guarantees the adaptation of the robot actions to the human ones in real-time. In this paper, the decision-making process is based on Nash Equilibrium and perfect-information extensive form from game theory. It can deal with collaborative interactions considering different performance metrics such as optimizing the time to complete the task, considering the probability of human errors, etc. Simulations and a real experimental study on "an assembly task" -i.e., a game based on a construction kit-illustrate the effectiveness of the proposed framework.

Generation of Tactile Data From 3D Vision and Target Robotic Grasps.

Tactile perception is a rich source of information for robotic grasping: it allows a robot to identify a grasped object and assess the stability of a grasp, among other things. However, the tactile sensor must come into contact with the target object in order to produce readings. As a result, tactile data can only be attained if a real contact is made. We propose to overcome this restriction by employing a method that models the behaviour of a tactile sensor using 3D vision and grasp information as a stimulus. Our system regresses the quantified tactile response that would be experienced if this grasp were performed on the object. We experiment with 16 items and 4 tactile data modalities to show that our proposal learns this task with low error.

Hierarchical, Dense and Dynamic 3D Reconstruction Based on VDB Data Structure for Robotic Manipulation Tasks.

This paper presents a novel approach to implement hierarchical, dense and dynamic reconstruction of 3D objects based on the VDB (Variational Dynamic B + Trees) data structure for robotic applications. The scene reconstruction is done by the integration of depth-images using the Truncated Signed Distance Field (TSDF). The proposed reconstruction method is based on dynamic trees in order to provide similar reconstruction results to the current state-of-the-art methods (i.e., complete volumes, hashing voxels and hierarchical volumes) in terms of execution time but with a direct multi-level representation that remains real-time. This representation provides two major advantages: it is a hierarchical and unbounded space representation. The proposed method is optimally implemented to be used on a GPU architecture, exploiting the parallelism skills of this hardware. A series of experiments will be presented to prove the performance of this approach in a robot arm platform.

Blind Manipulation of Deformable Objects Based on Force Sensing and Finite Element Modeling.

In this paper, we present a novel pipeline to simultaneously estimate and manipulate the deformation of an object using only force sensing and an FEM model. The pipeline is composed of a sensor model, a deformation model and a pose controller. The sensor model computes the contact forces that are used as input to the deformation model which updates the volumetric mesh of a manipulated object. The controller then deforms the object such that a given pose on the mesh reaches a desired pose. The proposed approach is thoroughly evaluated in real experiments using a robot manipulator and a force-torque sensor to show its accuracy in estimating and manipulating deformations without the use of vision sensors.

Dual filtering in operational and joint spaces for reaching and grasping.

To study human movement generation, as well as to develop efficient control algorithms for humanoid or dexterous manipulation robots, overcoming the limits and drawbacks of inverse-kinematics-based methods is needed. Adequate methods must deal with high dimensionality, uncertainty, and must perform in real time (constraints shared by robots and humans). This paper introduces a Bayesian filtering method, hierarchically applied in the operational and joint spaces to break down the complexity of the problem. The method is validated in simulation on a robotic arm in a cluttered environment, with up to 51 degrees of freedom.

Efficient iterative pose estimation using an invariant to rotations.

This paper deals with pose estimation using an iterative scheme. We show that using adequate visual information, pose estimation can be performed iteratively with only three independent unknowns, which are the translation parameters. Specifically, an invariant to rotational motion is used to estimate the camera position. In addition, an adequate transformation is applied to the proposed invariant to decrease the nonlinearities between the variations in image space and 3-D space. Once the camera position is estimated, we show that the rotation can be estimated efficiently using two different direct methods. The proposed approach is compared against two other methods from the literature. The results show that using our method, pose tracking in image sequences and the convergence rate for randomly generated poses are improved.

High level functions for the intuitive use of an assistive robot.

This document presents the research project ARMEN (Assistive Robotics to Maintain Elderly People in a Natural environment), aimed at the development of a user friendly robot with advanced functions for assistance to elderly or disabled persons at home. Focus is given to the robot SAM (Smart Autonomous Majordomo) and its new features of navigation, manipulation, object recognition, and knowledge representation developed for the intuitive supervision of the robot. The results of the technical evaluations show the value and potential of these functions for practical applications. The paper also documents the details of the clinical evaluations carried out with elderly and disabled persons in a therapeutic setting to validate the project.