A learning robot for cognitive camera control in minimally invasive surgery.

BACKGROUND: We demonstrate the first self-learning, context-sensitive, autonomous camera-guiding robot applicable to minimally invasive surgery. The majority of surgical robots nowadays are telemanipulators without autonomous capabilities. Autonomous systems have been developed for laparoscopic camera guidance, however following simple rules and not adapting their behavior to specific tasks, procedures, or surgeons. METHODS: The herein presented methodology allows different robot kinematics to perceive their environment, interpret it according to a knowledge base and perform context-aware actions. For training, twenty operations were conducted with human camera guidance by a single surgeon. Subsequently, we experimentally evaluated the cognitive robotic camera control. A VIKY EP system and a KUKA LWR 4 robot were trained on data from manual camera guidance after completion of the surgeon's learning curve. Second, only data from VIKY EP were used to train the LWR and finally data from training with the LWR were used to re-train the LWR. RESULTS: The duration of each operation decreased with the robot's increasing experience from 1704 s ± 244 s to 1406 s ± 112 s, and 1197 s. Camera guidance quality (good/neutral/poor) improved from 38.6/53.4/7.9 to 49.4/46.3/4.1% and 56.2/41.0/2.8%. CONCLUSIONS: The cognitive camera robot improved its performance with experience, laying the foundation for a new generation of cognitive surgical robots that adapt to a surgeon's needs.

3D Perception Technologies for Surgical Operating Theatres.

3D Perception technologies have been explored in various fields. This paper explores the application of such technologies for surgical operating theatres. Clinical applications can be found in workflow detection, tracking and analysis, collision avoidance with medical robots, perception of interaction between participants of the operation, training of the operation room crew, patient calibration and many more. In this paper a complete perception solution for the operating room is shown. The system is based on the ToF technology integrated to the Microsoft Kinect One implements a multi camera approach. Special emphasize is put on the tracking of the personnel and the evaluation of the system performance and accuracy.

Haptic feedback in OP:Sense - augmented reality in telemanipulated robotic surgery.

In current research, haptic feedback in robot assisted interventions plays an important role. However most approaches to haptic feedback only regard the mapping of the current forces at the surgical instrument to the haptic input devices, whereas surgeons demand a combination of medical imaging and telemanipulated robotic setups. In this paper we describe how this feature is integrated in our robotic research platform OP:Sense. The proposed method allows the automatic transfer of segmented imaging data to the haptic renderer and therefore allows enriching the haptic feedback with virtual fixtures based on imaging data. Anatomical structures are extracted from pre-operative generated medical images or virtual walls are defined by the surgeon inside the imaging data. Combining real forces with virtual fixtures can guide the surgeon to the regions of interest as well as helps to prevent the risk of damage to critical structures inside the patient. We believe that the combination of medical imaging and telemanipulation is a crucial step for the next generation of MIRS-systems.

Ex vivo accuracy evaluation for robot assisted laser bone ablation.

BACKGROUND: Cutting bony tissue using short-pulsed laser ablation enables contact-free processing in arbitrary shapes and with considerably smaller incision widths compared with mechanical tools. This precise method necessitates assistance by robotic surgery. METHODS: Using a prototype system for robot assisted laser bone ablation, the complete workflow was evaluated. Planning of cutting incisions was performed based on CT datasets of an ex vivo bone of a pig. After registration the preplanned cutting was executed autonomously by the robot assisted laser ablation system. RESULTS: Evaluation of post-operative measurements revealed an overall positioning accuracy of less than 0.5 mm. CONCLUSION: Robot assisted laser bone ablation has the potential to revolutionize surgery, especially in those interventions where the accuracy achievable manually is not sufficient.

Computer- and robot-aided head surgery.

In this paper new methods and devices for computer and robot based head surgery are presented. A computer based planning system for CMF-surgery allows the surgeon to plan complex trajectories on the head of the patient for operations where bone segments were cut out and shifted. Different registration methods have been developed and tested. A surgical robot system for bone cutting on the head has been developed and evaluated at the patient in the operating theatre. In future, laser cutting of bones with a robot will be seen as a new powerful method for robot based surgery. A 3D augmented reality system will assist the surgeon in the future by augmenting virtual anatomical structure into the situs.

Robot-assisted craniotomy.

In the Special Research Centre 414 of the German Research Funding (DFG, Bonn) a system for robot-assisted cranial surgery was developed. It is designed for the accurate and safe execution of craniotomies and repositioning of bone pieces. The system is intended for use in the surgical therapy of craniosynostosis. Preoperatively, CT imaging is performed. In a computerized planning system the position and shape of the intended craniotomy is intuitively planned on a virtual model of the patient's skull. Intraoperatively, after conventional removal of the covering soft tissue, the robot performs the craniotomy autonomously. Extensive testing in phantom studies and animal tests confirmed the reliability and accuracy of the system. A thorough risk analysis of the system was performed. In this paper, the first clinical use of the system on a patient is described and the clinical importance is discussed.

New augmented reality and robotic based methods for head-surgery.

Within the framework of the collaborative research centre "Information Technology in Medicine--Computer and Sensor-Aided Surgery" (SFB414) new methods for intraoperative computer assistance of surgical procedures are being developed. The developed tools will be controlled by an intraoperative host which provides interfaces to the electronic health record (EHR) and intraoperative computer assisted instruments.The interaction is based on standardised communication protocols. Plug & work functions will allow easy integration and configuration of new components. Intraoperative systems currently under development are intraoperative augmented reality (AR) using a projector and via a microscope, a planning system for definition of complex trajectories and a surgical robot system. The developed systems are under clinical evaluation and showing promising results in their application.

["Computer- and Sensor-Assisted Surgery" Special Research Area 414--goals and concepts]. / Sonderforschungsbereich 414 "Rechner- und Sensorgestützte Chirurgie"--Ziele und Konzepte.

This paper presents an overview of our research in computer assisted surgery within the "Sonderforschungsbereich 414--Information Technology in Medicine: Computer and Sensor Supported Surgery". The overall goal is to achieve improved operation methods including higher quality, more safety and also more economy due to shorter operation time and less postoperative treatment. The SFB 414 is divided into three different sections focusing on projects concerning the heart, projects supporting cranio-facial interventions and interconnecting projects that serve both categories.

[Processing medical image data for head surgery]. / Aufbereitung medizinischer Bilddaten für die Kopfchirurgie.

Three-dimensional models of the patient's anatomy are the basis for modern applications in computer-assisted surgery or radiology. To create these models the data is processed in a series of steps that transforms the initial raw data (generally CT/MRT volume images) into the three-dimensional models (e.g. triangle meshes). In doing so, a multitude of parameters adjustments, interactions and decisions are necessary to be made by the physician in order to create the models adapted to the specific needs of the patient and to the surgery. The quality of the models therefore strongly depends on that process chain. As a result of the high requirements on the quality and security of a clinical application the quality of the models itself and the capability to use them flexibly are essential. This work present a method to adjust and optimise the creation of the models and to asses their quality afterwards.

[Expanded reality in head surgery]. / Erweiterte Realitat in der Kopfchirurgie.

The preoperative planning of complex craniofacial surgical interventions is increasingly realized with the aid of appropriate planning systems in order to achieve high precision, minimal invasion and protection of risk areas. But the most important step from the planning to the actual intervention consists of providing the planning data intraoperatively in a reasonable and easy to handle way. In this regard, augmented reality is one of the most challenging and promising techniques to solve this problem. In this paper we present two different systems based on see-through glasses resp. a common video projector to directly visualize the planning data in the surgeon's field of view. The work is funded by the 'Deutsche Forschungsgemeinschaft (DFG)' and Stryker Leibinger Freiburg/Germany.

Analysing the variations of shapes based on surface-models.

In this paper we present a freely available application with which surface models, e.g. triangle meshes can be registered rigidly as well as by using similarity or affine transformations. Registered surfaces can be analysed, i.e. their difference can be calculated and visualised in various ways. The software is addressing the problem of evaluating shape differences of generated surface models due to mesh relaxation, decimation, object (in our case: patient) variation, different co-ordinate systems, etc. We invite all interested research personnel to download and use the software.

[3D-ultrasound: a valuable adjunct for therapy planning and follow-up of head and neck tumours]. / 3D-Ultraschall: Eine wertvolle Ergänzung bei Therapie-planung und Verlaufskontrolle von Kopf-Hals-Tumoren.

AIM: The three-dimensional visualization of orofacial tumours and adjacent organs at risk of infiltration is an important requirement for staging, therapy planning and follow-up. Artifacts from the mandible or metal implants often reduce the diagnostic power of computed tomography (CT). The value of 3D-ultrasound (3D-US) in respect to the standard methods CT and magnetic resonance imaging (MRI) was therefore analyzed. METHOD: Between 9/97 and 10/99 the visualization of orofacial tumours in 243 patients by 3D-US was examined, classified on a five-point scale and compared to the corresponding CT and MRI scans. RESULTS: Complete visualization of the tumours was possible in 85.6 % by 3D-US, whereas the rates of MRI and CT were lower with 77.4 % and 61.3 % respectively, mainly because of artifacts. The best combinations of methods, 3D-US + CT (96.7 %) and 3D-US + MRI (95.1 %) gave almost equivalent results, whereas CT + MRI (83.5 %) was inferior. In 2.5 % of the cases none of the methods produced adequate results. CONCLUSION: By free selection of sectional planes and direct correlation to the clinical findings 3D-ultrasound can improve staging, therapy planning and follow-up of orofacial tumours, especially in the case of small tumour size, dental crowns/inlays, metal implants or contraindications to MRI.

[Ultrasonography of tumors of the locomotor system]. / Sonographie von Tumoren des Bewegungsapparats.

Sonography is an integral part of primary tumor diagnosis and follow-up for the great majority of organ systems. However, its value in the field of space-occupying processes of the locomotor system, especially of malignant bone tumors, has mostly been underestimated. The sonographic examination has to investigate the whole tumor region and the corresponding lymph nodes statically and dynamically. The examination procedure should be standardized and the documentation reliable. Evaluation criteria are the localization, dimensions, and volume of the tumors, echogenicity and homogeneity, peri- and intratumoral vascularization (vessel density and architecture), borders of the tumor, and neighboring structures. Pathologic changes not only of the soft tissues but also of the bones can be evaluated sonographically. Even subtle analysis does not permit definitive assessment of tumor status. Taking its physical limitations into consideration, high-resolution sonography, enhanced by color/PowerDoppler and three-dimensional techniques, is a valuable adjunct to improve diagnosis, therapy planning, monitoring, and posttherapeutic care of tumors of the locomotor system.

Statistical analysis of the morphology of three-dimensional objects and pathologic structures using spherical harmonics.

To support diagnosis and therapy, it is a fundamental aim of medical image processing to describe morphological characteristics of pathological structures or image objects in general. Different authors propose quantitative methods of description like bounding boxes[1], fourier descriptors[2] or contour moments[3]. Unfortunately, these methods either don't supply a complete, respectively precise description of the object or only operate on two-dimensional images. Among the range of application are systems to classify lung nodules [4] or to help the diagnosis of brain tumors [5]. In this paper we present a method to analyze the morphology or shape of any three-dimensional object in order to describe it mathematically well-defined. We show how the description can be used to perform statistical operations on morphologies. The method presented in this paper was developed to assist the planning of craniofacial surgery. We analyze the shape of a given set of skull CT-data and use the mathematical description to statistically calculate the average shape of the skulls.

Virtual simulation system for collision avoidance for medical robot.

For the collision avoidance with a medical robot with 6 DOF a virtual simulation system is presented. Manipulator and obstacles are modelled by geometric primitives. Collisions are detected in the Cartesian workspace by hierarchical distance computation based on the given CAD model. The application initially being addressed is maxillofacial surgery, where the safety of the patient is the main requirement,because of the closeness to vital parts. The simulation system allows the surgeon to check up the trajectory of the robot before the current operation begins.