Heidelberg colorectal data set for surgical data science in the sensor operating room.

Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a specific emphasis on method robustness and generalization capabilities. Our data set comprises 30 laparoscopic videos and corresponding sensor data from medical devices in the operating room for three different types of laparoscopic surgery. Annotations include surgical phase labels for all video frames as well as information on instrument presence and corresponding instance-wise segmentation masks for surgical instruments (if any) in more than 10,000 individual frames. The data has successfully been used to organize international competitions within the Endoscopic Vision Challenges 2017 and 2019.

Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the ROBUST-MIS 2019 challenge.

Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions. In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).

Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy.

OBJECTIVE: Surgical data science is evolving into a research field that aims to observe everything occurring within and around the treatment process to provide situation-aware data-driven assistance. In the context of endoscopic video analysis, the accurate classification of organs in the field of view of the camera proffers a technical challenge. Herein, we propose a new approach to anatomical structure classification and image tagging that features an intrinsic measure of confidence to estimate its own performance with high reliability and which can be applied to both RGB and multispectral imaging (MI) data. METHODS: Organ recognition is performed using a superpixel classification strategy based on textural and reflectance information. Classification confidence is estimated by analyzing the dispersion of class probabilities. Assessment of the proposed technology is performed through a comprehensive in vivo study with seven pigs. RESULTS: When applied to image tagging, mean accuracy in our experiments increased from 65% (RGB) and 80% (MI) to 90% (RGB) and 96% (MI) with the confidence measure. CONCLUSION: Results showed that the confidence measure had a significant influence on the classification accuracy, and MI data are better suited for anatomical structure labeling than RGB data. SIGNIFICANCE: This paper significantly enhances the state of art in automatic labeling of endoscopic videos by introducing the use of the confidence metric, and by being the first study to use MI data for in vivo laparoscopic tissue classification. The data of our experiments will be released as the first in vivo MI dataset upon publication of this paper.

An ad hoc three dimensionally printed tool facilitates intraesophageal suturing in experimental surgery.

BACKGROUND: Three-dimensional printing (3DP) has become popular for development of anatomic models, preoperative planning, and production of tailored implants. A novel laparoscopic, transgastric procedure for distal esophageal mucosectomy was developed. During this procedure, a space holder had to be introduced into the distal esophagus for exposure during suturing. The production process and evaluation of a 3DP space holder are described herein. MATERIALS AND METHODS: Computer-aided design software was used to develop models printed from polylactic acid. The prototype was adapted after testing in a cadaveric model. Subsequently, the device was evaluated in a nonsurvival porcine model. A mucosal purse-string suture was placed as orally as possible in the esophagus, in the intervention group with and in the control group without use of the tool (n = 8 each). The distance of the stitches from the Z-line was measured. The variability of stitches indicated the suture quality. RESULTS: The median maximum distance from the Z-line to purse-string suture was larger in the intervention group (5.0 [3.3-6.4] versus 2.4 [2.0-4.1] cm; P = 0.013). The time taken to place the sutures was shorter in the control group (P < 0.001). Stitch variance tended to be greater in the intervention group (2.3 [0.9-2.5] versus 0.7 [0.2-0.4] cm; P = 0.051). The time required for design and production of a tailored tool was less than 24 h. CONCLUSIONS: 3DP in experimental surgery enables rapid production, permits repeated adaptation until a tailored tool is obtained, and ensures independence from industrial partners. With the aid of the space holder more orally located esophageal lesions came within reach.

Paradigm shift: cognitive surgery.

In the last hundred years surgery has experienced a dramatic increase of scientific knowledge and innovation. The need to consider best available evidence and to apply technical innovations, such as minimally invasive approaches, challenges the surgeon both intellectually and manually. In order to overcome this challenge, computer scientists and surgeons within the interdisciplinary field of "cognitive surgery" explore and innovate new ways of data processing and management. This article gives a general overview of the topic and outlines selected pre-, intra- and postoperative applications. It explores the possibilities of new intelligent devices and software across the entire treatment process of patients ending in the consideration of an "Intelligent Hospital" or "Hospital 4.0", in which the borders between IT infrastructures, medical devices, medical personnel and patients are bridged by technology. Thereby, the "Hospital 4.0" is an intelligent system, which gives the right information, at the right time, at the right place to the individual stakeholder and thereby helps to decrease complications and improve clinical processes as well as patient outcome.