Active learning for extracting surgomic features in robot-assisted minimally invasive esophagectomy: a prospective annotation study.

BACKGROUND: With Surgomics, we aim for personalized prediction of the patient's surgical outcome using machine-learning (ML) on multimodal intraoperative data to extract surgomic features as surgical process characteristics. As high-quality annotations by medical experts are crucial, but still a bottleneck, we prospectively investigate active learning (AL) to reduce annotation effort and present automatic recognition of surgomic features. METHODS: To establish a process for development of surgomic features, ten video-based features related to bleeding, as highly relevant intraoperative complication, were chosen. They comprise the amount of blood and smoke in the surgical field, six instruments, and two anatomic structures. Annotation of selected frames from robot-assisted minimally invasive esophagectomies was performed by at least three independent medical experts. To test whether AL reduces annotation effort, we performed a prospective annotation study comparing AL with equidistant sampling (EQS) for frame selection. Multiple Bayesian ResNet18 architectures were trained on a multicentric dataset, consisting of 22 videos from two centers. RESULTS: In total, 14,004 frames were tag annotated. A mean F1-score of 0.75 ± 0.16 was achieved for all features. The highest F1-score was achieved for the instruments (mean 0.80 ± 0.17). This result is also reflected in the inter-rater-agreement (1-rater-kappa > 0.82). Compared to EQS, AL showed better recognition results for the instruments with a significant difference in the McNemar test comparing correctness of predictions. Moreover, in contrast to EQS, AL selected more frames of the four less common instruments (1512 vs. 607 frames) and achieved higher F1-scores for common instruments while requiring less training frames. CONCLUSION: We presented ten surgomic features relevant for bleeding events in esophageal surgery automatically extracted from surgical video using ML. AL showed the potential to reduce annotation effort while keeping ML performance high for selected features. The source code and the trained models are published open source.

Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark.

PURPOSE: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data single-center video dataset. In this work we investigated the generalizability of phase recognition algorithms in a multicenter setting including more difficult recognition tasks such as surgical action and surgical skill. METHODS: To achieve this goal, a dataset with 33 laparoscopic cholecystectomy videos from three surgical centers with a total operation time of 22 h was created. Labels included framewise annotation of seven surgical phases with 250 phase transitions, 5514 occurences of four surgical actions, 6980 occurences of 21 surgical instruments from seven instrument categories and 495 skill classifications in five skill dimensions. The dataset was used in the 2019 international Endoscopic Vision challenge, sub-challenge for surgical workflow and skill analysis. Here, 12 research teams trained and submitted their machine learning algorithms for recognition of phase, action, instrument and/or skill assessment. RESULTS: F1-scores were achieved for phase recognition between 23.9% and 67.7% (n = 9 teams), for instrument presence detection between 38.5% and 63.8% (n = 8 teams), but for action recognition only between 21.8% and 23.3% (n = 5 teams). The average absolute error for skill assessment was 0.78 (n = 1 team). CONCLUSION: Surgical workflow and skill analysis are promising technologies to support the surgical team, but there is still room for improvement, as shown by our comparison of machine learning algorithms. This novel HeiChole benchmark can be used for comparable evaluation and validation of future work. In future studies, it is of utmost importance to create more open, high-quality datasets in order to allow the development of artificial intelligence and cognitive robotics in surgery.

Surgomics: personalized prediction of morbidity, mortality and long-term outcome in surgery using machine learning on multimodal data.

BACKGROUND: Personalized medicine requires the integration and analysis of vast amounts of patient data to realize individualized care. With Surgomics, we aim to facilitate personalized therapy recommendations in surgery by integration of intraoperative surgical data and their analysis with machine learning methods to leverage the potential of this data in analogy to Radiomics and Genomics. METHODS: We defined Surgomics as the entirety of surgomic features that are process characteristics of a surgical procedure automatically derived from multimodal intraoperative data to quantify processes in the operating room. In a multidisciplinary team we discussed potential data sources like endoscopic videos, vital sign monitoring, medical devices and instruments and respective surgomic features. Subsequently, an online questionnaire was sent to experts from surgery and (computer) science at multiple centers for rating the features' clinical relevance and technical feasibility. RESULTS: In total, 52 surgomic features were identified and assigned to eight feature categories. Based on the expert survey (n = 66 participants) the feature category with the highest clinical relevance as rated by surgeons was "surgical skill and quality of performance" for morbidity and mortality (9.0 ± 1.3 on a numerical rating scale from 1 to 10) as well as for long-term (oncological) outcome (8.2 ± 1.8). The feature category with the highest feasibility to be automatically extracted as rated by (computer) scientists was "Instrument" (8.5 ± 1.7). Among the surgomic features ranked as most relevant in their respective category were "intraoperative adverse events", "action performed with instruments", "vital sign monitoring", and "difficulty of surgery". CONCLUSION: Surgomics is a promising concept for the analysis of intraoperative data. Surgomics may be used together with preoperative features from clinical data and Radiomics to predict postoperative morbidity, mortality and long-term outcome, as well as to provide tailored feedback for surgeons.

A Platform and Multisided Market for Translational, Software-Defined Medical Procedures in the Operating Room (OP 4.1): Proof-of-Concept Study.

BACKGROUND: Although digital and data-based technologies are widespread in various industries in the context of Industry 4.0, the use of smart connected devices in health care is still in its infancy. Innovative solutions for the medical environment are affected by difficult access to medical device data and high barriers to market entry because of proprietary systems. OBJECTIVE: In the proof-of-concept project OP 4.1, we show the business viability of connecting and augmenting medical devices and data through software add-ons by giving companies a technical and commercial platform for the development, implementation, distribution, and billing of innovative software solutions. METHODS: The creation of a central platform prototype requires the collaboration of several independent market contenders, including medical users, software developers, medical device manufacturers, and platform providers. A dedicated consortium of clinical and scientific partners as well as industry partners was set up. RESULTS: We demonstrate the successful development of the prototype of a user-centric, open, and extensible platform for the intelligent support of processes starting with the operating room. By connecting heterogeneous data sources and medical devices from different manufacturers and making them accessible for software developers and medical users, the cloud-based platform OP 4.1 enables the augmentation of medical devices and procedures through software-based solutions. The platform also allows for the demand-oriented billing of apps and medical devices, thus permitting software-based solutions to fast-track their economic development and become commercially successful. CONCLUSIONS: The technology and business platform OP 4.1 creates a multisided market for the successful development, implementation, distribution, and billing of new software solutions in the operating room and in the health care sector in general. Consequently, software-based medical innovation can be translated into clinical routine quickly, efficiently, and cost-effectively, optimizing the treatment of patients through smartly assisted procedures.

A novel endoimaging system for endoscopic 3D reconstruction in bladder cancer patients.

INTRODUCTION: The methods employed to document cystoscopic findings in bladder cancer patients lack accuracy and are subject to observer variability. We propose a novel endoimaging system and an online documentation platform to provide post-procedural 3D bladder reconstructions for improved diagnosis, management and follow-up. MATERIAL AND METHODS: The RaVeNNA4pi consortium is comprised of five industrial partners, two university hospitals and two technical institutes. These are grouped into hardware, software and clinical partners according to their professional expertise. The envisaged endoimaging system consists of an innovative cystoscope that generates 3D bladder reconstructions allowing users to remotely access a cloud-based centralized database to visualize individualized 3D bladder models from previous cystoscopies archived in DICOM format. RESULTS: Preliminary investigations successfully tracked the endoscope's rotational and translational movements. The structure-from-motion pipeline was tested in a bladder phantom and satisfactorily demonstrated 3D reconstructions of the processing sequence. AI-based semantic image segmentation achieved a 0.67 dice-score-coefficient over all classes. An online-platform allows physicians and patients to digitally visualize endoscopic findings by navigating a 3D bladder model. CONCLUSIONS: Our work demonstrates the current developments of a novel endoimaging system equipped with the potential to generate 3D bladder reconstructions from cystoscopy videos and AI-assisted automated detection of bladder tumors.

Surgical data science - from concepts toward clinical translation.

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

Heidelberg colorectal data set for surgical data science in the sensor operating room.

Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a specific emphasis on method robustness and generalization capabilities. Our data set comprises 30 laparoscopic videos and corresponding sensor data from medical devices in the operating room for three different types of laparoscopic surgery. Annotations include surgical phase labels for all video frames as well as information on instrument presence and corresponding instance-wise segmentation masks for surgical instruments (if any) in more than 10,000 individual frames. The data has successfully been used to organize international competitions within the Endoscopic Vision Challenges 2017 and 2019.

Service-Oriented Medical Device Connectivity: Particular Standards for Endoscopic Surgery.

To translate recent advances in medical device interoperability research into clinical practice, standards are being developed that specify precise requirements towards the network representation of particular medical devices connecting through ISO/IEEE 11073 SDC. The present contribution supplements this protocol standard with specific models for endoscopic camera systems, light sources, insufflators, and pumps. Through industry consensus, these new standards provide modular means to describe the devices' capabilities and modes of interaction in a service-oriented medical device communication architecture. This enables seamless data exchange and the potential for new assistive systems to support the caregiver.

Prediction of laparoscopic procedure duration using unlabeled, multimodal sensor data.

PURPOSE: The course of surgical procedures is often unpredictable, making it difficult to estimate the duration of procedures beforehand. This uncertainty makes scheduling surgical procedures a difficult task. A context-aware method that analyses the workflow of an intervention online and automatically predicts the remaining duration would alleviate these problems. As basis for such an estimate, information regarding the current state of the intervention is a requirement. METHODS: Today, the operating room contains a diverse range of sensors. During laparoscopic interventions, the endoscopic video stream is an ideal source of such information. Extracting quantitative information from the video is challenging though, due to its high dimensionality. Other surgical devices (e.g., insufflator, lights, etc.) provide data streams which are, in contrast to the video stream, more compact and easier to quantify. Though whether such streams offer sufficient information for estimating the duration of surgery is uncertain. In this paper, we propose and compare methods, based on convolutional neural networks, for continuously predicting the duration of laparoscopic interventions based on unlabeled data, such as from endoscopic image and surgical device streams. RESULTS: The methods are evaluated on 80 recorded laparoscopic interventions of various types, for which surgical device data and the endoscopic video streams are available. Here the combined method performs best with an overall average error of 37% and an average halftime error of approximately 28%. CONCLUSION: In this paper, we present, to our knowledge, the first approach for online procedure duration prediction using unlabeled endoscopic video data and surgical device data in a laparoscopic setting. Furthermore, we show that a method incorporating both vision and device data performs better than methods based only on vision, while methods only based on tool usage and surgical device data perform poorly, showing the importance of the visual channel.

Amplitude and phasing of trunk motion is critical for the efficacy of gait training aimed at reducing ambulatory loads at the knee.

The purpose of this study was to determine the contribution of changes in amplitude and phasing of medio-lateral trunk sway to a change in the knee adduction moment when walking with increased medio-lateral trunk sway. Kinematic and kinetic data of walking trials with normal and with increased trunk sway were collected for 19 healthy volunteers using a standard motion analysis system. The relationship between the change in first peak knee adduction moment (ΔKAM) and change in trunk sway amplitude (ΔSA; difference between maximum contralateral trunk lean and maximum ipsilateral trunk lean) and phasing (SP; time of heel-strike relative to time of maximum contralateral and time of maximum ipsilateral trunk lean) was determined using nonlinear regression analysis. On average, subjects increased their SA by 9.7 ± 3.6 deg (P < 0.001) with an average SP of 98.8 ± 88.8 ms resulting in an average reduction in the first peak knee adduction moment of -55.2 ± 30.3% (P < 0.001). 64.3% of variability in change in peak knee adduction moment with the increased trunk sway condition was explained by both differences in SA and SP, and the relationship among these parameters was described by the regression equation ΔKAM = 27.220-4.128 [middle dot] ΔSA-64.785 [middle dot] cos(SP). Hence, not only the amplitude but also the phasing of trunk motion is critical. Not only lower limb movement but also lumbar and thoracic lateral flexion should be considered in the decision making process for an optimal intervention aimed at reducing the load on the medial compartment of the knee during walking. However, these promising findings originated from studies on healthy subjects and their relevance for gait training interventions in patients with presumably painful knee osteoarthritis remains to be determined.

Automatic generation of a subject-specific model for accurate markerless motion capture and biomechanical applications.

A novel approach for the automatic generation of a subject-specific model consisting of morphological and joint location information is described. The aim is to address the need for efficient and accurate model generation for markerless motion capture (MMC) and biomechanical studies. The algorithm applied and expanded on previous work on human shapes space by embedding location information for ten joint centers in a subject-specific free-form surface. The optimal locations of joint centers in the 3-D mesh were learned through linear regression over a set of nine subjects whose joint centers were known. The model was shown to be sufficiently accurate for both kinematic (joint centers) and morphological (shape of the body) information to allow accurate tracking with MMC systems. The automatic model generation algorithm was applied to 3-D meshes of different quality and resolution such as laser scans and visual hulls. The complete method was tested using nine subjects of different gender, body mass index (BMI), age, and ethnicity. Experimental training error and cross-validation errors were 19 and 25 mm, respectively, on average over the joints of the ten subjects analyzed in the study.

Predicting changes in knee adduction moment due to load-altering interventions from pressure distribution at the foot in healthy subjects.

The purpose of this pilot study of healthy subjects was to determine if changes in foot pressure patterns associated with a lateral wedge can predict the changes in the knee adduction moment. We tested two hypotheses: (1) increases or decreases in the knee adduction moment and ankle eversion moment due to load-altering footwear interventions can be predicted from foot pressure distribution and (2) changes in magnitude of the knee adduction moment and ankle eversion moment due to lateral wedges can be predicted from pressure distribution at the foot during walking. Fifteen healthy adults performed walking trials in three shoes: 0 degrees , 4 degrees , and 8 degrees laterally wedged. Maximum heel pressure ratio, first peak knee adduction moment, and peak ankle eversion moment were assessed using a pressure mat, motion capture system, and force plate. Increases or decreases in the knee adduction moment and ankle eversion moment were predicted well from foot pressure distribution. However, the magnitude of the pressure change did not predict the magnitude of the peak knee adduction moment change or peak ankle eversion moment change. Factors such as limb alignment or trunk motion may affect the knee adduction moment and override a direct relationship between the pressure distribution at the shoe-ground interface and the load distribution at the knee. However, changes (increases or decreases) in the peak knee adduction moment due to load-altering footwear interventions predicted from pressure distribution during walking can be important when evaluating these types of interventions from a clinical perspective.

Implications of increased medio-lateral trunk sway for ambulatory mechanics.

The purposes of this study was to test a mechanism to reduce the knee adduction moment by testing the hypothesis that increased medio-lateral trunk sway can reduce the knee adduction moment during ambulation in healthy subjects, and to examine the possibility that increasing medio-lateral trunk sway can produce similar potentially adverse secondary gait changes previously associated with reduced knee adduction moments in patients with knee osteoarthritis. Nineteen healthy adults performed walking trials with normal and increased medio-lateral trunk sway at a self-selected normal walking speed. Standard gait analysis was used to calculate three-dimensional lower extremity joint kinematics and kinetics. Knee and hip adduction moments were lower (-65.0% and -57.1%, respectively) for the increased medio-lateral trunk sway trials than for the normal trunk sway trials. Knee flexion angle at heel-strike was 3 degrees higher for the increased than for the normal trunk sway trials. Knee and hip abduction moments were higher for the increased medio-lateral trunk sway trials, and none of the other variables differed between the two conditions. Walking with increased medio-lateral trunk sway substantially reduces the knee adduction moment during walking in healthy subjects without some of the adverse secondary effects such as increased axial loading rates at the major joints of the lower extremity. This result supports the potential of using gait retraining for walking with increased medio-lateral trunk sway as treatment for patients with degenerative joint disease such as medial compartment knee osteoarthritis.

A framework for the functional identification of joint centers using markerless motion capture, validation for the hip joint.

The objective of the study was to develop a framework for the accurate identification of joint centers to be used for the calculation of human body kinematics and kinetics. The present work introduces a method for the functional identification of joint centers using markerless motion capture (MMC). The MMC system used 8 color VGA cameras. An automatic segmentation-registration algorithm was developed to identify the optimal joint center in a least-square sense. The method was applied to the hip joint center with a validation study conducted in a virtual environment. The results had an accuracy (6mm mean absolute error) below the current MMC system resolution (1cm voxel resolution). Direct experimental comparison with marker-based methods was carried out showing mean absolute deviations over the three anatomical directions of 11.9 and 15.3mm if compared with either a full leg or only thigh markers protocol, respectively. Those experimental results were presented only in terms of deviations between the two systems (marker-based and markerless) as no real gold standard was available. The methods presented in this paper provide an important enabling step towards the biomechanical and clinical applications of markerless motion capture.

The evolution of methods for the capture of human movement leading to markerless motion capture for biomechanical applications.

Over the centuries the evolution of methods for the capture of human movement has been motivated by the need for new information on the characteristics of normal and pathological human movement. This study was motivated in part by the need of new clinical approaches for the treatment and prevention of diseases that are influenced by subtle changes in the patterns movement. These clinical approaches require new methods to measure accurately patterns of locomotion without the risk of artificial stimulus producing unwanted artifacts that could mask the natural patterns of motion. Most common methods for accurate capture of three-dimensional human movement require a laboratory environment and the attachment of markers or fixtures to the body's segments. These laboratory conditions can cause unknown experimental artifacts. Thus, our understanding of normal and pathological human movement would be enhanced by a method that allows the capture of human movement without the constraint of markers or fixtures placed on the body. In this paper, the need for markerless human motion capture methods is discussed and the advancement of markerless approaches is considered in view of accurate capture of three-dimensional human movement for biomechanical applications. The role of choosing appropriate technical equipment and algorithms for accurate markerless motion capture is critical. The implementation of this new methodology offers the promise for simple, time-efficient, and potentially more meaningful assessments of human movement in research and clinical practice. The feasibility of accurately and precisely measuring 3D human body kinematics for the lower limbs using a markerless motion capture system on the basis of visual hulls is demonstrated.

Quantitative modeling of Arabidopsis development.

We present an empirical model of Arabidopsis (Arabidopsis thaliana), intended as a framework for quantitative understanding of plant development. The model simulates and realistically visualizes development of aerial parts of the plant from seedling to maturity. It integrates thousands of measurements, taken from several plants at frequent time intervals. These data are used to infer growth curves, allometric relations, and progression of shapes over time, which are incorporated into the final three-dimensional model. Through the process of model construction, we identify the key attributes required to characterize the development of Arabidopsis plant form over time. The model provides a basis for integrating experimental data and constructing mechanistic models.