[BIOPASS hybrid navigation for endoscopic sinus surgery - an assistance system]. / BIOPASS-Hybrid-Navigation für die endoskopische Nasennebenhöhlenchirurgie ­ ein Assistenzsystem.

Previous navigation systems can determine the position of the "tracked" surgical instrument in CT images in the context of functional endoscopic sinus surgery (FESS), but do not provide any assistance directly in the video endoscopic image of the surgeon. Developing this direct assistance for intraoperative orientation and risk reduction was the goal of the BIOPASS project (Bild Ontologie und prozessgestütztes Assistenzsystem). The Project pursues the development of a novel navigation system for FESS without markers. BIOPASS describes a hybrid system that integrates various sensor data and makes it available. The goal is to abandon tracking and exclusively provide navigation information directly in the video image. This paper describes the first step of the development by collecting and structuring the surgical phases (workflows), the video endoscopic landmarks and a first clinical evaluation of the model version. The results provide the important basis and platform for the next step of the project.

Ontology-based surgical workflow recognition and prediction.

BACKGROUND: Surgical context-aware systems can adapt to the current situation in the operating room and thus provide computer-aided assistance functionalities and intraoperative decision-support. To interact with the surgical team perceptively and assist the surgical process, the system needs to monitor the intraoperative activities, understand the current situation in the operating room at any time, and anticipate the following possible situations. METHODS: A structured representation of surgical process knowledge is a prerequisite for any applications in the intelligent operating room. For this purpose, a surgical process ontology, which is formally based on standard medical terminology (SNOMED CT) and an upper-level ontology (GFO), was developed and instantiated for a neurosurgical use case. A new ontology-based surgical workflow recognition and a novel prediction method are presented utilizing ontological reasoning, abstraction, and explication. This way, a surgical situation representation with combined phase, high-level task, and low-level task recognition and prediction was realized based on the currently used instrument as the only input information. RESULTS: The ontology-based approach performed efficiently, and decent accuracy was achieved for situation recognition and prediction. Especially during situation recognition, the missing sensor information were reasoned based on the situation representation provided by the process ontology, which resulted in improved recognition results compared to the state-of-the-art. CONCLUSIONS: In this work, a reference ontology was developed, which provides workflow support and a knowledge base for further applications in the intelligent operating room, for instance, context-aware medical device orchestration, (semi-) automatic documentation, and surgical simulation, education, and training.

Analysis of the paraspinal muscle morphology of the lumbar spine using a convolutional neural network (CNN).

PURPOSE: This single-center study aimed to develop a convolutional neural network to segment multiple consecutive axial magnetic resonance imaging (MRI) slices of the lumbar spinal muscles of patients with lower back pain and automatically classify fatty muscle degeneration. METHODS: We developed a fully connected deep convolutional neural network (CNN) with a pre-trained U-Net model trained on a dataset of 3,650 axial T2-weighted MRI images from 100 patients with lower back pain. We included all qualities of MRI; the exclusion criteria were fractures, tumors, infection, or spine implants. The training was performed using k-fold cross-validation (k = 10), and performance was evaluated using the dice similarity coefficient (DSC) and cross-sectional area error (CSA error). For clinical correlation, we used a simplified Goutallier classification (SGC) system with three classes. RESULTS: The mean DSC was high for overall muscle (0.91) and muscle tissue segmentation (0.83) but showed deficiencies in fatty tissue segmentation (0.51). The CSA error was small for the overall muscle area of 8.42%, and fatty tissue segmentation showed a high mean CSA error of 40.74%. The SGC classification was correctly predicted in 75% of the patients. CONCLUSION: Our fully connected CNN segmented overall muscle and muscle tissue with high precision and recall, as well as good DSC values. The mean predicted SGC values of all available patient axial slices showed promising results. With an overall Error of 25%, further development is needed for clinical implementation. Larger datasets and training of other model architectures are required to segment fatty tissue more accurately.

[Ethical, legal and social implications in the use of artificial intelligence-based technologies in surgery : Principles, implementation and importance for the user]. / Ethische, legale und soziale Implikationen bei der Anwendung künstliche-Intelligenz-gestützter Technologien in der Chirurgie : Grundlagen, Umsetzung und Bedeutung für den Anwender.

Ethical, legal and social aspects are gaining increasingly more attention in the development and during the initial clinical application of medical devices. The introduction of elements of artificial intelligence (AI) and systems which are using AI makes this already complex topic even more challenging. The introduction of so-called dynamic AI or dynamic machine learning (ML) algorithms in this respect represents a turning point. Unlike conventional medical devices, the development of systems using dynamic AI is not yet complete at the beginning of the clinical application. The aim of a dynamic AI system is to continuously improve through practical use and by the processing of usage data. This continuous evolution, along with the lack of transparency regarding internal work processes, could make it difficult to understand the underlying rationale for the assessments made by the algorithms. This aspect affects the acceptance of the technology both by clinicians and patients and furthermore questions the autonomy of patients and clinicians in the course of the treatment process. A way out of this ethical and regulatory dilemma must urgently be found and will require extreme efforts from all stakeholders. At present, no consensual solution is apparent. What is quite certain, however, is that users, i.e. in concrete terms surgeons, must play a much more active role than they have done in the past when dealing with AI-based medical devices and should prepare themselves to actively accompany the software life cycle of AI technologies.

A system for real-time multivariate feature combination of endoscopic mitral valve simulator training data.

PURPOSE: For an in-depth analysis of the learning benefits that a stereoscopic view presents during endoscopic training, surgeons required a custom surgical evaluation system enabling simulator independent evaluation of endoscopic skills. Automated surgical skill assessment is in dire need since supervised training sessions and video analysis of recorded endoscope data are very time-consuming. This paper presents a first step towards a multimodal training evaluation system, which is not restricted to certain training setups and fixed evaluation metrics. METHODS: With our system we performed data fusion of motion and muscle-action measurements during multiple endoscopic exercises. The exercises were performed by medical experts with different surgical skill levels, using either two or three-dimensional endoscopic imaging. Based on the multi-modal measurements, training features were calculated and their significance assessed by distance and variance analysis. Finally, the features were used automatic classification of the used endoscope modes. RESULTS: During the study, 324 datasets from 12 participating volunteers were recorded, consisting of spatial information from the participants' joint and right forearm electromyographic information. Feature significance analysis showed distinctive significance differences, with amplitude-related muscle information and velocity information from hand and wrist being among the most significant ones. The analyzed and generated classification models exceeded a correct prediction rate of used endoscope type accuracy rate of 90%. CONCLUSION: The results support the validity of our setup and feature calculation, while their analysis shows significant distinctions and can be used to identify the used endoscopic view mode, something not apparent when analyzing time tables of each exercise attempt. The presented work is therefore a first step toward future developments, with which multivariate feature vectors can be classified automatically in real-time to evaluate endoscopic training and track learning progress.

Language-based translation and prediction of surgical navigation steps for endoscopic wayfinding assistance in minimally invasive surgery.

PURPOSE: In the context of aviation and automotive navigation technology, assistance functions are associated with predictive planning and wayfinding tasks. In endoscopic minimally invasive surgery, however, assistance so far relies primarily on image-based localization and classification. We show that navigation workflows can be described and used for the prediction of navigation steps. METHODS: A natural description vocabulary for observable anatomical landmarks in endoscopic images was defined to create 3850 navigation workflow sentences from 22 annotated functional endoscopic sinus surgery (FESS) recordings. Resulting FESS navigation workflows showed an imbalanced data distribution with over-represented landmarks in the ethmoidal sinus. A transformer model was trained to predict navigation sentences in sequence-to-sequence tasks. The training was performed with the Adam optimizer and label smoothing in a leave-one-out cross-validation study. The sentences were generated using an adapted beam search algorithm with exponential decay beam rescoring. The transformer model was compared to a standard encoder-decoder-model, as well as HMM and LSTM baseline models. RESULTS: The transformer model reached the highest prediction accuracy for navigation steps at 0.53, followed by 0.35 of the LSTM and 0.32 for the standard encoder-decoder-network. With an accuracy of sentence generation of 0.83, the prediction of navigation steps at sentence-level benefits from the additional semantic information. While standard class representation predictions suffer from an imbalanced data distribution, the attention mechanism also considered underrepresented classes reasonably well. CONCLUSION: We implemented a natural language-based prediction method for sentence-level navigation steps in endoscopic surgery. The sentence-level prediction method showed a potential that word relations to navigation tasks can be learned and used for predicting future steps. Further studies are needed to investigate the functionality of path prediction. The prediction approach is a first step in the field of visuo-linguistic navigation assistance for endoscopic minimally invasive surgery.

Processual Reasoning over Sequences of Situations in Endoscopic Surgery.

Minimally invasive surgery is a highly complex and technically demanding alternative to open surgery. Surgical procedures based on this method are characterized by small incisions and allow for a fast recovery of the patient. Such techniques are challenging for surgeons since they do not have a direct view of the surgical area. Systems that provide surgical navigation are well established in clinical practice but depend on external markers allowing a mapping between a surgeon's tools and a patient's medical images. As of today, these systems are prone to inaccuracies, the reasons of which lie in their extensive technical requirements. The BIOPASS project aims to develop an alternative that works without external markers and indirect computation of locations. An ontology has been used to provide an adequate vocabulary describing situations and their temporal relationship. This ontology is expected to relate real time multimodal sensor data and static surgical process models in order to infer movement directions, subsequent actions and hidden anatomical structures that inhere risk for surgical interventions. However, the Web Ontology Language is not capable of modelling temporal conditions, which are necessary to provide such exhaustive situational descriptions as expected by a surgeon. This paper concerns an ontology design pattern developed to overcome this issue by the integration of dynamic ontological classes that are assigned according to the temporal relations between situations.

Radiographic reconstruction of lower-extremity bone fragments: a first trial.

PURPOSE: The correct rotational alignment of the proximal and the distal bone fragments is an essential step in a long-bone deformity correction process. In order to plan the deformity correction, plain radiographs are conventionally used. But as three-dimensional information of the complex situation is not available, the correct amount of rotation can only be approximated. Thus, the objective of this study was to develop a system to assess the rotational relationship between the proximal and distal fragments of a long bone (tibia or femur) based on a set of two calibrated X-ray radiographs. METHODS: In order to robustly determine the rotational relationship of proximal and distal bone fragments, a statistical shape model-based 2D/3D reconstruction approach was employed. The resulting fragment models were used to determine the angle between its anatomical axes and the rotation around its particular axes. Two different studies were performed to evaluate the accuracy of the proposed system. RESULTS: The accuracy of the complete system was evaluated in terms of major bone axis and in-plane rotational difference. The angle between the anatomical fragment axes could be measured with an average error of 0.33[Formula: see text] ± 0.27[Formula: see text], while an average in-plane rotational error of 2.27[Formula: see text] ± 1.76[Formula: see text]  and 2.67[Formula: see text]  ± 1.80[Formula: see text]  was found for the proximal and distal fragments, respectively. The overall mean surface reconstruction error was 0.81  ± 0.59 mm when the present technique was applied to the tibia and 1.12 ± 0.87 mm when it was applied to the femur. CONCLUSIONS: A new approach for estimating the rotational parameters of long-bone fragments has been proposed. This approach is based on two conventional radiographs and 2D/3D reconstruction technology. It is generally applicable to the alignment of any long-bone fragments and could provide an important means for achieving accurate rotational alignment.

Quantification of osseointegration of plasma-polymer coated titanium alloyed implants by means of microcomputed tomography versus histomorphometry.

A common method to derive both qualitative and quantitative data to evaluate osseointegration of implants is histomorphometry. The present study describes a new image reconstruction algorithm comparing the results of bone-to-implant contact (BIC) evaluated by means of µCT with histomorphometry data. Custom-made conical titanium alloyed (Ti6Al4V) implants were inserted in the distal tibial bone of female Sprague-Dawley rats. Different surface configurations were examined: Ti6Al4V implants with plasma-polymerized allylamine (PPAAm) coating and plasma-polymerized ethylenediamine (PPEDA) coating as well as implants without surface coating. After six weeks postoperatively, tibiae were explanted and BIC was determined by µCT (3D) and afterwards by histomorphometry (2D). In comparison to uncoated Ti6Al4V implants demonstrating low BIC of 32.4% (histomorphometry) and 51.3% (µCT), PPAAm and PPEDA coated implants showed a nonsignificant increase in BIC (histomorphometry: 45.7% and 53.5% and µCT: 51.8% and 62.0%, resp.). Mean BIC calculated by µCT was higher for all surface configurations compared to BIC detected by histomorphometry. Overall, a high correlation coefficient of 0.70 (p < 0.002) was found between 3D and 2D quantification of BIC. The µCT analysis seems to be suitable as a nondestructive and accurate 3D imaging method for the evaluation of the bone-implant interface.