NeuroIGN: Explainable Multimodal Image-Guided System for Precise Brain Tumor Surgery.

Precise neurosurgical guidance is critical for successful brain surgeries and plays a vital role in all phases of image-guided neurosurgery (IGN). Neuronavigation software enables real-time tracking of surgical tools, ensuring their presentation with high precision in relation to a virtual patient model. Therefore, this work focuses on the development of a novel multimodal IGN system, leveraging deep learning and explainable AI to enhance brain tumor surgery outcomes. The study establishes the clinical and technical requirements of the system for brain tumor surgeries. NeuroIGN adopts a modular architecture, including brain tumor segmentation, patient registration, and explainable output prediction, and integrates open-source packages into an interactive neuronavigational display. The NeuroIGN system components underwent validation and evaluation in both laboratory and simulated operating room (OR) settings. Experimental results demonstrated its accuracy in tumor segmentation and the success of ExplainAI in increasing the trust of medical professionals in deep learning. The proposed system was successfully assembled and set up within 11 min in a pre-clinical OR setting with a tracking accuracy of 0.5 (± 0.1) mm. NeuroIGN was also evaluated as highly useful, with a high frame rate (19 FPS) and real-time ultrasound imaging capabilities. In conclusion, this paper describes not only the development of an open-source multimodal IGN system but also demonstrates the innovative application of deep learning and explainable AI algorithms in enhancing neuronavigation for brain tumor surgeries. By seamlessly integrating pre- and intra-operative patient image data with cutting-edge interventional devices, our experiments underscore the potential for deep learning models to improve the surgical treatment of brain tumors and long-term post-operative outcomes.

Explainable hybrid vision transformers and convolutional network for multimodal glioma segmentation in brain MRI.

Accurate localization of gliomas, the most common malignant primary brain cancer, and its different sub-region from multimodal magnetic resonance imaging (MRI) volumes are highly important for interventional procedures. Recently, deep learning models have been applied widely to assist automatic lesion segmentation tasks for neurosurgical interventions. However, these models are often complex and represented as "black box" models which limit their applicability in clinical practice. This article introduces new hybrid vision Transformers and convolutional neural networks for accurate and robust glioma segmentation in Brain MRI scans. Our proposed method, TransXAI, provides surgeon-understandable heatmaps to make the neural networks transparent. TransXAI employs a post-hoc explanation technique that provides visual interpretation after the brain tumor localization is made without any network architecture modifications or accuracy tradeoffs. Our experimental findings showed that TransXAI achieves competitive performance in extracting both local and global contexts in addition to generating explainable saliency maps to help understand the prediction of the deep network. Further, visualization maps are obtained to realize the flow of information in the internal layers of the encoder-decoder network and understand the contribution of MRI modalities in the final prediction. The explainability process could provide medical professionals with additional information about the tumor segmentation results and therefore aid in understanding how the deep learning model is capable of processing MRI data successfully. Thus, it enables the physicians' trust in such deep learning systems towards applying them clinically. To facilitate TransXAI model development and results reproducibility, we will share the source code and the pre-trained models after acceptance at https://github.com/razeineldin/TransXAI .

Toward an interoperable, intraoperative situation recognition system via process modeling, execution, and control using the standards BPMN and CMMN.

PURPOSE: For the modeling, execution, and control of complex, non-standardized intraoperative processes, a modeling language is needed that reflects the variability of interventions. As the established Business Process Model and Notation (BPMN) reaches its limits in terms of flexibility, the Case Management Model and Notation (CMMN) was considered as it addresses weakly structured processes. METHODS: To analyze the suitability of the modeling languages, BPMN and CMMN models of a Robot-Assisted Minimally Invasive Esophagectomy and Cochlea Implantation were derived and integrated into a situation recognition workflow. Test cases were used to contrast the differences and compare the advantages and disadvantages of the models concerning modeling, execution, and control. Furthermore, the impact on transferability was investigated. RESULTS: Compared to BPMN, CMMN allows flexibility for modeling intraoperative processes while remaining understandable. Although more effort and process knowledge are needed for execution and control within a situation recognition system, CMMN enables better transferability of the models and therefore the system. Concluding, CMMN should be chosen as a supplement to BPMN for flexible process parts that can only be covered insufficiently by BPMN, or otherwise as a replacement for the entire process. CONCLUSION: CMMN offers the flexibility for variable, weakly structured process parts, and is thus suitable for surgical interventions. A combination of both notations could allow optimal use of their advantages and support the transferability of the situation recognition system.

A Toolchain for Big Data Analyses in the Intelligent Cognitive Operating Room.

This project aims to evaluate existing big data infrastructures for their applicability in the operating room to support medical staff with context-sensitive systems. Requirements for the system design were generated. The project compares different data mining technologies, interfaces, and software system infrastructures with a focus on their usefulness in the peri-operative setting. The lambda architecture was chosen for the proposed system design, which will provide data for both postoperative analysis and real-time support during surgery.

Surgomics: personalized prediction of morbidity, mortality and long-term outcome in surgery using machine learning on multimodal data.

BACKGROUND: Personalized medicine requires the integration and analysis of vast amounts of patient data to realize individualized care. With Surgomics, we aim to facilitate personalized therapy recommendations in surgery by integration of intraoperative surgical data and their analysis with machine learning methods to leverage the potential of this data in analogy to Radiomics and Genomics. METHODS: We defined Surgomics as the entirety of surgomic features that are process characteristics of a surgical procedure automatically derived from multimodal intraoperative data to quantify processes in the operating room. In a multidisciplinary team we discussed potential data sources like endoscopic videos, vital sign monitoring, medical devices and instruments and respective surgomic features. Subsequently, an online questionnaire was sent to experts from surgery and (computer) science at multiple centers for rating the features' clinical relevance and technical feasibility. RESULTS: In total, 52 surgomic features were identified and assigned to eight feature categories. Based on the expert survey (n = 66 participants) the feature category with the highest clinical relevance as rated by surgeons was "surgical skill and quality of performance" for morbidity and mortality (9.0 ± 1.3 on a numerical rating scale from 1 to 10) as well as for long-term (oncological) outcome (8.2 ± 1.8). The feature category with the highest feasibility to be automatically extracted as rated by (computer) scientists was "Instrument" (8.5 ± 1.7). Among the surgomic features ranked as most relevant in their respective category were "intraoperative adverse events", "action performed with instruments", "vital sign monitoring", and "difficulty of surgery". CONCLUSION: Surgomics is a promising concept for the analysis of intraoperative data. Surgomics may be used together with preoperative features from clinical data and Radiomics to predict postoperative morbidity, mortality and long-term outcome, as well as to provide tailored feedback for surgeons.

[Development and evaluation of a simulator for endoscopic sinus surgery]. / Entwicklung und Evaluation eines Simulators für die endoskopische Nasennebenhöhlenoperation.

BACKGROUND: Endoscopic surgical procedures have been established as gold standard in sinus surgery. Challenges for surgical training have been addressed by the use of virtual reality (VR) simulators. To date, a number of simulators have been developed. However, previous studies regarding their training effects investigated only medically pretrained subjects or the time course of training outcomes has not been reported. METHODS: A computer tomography (CT) dataset was segmented manually. A three-dimensional polygonal surface model was generated and textured using original photographic material. Interaction with the virtual environment was performed using a haptic input device. For the investigation of training outcomes with the simulator, the parameters duration and the number of errors were recorded. Ten subjects completed a training consisting of five runs on ten consecutive days. RESULTS: Within the whole exercise period, four subjects reduced the duration of intervention by more than 60%. Four subjects reduced the number of errors by more than 60%. Eight out of 10 subjects showed an improvement with respect to both parameters. On median, the duration of the procedure was reduced by 46 seconds and the number of errors by 191. The statistical analysis between the two parameters showed a positive correlation. CONCLUSION: Our data suggests that training on the FESS-simulator considerably improves the performance even in inexperienced subjects, both in terms of duration and accuracy of the procedure.

Assessment of the Body Posture of Interventional Radiologists.

Physicians in interventional radiology are exposed to high physical stress. To avoid negative long-term effects resulting from unergonomic working conditions, we demonstrated the feasibility of a system that gives feedback about unergonomic situations arising during the intervention based on the Azure Kinect camera. The overall feasibility of the approach could be shown.

Service-oriented Device Connectivity interface for a situation recognition system in the OR.

PURPOSE: Context awareness in the operating room (OR) is important to realize targeted assistance to support actors during surgery. A situation recognition system (SRS) is used to interpret intraoperative events and derive an intraoperative situation from these. To achieve a modular system architecture, it is desirable to de-couple the SRS from other system components. This leads to the need of an interface between such an SRS and context-aware systems (CAS). This work aims to provide an open standardized interface to enable loose coupling of the SRS with varying CAS to allow vendor-independent device orchestrations. METHODS: A requirements analysis investigated limiting factors that currently prevent the integration of CAS in today's ORs. These elicited requirements enabled the selection of a suitable base architecture. We examined how to specify this architecture with the constraints of an interoperability standard. The resulting middleware was integrated into a prototypic SRS and our system for intraoperative support, the OR-Pad, as exemplary CAS for evaluating whether our solution can enable context-aware assistance during simulated orthopedical interventions. RESULTS: The emerging Service-oriented Device Connectivity (SDC) standard series was selected to specify and implement a middleware for providing the interpreted contextual information while the SRS and CAS are loosely coupled. The results were verified within a proof of concept study using the OR-Pad demonstration scenario. The fulfillment of the CAS' requirements to act context-aware, conformity to the SDC standard series, and the effort for integrating the middleware in individual systems were evaluated. The semantically unambiguous encoding of contextual information depends on the further standardization process of the SDC nomenclature. The discussion of the validity of these results proved the applicability and transferability of the middleware. CONCLUSION: The specified and implemented SDC-based middleware shows the feasibility of loose coupling an SRS with unknown CAS to realize context-aware assistance in the OR.

Explainability of deep neural networks for MRI analysis of brain tumors.

PURPOSE: Artificial intelligence (AI), in particular deep neural networks, has achieved remarkable results for medical image analysis in several applications. Yet the lack of explainability of deep neural models is considered the principal restriction before applying these methods in clinical practice. METHODS: In this study, we propose a NeuroXAI framework for explainable AI of deep learning networks to increase the trust of medical experts. NeuroXAI implements seven state-of-the-art explanation methods providing visualization maps to help make deep learning models transparent. RESULTS: NeuroXAI has been applied to two applications of the most widely investigated problems in brain imaging analysis, i.e., image classification and segmentation using magnetic resonance (MR) modality. Visual attention maps of multiple XAI methods have been generated and compared for both applications. Another experiment demonstrated that NeuroXAI can provide information flow visualization on internal layers of a segmentation CNN. CONCLUSION: Due to its open architecture, ease of implementation, and scalability to new XAI methods, NeuroXAI could be utilized to assist radiologists and medical professionals in the detection and diagnosis of brain tumors in the clinical routine of cancer patients. The code of NeuroXAI is publicly accessible at https://github.com/razeineldin/NeuroXAI .

Detection of adverse events leading to inadvertent injury during laparoscopic cholecystectomy using convolutional neural networks.

Uncontrolled movements of laparoscopic instruments can lead to inadvertent injury of adjacent structures. The risk becomes evident when the dissecting instrument is located outside the field of view of the laparoscopic camera. Technical solutions to ensure patient safety are appreciated. The present work evaluated the feasibility of an automated binary classification of laparoscopic image data using Convolutional Neural Networks (CNN) to determine whether the dissecting instrument is located within the laparoscopic image section. A unique record of images was generated from six laparoscopic cholecystectomies in a surgical training environment to configure and train the CNN. By using a temporary version of the neural network, the annotation of the training image files could be automated and accelerated. A combination of oversampling and selective data augmentation was used to enlarge the fully labeled image data set and prevent loss of accuracy due to imbalanced class volumes. Subsequently the same approach was applied to the comprehensive, fully annotated Cholec80 database. The described process led to the generation of extensive and balanced training image data sets. The performance of the CNN-based binary classifiers was evaluated on separate test records from both databases. On our recorded data, an accuracy of 0.88 with regard to the safety-relevant classification was achieved. The subsequent evaluation on the Cholec80 data set yielded an accuracy of 0.84. The presented results demonstrate the feasibility of a binary classification of laparoscopic image data for the detection of adverse events in a surgical training environment using a specifically configured CNN architecture.

DeepSeg: deep neural network framework for automatic brain tumor segmentation using magnetic resonance FLAIR images.

PURPOSE: Gliomas are the most common and aggressive type of brain tumors due to their infiltrative nature and rapid progression. The process of distinguishing tumor boundaries from healthy cells is still a challenging task in the clinical routine. Fluid-attenuated inversion recovery (FLAIR) MRI modality can provide the physician with information about tumor infiltration. Therefore, this paper proposes a new generic deep learning architecture, namely DeepSeg, for fully automated detection and segmentation of the brain lesion using FLAIR MRI data. METHODS: The developed DeepSeg is a modular decoupling framework. It consists of two connected core parts based on an encoding and decoding relationship. The encoder part is a convolutional neural network (CNN) responsible for spatial information extraction. The resulting semantic map is inserted into the decoder part to get the full-resolution probability map. Based on modified U-Net architecture, different CNN models such as residual neural network (ResNet), dense convolutional network (DenseNet), and NASNet have been utilized in this study. RESULTS: The proposed deep learning architectures have been successfully tested and evaluated on-line based on MRI datasets of brain tumor segmentation (BraTS 2019) challenge, including s336 cases as training data and 125 cases for validation data. The dice and Hausdorff distance scores of obtained segmentation results are about 0.81 to 0.84 and 9.8 to 19.7 correspondingly. CONCLUSION: This study showed successful feasibility and comparative performance of applying different deep learning models in a new DeepSeg framework for automated brain tumor segmentation in FLAIR MR images. The proposed DeepSeg is open source and freely available at https://github.com/razeineldin/DeepSeg/.

Design, Implementation and Operation of a Reading Center Platform for Clinical Studies.

Clinical reading centers provide expertise for consistent, centralized analysis of medical data gathered in a distributed context. Accordingly, appropriate software solutions are required for the involved communication and data management processes. In this work, an analysis of general requirements and essential architectural and software design considerations for reading center information systems is provided. The identified patterns have been applied to the implementation of the reading center platform which is currently operated at the Center of Ophthalmology of the University Hospital of Tübingen.

A simple and accurate method for computer-aided transapical aortic valve replacement.

BACKGROUND AND PURPOSE: Transapical aortic valve replacement (TAVR) is a recent minimally invasive surgical treatment technique for elderly and high-risk patients with severe aortic stenosis. In this paper, a simple and accurate image-based method is introduced to aid the intra-operative guidance of TAVR procedure under 2-D X-ray fluoroscopy. METHODS: The proposed method fuses a 3-D aortic mesh model and anatomical valve landmarks with live 2-D fluoroscopic images. The 3-D aortic mesh model and landmarks are reconstructed from interventional X-ray C-arm CT system, and a target area for valve implantation is automatically estimated using these aortic mesh models. Based on template-based tracking approach, the overlay of visualized 3-D aortic mesh model, landmarks and target area of implantation is updated onto fluoroscopic images by approximating the aortic root motion from a pigtail catheter motion without contrast agent. Also, a rigid intensity-based registration algorithm is used to track continuously the aortic root motion in the presence of contrast agent. Furthermore, a sensorless tracking of the aortic valve prosthesis is provided to guide the physician to perform the appropriate placement of prosthesis into the estimated target area of implantation. RESULTS: Retrospective experiments were carried out on fifteen patient datasets from the clinical routine of the TAVR. The maximum displacement errors were less than 2.0mm for both the dynamic overlay of aortic mesh models and image-based tracking of the prosthesis, and within the clinically accepted ranges. Moreover, high success rates of the proposed method were obtained above 91.0% for all tested patient datasets. CONCLUSION: The results showed that the proposed method for computer-aided TAVR is potentially a helpful tool for physicians by automatically defining the accurate placement position of the prosthesis during the surgical procedure.

Touch-free, gesture-based control of medical devices and software based on the leap motion controller.

There are several intra-operative use cases which require the surgeon to interact with medical devices. We used the Leap Motion Controller as input device and implemented two use-cases: 2D-Interaction (e.g. advancing EPR data) and selection of a value (e.g. room illumination brightness). The gesture detection was successful and we mapped its output to several devices and systems.

DICOM for implantations--overview and application.

Surgeons have to deal with many devices from different vendors within the operating room during surgery. Independent communication standards are necessary for the system integration of these devices. For implantations, three new extensions of the Digital Imaging and Communications in Medicine (DICOM) standard make use of a common communication standard that may optimise one of the surgeon's presently very time-consuming daily tasks. The paper provides a brief description of these DICOM Supplements and gives recommendations to their application in practice based on workflows that are proposed to be covered by the new standard extension. Two of the workflows are described in detail and separated into phases that are supported by the new data structures. Examples for the application of the standard within these phases give an impression of the potential usage. Even if the presented workflows are from different domains, we identified a generic core that may benefit from the surgical DICOM Supplements. In some steps of the workflows, the surgical DICOM Supplements are able to replace or optimise conventional methods. Standardisation can only be a means for integration and interoperability. Thus, it can be used as the basis for new applications and system architectures. The influence on current applications and communication processes is limited. Additionally, the supplements provide the basis for further applications, such as the support of surgical navigation systems. Given the support of all involved stakeholders, it is possible to provide a benefit for surgeons and patients.

Laparoscopic versus robot-assisted Nissen fundoplication in an infant pig model.

PURPOSE: Surgical robots are designed to facilitate dissection and suturing, although objective data on their superiority are lacking. This study compares conventional laparoscopic Nissen fundoplication (CLNF) to robot-assisted Nissen fundoplication (RANF) using computer-based workflow analysis in an infant pig model. METHODS: CLNF and RANF were performed in 12 pigs. Surgical workflow was segmented into phases. Time required to perform specific actions was compared by t test. The quality of knot-tying was evaluated by a skill scoring system. Cardia yield pressure (CYP) was determined to test the efficacy of the fundoplications, and the incidence of complications was compared. RESULTS: There was no difference in average times to complete the various phases, despite faster robotic knot-tying (p = 0.001). Suturing quality was superior in CLNF (p = 0.02). CYP increased similarly in both groups. Workflow-interrupting hemorrhage and pneumothorax occurred more frequently during CLNF (p = 0.040 and 0.044, respectively), while more sutures broke during RANF (p = 0.001). CONCLUSION: The robot provides no clear temporal advantage compared to conventional laparoscopy for fundoplication, although suturing was faster in RANF. Fewer complications were noted using the robot. RANF and CLNF were equally efficient anti-reflux procedures. For robotic surgery to manifest its full potential, more complex operations may have to be evaluated.

Analysis of surgical intervention populations using generic surgical process models.

PURPOSE: According to differences in patient characteristics, surgical performance, or used surgical technological resources, surgical interventions have high variability. No methods for the generation and comparison of statistical 'mean' surgical procedures are available. The convenience of these models is to provide increased evidence for clinical, technical, and administrative decision-making. METHODS: Based on several measurements of patient individual surgical treatments, we present a method of how to calculate a statistical 'mean' intervention model, called generic Surgical Process Model (gSPM), from a number of interventions. In a proof-of-concept study, we show how statistical 'mean' procedure courses can be computed and how differences between several of these models can be quantified. Patient individual surgical treatments of 102 cataract interventions from eye surgery were allocated to an ambulatory or inpatient sample, and the gSPMs for each of the samples were computed. Both treatment strategies are exemplary compared for the interventional phase Capsulorhexis. RESULTS: Statistical differences between the gSPMs of ambulatory and inpatient procedures of performance times for surgical activities and activity sequences were identified. Furthermore, the work flow that corresponds to the general recommended clinical treatment was recovered out of the individual Surgical Process Models. CONCLUSION: The computation of gSPMs is a new approach in medical engineering and medical informatics. It supports increased evidence, e.g. for the application of alternative surgical strategies, investments for surgical technology, optimization protocols, or surgical education. Furthermore, this may be applicable in more technical research fields, as well, such as the development of surgical workflow management systems for the operating room of the future.

Image-guided transapical aortic valve implantation: sensorless tracking of stenotic valve landmarks in live fluoroscopic images.

OBJECTIVE: Aortic valve stenosis is one of the most frequently acquired valvular heart diseases, accounting for almost 70% of valvular cardiac surgery. Transapical transcatheter aortic valve implantation has recently become a suitable minimally invasive technique for high-risk and elderly patients with severe aortic stenosis. In this article, we aim to automatically define a target area of valve implantation, namely, the area between the coronary ostia and the lowest points of two aortic valve cusps. Therefore, we present a new image-based tracking method of these aortic landmarks to assist in the placement of aortic valve prosthesis under live 2D fluoroscopy guidance. METHODS: We propose a rigid intensity-based image registration technique for tracking valve landmarks in 2D fluoroscopic image sequences, based on a real-time alignment of a contrast image including the initialized manual valve landmarks to each image of sequence. The contrast image is automatically detected to visualize aortic valve features when the aortic root is filled with a contrast agent. RESULTS: Our registration-based tracking method has been retrospectively applied to 10 fluoroscopic image sequences from routine transapical aortic valve implantation procedures. Most of all tested fluoroscopic images showed a successful tracking of valve landmarks, especially for the images without contrast agent injections. CONCLUSIONS: A new intraoperative image-based method has been developed for tracking aortic valve landmarks in live 2D fluoroscopic images to assist transapical aortic valve implantations and to increase the overall safety of surgery as well.

Recording of surgical processes: a study comparing senior and junior neurosurgeons during lumbar disc herniation surgery.

BACKGROUND: Evaluating surgical practice in the operating room is difficult, and its assessment is largely subjective. OBJECTIVE: Recording of standardized spine surgery processes was conducted to ascertain whether any significant differences in surgical practice could be observed between senior and junior neurosurgeons. METHODS: Twenty-four procedures of lumbar discectomies were consecutively recorded by a senior neurosurgeon. In 12 cases, surgery was entirely performed by a senior neurosurgeon with the aid of a resident, and in the 12 remaining cases, surgery was performed by a resident with the aid of a senior neurosurgeon. The data recorded were general parameters (operating time for the whole procedure and for each step), and general and specific parameters of the surgeon's activities (number of manual gestures, number and duration of actions performed, use of the instruments, and use of interventions on anatomic structures). The Mann-Whitney U test was used for comparison between the 2 groups of neurosurgeons. RESULTS: The operating time was statistically lower for the group of senior surgeons. The seniors statistically demonstrated greater economy in time and in gestures during the closure step, for sewing and for the use of scissors, needle holders, and forceps. The senior surgeons statistically worked for a shorter time on the skin and used fewer manual gestures on the thoracolumbalis fascia. The number of changes in microscope position was also statistically lower for this group. CONCLUSION: There is a relationship between surgical practice, as determined by a method of objective measurement using observation software, and surgical experience: gesture economy evolves with seniority.

A modular video streaming method for surgical assistance in operating room networks.

PURPOSE: Continuous video is used with increasing frequency in the operating room for minimally invasive laparoscopic and endoscopic procedures. Video data communication in the OR requires device interoperability, efficient data transfer methods, and specialized IT infrastructure. METHODS: A framework for digital video communication based on a two channel client-server architecture was developed and tested. One channel is used for stream handling and the second channel is used for data streaming. A video stream description (VSD) specification is defined to negotiate video stream characteristics and ensure semantic interoperability. Quality assessment of the streamed data employs an image-based structural quality measure called the Structural Similarity (SSIM) Index. By introducing the stream description and a quality metric, the stream parameters can be modified as needed. The video communication framework ensures interoperability by defining interfaces for each of the streaming architecture modules. RESULTS: To prove the framework's feasibility, two prototype applications were developed and performance tests were performed on a dedicated OR network. The results showed acceptable network performance for streaming video in the OR network under clinically realistic conditions. CONCLUSION: An OR video communications framework was developed that uses existing OR network infrastructure as an economical alternative to dedicated integrated OR solutions. This framework provides functional and semantic interoperability among imaging modalities for continuous video data communication.