Stereo reconstruction from microscopic images for computer-assisted ophthalmic surgery.

PURPOSE: This work presents a novel platform for stereo reconstruction in anterior segment ophthalmic surgery to enable enhanced scene understanding, especially depth perception, for advanced computer-assisted eye surgery by effectively addressing the lack of texture and corneal distortions artifacts in the surgical scene. METHODS: The proposed platform for stereo reconstruction uses a two-step approach: generating a sparse 3D point cloud from microscopic images, deriving a dense 3D representation by fitting surfaces onto the point cloud, and considering geometrical priors of the eye anatomy. We incorporate a pre-processing step to rectify distortion artifacts induced by the cornea's high refractive power, achieved by aligning a 3D phenotypical cornea geometry model to the images and computing a distortion map using ray tracing. RESULTS: The accuracy of 3D reconstruction is evaluated on stereo microscopic images of ex vivo porcine eyes, rigid phantom eyes, and synthetic photo-realistic images. The results demonstrate the potential of the proposed platform to enhance scene understanding via an accurate 3D representation of the eye and enable the estimation of instrument to layer distances in porcine eyes with a mean average error of 190  µ m , comparable to the scale of surgeons' hand tremor. CONCLUSION: This work marks a significant advancement in stereo reconstruction for ophthalmic surgery by addressing corneal distortions, a previously often overlooked aspect in such surgical scenarios. This could improve surgical outcomes by allowing for intra-operative computer assistance, e.g., in the form of virtual distance sensors.

NeuroIGN: Explainable Multimodal Image-Guided System for Precise Brain Tumor Surgery.

Precise neurosurgical guidance is critical for successful brain surgeries and plays a vital role in all phases of image-guided neurosurgery (IGN). Neuronavigation software enables real-time tracking of surgical tools, ensuring their presentation with high precision in relation to a virtual patient model. Therefore, this work focuses on the development of a novel multimodal IGN system, leveraging deep learning and explainable AI to enhance brain tumor surgery outcomes. The study establishes the clinical and technical requirements of the system for brain tumor surgeries. NeuroIGN adopts a modular architecture, including brain tumor segmentation, patient registration, and explainable output prediction, and integrates open-source packages into an interactive neuronavigational display. The NeuroIGN system components underwent validation and evaluation in both laboratory and simulated operating room (OR) settings. Experimental results demonstrated its accuracy in tumor segmentation and the success of ExplainAI in increasing the trust of medical professionals in deep learning. The proposed system was successfully assembled and set up within 11 min in a pre-clinical OR setting with a tracking accuracy of 0.5 (± 0.1) mm. NeuroIGN was also evaluated as highly useful, with a high frame rate (19 FPS) and real-time ultrasound imaging capabilities. In conclusion, this paper describes not only the development of an open-source multimodal IGN system but also demonstrates the innovative application of deep learning and explainable AI algorithms in enhancing neuronavigation for brain tumor surgeries. By seamlessly integrating pre- and intra-operative patient image data with cutting-edge interventional devices, our experiments underscore the potential for deep learning models to improve the surgical treatment of brain tumors and long-term post-operative outcomes.

Explainable hybrid vision transformers and convolutional network for multimodal glioma segmentation in brain MRI.

Accurate localization of gliomas, the most common malignant primary brain cancer, and its different sub-region from multimodal magnetic resonance imaging (MRI) volumes are highly important for interventional procedures. Recently, deep learning models have been applied widely to assist automatic lesion segmentation tasks for neurosurgical interventions. However, these models are often complex and represented as "black box" models which limit their applicability in clinical practice. This article introduces new hybrid vision Transformers and convolutional neural networks for accurate and robust glioma segmentation in Brain MRI scans. Our proposed method, TransXAI, provides surgeon-understandable heatmaps to make the neural networks transparent. TransXAI employs a post-hoc explanation technique that provides visual interpretation after the brain tumor localization is made without any network architecture modifications or accuracy tradeoffs. Our experimental findings showed that TransXAI achieves competitive performance in extracting both local and global contexts in addition to generating explainable saliency maps to help understand the prediction of the deep network. Further, visualization maps are obtained to realize the flow of information in the internal layers of the encoder-decoder network and understand the contribution of MRI modalities in the final prediction. The explainability process could provide medical professionals with additional information about the tumor segmentation results and therefore aid in understanding how the deep learning model is capable of processing MRI data successfully. Thus, it enables the physicians' trust in such deep learning systems towards applying them clinically. To facilitate TransXAI model development and results reproducibility, we will share the source code and the pre-trained models after acceptance at https://github.com/razeineldin/TransXAI .

Design criteria for AI-based IT systems.

PURPOSE: This editorial relates to a panel discussion during the CARS 2023 congress that addressed the question on how AI-based IT systems should be designed that record and (transparently) display a reproducible path on clinical decision making. Even though the software engineering approach suggested for this endeavor is of a generic nature, it is assumed that the listed design criteria are applicable to IT system development also for the domain of radiology and surgery. METHODS: An example of a possible design approach is outlined by illustrating on how to move from data, information, knowledge and models to wisdom-based decision making in the context of a conceptual GPT system design. In all these design steps, the essential requirements for system quality, information quality, and service quality may be realized by following the design cycle as suggested by A.R. Hevner, appropriately applied to AI-based IT systems design. RESULTS: It can be observed that certain state-of-the-art AI algorithms and systems, such as large language models or generative pre-trained transformers (GPTs), are becoming increasingly complex and, therefore, need to be rigorously examined to render them transparent and comprehensible in their usage for all stakeholders involved in health care. Further critical questions that need to be addressed are outlined and complemented with some suggestions, that a possible design framework for a stakeholder specific AI system could be a (modest) GPT based on a small language model. DISCUSSION: A fundamental question for the future remains whether society wants a quasi-wisdom-oriented healthcare system, based on data-driven intelligence with AI, or a human curated wisdom based on model-driven intelligence (with and without AI). Special CARS workshops and think tanks are planned to address this challenging question and possible new direction for assisting selected medical disciplines, e.g., radiology and surgery.

Augmented Reality-Assisted versus Freehand Ventriculostomy in a Head Model.

BACKGROUND: Ventriculostomy (VST) is a frequent neurosurgical procedure. Freehand catheter placement represents the standard current practice. However, multiple attempts are often required. We present augmented reality (AR) headset guided VST with in-house developed head models. We conducted a proof of concept study in which we tested AR-guided as well as freehand VST. Repeated AR punctures were conducted to investigate if a learning curve can be derived. METHODS: Five custom-made 3D-printed head models, each holding an anatomically different ventricular system, were filled with agarose gel. Eleven surgeons placed two AR-guided as well as two freehand ventricular drains per head. A subgroup of four surgeons did a total of three series of AR-guided punctures each to test for a learning curve. A Microsoft HoloLens served as the hardware platform. The marker-based tracking did not require rigid head fixation. Catheter tip position was evaluated in computed tomography scans. RESULTS: Marker-tracking, image segmentation, and holographic display worked satisfactorily. In freehand VST, a success rate of 72.7% was achieved, which was higher than under AR guidance (68.2%, difference not statistically significant). Repeated AR-guided punctures increased the success rate from 65 to 95%. We assume a steep learning curve as repeated AR-guided punctures led to an increase in successful attempts. Overall user experience showed positive feedback. CONCLUSIONS: We achieved promising results that encourage the continued development and technical improvement. However, several more developmental steps have to be taken before an application in humans can be considered. In the future, AR headset-based holograms have the potential to serve as a compact navigational help inside and outside the operating room.

Recurrent neural networks for generalization towards the vessel geometry in autonomous endovascular guidewire navigation in the aortic arch.

PURPOSE: Endovascular intervention is the state-of-the-art treatment for common cardiovascular diseases, such as heart attack and stroke. Automation of the procedure may improve the working conditions of physicians and provide high-quality care to patients in remote areas, posing a major impact on overall treatment quality. However, this requires the adaption to individual patient anatomies, which currently poses an unsolved challenge. METHODS: This work investigates an endovascular guidewire controller architecture based on recurrent neural networks. The controller is evaluated in-silico on its ability to adapt to new vessel geometries when navigating through the aortic arch. The controller's generalization capabilities are examined by reducing the number of variations seen during training. For this purpose, an endovascular simulation environment is introduced, which allows guidewire navigation in a parametrizable aortic arch. RESULTS: The recurrent controller achieves a higher navigation success rate of 75.0% after 29,200 interventions compared to 71.6% after 156,800 interventions for a feedforward controller. Furthermore, the recurrent controller generalizes to previously unseen aortic arches and is robust towards size changes of the aortic arch. Being trained on 2048 aortic arch geometries gives the same results as being trained with full variation when evaluated on 1000 different geometries. For interpolation a gap of 30% of the scaling range and for extrapolation additional 10% of the scaling range can be navigated successfully. CONCLUSION: Adaption to new vessel geometries is essential in the navigation of endovascular instruments. Therefore, the intrinsic generalization to new vessel geometries poses an essential step towards autonomous endovascular robotics.

A sensorized modular training platform to reduce vascular damage in endovascular surgery.

PURPOSE: Endovascular interventions require intense practice to develop sufficient dexterity in catheter handling within the human body. Therefore, we present a modular training platform, featuring 3D-printed vessel phantoms with patient-specific anatomy and integrated piezoresistive impact force sensing of instrument interaction at clinically relevant locations for feedback-based skill training to detect and reduce damage to the delicate vascular wall. METHODS: The platform was fabricated and then evaluated in a user study by medical ([Formula: see text]) and non-medical ([Formula: see text]) users. The users had to navigate a set of guidewire and catheter through a parkour of 3 modules including an aneurismatic abdominal aorta, while impact force and completion time were recorded. Eventually, a questionnaire was conducted. RESULTS: The platform allowed to perform more than 100 runs in which it proved capable to distinguish between users of different experience levels. Medical experts in the fields of vascular and visceral surgery had a strong performance assessment on the platform. It could be shown, that medical students could improve runtime and impact over 5 runs. The platform was well received and rated as promising for medical education despite the experience of higher friction compared to real human vessels. CONCLUSION: We investigated an authentic patient-specific training platform with integrated sensor-based feedback functionality for individual skill training in endovascular surgery. The presented method for phantom manufacturing is easily applicable to arbitrary patient-individual imaging data. Further work shall address the implementation of smaller vessel branches, as well as real-time feedback and camera imaging for further improved training experience.

Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark.

PURPOSE: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data single-center video dataset. In this work we investigated the generalizability of phase recognition algorithms in a multicenter setting including more difficult recognition tasks such as surgical action and surgical skill. METHODS: To achieve this goal, a dataset with 33 laparoscopic cholecystectomy videos from three surgical centers with a total operation time of 22 h was created. Labels included framewise annotation of seven surgical phases with 250 phase transitions, 5514 occurences of four surgical actions, 6980 occurences of 21 surgical instruments from seven instrument categories and 495 skill classifications in five skill dimensions. The dataset was used in the 2019 international Endoscopic Vision challenge, sub-challenge for surgical workflow and skill analysis. Here, 12 research teams trained and submitted their machine learning algorithms for recognition of phase, action, instrument and/or skill assessment. RESULTS: F1-scores were achieved for phase recognition between 23.9% and 67.7% (n = 9 teams), for instrument presence detection between 38.5% and 63.8% (n = 8 teams), but for action recognition only between 21.8% and 23.3% (n = 5 teams). The average absolute error for skill assessment was 0.78 (n = 1 team). CONCLUSION: Surgical workflow and skill analysis are promising technologies to support the surgical team, but there is still room for improvement, as shown by our comparison of machine learning algorithms. This novel HeiChole benchmark can be used for comparable evaluation and validation of future work. In future studies, it is of utmost importance to create more open, high-quality datasets in order to allow the development of artificial intelligence and cognitive robotics in surgery.

Learning-based autonomous vascular guidewire navigation without human demonstration in the venous system of a porcine liver.

PURPOSE: The navigation of endovascular guidewires is a dexterous task where physicians and patients can benefit from automation. Machine learning-based controllers are promising to help master this task. However, human-generated training data are scarce and resource-intensive to generate. We investigate if a neural network-based controller trained without human-generated data can learn human-like behaviors. METHODS: We trained and evaluated a neural network-based controller via deep reinforcement learning in a finite element simulation to navigate the venous system of a porcine liver without human-generated data. The behavior is compared to manual expert navigation, and real-world transferability is evaluated. RESULTS: The controller achieves a success rate of 100% in simulation. The controller applies a wiggling behavior, where the guidewire tip is continuously rotated alternately clockwise and counterclockwise like the human expert applies. In the ex vivo porcine liver, the success rate drops to 30%, because either the wrong branch is probed, or the guidewire becomes entangled. CONCLUSION: In this work, we prove that a learning-based controller is capable of learning human-like guidewire navigation behavior without human-generated data, therefore, mitigating the requirement to produce resource-intensive human-generated training data. Limitations are the restriction to one vessel geometry, the neglected safeness of navigation, and the reduced transferability to the real world.

Explainability of deep neural networks for MRI analysis of brain tumors.

PURPOSE: Artificial intelligence (AI), in particular deep neural networks, has achieved remarkable results for medical image analysis in several applications. Yet the lack of explainability of deep neural models is considered the principal restriction before applying these methods in clinical practice. METHODS: In this study, we propose a NeuroXAI framework for explainable AI of deep learning networks to increase the trust of medical experts. NeuroXAI implements seven state-of-the-art explanation methods providing visualization maps to help make deep learning models transparent. RESULTS: NeuroXAI has been applied to two applications of the most widely investigated problems in brain imaging analysis, i.e., image classification and segmentation using magnetic resonance (MR) modality. Visual attention maps of multiple XAI methods have been generated and compared for both applications. Another experiment demonstrated that NeuroXAI can provide information flow visualization on internal layers of a segmentation CNN. CONCLUSION: Due to its open architecture, ease of implementation, and scalability to new XAI methods, NeuroXAI could be utilized to assist radiologists and medical professionals in the detection and diagnosis of brain tumors in the clinical routine of cancer patients. The code of NeuroXAI is publicly accessible at https://github.com/razeineldin/NeuroXAI .

Continuous Feature-Based Tracking of the Inner Ear for Robot-Assisted Microsurgery.

Robotic systems for surgery of the inner ear must enable highly precise movement in relation to the patient. To allow for a suitable collaboration between surgeon and robot, these systems should not interrupt the surgical workflow and integrate well in existing processes. As the surgical microscope is a standard tool, present in almost every microsurgical intervention and due to it being in close proximity to the situs, it is predestined to be extended by assistive robotic systems. For instance, a microscope-mounted laser for ablation. As both, patient and microscope are subject to movements during surgery, a well-integrated robotic system must be able to comply with these movements. To solve the problem of on-line registration of an assistance system to the situs, the standard of care often utilizes marker-based technologies, which require markers being rigidly attached to the patient. This not only requires time for preparation but also increases invasiveness of the procedure and the line of sight of the tracking system may not be obstructed. This work aims at utilizing the existing imaging system for detection of relative movements between the surgical microscope and the patient. The resulting data allows for maintaining registration. Hereby, no artificial markers or landmarks are considered but an approach for feature-based tracking with respect to the surgical environment in otology is presented. The images for tracking are obtained by a two-dimensional RGB stream of a surgical microscope. Due to the bony structure of the surgical site, the recorded cochleostomy scene moves nearly rigidly. The goal of the tracking algorithm is to estimate motion only from the given image stream. After preprocessing, features are detected in two subsequent images and their affine transformation is computed by a random sample consensus (RANSAC) algorithm. The proposed method can provide movement feedback with up to 93.2 µm precision without the need for any additional hardware in the operating room or attachment of fiducials to the situs. In long term tracking, an accumulative error occurs.

Assistive Device for Efficient Intravitreal Injections.

Intravitreal therapy is the most common treatment for many chronic ophthalmic diseases, such as age-related macular degeneration. Due to the increasing worldwide demand for intravitreal injections, there exists a need to render this medical procedure more time- and cost-efficient while increasing patient safety. The authors propose a medical assistive device that injects medication intravitreally. Compared to the manual intravitreal injection procedure, an automated device has the potential to increase safety for patients, decrease procedure times, allow for integrated data storage and documentation, and reduce costs for medical staff and expensive operating rooms. This work demonstrates the development of an assistive injection system that is coarsely positioned over the patient's head by the human operator, followed by automatic fine positioning and intravitreal injection through the pars plana. Several safety features, such as continuous eye tracking and iris recognition, have been implemented. The functioning system is demonstrated through ex vivo experiments with porcine eyes. [Ophthalmic Surg Lasers Imaging Retina. 2016;47:752-762.].

Perforation forces of the intact porcine anterior lens capsule.

During the first step of cataract surgery, the lens capsule is perforated and a circular hole is created with a sharp instrument, a procedure called capsulorhexis. To develop automated systems that can assist ophthalmologists during capsulorhexis, the forces required must be quantified. This study investigates perforation forces of the central anterior lens capsule in porcine eyes, which are used as a conservative model for the human eye. A micro-mechanical characterisation method is presented that measures capsular bag perforation forces with a high precision positioning and high-resolution force sensing system. The force during perforation of the anterior lens capsule was measured with various sized needles and indentation speeds and is found to be 15-35mN. A bio-mechanical model is identified that describes an exponential correlation between indentation force and depth, indicating strain hardening behaviour of the porcine anterior lens capsule.

Electroforming of implantable tubular magnetic microrobots for wireless ophthalmologic applications.

Magnetic tubular implantable micro-robots are batch fabricated by electroforming. These microdevices can be used in targeted drug delivery and minimally invasive surgery for ophthalmologic applications. These tubular shapes are fitted into a 23-gauge needle enabling sutureless injections. Using a 5-degree-of-freedom magnetic manipulation system, the microimplants are conveniently maneuvered in biological environments. To increase their functionality, the tubes are coated with biocompatible films and can be successfully filled with drugs.

Mobility experiments with microrobots for minimally invasive intraocular surgery.

PURPOSE: To investigate microrobots as an assistive tool for minimally invasive intraocular surgery and to demonstrate mobility and controllability inside the living rabbit eye. METHODS: A system for wireless magnetic control of untethered microrobots was developed. Mobility and controllability of a microrobot are examined in different media, specifically vitreous, balanced salt solution (BSS), and silicone oil. This is demonstrated through ex vivo and in vivo animal experiments. RESULTS: The developed electromagnetic system enables precise control of magnetic microrobots over a workspace that covers the posterior eye segment. The system allows for rotation and translation of the microrobot in different media (vitreous, BSS, silicone oil) inside the eye. CONCLUSIONS: Intravitreal introduction of untethered mobile microrobots can enable sutureless and precise ophthalmic procedures. Ex vivo and in vivo experiments demonstrate that microrobots can be manipulated inside the eye. Potential applications are targeted drug delivery for maculopathies such as AMD, intravenous deployment of anticoagulation agents for retinal vein occlusion (RVO), and mechanical applications, such as manipulation of epiretinal membrane peeling (ERM). The technology has the potential to reduce the invasiveness of ophthalmic surgery and assist in the treatment of a variety of ophthalmic diseases.

Detection of WU polyomavirus DNA by real-time PCR in nasopharyngeal aspirates, serum, and stool samples.

BACKGROUND: The human WU polyomavirus (WUPyV) has been recently described as a novel virus in respiratory tract samples. OBJECTIVE: To investigate the viral load of WUPyV in nasopharyngeal aspirates (NPAs), stool, and serum samples of pediatric patients with acute respiratory tract diseases. STUDY DESIGN: We established a real-time PCR for WUPyV DNA and tested NPA obtained between 2002 and 2007 from pediatric in-patients with acute respiratory tract diseases. In addition, 14 stool and 14 serum samples of children with WUPyV DNA positive NPA were analysed. RESULTS: WUPyV DNA was found in 5.2% of 1232 NPA. The median viral load in the NPA was 950 copies/ml (maximum 3.4 x 10(10) copies/ml). The WUPyV load in NPA was neither associated with the coinfection status nor with the clinical diagnoses. WUPyV DNA was found in 3 of 14 serum samples and in 2 of 14 stool samples. The WUPyV load in NPA tended to be higher in viremic children. CONCLUSION: WUPyV DNA was found in NPA, serum, and stool of hospitalised children with acute respiratory tract diseases. Further studies are necessary to determine whether WUPyV is a human pathogen.