Mobile, real-time, and point-of-care augmented reality is robust, accurate, and feasible: a prospective pilot study.

BACKGROUND: Augmented reality (AR) systems are currently being explored by a broad spectrum of industries, mainly for improving point-of-care access to data and images. Especially in surgery and especially for timely decisions in emergency cases, a fast and comprehensive access to images at the patient bedside is mandatory. Currently, imaging data are accessed at a distance from the patient both in time and space, i.e., at a specific workstation. Mobile technology and 3-dimensional (3D) visualization of radiological imaging data promise to overcome these restrictions by making bedside AR feasible. METHODS: In this project, AR was realized in a surgical setting by fusing a 3D-representation of structures of interest with live camera images on a tablet computer using marker-based registration. The intent of this study was to focus on a thorough evaluation of AR. Feasibility, robustness, and accuracy were thus evaluated consecutively in a phantom model and a porcine model. Additionally feasibility was evaluated in one male volunteer. RESULTS: In the phantom model (n = 10), AR visualization was feasible in 84% of the visualization space with high accuracy (mean reprojection error ± standard deviation (SD): 2.8 ± 2.7 mm; 95th percentile = 6.7 mm). In a porcine model (n = 5), AR visualization was feasible in 79% with high accuracy (mean reprojection error ± SD: 3.5 ± 3.0 mm; 95th percentile = 9.5 mm). Furthermore, AR was successfully used and proved feasible within a male volunteer. CONCLUSIONS: Mobile, real-time, and point-of-care AR for clinical purposes proved feasible, robust, and accurate in the phantom, animal, and single-trial human model shown in this study. Consequently, AR following similar implementation proved robust and accurate enough to be evaluated in clinical trials assessing accuracy, robustness in clinical reality, as well as integration into the clinical workflow. If these further studies prove successful, AR might revolutionize data access at patient bedside.

Deep Learning-based Assessment of Internal Carotid Artery Anatomy to Predict Difficult Intracranial Access in Endovascular Recanalization of Acute Ischemic Stroke.

BACKGROUND: Endovascular thrombectomy (EVT) duration is an important predictor for neurological outcome. Recently it was shown that an angle of ≤ 90° of the internal carotid artery (ICA) is predictive for longer EVT duration. As manual angle measurement is not trivial and time-consuming, deep learning (DL) could help identifying difficult EVT cases in advance. METHODS: We included 379 CT angiographies (CTA) of patients who underwent EVT between January 2016 and December 2020. Manual segmentation of 121 CTAs was performed for the aortic arch, common carotid artery (CCA) and ICA. These were used to train a nnUNet. The remaining 258 CTAs were segmented using the trained nnUNet with manual verification afterwards. Angles of left and right ICAs were measured resulting in two classes: acute angle ≤ 90° and > 90°. The segmentations together with angle measurements were used to train a convolutional neural network (CNN) determining the ICA angle. The performance was evaluated using Dice scores. The classification was evaluated using AUC and accuracy. Associations of ICA angle and procedural times was explored using median and Whitney­U test. RESULTS: Median EVT duration for cases with ICA angle > 90° was 48 min and with ≤ 90° was 64 min (p = 0.001). Segmentation evaluation showed Dice scores of 0.94 for the aorta and 0.86 for CCA/ICA, respectively. Evaluation of ICA angle determination resulted in an AUC of 0.92 and accuracy of 0.85. CONCLUSION: The association between ICA angle and EVT duration could be verified and a DL-based method for semi-automatic assessment with the potential for full automation was developed. More anatomical features of interest could be examined in a similar fashion.

Computer-assisted trajectory planning for percutaneous needle insertions.

PURPOSE: Computed tomography (CT) guided minimally invasive interventions such as biopsies or ablation therapies often involve insertion of a needle-shaped instrument into the target organ (e.g., the liver). Today, these interventions still require manual planning of a suitable trajectory to the target (e.g., the tumor) based on the slice data provided by the imaging modality. However, taking into account the critical structures and other parameters crucial to the success of the intervention--such as instrument shape and penetration angle--is challenging and requires a lot of experience. METHODS: To overcome these problems, we present a system for the automatic or semiautomatic planning of optimal trajectories to a target, based on 3D reconstructions of all relevant structures. The system determines possible insertion zones based on so-called hard constraints and rates the quality of these zones by so-called soft constraints. The concept of pareto optimality is utilized to allow for a weight-independent proposal of insertion trajectories. In order to demonstrate the benefits of our method, automatic trajectory planning was applied retrospectively to n = 10 data sets from interventions in which complications occurred. RESULTS: The efficient (graphics processing unit-based) implementation of the constraints results in a mean overall planning time of about 9 s. The examined trajectories, originally chosen by the physician, have been rated as follows: in six cases, the insertion point was labeled invalid by the planning system. For two cases, the system would have proposed points with a better rating according to the soft constraints. For the remaining two cases the system would have indicated poor rating with respect to one of the soft constraints. The paths proposed by our system were rated feasible and qualitatively good by experienced interventional radiologists. CONCLUSIONS: The proposed computer-assisted trajectory planning system is able to detect unsafe and propose safe insertion trajectories and may especially be helpful for interventional radiologist at the beginning or during their interventional training.

Three-Dimensional Reconstruction of Preoperative Imaging Improves Surgical Success in Laparoscopy.

PURPOSE: To show the benefit of three-dimensional (3D) reconstructions of preoperative imaging for surgical performance. METHODS: A laparoscopic training environment with 15 hidden lymph nodes was designed. Three of them were marked with radiographic contrast agent and were only distinguishable from unmarked nodes via CT imaging. Thirty-six surgeons were divided into two groups. To group 1 the unprocessed CT data were shown. Group 2 was additionally shown a 3D reconstruction of the anatomy. Time of studying the imaging was recorded. All surgeons had to find the three target lymph nodes laparoscopically. Time to fulfill this task and errors was measured. Afterward, the 3D reconstruction was also shown to group 1. Then, all participants completed a questionnaire. Furthermore, 3D reconstructions were used in 15 clinical cases of partial nephrectomy or lymphadenectomy, and surgeons' opinion was evaluated with an additional questionnaire. The imaging and 3D reconstructions were available on a mobile device. RESULTS: The time of studying the imaging to gain confidence was significantly shorter with the 3D reconstruction. Laparoscopic intervention time was shortened and errors were reduced significantly within group 2. The clinical application of 3D reconstructions in difficult cases was believed to be helpful. CONCLUSIONS: 3D reconstructions of preoperative imaging lead to better surgical performance in a difficult laparoscopic training environment. Surgeons gain a 3D impression of patients' individual anatomy easier, faster, and more reliable. Providing 3D reconstructions previous to surgery should be routinely implemented for patients with complex anatomical situations.

Mobile markerless augmented reality and its application in forensic medicine.

PURPOSE: During autopsy, forensic pathologists today mostly rely on visible indication, tactile perception and experience to determine the cause of death. Although computed tomography (CT) data is often available for the bodies under examination, these data are rarely used due to the lack of radiological workstations in the pathological suite. The data may prevent the forensic pathologist from damaging evidence by allowing him to associate, for example, external wounds to internal injuries. To facilitate this, we propose a new multimodal approach for intuitive visualization of forensic data and evaluate its feasibility. METHODS: A range camera is mounted on a tablet computer and positioned in a way such that the camera simultaneously captures depth and color information of the body. A server estimates the camera pose based on surface registration of CT and depth data to allow for augmented reality visualization of the internal anatomy directly on the tablet. Additionally, projection of color information onto the CT surface is implemented. RESULTS: We validated the system in a postmortem pilot study using fiducials attached to the skin for quantification of a mean target registration error of [Formula: see text] mm. CONCLUSIONS: The system is mobile, markerless, intuitive and real-time capable with sufficient accuracy. It can support the forensic pathologist during autopsy with augmented reality and textured surfaces. Furthermore, the system enables multimodal documentation for presentation in court. Despite its preliminary prototype status, it has high potential due to its low price and simplicity.

The Medical Imaging Interaction Toolkit: challenges and advances : 10 years of open-source development.

PURPOSE: The Medical Imaging Interaction Toolkit (MITK) has been available as open-source software for almost 10 years now. In this period the requirements of software systems in the medical image processing domain have become increasingly complex. The aim of this paper is to show how MITK evolved into a software system that is able to cover all steps of a clinical workflow including data retrieval, image analysis, diagnosis, treatment planning, intervention support, and treatment control. METHODS: MITK provides modularization and extensibility on different levels. In addition to the original toolkit, a module system, micro services for small, system-wide features, a service-oriented architecture based on the Open Services Gateway initiative (OSGi) standard, and an extensible and configurable application framework allow MITK to be used, extended and deployed as needed. A refined software process was implemented to deliver high-quality software, ease the fulfillment of regulatory requirements, and enable teamwork in mixed-competence teams. RESULTS: MITK has been applied by a worldwide community and integrated into a variety of solutions, either at the toolkit level or as an application framework with custom extensions. The MITK Workbench has been released as a highly extensible and customizable end-user application. Optional support for tool tracking, image-guided therapy, diffusion imaging as well as various external packages (e.g. CTK, DCMTK, OpenCV, SOFA, Python) is available. MITK has also been used in several FDA/CE-certified applications, which demonstrates the high-quality software and rigorous development process. CONCLUSIONS: MITK provides a versatile platform with a high degree of modularization and interoperability and is well suited to meet the challenging tasks of today's and tomorrow's clinically motivated research.

Convergent iterative closest-point algorithm to accomodate anisotropic and inhomogenous localization error.

Since its introduction in the early 1990s, the Iterative Closest Point (ICP) algorithm has become one of the most well-known methods for geometric alignment of 3D models. Given two roughly aligned shapes represented by two point sets, the algorithm iteratively establishes point correspondences given the current alignment of the data and computes a rigid transformation accordingly. From a statistical point of view, however, it implicitly assumes that the points are observed with isotropic Gaussian noise. In this paper, we show that this assumption may lead to errors and generalize the ICP such that it can account for anisotropic and inhomogenous localization errors. We 1) provide a formal description of the algorithm, 2) extend it to registration of partially overlapping surfaces, 3) prove its convergence, 4) derive the required covariance matrices for a set of selected applications, and 5) present means for optimizing the runtime. An evaluation on publicly available surface meshes as well as on a set of meshes extracted from medical imaging data shows a dramatic increase in accuracy compared to the original ICP, especially in the case of partial surface registration. As point-based surface registration is a central component in various applications, the potential impact of the proposed method is high.