Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery.

Laparoscopic surgery has evolved as a key technique for cancer diagnosis and therapy. While characterization of the tissue perfusion is crucial in various procedures, such as partial nephrectomy, doing so by means of visual inspection remains highly challenging. We developed a laparoscopic real-time multispectral imaging system featuring a compact and lightweight multispectral camera and the possibility to complement the conventional surgical view of the patient with functional information at a video rate of 25 Hz. To enable contrast agent-free ischemia monitoring during laparoscopic partial nephrectomy, we phrase the problem of ischemia detection as an out-of-distribution detection problem that does not rely on data from any other patient and uses an ensemble of invertible neural networks at its core. An in-human trial demonstrates the feasibility of our approach and highlights the potential of spectral imaging combined with advanced deep learning-based analysis tools for fast, efficient, reliable, and safe functional laparoscopic imaging.

Robust hand tracking for surgical telestration.

PURPOSE: As human failure has been shown to be one primary cause for post-operative death, surgical training is of the utmost socioeconomic importance. In this context, the concept of surgical telestration has been introduced to enable experienced surgeons to efficiently and effectively mentor trainees in an intuitive way. While previous approaches to telestration have concentrated on overlaying drawings on surgical videos, we explore the augmented reality (AR) visualization of surgical hands to imitate the direct interaction with the situs. METHODS: We present a real-time hand tracking pipeline specifically designed for the application of surgical telestration. It comprises three modules, dedicated to (1) the coarse localization of the expert's hand and the subsequent (2) segmentation of the hand for AR visualization in the field of view of the trainee and (3) regression of keypoints making up the hand's skeleton. The semantic representation is obtained to offer the ability for structured reporting of the motions performed as part of the teaching. RESULTS: According to a comprehensive validation based on a large data set comprising more than 14,000 annotated images with varying application-relevant conditions, our algorithm enables real-time hand tracking and is sufficiently accurate for the task of surgical telestration. In a retrospective validation study, a mean detection accuracy of 98%, a mean keypoint regression accuracy of 10.0 px and a mean Dice Similarity Coefficient of 0.95 were achieved. In a prospective validation study, it showed uncompromised performance when the sensor, operator or gesture varied. CONCLUSION: Due to its high accuracy and fast inference time, our neural network-based approach to hand tracking is well suited for an AR approach to surgical telestration. Future work should be directed to evaluating the clinical value of the approach.

Reattachable fiducial skin marker for automatic multimodality registration.

PURPOSE: Fusing image information has become increasingly important for optimal diagnosis and treatment of the patient. Despite intensive research towards markerless registration approaches, fiducial marker-based methods remain the default choice for a wide range of applications in clinical practice. However, as especially non-invasive markers cannot be positioned reproducibly in the same pose on the patient, pre-interventional imaging has to be performed immediately before the intervention for fiducial marker-based registrations. METHODS: We propose a new non-invasive, reattachable fiducial skin marker concept for multi-modal registration approaches including the use of electromagnetic or optical tracking technologies. We furthermore describe a robust, automatic fiducial marker localization algorithm for computed tomography (CT) and magnetic resonance imaging (MRI) images. Localization of the new fiducial marker has been assessed for different marker configurations using both CT and MRI. Furthermore, we applied the marker in an abdominal phantom study. For this, we attached the marker at three poses to the phantom, registered ten segmented targets of the phantom's CT image to live ultrasound images and determined the target registration error (TRE) for each target and each marker pose. RESULTS: Reattachment of the marker was possible with a mean precision of 0.02 mm ± 0.01 mm. Our algorithm successfully localized the marker automatically in all ([Formula: see text]) evaluated CT/MRI images. Depending on the marker pose, the mean ([Formula: see text]) TRE of the abdominal phantom study ranged from 1.51 ± 0.75 mm to 4.65 ± 1.22 mm. CONCLUSIONS: The non-invasive, reattachable skin marker concept allows reproducible positioning of the marker and automatic localization in different imaging modalities. The low TREs indicate the potential applicability of the marker concept for clinical interventions, such as the puncture of abdominal lesions, where current image-based registration approaches still lack robustness and existing marker-based methods are often impractical.

Surgical data science - from concepts toward clinical translation.

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

Tattoo tomography: Freehand 3D photoacoustic image reconstruction with an optical pattern.

PURPOSE: Photoacoustic tomography (PAT) is a novel imaging technique that can spatially resolve both morphological and functional tissue properties, such as vessel topology and tissue oxygenation. While this capacity makes PAT a promising modality for the diagnosis, treatment, and follow-up of various diseases, a current drawback is the limited field of view provided by the conventionally applied 2D probes. METHODS: In this paper, we present a novel approach to 3D reconstruction of PAT data (Tattoo tomography) that does not require an external tracking system and can smoothly be integrated into clinical workflows. It is based on an optical pattern placed on the region of interest prior to image acquisition. This pattern is designed in a way that a single tomographic image of it enables the recovery of the probe pose relative to the coordinate system of the pattern, which serves as a global coordinate system for image compounding. RESULTS: To investigate the feasibility of Tattoo tomography, we assessed the quality of 3D image reconstruction with experimental phantom data and in vivo forearm data. The results obtained with our prototype indicate that the Tattoo method enables the accurate and precise 3D reconstruction of PAT data and may be better suited for this task than the baseline method using optical tracking. CONCLUSIONS: In contrast to previous approaches to 3D ultrasound (US) or PAT reconstruction, the Tattoo approach neither requires complex external hardware nor training data acquired for a specific application. It could thus become a valuable tool for clinical freehand PAT.

Model-based registration of preprocedure MR and intraprocedure US of the lumbar spine.

PURPOSE: Epidural and spinal needle insertions, as well as facet joint denervation and injections are widely performed procedures on the lumbar spine for delivering anesthesia and analgesia. Ultrasound (US)-based approaches have gained popularity for accurate needle placement, as they use a non-ionizing, inexpensive and accessible modality for guiding these procedures. However, due to the inherent difficulties in interpreting spinal US, they yet to become the clinical standard-of-care. METHODS: A novel statistical shape [Formula: see text] pose [Formula: see text] scale (s [Formula: see text] p [Formula: see text] s) model of the lumbar spine is jointly registered to preoperative magnetic resonance (MR) and US images. An instance of the model is created for each modality. The shape and scale model parameters are jointly computed, while the pose parameters are estimated separately for each modality. RESULTS: The proposed method is successfully applied to nine pairs of preoperative clinical MR volumes and their corresponding US images. The results are assessed using the target registration error (TRE) metric in both MR and US domains. The s [Formula: see text] p [Formula: see text] s model in the proposed joint registration framework results in a mean TRE of 2.62 and 4.20 mm for MR and US images, respectively, on different landmarks. CONCLUSION: The joint framework benefits from the complementary features in both modalities, leading to significantly smaller TREs compared to a model-to-US registration approach. The s [Formula: see text] p [Formula: see text] s model also outperforms our previous shape [Formula: see text] pose model of the lumbar spine, as separating scale from pose allows to better capture pose and guarantees equally-sized vertebrae in both modalities. Furthermore, the simultaneous visualization of the patient-specific models on the MR and US domains makes it possible for clinicians to better evaluate the local registration accuracy.

Anser EMT: the first open-source electromagnetic tracking platform for image-guided interventions.

PURPOSE: Electromagnetic tracking is the gold standard for instrument tracking and navigation in the clinical setting without line of sight. Whilst clinical platforms exist for interventional bronchoscopy and neurosurgical navigation, the limited flexibility and high costs of electromagnetic tracking (EMT) systems for research investigations mitigate against a better understanding of the technology's characterisation and limitations. The Anser project provides an open-source implementation for EMT with particular application to image-guided interventions. METHODS: This work provides implementation schematics for our previously reported EMT system which relies on low-cost acquisition and demodulation techniques using both National Instruments and Arduino hardware alongside MATLAB support code. The system performance is objectively compared to other commercial tracking platforms using the Hummel assessment protocol. RESULTS: Positional accuracy of 1.14 mm and angular rotation accuracy of [Formula: see text] are reported. Like other EMT platforms, Anser is susceptible to tracking errors due to eddy current and ferromagnetic distortion. The system is compatible with commercially available EMT sensors as well as the Open Network Interface for image-guided therapy (OpenIGTLink) for easy communication with visualisation and medical imaging toolkits such as MITK and 3D Slicer. CONCLUSIONS: By providing an open-source platform for research investigations, we believe that novel and collaborative approaches can overcome the limitations of current EMT technology.

First clinical use of the EchoTrack guidance approach for radiofrequency ablation of thyroid gland nodules.

PURPOSE: Percutaneous radiofrequency ablation (RFA) of thyroid nodules is an alternative to surgical resection that offers the benefits of minimal scars for the patient, lower complication rates, and shorter treatment times. Ultrasound (US) is the preferred modality for guiding these procedures. The needle is usually kept within the US scanning plane to ensure needle visibility. However, this restricts flexibility in both transducer and needle movement and renders the procedure difficult, especially for inexperienced users. Existing navigation solutions often involve electromagnetic (EM) tracking, which requires placement of an external field generator (FG) in close proximity of the intervention site in order to avoid distortion of the EM field. This complicates the clinical workflow as placing the FG while ensuring that it neither restricts the physician's workspace nor affects tracking accuracy is awkward and time-consuming. METHODS: The EchoTrack concept overcomes these issues by combining the US probe and the EM FG in one modality, simultaneously providing both real-time US and tracking data without requiring the placement of an external FG for tracking. We propose a system and workflow to use EchoTrack for RFA of thyroid nodules. RESULTS: According to our results, the overall error of the EchoTrack system resulting from errors related to tracking and calibration is below 2 mm. Navigated thyroid RFA with the proposed concept is clinically feasible. Motion of internal critical structures relative to external markers can be up to several millimeters in extreme cases. CONCLUSIONS: The EchoTrack concept with its simple setup, flexibility, improved needle visualization, and additional guidance information has high potential to be clinically used for thyroid RFA.

Ultrasound-navigated radiofrequency ablation of thyroid nodules with integrated electromagnetic tracking: comparison with conventional ultrasound guidance in gelatin models.

PURPOSE: A thyroid-like gelatin model was used to determine potential superiority of a new navigation system for ultrasound (US)-guided electrode insertion called EchoTrack, featuring a US probe with an integrated electromagnetic field generator, in comparison with conventional US when performing radiofrequency ablation. METHODS: In order to compare 20 navigated ablations with 20 ablations under conventional US guidance, a thyroid-like gelatin model was used. In each group, 10 in-plane and 10 out-of-plane punctures were performed. Metal seeds measuring 8.5 [Formula: see text] 1.8 mm served as ablation targets. The number of redirections until final electrode placement, targeting accuracy and electrode placement time were measured. RESULTS: The number of redirections could be significantly ([Formula: see text]) reduced from 2.7 ± 1.3 in the conventional group to 0.2 ± 0.5 in the EchoTrack group. Accuracy increased from 3.9 ± 4.7 to 2.0 ± 1.9 mm. The total placement time increased from 39 ± 20.5 to 79.2 ± 26 s. CONCLUSIONS: EchoTrack is able to reduce the redirections needed to place the electrode in comparison with conventional US and provides high placement accuracy. Our new navigation system has high potential to reduce the risk of harming critical structures and to improve guidance during ablation of difficult nodules, as treatment planning as well as the safety of out-of-plane punctures are improved.

MITK-OpenIGTLink for combining open-source toolkits in real-time computer-assisted interventions.

PURPOSE: Due to rapid developments in the research areas of medical imaging, medical image processing and robotics, computer-assisted interventions (CAI) are becoming an integral part of modern patient care. From a software engineering point of view, these systems are highly complex and research can benefit greatly from reusing software components. This is supported by a number of open-source toolkits for medical imaging and CAI such as the medical imaging interaction toolkit (MITK), the public software library for ultrasound imaging research (PLUS) and 3D Slicer. An independent inter-toolkit communication such as the open image-guided therapy link (OpenIGTLink) can be used to combine the advantages of these toolkits and enable an easier realization of a clinical CAI workflow. METHODS: MITK-OpenIGTLink is presented as a network interface within MITK that allows easy to use, asynchronous two-way messaging between MITK and clinical devices or other toolkits. Performance and interoperability tests with MITK-OpenIGTLink were carried out considering the whole CAI workflow from data acquisition over processing to visualization. RESULTS: We present how MITK-OpenIGTLink can be applied in different usage scenarios. In performance tests, tracking data were transmitted with a frame rate of up to 1000 Hz and a latency of 2.81 ms. Transmission of images with typical ultrasound (US) and greyscale high-definition (HD) resolutions of [Formula: see text] and [Formula: see text] is possible at up to 512 and 128 Hz, respectively. CONCLUSION: With the integration of OpenIGTLink into MITK, this protocol is now supported by all established open-source toolkits in the field. This eases interoperability between MITK and toolkits such as PLUS or 3D Slicer and facilitates cross-toolkit research collaborations. MITK and its submodule MITK-OpenIGTLink are provided open source under a BSD-style licence ( http://mitk.org ).

Registration of a statistical model to intraoperative ultrasound for scaphoid screw fixation.

PURPOSE: Volar percutaneous scaphoid fracture fixation is conventionally performed under fluoroscopy-based guidance, where surgeons need to mentally determine a trajectory for the insertion of the screw and its depth based on a series of 2D projection images. In addition to challenges associated with mapping 2D information to a 3D space, the process involves exposure to ionizing radiation. Three-dimensional ultrasound has been suggested as an alternative imaging tool for this procedure; however, it has not yet been integrated into clinical routine since ultrasound only provides a limited view of the scaphoid and its surrounding anatomy. METHODS: We propose a registration of a statistical wrist shape + scale + pose model to a preoperative CT and intraoperative ultrasound to derive a patient-specific 3D model for guiding scaphoid fracture fixation. The registered model is then used to determine clinically important intervention parameters, including the screw length and the trajectory of screw insertion in the scaphoid bone. RESULTS: Feasibility experiments are performed using 13 cadaver wrists. In 10 out of 13 cases, the trajectory of screw suggested by the registered model meets all clinically important intervention parameters. Overall, an average 94 % of maximum allowable screw length is obtained based on the measurements from gold standard CT. Also, we obtained an average 92 % successful volar accessibility, which indicates that the trajectory is not obstructed by the surrounding trapezium bone. CONCLUSIONS: These promising results indicate that determining clinically important screw insertion parameters for scaphoid fracture fixation is feasible using 3D ultrasound imaging. This suggests the potential of this technology in replacing fluoroscopic guidance for this procedure in future applications.

Joint registration of ultrasound, CT and a shape+pose statistical model of the lumbar spine for guiding anesthesia.

PURPOSE: Facet joint injections and epidural needle insertions are widely used for spine anesthesia. Accurate needle placement is important for effective therapy delivery and avoiding complications arising from damage of soft tissue and nerves. Needle guidance is usually performed by fluoroscopy or palpation, resulting in radiation exposure and multiple needle re-insertions. Several ultrasound (US)-based approaches have been proposed but have not found wide acceptance in clinical routine. This is mainly due to difficulties in interpretation of the complex spinal anatomy in US, which leads to clinicians' lack of confidence in relying only on information derived from US for needle guidance. METHODS: We introduce a multimodal joint registration technique that takes advantage of easy-to-interpret preprocedure computed topography (CT) scans of the lumbar spine to concurrently register a shape+pose model to the intraprocedure 3D US. Common shape coefficients are assumed between two modalities, while pose coefficients are specific to each modality. RESULTS: The joint method was evaluated on patient data consisting of ten pairs of US and CT scans of the lumbar spine. It was successfully applied in all cases and yielded an RMS shape error of 2.1 mm compared to the CT ground truth. The joint registration technique was compared to a previously proposed method of statistical model to US registration Rasoulian et al. (Information processing in computer-assisted interventions. Springer, Berlin, pp 51-60, 2013). The joint framework improved registration accuracy to US in 7 out of 17 visible vertebrae, belonging to four patients. In the remaining cases, the two methods were equally accurate. CONCLUSION: The joint registration allows visualization and augmentation of important anatomy in both the US and CT domain and improves the registration accuracy in both modalities. Observing the patient-specific model in the CT domain allows the clinicians to assess the local registration accuracy qualitatively, which is likely to increase their confidence in using the US model for deriving needle guidance decisions.

Towards markerless navigation for percutaneous needle insertions.

PURPOSE: Percutaneous needle insertions are increasingly used for diagnosis and treatment of abdominal lesions. The challenging part of computed tomography (CT)-guided punctures is the transfer of the insertion trajectory planned in the CT image to the patient. Conventionally, this often results in several needle repositionings and control CT scans. To address this issue, several navigation systems for percutaneous needle insertions have been presented; however, none of them has thus far become widely accepted in clinical routine. Their benefit for the patient could not exceed the additional higher costs and the increased complexity in terms of bulky tracking systems and specialized markers for registration and tracking. METHODS: We present the first markerless and trackerless navigation concept for real-time patient localization and instrument guidance. It has specifically been designed to be integrated smoothly into the clinical workflow and does not require markers or an external tracking system. The main idea is the utilization of a range imaging device that allows for contactless and radiation-free acquisition of both range and color information used for patient localization and instrument guidance. RESULTS: A first feasibility study in phantom and porcine models yielded a median targeting accuracy of 6.9 and 19.4 mm, respectively. CONCLUSIONS: Although system performance remains to be improved for clinical use, expected advances in camera technology as well as consideration of respiratory motion and automation of the individual steps will make this approach an interesting alternative for guiding percutaneous needle insertions.

Ultrasound-guided spinal injections: a feasibility study of a guidance system.

PURPOSE: Facet joint injections of analgesic agents are widely used to treat patients with lower back pain. The current standard-of-care for guiding the injection is fluoroscopy, which exposes the patient and physician to significant radiation. As an alternative, several ultrasound guidance systems have been proposed, but have not become the standard-of-care, mainly because of the difficulty in image interpretation by the anesthesiologist unfamiliar with the complex spinal sonography. METHODS: We introduce an ultrasound-based navigation system that allows for live 2D ultrasound images augmented with a patient-specific statistical model of the spine and relating this information to the position of the tracked injection needle. The model registration accuracy is assessed on ultrasound data obtained from nine subjects who had prior CT images as the gold standard for the statistical model. The clinical validity of our method is evaluated on four subjects (of an ongoing in vivo study) which underwent facet joint injections. RESULTS: The statistical model could be registered to the bone structures in the ultrasound volume with an average RMS accuracy of 2.3±0.4 mm. The shape of the individual vertebrae could be estimated from the US volume with an average RMS surface distance error of 1.5±0.4 mm. The facet joints could be identified by the statistical model with an average accuracy of 5.1 ± 1.5 mm. CONCLUSIONS: The results of this initial feasibility assessment suggest that this ultrasound-based system is capable of providing information sufficient to guide facet joint injections. Further clinical studies are warranted.

Towards real-time, tracker-less 3D ultrasound guidance for spine anaesthesia.

PURPOSE: Epidural needle insertions and facet joint injections play an important role in spine anaesthesia. The main challenge of safe needle insertion is the deep location of the target, resulting in a narrow and small insertion channel close to sensitive anatomy. Recent approaches utilizing ultrasound (US) as a low-cost and widely available guiding modality are promising but have yet to become routinely used in clinical practice due to the difficulty in interpreting US images, their limited view of the internal anatomy of the spine, and/or inclusion of cost-intensive tracking hardware which impacts the clinical workflow. METHODS: We propose a novel guidance system for spine anaesthesia. An efficient implementation allows us to continuously align and overlay a statistical model of the lumbar spine on the live 3D US stream without making use of additional tracking hardware. The system is evaluated in vivo on 12 volunteers. RESULTS: The in vivo study showed that the anatomical features of the epidural space and the facet joints could be continuously located, at a volume rate of 0.5 Hz, within an accuracy of 3 and 7 mm, respectively. CONCLUSIONS: A novel guidance system for spine anaesthesia has been presented which augments a live 3D US stream with detailed anatomical information of the spine. Results from an in vivo study indicate that the proposed system has potential for assisting the physician in quickly finding the target structure and planning a safe insertion trajectory in the spine.

Bone enhancement in ultrasound using local spectrum variations for guiding percutaneous scaphoid fracture fixation procedures.

PURPOSE: The scaphoid bone is the most frequently fractured bone in the wrist. When fracture fixation is indicated, a screw is inserted into the bone either in an open surgical procedure or percutaneously under fluoroscopic guidance. Due to the complex geometry of the wrist, fracture fixation is a challenging task. Fluoroscopic guidance exposes both the patient and the physician to ionizing radiation. Ultrasound-based guidance has been suggested as a real-time, radiation-free alternative. The main challenge of using ultrasound is the difficulty in interpreting the images due to the low contrast and noisy nature of the data. METHODS: We propose a bone enhancement method that exploits local spectrum features of the ultrasound image. These features are utilized to design a set of quadrature band-pass filters and subsequently estimate the local phase symmetry, where high symmetry is expected at the bone locations. We incorporate the shadow information below the bone surfaces to further enhance the bone responses. The extracted bone surfaces are then used to register a statistical wrist model to ultrasound volumes, allowing the localization and interpretation of the scaphoid bone in the volumes. RESULTS: Feasibility experiments were performed using phantom and in vivo data. For phantoms, we obtain a surface distance error 1.08 mm and an angular deviation from the main axis of the scaphoid bone smaller than 5°, which are better compared to previously presented approaches. CONCLUSION: The results are promising for further development of a surgical guidance system to enable accurate anatomy localization for guiding percutaneous scaphoid fracture fixations.

Mobile markerless augmented reality and its application in forensic medicine.

PURPOSE: During autopsy, forensic pathologists today mostly rely on visible indication, tactile perception and experience to determine the cause of death. Although computed tomography (CT) data is often available for the bodies under examination, these data are rarely used due to the lack of radiological workstations in the pathological suite. The data may prevent the forensic pathologist from damaging evidence by allowing him to associate, for example, external wounds to internal injuries. To facilitate this, we propose a new multimodal approach for intuitive visualization of forensic data and evaluate its feasibility. METHODS: A range camera is mounted on a tablet computer and positioned in a way such that the camera simultaneously captures depth and color information of the body. A server estimates the camera pose based on surface registration of CT and depth data to allow for augmented reality visualization of the internal anatomy directly on the tablet. Additionally, projection of color information onto the CT surface is implemented. RESULTS: We validated the system in a postmortem pilot study using fiducials attached to the skin for quantification of a mean target registration error of [Formula: see text] mm. CONCLUSIONS: The system is mobile, markerless, intuitive and real-time capable with sufficient accuracy. It can support the forensic pathologist during autopsy with augmented reality and textured surfaces. Furthermore, the system enables multimodal documentation for presentation in court. Despite its preliminary prototype status, it has high potential due to its low price and simplicity.

Lack of centrioles and primary cilia in STIL(-/-) mouse embryos.

Although most animal cells contain centrosomes, consisting of a pair of centrioles, their precise contribution to cell division and embryonic development is unclear. Genetic ablation of STIL, an essential component of the centriole replication machinery in mammalian cells, causes embryonic lethality in mice around mid gestation associated with defective Hedgehog signaling. Here, we describe, by focused ion beam scanning electron microscopy, that STIL(-/-) mouse embryos do not contain centrioles or primary cilia, suggesting that these organelles are not essential for mammalian development until mid gestation. We further show that the lack of primary cilia explains the absence of Hedgehog signaling in STIL(-/-) cells. Exogenous re-expression of STIL or STIL microcephaly mutants compatible with human survival, induced non-templated, de novo generation of centrioles in STIL(-/-) cells. Thus, while the abscence of centrioles is compatible with mammalian gastrulation, lack of centrioles and primary cilia impairs Hedgehog signaling and further embryonic development.