ABANICCO: A New Color Space for Multi-Label Pixel Classification and Color Analysis.

Classifying pixels according to color, and segmenting the respective areas, are necessary steps in any computer vision task that involves color images. The gap between human color perception, linguistic color terminology, and digital representation are the main challenges for developing methods that properly classify pixels based on color. To address these challenges, we propose a novel method combining geometric analysis, color theory, fuzzy color theory, and multi-label systems for the automatic classification of pixels into 12 conventional color categories, and the subsequent accurate description of each of the detected colors. This method presents a robust, unsupervised, and unbiased strategy for color naming, based on statistics and color theory. The proposed model, "ABANICCO" (AB ANgular Illustrative Classification of COlor), was evaluated through different experiments: its color detection, classification, and naming performance were assessed against the standardized ISCC-NBS color system; its usefulness for image segmentation was tested against state-of-the-art methods. This empirical evaluation provided evidence of ABANICCO's accuracy in color analysis, showing how our proposed model offers a standardized, reliable, and understandable alternative for color naming that is recognizable by both humans and machines. Hence, ABANICCO can serve as a foundation for successfully addressing a myriad of challenges in various areas of computer vision, such as region characterization, histopathology analysis, fire detection, product quality prediction, object description, and hyperspectral imaging.

HoloLens 1 vs. HoloLens 2: Improvements in the New Model for Orthopedic Oncological Interventions.

This work analyzed the use of Microsoft HoloLens 2 in orthopedic oncological surgeries and compares it to its predecessor (Microsoft HoloLens 1). Specifically, we developed two equivalent applications, one for each device, and evaluated the augmented reality (AR) projection accuracy in an experimental scenario using phantoms based on two patients. We achieved automatic registration between virtual and real worlds using patient-specific surgical guides on each phantom. They contained a small adaptor for a 3D-printed AR marker, the characteristic patterns of which were easily recognized using both Microsoft HoloLens devices. The newest model improved the AR projection accuracy by almost 25%, and both of them yielded an RMSE below 3 mm. After ascertaining the enhancement of the second model in this aspect, we went a step further with Microsoft HoloLens 2 and tested it during the surgical intervention of one of the patients. During this experience, we collected the surgeons' feedback in terms of comfortability, usability, and ergonomics. Our goal was to estimate whether the improved technical features of the newest model facilitate its implementation in actual surgical scenarios. All of the results point to Microsoft HoloLens 2 being better in all the aspects affecting surgical interventions and support its use in future experiences.

Deep Learning-Based Segmentation of Head and Neck Organs-at-Risk with Clinical Partially Labeled Data.

Radiotherapy is one of the main treatments for localized head and neck (HN) cancer. To design a personalized treatment with reduced radio-induced toxicity, accurate delineation of organs at risk (OAR) is a crucial step. Manual delineation is time- and labor-consuming, as well as observer-dependent. Deep learning (DL) based segmentation has proven to overcome some of these limitations, but requires large databases of homogeneously contoured image sets for robust training. However, these are not easily obtained from the standard clinical protocols as the OARs delineated may vary depending on the patient's tumor site and specific treatment plan. This results in incomplete or partially labeled data. This paper presents a solution to train a robust DL-based automated segmentation tool exploiting a clinical partially labeled dataset. We propose a two-step workflow for OAR segmentation: first, we developed longitudinal OAR-specific 3D segmentation models for pseudo-contour generation, completing the missing contours for some patients; with all OAR available, we trained a multi-class 3D convolutional neural network (nnU-Net) for final OAR segmentation. Results obtained in 44 independent datasets showed superior performance of the proposed methodology for the segmentation of fifteen OARs, with an average Dice score coefficient and surface Dice similarity coefficient of 80.59% and 88.74%. We demonstrated that the model can be straightforwardly integrated into the clinical workflow for standard and adaptive radiotherapy.

Conceptual evolution of 3D printing in orthopedic surgery and traumatology: from "do it yourself" to "point of care manufacturing".

BACKGROUND: 3D printing technology in hospitals facilitates production models such as point-of-care manufacturing. Orthopedic Surgery and Traumatology is the specialty that can most benefit from the advantages of these tools. The purpose of this study is to present the results of the integration of 3D printing technology in a Department of Orthopedic Surgery and Traumatology and to identify the productive model of the point-of-care manufacturing as a paradigm of personalized medicine. METHODS: Observational, descriptive, retrospective and monocentric study of a total of 623 additive manufacturing processes carried out in a Department of Orthopedic Surgery and Traumatology from November 2015 to March 2020. Variables such as product type, utility, time or materials for manufacture were analyzed. RESULTS: The areas of expertise that have performed more processes are Traumatology, Reconstructive and Orthopedic Oncology. Pre-operative planning is their primary use. Working and 3D printing hours, as well as the amount of 3D printing material used, vary according to the type of product or material delivered to perform the process. The most commonly used 3D printing material for manufacturing is polylactic acid, although biocompatible resin has been used to produce surgical guides. In addition, the hospital has worked on the co-design of customized implants with manufacturing companies. CONCLUSIONS: The integration of 3D printing in a Department of Orthopedic Surgery and Traumatology allows identifying the conceptual evolution from "Do-It-Yourself" to "POC manufacturing".

Combining Augmented Reality and 3D Printing to Improve Surgical Workflows in Orthopedic Oncology: Smartphone Application and Clinical Evaluation.

During the last decade, orthopedic oncology has experienced the benefits of computerized medical imaging to reduce human dependency, improving accuracy and clinical outcomes. However, traditional surgical navigation systems do not always adapt properly to this kind of interventions. Augmented reality (AR) and three-dimensional (3D) printing are technologies lately introduced in the surgical environment with promising results. Here we present an innovative solution combining 3D printing and AR in orthopedic oncological surgery. A new surgical workflow is proposed, including 3D printed models and a novel AR-based smartphone application (app). This app can display the patient's anatomy and the tumor's location. A 3D-printed reference marker, designed to fit in a unique position of the affected bone tissue, enables automatic registration. The system has been evaluated in terms of visualization accuracy and usability during the whole surgical workflow. Experiments on six realistic phantoms provided a visualization error below 3 mm. The AR system was tested in two clinical cases during surgical planning, patient communication, and surgical intervention. These results and the positive feedback obtained from surgeons and patients suggest that the combination of AR and 3D printing can improve efficacy, accuracy, and patients' experience.

Augmented Reality as a Tool to Guide PSI Placement in Pelvic Tumor Resections.

Patient-specific instruments (PSIs) have become a valuable tool for osteotomy guidance in complex surgical scenarios such as pelvic tumor resection. They provide similar accuracy to surgical navigation systems but are generally more convenient and faster. However, their correct placement can become challenging in some anatomical regions, and it cannot be verified objectively during the intervention. Incorrect installations can result in high deviations from the planned osteotomy, increasing the risk of positive resection margins. In this work, we propose to use augmented reality (AR) to guide and verify PSIs placement. We designed an experiment to assess the accuracy provided by the system using a smartphone and the HoloLens 2 and compared the results with the conventional freehand method. The results showed significant differences, where AR guidance prevented high osteotomy deviations, reducing maximal deviation of 54.03 mm for freehand placements to less than 5 mm with AR guidance. The experiment was performed in two versions of a plastic three-dimensional (3D) printed phantom, one including a silicone layer to simulate tissue, providing more realism. We also studied how differences in shape and location of PSIs affect their accuracy, concluding that those with smaller sizes and a homogeneous target surface are more prone to errors. Our study presents promising results that prove AR's potential to overcome the present limitations of PSIs conveniently and effectively.

Real-Time Tool Detection for Workflow Identification in Open Cranial Vault Remodeling.

Deep learning is a recent technology that has shown excellent capabilities for recognition and identification tasks. This study applies these techniques in open cranial vault remodeling surgeries performed to correct craniosynostosis. The objective was to automatically recognize surgical tools in real-time and estimate the surgical phase based on those predictions. For this purpose, we implemented, trained, and tested three algorithms based on previously proposed Convolutional Neural Network architectures (VGG16, MobileNetV2, and InceptionV3) and one new architecture with fewer parameters (CranioNet). A novel 3D Slicer module was specifically developed to implement these networks and recognize surgical tools in real time via video streaming. The training and test data were acquired during a surgical simulation using a 3D printed patient-based realistic phantom of an infant's head. The results showed that CranioNet presents the lowest accuracy for tool recognition (93.4%), while the highest accuracy is achieved by the MobileNetV2 model (99.6%), followed by VGG16 and InceptionV3 (98.8% and 97.2%, respectively). Regarding phase detection, InceptionV3 and VGG16 obtained the best results (94.5% and 94.4%), whereas MobileNetV2 and CranioNet presented worse values (91.1% and 89.8%). Our results prove the feasibility of applying deep learning architectures for real-time tool detection and phase estimation in craniosynostosis surgeries.

Performance of ultra-high-frequency ultrasound in the evaluation of skin involvement in systemic sclerosis: a preliminary report.

OBJECTIVE: High frequency ultrasound allows visualization of epidermis, dermis and hypodermis, precise measurement of skin thickness, as well as assessment of skin oedema, fibrosis and atrophy. The aim of this pilot cross-sectional observational study was to assess the performance and multiobserver variability of ultra-high-frequency (UHF) (50 MHz) ultrasound (US) in measuring skin thickness as well as the capacity of UHF-derived skin features to differentiate SSc patients from healthy controls. METHODS: Twenty-one SSc patients (16 limited and five diffuse SSc) and six healthy controls were enrolled. All subjects underwent US evaluation by three experts at three anatomical sites (forearm, hand and finger). Dermal thickness was measured and two rectangular regions of interest, one in dermis and one in hypodermis, were established for texture feature analysis. RESULTS: UHF-US allowed a precise identification and measurement of the thickness of the dermis. The dermal thickness in the finger was significantly higher in patients than in controls (P < 0.05), while in the forearm it was significantly lower in patients than in controls (P < 0.001). Interobserver variability for dermal thickness was good to excellent [forearm intraclass correlation coefficient (ICC) = 0.754; finger ICC = 0.699; hand ICC = 0.602]. Texture computed analysis of dermis and hypodermis was able to discriminate between SSc and healthy subjects (area under the curve >0.7). CONCLUSION: These preliminary data show that skin UHF-US allows a very detailed imaging of skin layers, a reliable measurement of dermal thickness, and a discriminative capacity between dermis and hypodermis texture features in SSc and healthy subjects.

Nonlinear Image Registration and Pixel Classification Pipeline for the Study of Tumor Heterogeneity Maps.

We present a novel method to assess the variations in protein expression and spatial heterogeneity of tumor biopsies with application in computational pathology. This was done using different antigen stains for each tissue section and proceeding with a complex image registration followed by a final step of color segmentation to detect the exact location of the proteins of interest. For proper assessment, the registration needs to be highly accurate for the careful study of the antigen patterns. However, accurate registration of histopathological images comes with three main problems: the high amount of artifacts due to the complex biopsy preparation, the size of the images, and the complexity of the local morphology. Our method manages to achieve an accurate registration of the tissue cuts and segmentation of the positive antigen areas.

Spatiotemporal alignment of in utero BOLD-MRI series.

PURPOSE: To present a method for spatiotemporal alignment of in-utero magnetic resonance imaging (MRI) time series acquired during maternal hyperoxia for enabling improved quantitative tracking of blood oxygen level-dependent (BOLD) signal changes that characterize oxygen transport through the placenta to fetal organs. MATERIALS AND METHODS: The proposed pipeline for spatiotemporal alignment of images acquired with a single-shot gradient echo echo-planar imaging includes 1) signal nonuniformity correction, 2) intravolume motion correction based on nonrigid registration, 3) correction of motion and nonrigid deformations across volumes, and 4) detection of the outlier volumes to be discarded from subsequent analysis. BOLD MRI time series collected from 10 pregnant women during 3T scans were analyzed using this pipeline. To assess pipeline performance, signal fluctuations between consecutive timepoints were examined. In addition, volume overlap and distance between manual region of interest (ROI) delineations in a subset of frames and the delineations obtained through propagation of the ROIs from the reference frame were used to quantify alignment accuracy. A previously demonstrated rigid registration approach was used for comparison. RESULTS: The proposed pipeline improved anatomical alignment of placenta and fetal organs over the state-of-the-art rigid motion correction methods. In particular, unexpected temporal signal fluctuations during the first normoxia period were significantly decreased (P < 0.01) and volume overlap and distance between region boundaries measures were significantly improved (P < 0.01). CONCLUSION: The proposed approach to align MRI time series enables more accurate quantitative studies of placental function by improving spatiotemporal alignment across placenta and fetal organs. LEVEL OF EVIDENCE: 1 Technical Efficacy: Stage 1 J. MAGN. RESON. IMAGING 2017;46:403-412.

Bibliometrics of intraoperative radiotherapy: analysis of technology, practice and publication tendencies.

PURPOSE: To analyze the performance and quality of intraoperative radiation therapy (IORT) publications identified in medical databases during a recent period in terms of bibliographic metrics. MATERIALS AND METHODS: A bibliometric search was conducted for IORT papers published in the PubMed database between 1997 and 2013. Publication rate was used as a quantity indicator; the 2012 Science Citation Index Impact Factor as a quality indicator. Furthermore, the publications were stratified in terms of study type, scientific topic reported, year of publication, tumor type and journal specialty. We performed a one-way analysis of variance (ANOVA) to determine differences between the means of the analyzed groups. RESULTS: Among the total of 207 journals, articles were reported significantly more frequently in surgery (n = 399, 41 %) and radiotherapy journals (n = 273, 28 %; p < 0.01). The highest impact factor was achieved by clinical oncology journals (p < 0.01). The majority of identified articles were retrospective cohort reports (n = 622, 64 %), followed by review articles (n = 204, 21 %; p < 0.001). Regarding primary topic, reports on cancer outcome following specific tumor therapy were most frequently published (n = 661, 68 %; p < 0.001) and gained the highest mean impact factor (p < 0.01). Gastrointestinal tumor reports were represented most frequently (n = 456, 47 %; p < 0.001) and the mean superior impact factor was earned by breast and gynecologic publications (p < 0.01). CONCLUSION: We identified a consistent and sustained scientific productivity of international IORT expert groups. Most publications appeared in journals with surgical and radiooncological content. The highest impact factor was achieved by medical oncology journals.

Is a greater degree of constraint really harmful? Clinical biomechanical comparative study between condylar constrained knee and rotating hinge prosthesis.

BACKGROUND: The real degree of constraint of rotating hinge knee and condylar constrained prostheses is a matter of discussion in revision knee arthroplasty. The objective of this study is to compare the tibial rotation between implants in the clinical settings. METHODS: An investigator blinded experimental study was designed including 20 patients: in 10 of them a rotating hinge knee prosthesis (Endomodel®, LINK) was implanted and in the remaining 10 a constrained condylar knee prosthesis (LCCK®, Zimmer) was used. A medial parapatellar approach was performed and implantation was performed according to conventional surgical technique. Tibial rotation was measured with two accelerometers in full extension and at 30°, 60° and 90° of flexion. Pre and postoperative Knee Injury and Osteoarthritis Outcome Score was recorded. FINDINGS: Both groups were homogenous in age (73.4 years in rotating hinge knee prosthesis vs 74 years in constrained condylar knee group), sex, laterality and preoperative Knee Injury and Osteoarthritis Outcome Score (p > 0.05). The postoperative Knee Injury and Osteoarthritis Outcome Score was significantly higher in the rotating hinge knee prosthesis group (80.98 vs 76.28). The degrees of tibial rotation measured by inertial sensors in the rotating hinge knee prosthesis group were also significantly higher than those measured in the constrained condylar knee group (5.66° vs 2.1°) with p = 0.001. INTERPRETATION: Rotating hinge knee prosthesis appears to represent a lower rotational constraint degree than constrained condylar knee systems in clinical practice and it may increase the clinical satisfaction. The clinical significance: Rotating hinge knee prosthesis appears to represent a lower constraint degree than constrained condylar knee systems in clinical practice.

Tablet-based Augmented reality and 3D printed templates in fully guided Microtia Reconstruction: a clinical workflow.

BACKGROUND: Microtia is a congenital malformation of the auricle that affects approximately 4 of every 10,000 live newborns. Radiographic film paper is traditionally employed to bidimensionally trace the structures of the contralateral healthy ear in a quasi-artistic manner. Anatomical points provide linear and angular measurements. However, this technique proves time-consuming, subjectivity-rich, and greatly dependent on surgeon expertise. Hence, it's susceptible to shape errors and misplacement. METHODS: We present an innovative clinical workflow that combines 3D printing and augmented reality (AR) to increase objectivity and reproducibility of these procedures. Specifically, we introduce patient-specific 3D cutting templates and remodeling molds to carve and construct the cartilaginous framework that will conform the new ear. Moreover, we developed an in-house AR application compatible with any commercial Android tablet. It precisely guides the positioning of the new ear during surgery, ensuring symmetrical alignment with the healthy one and avoiding time-consuming intraoperative linear or angular measurements. Our solution was evaluated in one case, first with controlled experiments in a simulation scenario and finally during surgery. RESULTS: Overall, the ears placed in the simulation scenario had a mean absolute deviation of 2.2 ± 1.7 mm with respect to the reference plan. During the surgical intervention, the reconstructed ear was 3.1 mm longer and 1.3 mm wider with respect to the ideal plan and had a positioning error of 2.7 ± 2.4 mm relative to the contralateral side. Note that in this case, additional morphometric variations were induced from inflammation and other issues intended to be addressed in a subsequent stage of surgery, which are independent of our proposed solution. CONCLUSIONS: In this work we propose an innovative workflow that combines 3D printing and AR to improve ear reconstruction and positioning in microtia correction procedures. Our implementation in the surgical workflow showed good accuracy, empowering surgeons to attain consistent and objective outcomes.

Real-time integration between Microsoft HoloLens 2 and 3D Slicer with demonstration in pedicle screw placement planning.

PURPOSE: Up to date, there has been a lack of software infrastructure to connect 3D Slicer to any augmented reality (AR) device. This work describes a novel connection approach using Microsoft HoloLens 2 and OpenIGTLink, with a demonstration in pedicle screw placement planning. METHODS: We developed an AR application in Unity that is wirelessly rendered onto Microsoft HoloLens 2 using Holographic Remoting. Simultaneously, Unity connects to 3D Slicer using the OpenIGTLink communication protocol. Geometrical transform and image messages are transferred between both platforms in real time. Through the AR glasses, a user visualizes a patient's computed tomography overlaid onto virtual 3D models showing anatomical structures. We technically evaluated the system by measuring message transference latency between the platforms. Its functionality was assessed in pedicle screw placement planning. Six volunteers planned pedicle screws' position and orientation with the AR system and on a 2D desktop planner. We compared the placement accuracy of each screw with both methods. Finally, we administered a questionnaire to all participants to assess their experience with the AR system. RESULTS: The latency in message exchange is sufficiently low to enable real-time communication between the platforms. The AR method was non-inferior to the 2D desktop planner, with a mean error of 2.1 ± 1.4 mm. Moreover, 98% of the screw placements performed with the AR system were successful, according to the Gertzbein-Robbins scale. The average questionnaire outcomes were 4.5/5. CONCLUSIONS: Real-time communication between Microsoft HoloLens 2 and 3D Slicer is feasible and supports accurate planning for pedicle screw placement.

Clinical feasibility of combining intraoperative electron radiation therapy with minimally invasive surgery: a potential for electron-FLASH clinical development.

BACKGROUND: Local cancer therapy by combining real-time surgical exploration and resection with delivery of a single dose of high-energy electron irradiation entails a very precise and effective local therapeutic approach. Integrating the benefits from minimally invasive surgical techniques with the very precise delivery of intraoperative electron irradiation results in an efficient combined modality therapy. METHODS: Patients with locally advanced disease, who are candidates for laparoscopic and/or thoracoscopic surgery, received an integrated multimodal management. Preoperative treatment included induction chemotherapy and/or chemoradiation, followed by laparoscopic surgery and intraoperative electron radiation therapy. RESULTS: In a period of 5 consecutive years, 125 rectal cancer patients were treated, of which 35% underwent a laparoscopic approach. We found no differences in cancer outcomes and tolerance between the open and laparoscopic groups. Two esophageal cancer patients were treated with IOeRT during thoracoscopic resection, with the resection specimens showing intense downstaging effects. Two oligo-recurrent prostatic cancer patients (isolated nodal progression) had a robotic-assisted surgical resection and post-lymphadenectomy electron boost on the vascular and lateral pelvic wall. CONCLUSIONS: Minimally invasive and robotic-assisted surgery is feasible to combine with intraoperative electron radiation therapy and offers a new model explored with electron-FLASH beams.

Deep learning-based segmentation of prostatic urethra on computed tomography scans for treatment planning.

Background and purpose: The intraprostatic urethra is an organ at risk in prostate cancer radiotherapy, but its segmentation in computed tomography (CT) is challenging. This work sought to: i) propose an automatic pipeline for intraprostatic urethra segmentation in CT, ii) analyze the dose to the urethra, iii) compare the predictions to magnetic resonance (MR) contours. Materials and methods: First, we trained Deep Learning networks to segment the rectum, bladder, prostate, and seminal vesicles. Then, the proposed Deep Learning Urethra Segmentation model was trained with the bladder and prostate distance transforms and 44 labeled CT with visible catheters. The evaluation was performed on 11 datasets, calculating centerline distance (CLD) and percentage of centerline within 3.5 and 5 mm. We applied this method to a dataset of 32 patients treated with intensity-modulated radiation therapy (IMRT) to quantify the urethral dose. Finally, we compared predicted intraprostatic urethra contours to manual delineations in MR for 15 patients without catheter. Results: A mean CLD of 1.6 ± 0.8 mm for the whole urethra and 1.7 ± 1.4, 1.5 ± 0.9, and 1.7 ± 0.9 mm for the top, middle, and bottom thirds were obtained in CT. On average, 94% and 97% of the segmented centerlines were within a 3.5 mm and 5 mm radius, respectively. In IMRT, the urethra received a higher dose than the overall prostate. We also found a slight deviation between the predicted and manual MR delineations. Conclusion: A fully-automatic segmentation pipeline was validated to delineate the intraprostatic urethra in CT images.

Selective sparing of bladder and rectum sub-regions in radiotherapy of prostate cancer combining knowledge-based automatic planning and multicriteria optimization.

Background and Purpose: The association between dose to selected bladder and rectum symptom-related sub-regions (SRS) and late toxicity after prostate cancer radiotherapy has been evidenced by voxel-wise analyses. The aim of the current study was to explore the feasibility of combining knowledge-based (KB) and multi-criteria optimization (MCO) to spare SRSs without compromising planning target volume (PTV) dose delivery, including pelvic-node irradiation. Materials and Methods: Forty-five previously treated patients (74.2 Gy/28fr) were selected and SRSs (in the bladder, associated with late dysuria/hematuria/retention; in the rectum, associated with bleeding) were generated using deformable registration. A KB model was used to obtain clinically suitable plans (KB-plan). KB-plans were further optimized using MCO, aiming to reduce dose to the SRSs while safeguarding target dose coverage, homogeneity and avoiding worsening dose volume histograms of the whole bladder, rectum and other organs at risk. The resulting MCO-generated plans were examined to identify the best-compromise plan (KB + MCO-plan). Results: The mean SRS dose decreased in almost all patients for each SRS. D1% also decreased in the large majority, less frequently for dysuria/bleeding SRS. Mean differences were statistically significant (p < 0.05) and ranged between 1.3 and 2.2 Gy with maximum reduction of mean dose up to 3-5 Gy for the four SRSs. The better sparing of SRSs was obtained without compromising PTVs coverage. Conclusions: Selectively sparing SRSs without compromising PTV coverage is feasible and has the potential to reduce toxicities in prostate cancer radiotherapy. Further investigation to better quantify the expected risk reduction of late toxicities is warranted.

Comparative evaluation of autofocus algorithms for a real-time system for automatic detection of Mycobacterium tuberculosis.

Microscopy images must be acquired at the optimal focal plane for the objects of interest in a scene. Although manual focusing is a standard task for a trained observer, automatic systems often fail to properly find the focal plane under different microscope imaging modalities such as bright field microscopy or phase contrast microscopy. This article assesses several autofocus algorithms applied in the study of fluorescence-labeled tuberculosis bacteria. The goal of this work was to find the optimal algorithm in order to build an automatic real-time system for diagnosing sputum smear samples, where both accuracy and computational time are important. We analyzed 13 focusing methods, ranging from well-known algorithms to the most recently proposed functions. We took into consideration criteria that are inherent to the autofocus function, such as accuracy, computational cost, and robustness to noise and to illumination changes. We also analyzed the additional benefit provided by preprocessing techniques based on morphological operators and image projection profiling.

UltrARsound: in situ visualization of live ultrasound images using HoloLens 2.

PURPOSE: Augmented Reality (AR) has the potential to simplify ultrasound (US) examinations which usually require a skilled and experienced sonographer to mentally align narrow 2D cross-sectional US images in the 3D anatomy of the patient. This work describes and evaluates a novel approach to track retroreflective spheres attached to the US probe using an inside-out technique with the AR glasses HoloLens 2. Finally, live US images are displayed in situ on the imaged anatomy. METHODS: The Unity application UltrARsound performs spatial tracking of the US probe and attached retroreflective markers using the depth camera integrated into the AR glasses-thus eliminating the need for an external tracking system. Additionally, a Kalman filter is implemented to improve the noisy measurements of the camera. US images are streamed wirelessly via the PLUS toolkit to HoloLens 2. The technical evaluation comprises static and dynamic tracking accuracy, frequency and latency of displayed images. RESULTS: Tracking is performed with a median accuracy of 1.98 mm/1.81[Formula: see text] for the static setting when using the Kalman filter. In a dynamic scenario, the median error was 2.81 mm/1.70[Formula: see text]. The tracking frequency is currently limited to 20 Hz. 83% of the displayed US images had a latency lower than 16 ms. CONCLUSIONS: In this work, we showed that spatial tracking of retroreflective spheres with the depth camera of HoloLens 2 is feasible, achieving a promising accuracy for in situ visualization of live US images. For tracking, no additional hardware nor modifications to HoloLens 2 are required making it a cheap and easy-to-use approach. Moreover, a minimal latency of displayed images enables a real-time perception for the sonographer.

Evaluation of optical tracking and augmented reality for needle navigation in sacral nerve stimulation.

BACKGROUND AND OBJECTIVE: Sacral nerve stimulation (SNS) is a minimally invasive procedure where an electrode lead is implanted through the sacral foramina to stimulate the nerve modulating colonic and urinary functions. One of the most crucial steps in SNS procedures is the placement of the tined lead close to the sacral nerve. However, needle insertion is very challenging for surgeons. Several x-ray projections are required to interpret the needle position correctly. In many cases, multiple punctures are needed, causing an increase in surgical time and patient's discomfort and pain. In this work we propose and evaluate two different navigation systems to guide electrode placement in SNS surgeries designed to reduce surgical time, minimize patient discomfort and improve surgical outcomes. METHODS: We developed, for the first alternative, an open-source navigation software to guide electrode placement by real-time needle tracking with an optical tracking system (OTS). In the second method, we present a smartphone-based AR application that displays virtual guidance elements directly on the affected area, using a 3D printed reference marker placed on the patient. This guidance facilitates needle insertion with a predefined trajectory. Both techniques were evaluated to determine which one obtained better results than the current surgical procedure. To compare the proposals with the clinical method, we developed an x-ray software tool that calculates a digitally reconstructed radiograph, simulating the fluoroscopy acquisitions during the procedure. Twelve physicians (inexperienced and experienced users) performed needle insertions through several specific targets to evaluate the alternative SNS guidance methods on a realistic patient-based phantom. RESULTS: With each navigation solution, we observed that users took less average time to complete each insertion (36.83 s and 44.43 s for the OTS and AR methods, respectively) and needed fewer average punctures to reach the target (1.23 and 1.96 for the OTS and AR methods respectively) than following the standard clinical method (189.28 s and 3.65 punctures). CONCLUSIONS: To conclude, we have shown two navigation alternatives that could improve surgical outcome by significantly reducing needle insertions, surgical time and patient's pain in SNS procedures. We believe that these solutions are feasible to train surgeons and even replace current SNS clinical procedures.