Tracking and mapping in medical computer vision: A review.

As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.

Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping.

Quantifying performance of methods for tracking and mapping tissue in endoscopic environments is essential for enabling image guidance and automation of medical interventions and surgery. Datasets developed so far either use rigid environments, visible markers, or require annotators to label salient points in videos after collection. These are respectively: not general, visible to algorithms, or costly and error-prone. We introduce a novel labeling methodology along with a dataset that uses said methodology, Surgical Tattoos in Infrared (STIR). STIR has labels that are persistent but invisible to visible spectrum algorithms. This is done by labelling tissue points with IR-fluorescent dye, indocyanine green (ICG), and then collecting visible light video clips. STIR comprises hundreds of stereo video clips in both in vivo and ex vivo scenes with start and end points labelled in the IR spectrum. With over 3,000 labelled points, STIR will help to quantify and enable better analysis of tracking and mapping methods. After introducing STIR, we analyze multiple different frame-based tracking methods on STIR using both 3D and 2D endpoint error and accuracy metrics. STIR is available at https://dx.doi.org/10.21227/w8g4-g548.

Arc-to-line frame registration method for ultrasound and photoacoustic image-guided intraoperative robot-assisted laparoscopic prostatectomy.

PURPOSE: To achieve effective robot-assisted laparoscopic prostatectomy, the integration of transrectal ultrasound (TRUS) imaging system which is the most widely used imaging modality in prostate imaging is essential. However, manual manipulation of the ultrasound transducer during the procedure will significantly interfere with the surgery. Therefore, we propose an image co-registration algorithm based on a photoacoustic marker (PM) method, where the ultrasound/photoacoustic (US/PA) images can be registered to the endoscopic camera images to ultimately enable the TRUS transducer to automatically track the surgical instrument. METHODS: An optimization-based algorithm is proposed to co-register the images from the two different imaging modalities. The principle of light propagation and an uncertainty in PM detection were assumed in this algorithm to improve the stability and accuracy of the algorithm. The algorithm is validated using the previously developed US/PA image-guided system with a da Vinci surgical robot. RESULTS: The target-registration-error (TRE) is measured to evaluate the proposed algorithm. In both simulation and experimental demonstration, the proposed algorithm achieved a sub-centimeter accuracy which is acceptable in practical clinics (i.e., 1.15 ± 0.29 mm from the experimental evaluation). The result is also comparable with our previous approach (i.e., 1.05 ± 0.37 mm), and the proposed method can be implemented with a normal white light stereo camera and does not require highly accurate localization of the PM. CONCLUSION: The proposed frame registration algorithm enabled a simple yet efficient integration of commercial US/PA imaging system into laparoscopic surgical setting by leveraging the characteristic properties of acoustic wave propagation and laser excitation, contributing to automated US/PA image-guided surgical intervention applications.

Automatic search for photoacoustic marker using automated transrectal ultrasound.

Real-time transrectal ultrasound (TRUS) image guidance during robot-assisted laparoscopic radical prostatectomy has the potential to enhance surgery outcomes. Whether conventional or photoacoustic TRUS is used, the robotic system and the TRUS must be registered to each other. Accurate registration can be performed using photoacoustic (PA markers). However, this requires a manual search by an assistant [IEEE Robot. Autom. Lett8, 1287 (2023).10.1109/LRA.2022.3191788]. This paper introduces the first automatic search for PA markers using a transrectal ultrasound robot. This effectively reduces the challenges associated with the da Vinci-TRUS registration. This paper investigated the performance of three search algorithms in simulation and experiment: Weighted Average (WA), Golden Section Search (GSS), and Ternary Search (TS). For validation, a surgical prostate scenario was mimicked and various ex vivo tissues were tested. As a result, the WA algorithm can achieve 0.53°±0.30° average error after 9 data acquisitions, while the TS and GSS algorithm can achieve 0.29∘±0.31∘ and 0.48°±0.32° average errors after 28 data acquisitions.

Real-time intraoperative surgical guidance system in the da Vinci surgical robot based on transrectal ultrasound/photoacoustic imaging with photoacoustic markers: an ex vivo demonstration.

This paper introduces the first integrated real-time intraoperative surgical guidance system, in which an endoscope camera of da Vinci surgical robot and a transrectal ultrasound (TRUS) transducer are co-registered using photoacoustic markers that are detected in both fluorescence (FL) and photoacoustic (PA) imaging. The co-registered system enables the TRUS transducer to track the laser spot illuminated by a pulsed-laser-diode attached to the surgical instrument, providing both FL and PA images of the surgical region-of-interest (ROI). As a result, the generated photoacoustic marker is visualized and localized in the da Vinci endoscopic FL images, and the corresponding tracking can be conducted by rotating the TRUS transducer to display the PA image of the marker. A quantitative evaluation revealed that the average registration and tracking errors were 0.84 mm and 1.16°, respectively. This study shows that the co-registered photoacoustic marker tracking can be effectively deployed intraoperatively using TRUS+PA imaging providing functional guidance of the surgical ROI.

Robotic ultrasound imaging: State-of-the-art and future perspectives.

Ultrasound (US) is one of the most widely used modalities for clinical intervention and diagnosis due to the merits of providing non-invasive, radiation-free, and real-time images. However, free-hand US examinations are highly operator-dependent. Robotic US System (RUSS) aims at overcoming this shortcoming by offering reproducibility, while also aiming at improving dexterity, and intelligent anatomy and disease-aware imaging. In addition to enhancing diagnostic outcomes, RUSS also holds the potential to provide medical interventions for populations suffering from the shortage of experienced sonographers. In this paper, we categorize RUSS as teleoperated or autonomous. Regarding teleoperated RUSS, we summarize their technical developments, and clinical evaluations, respectively. This survey then focuses on the review of recent work on autonomous robotic US imaging. We demonstrate that machine learning and artificial intelligence present the key techniques, which enable intelligent patient and process-specific, motion and deformation-aware robotic image acquisition. We also show that the research on artificial intelligence for autonomous RUSS has directed the research community toward understanding and modeling expert sonographers' semantic reasoning and action. Here, we call this process, the recovery of the "language of sonography". This side result of research on autonomous robotic US acquisitions could be considered as valuable and essential as the progress made in the robotic US examination itself. This article will provide both engineers and clinicians with a comprehensive understanding of RUSS by surveying underlying techniques. Additionally, we present the challenges that the scientific community needs to face in the coming years in order to achieve its ultimate goal of developing intelligent robotic sonographer colleagues. These colleagues are expected to be capable of collaborating with human sonographers in dynamic environments to enhance both diagnostic and intraoperative imaging.

Artificial intelligence meets medical robotics.

Artificial intelligence (AI) applications in medical robots are bringing a new era to medicine. Advanced medical robots can perform diagnostic and surgical procedures, aid rehabilitation, and provide symbiotic prosthetics to replace limbs. The technology used in these devices, including computer vision, medical image analysis, haptics, navigation, precise manipulation, and machine learning (ML) , could allow autonomous robots to carry out diagnostic imaging, remote surgery, surgical subtasks, or even entire surgical procedures. Moreover, AI in rehabilitation devices and advanced prosthetics can provide individualized support, as well as improved functionality and mobility (see the figure). The combination of extraordinary advances in robotics, medicine, materials science, and computing could bring safer, more efficient, and more widely available patient care in the future. -Gemma K. Alderton.

Multi-Frequency 3D Shear Wave Absolute Vibro-Elastography (S-WAVE) System for the Prostate.

This article describes a novel system for quantitative and volumetric measurement of tissue elasticity in the prostate using simultaneous multi-frequency tissue excitation. Elasticity is computed by using a local frequency estimator to measure the three-dimensional local wavelengths of steady-state shear waves within the prostate gland. The shear wave is created using a mechanical voice coil shaker which transmits simultaneous multi-frequency vibrations transperineally. Radio frequency data is streamed directly from a BK Medical 8848 transrectal ultrasound transducer to an external computer where tissue displacement due to the excitation is measured using a speckle tracking algorithm. Bandpass sampling is used that eliminates the need for an ultra-fast frame rate to track the tissue motion and allows for accurate reconstruction at a sampling frequency that is below the Nyquist rate. A roll motor with computer control is used to rotate the transducer and obtain 3D data. Two commercially available phantoms were used to validate both the accuracy of the elasticity measurements as well as the functional feasibility of using the system for in vivo prostate imaging. The phantom measurements were compared with 3D Magnetic Resonance Elastography (MRE), where a high correlation of 96% was achieved. In addition, the system has been used in two separate clinical studies as a method for cancer identification. Qualitative and quantitative results of 11 patients from these clinical studies are presented here. Furthermore, an AUC of 0.87±0.12 was achieved for malignant vs. benign classification using a binary support vector machine classifier trained with data from the latest clinical study with leave one patient out cross-validation.

Laser diode photoacoustic point source detection: machine learning-based denoising and reconstruction.

A new development in photoacoustic (PA) imaging has been the use of compact, portable and low-cost laser diodes (LDs), but LD-based PA imaging suffers from low signal intensity recorded by the conventional transducers. A common method to improve signal strength is temporal averaging, which reduces frame rate and increases laser exposure to patients. To tackle this problem, we propose a deep learning method that will denoise point source PA radio-frequency (RF) data before beamforming with a very few frames, even one. We also present a deep learning method to automatically reconstruct point sources from noisy pre-beamformed data. Finally, we employ a strategy of combined denoising and reconstruction, which can supplement the reconstruction algorithm for very low signal-to-noise ratio inputs.

3-D Ultrafast Shear Wave Absolute Vibro-Elastography Using a Matrix Array Transducer.

Real-time ultrasound imaging plays an important role in ultrasound-guided interventions. The 3-D imaging provides more spatial information compared to conventional 2-D frames by considering the volumes of data. One of the main bottlenecks of 3-D imaging is the long data acquisition time, which reduces practicality and can introduce artifacts from unwanted patient or sonographer motion. This article introduces the first shear wave absolute vibro-elastography (S-WAVE) method with real-time volumetric acquisition using a matrix array transducer. In S-WAVE, an external vibration source generates mechanical vibrations inside the tissue. The tissue motion is then estimated and used in solving a wave equation inverse problem to provide the tissue elasticity. A matrix array transducer is used with a Verasonics ultrasound machine and a frame rate of 2000 volumes/s to acquire 100 radio frequency (RF) volumes in 0.05 s. Using plane wave (PW) and compounded diverging wave (CDW) imaging methods, we estimate axial, lateral, and elevational displacements over 3-D volumes. The curl of the displacements is used with local frequency estimation to estimate elasticity in the acquired volumes. Ultrafast acquisition extends substantially the possible S-WAVE excitation frequency range, now up to 800 Hz, enabling new tissue modeling and characterization. The method was validated on three homogeneous liver fibrosis phantoms and on four different inclusions within a heterogeneous phantom. The homogeneous phantom results show less than 8% (PW) and 5% (CDW) difference between the manufacturer values and the corresponding estimated values over a frequency range of 80-800 Hz. The estimated elasticity values for the heterogeneous phantom at 400-Hz excitation frequency show the average errors of 9% (PW) and 6% (CDW) compared to the provided average values by magnetic resonance elastography (MRE). Furthermore, both imaging methods were able to detect the inclusions within the elasticity volumes. An ex vivo study on a bovine liver sample shows less than 11% (PW) and 9% (CDW) difference between the estimated elasticity ranges by the proposed method and the elasticity ranges provided by MRE and acoustic radiation force impulse (ARFI).

Towards transcervical ultrasound image guidance for transoral robotic surgery.

PURPOSE: Trans-oral robotic surgery (TORS) using the da Vinci surgical robot is a new minimally-invasive surgery method to treat oropharyngeal tumors, but it is a challenging operation. Augmented reality (AR) based on intra-operative ultrasound (US) has the potential to enhance the visualization of the anatomy and cancerous tumors to provide additional tools for decision-making in surgery. METHODS: We propose a US-guided AR system for TORS, with the transducer placed on the neck for a transcervical view. Firstly, we perform a novel MRI-to-transcervical 3D US registration study, comprising (i) preoperative MRI to preoperative US registration, and (ii) preoperative to intraoperative US registration to account for tissue deformation due to retraction. Secondly, we develop a US-robot calibration method with an optical tracker and demonstrate its use in an AR system that displays anatomy models in the surgeon's console in real-time. RESULTS: Our AR system achieves a projection error from the US to the stereo cameras of 27.14 and 26.03 pixels (image is 540[Formula: see text]960) in a water bath experiment. The average target registration error (TRE) for MRI to 3D US is 8.90 mm for the 3D US transducer and 5.85 mm for freehand 3D US, and the TRE for pre-intra operative US registration is 7.90 mm. CONCLUSION: We demonstrate the feasibility of each component of the first complete pipeline for MRI-US-robot-patient registration for a proof-of-concept transcervical US-guided AR system for TORS. Our results show that trans-cervical 3D US is a promising technique for TORS image guidance.

Human-as-a-robot performance in mixed reality teleultrasound.

PURPOSE: In "human teleoperation" (HT), mixed reality (MR) and haptics are used to tightly couple an expert leader to a human follower [1]. To determine the feasibility of HT for teleultrasound, we quantify the ability of humans to track a position and/or force trajectory via MR cues. The human response time, precision, frequency response, and step response were characterized, and several rendering methods were compared. METHODS: Volunteers (n=11) performed a series of tasks as the follower in our HT system. The tasks involved tracking pre-recorded series of motions and forces while pose and force were recorded. The volunteers then performed frequency response tests and filled out a questionnaire. RESULTS: Following force and pose simultaneously was more difficult but did not lead to significant performance degradation versus following one at a time. On average, subjects tracked positions, orientations, and forces with RMS tracking errors of [Formula: see text] mm, [Formula: see text], [Formula: see text] N, steady-state errors of [Formula: see text] mm, [Formula: see text] N, and lags of [Formula: see text] ms, respectively. Performance decreased with input frequency, depending on the input amplitude. CONCLUSION: Teleoperating a person through MR is a novel concept with many possible applications. However, it is unknown what performance is achievable and which approaches work best. This paper thus characterizes human tracking ability in MR HT for teleultrasound, which is important for designing future tightly coupled guidance and training systems using MR.

3-D Transducer Mounted Shear Wave Absolute Vibro-Elastography: Proof of Concept.

Quantitative tissue stiffness characterization using ultrasound (US) has been shown to improve prostate cancer (PCa) detection in multiple studies. Shear wave absolute vibro-elastography (SWAVE) allows quantitative and volumetric assessment of tissue stiffness using external multifrequency excitation. This article presents a proof of concept of a first-of-a-kind 3-D hand-operated endorectal SWAVE system designed to be used during systematic prostate biopsy. The system is developed with a clinical US machine, requiring only an external exciter that can be mounted directly to the transducer. Subsector acquisition of radio frequency (RF) data allows imaging of shear waves with a high effective frame rate (up to 250 Hz). The system was characterized using eight different quality assurance phantoms. Due to the invasive nature of prostate imaging, at this early stage of development, validation of in vivo human tissue was instead carried out by intercostally scanning the livers of n = 7 healthy volunteers. The results are compared with 3-D magnetic resonance elastography (MRE) and an existing 3-D SWAVE system with a matrix array transducer (M-SWAVE). High correlations were found with MRE (99% in phantoms, 94% in liver data) and with M-SWAVE (99% in phantoms, 98% in liver data).

Robotically controlled three-dimensional micro-ultrasound for prostate biopsy guidance.

PURPOSE: Prostate imaging to guide biopsy remains unsatisfactory, with current solutions suffering from high complexity and poor accuracy and reliability. One novel entrant into this field is micro-ultrasound (microUS), which uses a high-frequency imaging probe to achieve very high spatial resolution, and achieves prostate cancer detection rates equivalent to multiparametric magnetic resonance imaging (mpMRI). However, the ExactVu transrectal microUS probe has a unique geometry that makes it challenging to acquire controlled, repeatable three-dimensional (3D) transrectal ultrasound (TRUS) volumes. We describe the design, fabrication, and validation of a 3D acquisition system that allows for the accurate use of the ExactVu microUS device for volumetric prostate imaging. METHODS: The design uses a motorized, computer-controlled brachytherapy stepper to rotate the ExactVu transducer about its axis. We perform geometric validation using a phantom with known dimensions and compare performance with magnetic resonance imaging (MRI) using a commercial quality assurance anthropomorphic prostate phantom. RESULTS: Our geometric validation shows accuracy of 1 mm or less in all three directions, and images of an anthropomorphic phantom qualitatively match those acquired using MRI and show good agreement quantitatively. CONCLUSION: We describe the first system to acquire robotically controlled 3D microUS images using the ExactVu microUS system. The reconstructed 3D microUS images are accurate, which will allow for future applications of the ExactVu microUS system in prostate specimen and in vivo imaging.

Phase-regularized and displacement-regularized compressed sensing for fast magnetic resonance elastography.

Liver magnetic resonance elastography (MRE) is a noninvasive stiffness measurement technique that captures the tissue displacement in the phase of the signal. To limit the scanning time to a single breath-hold, liver MRE usually involves advanced readout techniques such as simultaneous multislice (SMS) or multishot methods. Furthermore, all these readout techniques require additional in-plane acceleration using either parallel imaging capabilities, such as sensitivity encoding (SENSE), or k -space undersampling, such as compressed sensing (CS). However, these methods apply a single regularization function on the complex image. This study aims to design and evaluate methods that use separate regularization on the magnitude and phase of MRE to exploit their distinct spatiotemporal characteristics. Specifically, we introduce two compressed sensing methods. The first method, termed phase-regularized compressed sensing (PRCS), applies a two-dimensional total variation (TV) prior to the magnitude and two-dimensional wavelet regularization to the phase. The second method, termed displacement-regularized compressed sensing (DRCS), exploits the spatiotemporal redundancy using 3D total variation on the magnitude. Additionally, DRCS includes a displacement fitting function to apply wavelet regularization to the displacement phasor. Both DRCS and PRCS were evaluated with different levels of compression factors in three datasets: an in silico abdomen dataset, an in vitro tissue-mimicking phantom, and an in vivo liver dataset. The reconstructed images were compared with the full sampled reconstruction, zero-filling reconstruction, wavelet-regularized compressed sensing, and a low rank plus sparse reconstruction. The metrics used for quantitative evaluation were the structural similarity index (SSIM) of magnitude (M-SSIM), displacement (D-SSIM), and shear modulus (S-SSIM), and mean shear modulus. Results from highly undersampled in silico and in vitro datasets demonstrate that the DRCS method provides higher reconstruction quality than the conventional compressed sensing method for a wide range of stiffness values. Notably, DRCS provides 24% and 22% increase in D-SSIM compared with CS for the in silico and in vitro datasets, respectively. Comparison with liver stiffness measured from full sampled data and highly undersampled data (CR=4) demonstrates that the DRCS method provided the strongest correlation ( R 2 =0.95), second-lowest mean bias (-0.18 kPa, lowest for CS with -0.16 kPa), and lowest coefficient of variation (CV=3.6%). Our results demonstrate the potential of using DRCS to improve the reconstruction quality of accelerated MRE.

SWAVE 2.0 Imaging of Placental Elasticity and Viscosity: Potential Biomarkers for Placenta-Mediated Disease Detection.

Pregnancy complications such as pre-eclampsia (PE) and intrauterine growth restriction (IUGR) are associated with structural and functional changes in the placenta. Different elastography techniques with an ability to assess the mechanical properties of tissue can identify and monitor the pathological state of the placenta. Currently available elastography techniques have been used with promising results to detect placenta abnormalities; however, limitations include inadequate measurement depth and safety concerns from high negative pressure pulses. Previously, we described a shear wave absolute vibro-elastography (SWAVE) method by applying external low-frequency mechanical vibrations to generate shear waves and studied 61 post-delivery clinically normal placentas to explore the feasibility of SWAVE for placental assessment and establish a measurement baseline. This next phase of the study, namely, SWAVE 2.0, improves the previous system and elasticity reconstruction by incorporating a multi-frequency acquisition system and using a 3-D local frequency estimation (LFE) method. Compared with its 2-D counterpart, the proposed system using 3-D LFE was found to reduce the bias and variance in elasticity measurements in tissue-mimicking phantoms. In the aim of investigating the potential of improved SWAVE 2.0 measurements to identify placental abnormalities, we studied 46 post-delivery placentas, including 26 diseased (16 IUGR and 10 PE) and 20 normal control placentas. By use of a 3.33-MHz motorized curved-array transducer, multi-frequency (80,100 and 120 Hz) elasticity measures were obtained with 3-D LFE, and both IUGR (15.30 ± 2.96 kPa, p = 3.35e-5) and PE (12.33 ± 4.88 kPa, p = 0.017) placentas were found to be significantly stiffer compared with the control placentas (8.32 ± 3.67 kPa). A linear discriminant analysis (LDA) classifier was able to classify between healthy and diseased placentas with a sensitivity, specificity and accuracy of 87%, 78% and 83% and an area under the receiver operating curve of 0.90 (95% confidence interval: 0.8-0.99). Further, the pregnancy outcome in terms of neonatal intensive care unit admission was predicted with a sensitivity, specificity and accuracy of 70%, 71%, 71%, respectively, and area under the receiver operating curve of 0.78 (confidence interval: 0.62-0.93). A viscoelastic characterization of placentas using a fractional rheological model revealed that the viscosity measures in terms of viscosity parameter n were significantly higher in IUGR (2.3 ± 0.21) and PE (2.11 ± 0.52) placentas than in normal placentas (1.45 ± 0.65). This work illustrates the potential relevance of elasticity and viscosity imaging using SWAVE 2.0 as a non-invasive technology for detection of placental abnormalities and the prediction of pregnancy outcomes.

Project SWAVE 2.0: An overview of the study design for multimodal placental image acquisition and alignment.

Development of non-invasive and in utero placenta imaging techniques can potentially identify biomarkers of placental health. Correlative imaging using multiple multiscale modalities is particularly important to advance the understanding of placenta structure, function and their relationship. The objective of the project SWAVE 2.0 was to understand human placental structure and function and thereby identify quantifiable measures of placental health using a multimodal correlative approach. In this paper, we present a multimodal image acquisition protocol designed to acquire and align data from ex vivo placenta specimens derived from both healthy and complicated pregnancies. Qualitative and quantitative validation of the alignment method were performed. The qualitative analysis showed good correlation between findings in the MRI, ultrasound and histopathology images. The proposed protocol would enable future studies on comprehensive analysis of placental anatomy, function and their relationship. ● An overview of a novel multimodal placental image acquisition protocol is presented. ● A co-registration method using surface markers and external fiducials is described. ● A preliminary correlative imaging analysis for a placenta specimen is presented.

Model-Based Quantitative Elasticity Reconstruction Using ADMM.

We introduce two model-based iterative methods to obtain shear modulus images of tissue using magnetic resonance elastography. The first method jointly finds the displacement field that best fits tissue displacement data and the corresponding shear modulus. The displacement satisfies a viscoelastic wave equation constraint, discretized using the finite element method. Sparsifying regularization terms in both shear modulus and displacement are used in the cost function minimized for the best fit. The second method extends the first method for multifrequency tissue displacement data. The formulated problems are bi-convex. Their solution can be obtained iteratively by using the alternating direction method of multipliers. Sparsifying regularizations and the wave equation constraint filter out sensor noise and compressional waves. Our methods do not require bandpass filtering as a preprocessing step and converge fast irrespective of the initialization. We evaluate our new methods in multiple in silico and phantom experiments, with comparisons with existing methods, and we show improvements in contrast to noise and signal-to-noise ratios. Results from an in vivo liver imaging study show elastograms with mean elasticity comparable to other values reported in the literature.

Registration of trans-perineal template mapping biopsy cores to volumetric ultrasound.

PURPOSE: Pathology from trans-perineal template mapping biopsy (TTMB) can be used as labels to train prostate cancer classifiers. In this work, we propose a framework to register TTMB cores to advanced volumetric ultrasound data such as multi-parametric transrectal ultrasound (mpTRUS). METHODS: The framework has mainly two steps. First, needle trajectories are calculated with respect to the needle guiding template-considering deflections in their paths. In standard TTMB, a sparsely sampled ultrasound volume is taken prior to the procedure which contains the template overlaid on top of it. The position of this template is detected automatically, and the cores are mapped following the calculated needle trajectories. Second, the TTMB volume is aligned to the mpTRUS volume by a two-step registration method. Using the same transformations from the registration step, the cores are registered from the TTMB volume to the mpTRUS volume. RESULTS: TTMB and mpTRUS of 10 patients were available for this work. The target registration errors (TRE) of the volumes using landmarks picked by three research assistants (RA) and one radiation oncologist (RO) were on average 1.32 ± 0.7 mm and 1.03 ± 0.6 mm, respectively. Additionally, on average, our framework takes only 97 s to register the cores. CONCLUSION: Our proposed framework allows a quick way to find the spatial location of the cores with respect to volumetric ultrasound. Furthermore, knowing the correct location of the pathology will facilitate focal treatment and will aid in training imaging-based cancer classifiers.

A Unified Representation of Control Logic in Human-Ultrasound Machine Interaction.

Advances in human-computer interaction (HCI) technologies have granted sonographers and radiologists a much improved user experience when operating different ultrasound (US) machines. Continued HCI improvements in US would benefit from a systematic study of the HCI control logic used in this domain. Such a study has not been presented previously and is the subject of this paper. We surveyed sonographers to determine the most frequently used controls in US machines. We standardized the representation of the US machine HCI control logic by using the unified modelling language (UML). We used UML diagrams to analyze the HCI control logic of 10 different cart-based US machines from several major manufacturers, and we discovered that the control logic for the most frequently used functions are identical. While this control logic does not follow an established standard, it has been commonly adopted. Using the UML for the visualization and formulation of control logic, we can target logically optimal interactions (whose operation steps cannot be further reduced), e.g., adjustment of B-mode gain, frequency and depth, and can derive methods to simplify logically sub-optimal interactions, e.g., the pointing and selecting operation, as well as image measurements.Our study provides insights into existing HCI approaches used in US machines and establishes a rigorous UML-based framework for future US machine design to improve interoperability, efficiency and ease-of-use.