Laser diode photoacoustic point source detection: machine learning-based denoising and reconstruction.

A new development in photoacoustic (PA) imaging has been the use of compact, portable and low-cost laser diodes (LDs), but LD-based PA imaging suffers from low signal intensity recorded by the conventional transducers. A common method to improve signal strength is temporal averaging, which reduces frame rate and increases laser exposure to patients. To tackle this problem, we propose a deep learning method that will denoise point source PA radio-frequency (RF) data before beamforming with a very few frames, even one. We also present a deep learning method to automatically reconstruct point sources from noisy pre-beamformed data. Finally, we employ a strategy of combined denoising and reconstruction, which can supplement the reconstruction algorithm for very low signal-to-noise ratio inputs.

Phase-regularized and displacement-regularized compressed sensing for fast magnetic resonance elastography.

Liver magnetic resonance elastography (MRE) is a noninvasive stiffness measurement technique that captures the tissue displacement in the phase of the signal. To limit the scanning time to a single breath-hold, liver MRE usually involves advanced readout techniques such as simultaneous multislice (SMS) or multishot methods. Furthermore, all these readout techniques require additional in-plane acceleration using either parallel imaging capabilities, such as sensitivity encoding (SENSE), or k -space undersampling, such as compressed sensing (CS). However, these methods apply a single regularization function on the complex image. This study aims to design and evaluate methods that use separate regularization on the magnitude and phase of MRE to exploit their distinct spatiotemporal characteristics. Specifically, we introduce two compressed sensing methods. The first method, termed phase-regularized compressed sensing (PRCS), applies a two-dimensional total variation (TV) prior to the magnitude and two-dimensional wavelet regularization to the phase. The second method, termed displacement-regularized compressed sensing (DRCS), exploits the spatiotemporal redundancy using 3D total variation on the magnitude. Additionally, DRCS includes a displacement fitting function to apply wavelet regularization to the displacement phasor. Both DRCS and PRCS were evaluated with different levels of compression factors in three datasets: an in silico abdomen dataset, an in vitro tissue-mimicking phantom, and an in vivo liver dataset. The reconstructed images were compared with the full sampled reconstruction, zero-filling reconstruction, wavelet-regularized compressed sensing, and a low rank plus sparse reconstruction. The metrics used for quantitative evaluation were the structural similarity index (SSIM) of magnitude (M-SSIM), displacement (D-SSIM), and shear modulus (S-SSIM), and mean shear modulus. Results from highly undersampled in silico and in vitro datasets demonstrate that the DRCS method provides higher reconstruction quality than the conventional compressed sensing method for a wide range of stiffness values. Notably, DRCS provides 24% and 22% increase in D-SSIM compared with CS for the in silico and in vitro datasets, respectively. Comparison with liver stiffness measured from full sampled data and highly undersampled data (CR=4) demonstrates that the DRCS method provided the strongest correlation ( R 2 =0.95), second-lowest mean bias (-0.18 kPa, lowest for CS with -0.16 kPa), and lowest coefficient of variation (CV=3.6%). Our results demonstrate the potential of using DRCS to improve the reconstruction quality of accelerated MRE.

The Neyman Pearson detection of microsaccades with maximum likelihood estimation of parameters.

Despite the fact that the velocity threshold method is widely applied, the detection of microsaccades continues to be a challenging problem, due to gaze-tracking inaccuracy and the transient nature of microsaccades. Important parameters associated with a saccadic event, e.g., saccade duration, amplitude, and maximum velocity, are sometimes imprecisely estimated, which may lead to biases in inferring the roles of microsaccades in perception and cognition. To overcome the biases and have a better detection algorithm for microsaccades, we propose a novel statistical model for the tracked gaze positions during eye fixations. In this model, we incorporate a parametrization that has been previously applied to model saccades, which allows us to veridically capture the velocity profile of saccadic eye movements. Based on our model, we derive the Neyman Pearson Detector (NPD) for saccadic events. Implemented in conjunction with the maximum likelihood estimation method, our NPD can detect a saccadic event and estimate all parameters simultaneously. Because of its adaptive nature and its statistical optimality, our NPD method was able to better detect microsaccades in some datasets when compared with a recently proposed state-of-the-art method based on convolutional neural networks. NPD also yielded comparable performance with a recently developed Bayesian algorithm, with the added benefit of modeling a more biologically veridical velocity profile of the saccade. As opposed to these algorithms, NPD can lend itself better to online saccade detection, and thus has potential for human-computer interaction applications. Our algorithm is publicly available at https://github.com/hz-zhu/NPD-micro-saccade-detection.

A framework for optimization-based design of motion encoding in magnetic resonance elastography.

PURPOSE: In conventional three-dimensional magnetic resonance elastography, motion encoding gradients (MEGs) synchronized to a mechanical excitation are applied separately in each direction to encode tissue displacement generated by the corresponding waves. This requires long acquisition times that introduce errors due to patient motion and may hinder clinical deployment of magnetic resonance elastography. In this article, a framework for MEGs sequence design is proposed to reduce scanning time and increase signal-to-noise ratio. THEORY AND METHODS: The approach is based on applying MEGs in all three directions simultaneously with varying parameters, and formulation of the problem as a linear estimation of the wave properties. Multidirectional MEGs sequences are derived by setting the problem in an experimental design framework. Such designs are implemented and evaluated on simulation and phantom data. RESULTS: Estimation error of the displacement using the proposed MEGs designs is reduced up to a factor of two in comparison with a unidirectional design with a same number of acquisitions. Alternatively, for the same error, scanning time is reduced up to a factor of three using the multidirectional designs. CONCLUSION: The proposed framework generalizes acquisition of magnetic resonance elastography, and allows quantification of design performance, and optimization-based derivation of designs.

Intraoperative registered transrectal ultrasound guidance for robot-assisted laparoscopic radical prostatectomy.

PURPOSE: To provide unencumbered real-time ultrasound image guidance during robot-assisted laparoscopic radical prostatectomy, we developed a robotic transrectal ultrasound system that tracks the da Vinci® Surgical System instruments. We describe our initial clinical experience with this system. MATERIALS AND METHODS: After an evaluation in a canine model, 20 patients were enrolled in the study. During each procedure the transrectal ultrasound transducer was manually positioned using a brachytherapy stabilizer to provide good imaging of the prostate. Then the transrectal ultrasound was registered to the da Vinci robot by a previously validated procedure. Finally, automatic rotation of the transrectal ultrasound was enabled such that the transrectal ultrasound imaging plane safely tracked the tip of the da Vinci instrument controlled by the surgeon, while real-time transrectal ultrasound images were relayed to the surgeon at the da Vinci console. Tracking was activated during all critical stages of the surgery. RESULTS: The transrectal ultrasound robot was easy to set up and use, adding 7 minutes (range 5 to 14) to the procedure. It did not require an assistant or additional control devices. Qualitative feedback was acquired from the surgeons, who found transrectal ultrasound useful in identifying the urethra while passing the dorsal venous complex suture, defining the prostate-bladder interface during bladder neck dissection, identifying the seminal vesicles and their location with respect to the rectal wall, and identifying the distal prostate boundary at the apex. CONCLUSIONS: Real-time, registered robotic transrectal ultrasound guidance with automatic instrument tracking during robot-assisted laparoscopic radical prostatectomy is feasible and potentially useful. The results justify further studies to establish whether the approach can improve procedure outcomes.

MR elastography and diffusion-weighted imaging of ex vivo prostate cancer: quantitative comparison to histopathology.

The purpose of this work was (1) to develop a magnetic resonance elastography (MRE) system for imaging of the ex vivo human prostate and (2) to assess the diagnostic power of mono-frequency and multi-frequency MRE and diffusion weighted imaging (DWI) alone and combined as correlated with histopathology in a patient study. An electromagnetic driver was designed specifically for MRE studies in small-bore MR scanners. Ex vivo prostate specimens (post-fixation) of 14 patients who underwent radical prostatectomy were imaged with MRE at 7 T (nine cases had DWI). In six patients, the MRE examination was performed at three frequencies (600, 800, 1000 Hz) to extract the power-law exponent Gamma. The images were registered to wholemount pathology slides marked with the Gleason score. The areas under the receiver-operator-characteristic curves (AUC) were calculated. The methods were validated in a phantom study and it was demonstrated that (i) the driver does not interfere with the acquisition process and (ii) the driver can generate amplitudes greater than 100 µm for frequencies less than 1 kHz. In the quantitative study, cancerous tissue with Gleason score at least 3 + 3 was distinguished from normal tissue in the peripheral zone (PZ) with an average AUC of 0.75 (Gd ), 0.75 (Gl ), 0.70 (Gamma-Gd ), 0.68 (apparent diffusion coefficient, ADC), and 0.82 (Gd + Gl + ADC). The differentiation between PZ and central gland was modest for Gd (p < 0.07), Gl (p < 0.06) but not significant for Gamma (p < 0.2). A correlation of 0.4 kPa/h was found between the fixation time of the prostate specimen and the stiffness of the tissue, which could affect the diagnostic power results. DWI and MRE may provide complementary information; in fact MRE performed better than ADC in distinguishing normal from cancerous tissue in some cases. Multi-frequency (Gamma) analysis did not appear to improve the results. However, in light of the effect of tissue fixation, the clinical implication of our results may be inconclusive and more experiments are needed.

MR elastography of prostate cancer: quantitative comparison with histopathology and repeatability of methods.

The purpose of this work was to assess trans-perineal prostate magnetic resonance elastography (MRE) for (1) repeatability in phantoms/volunteers and (2) diagnostic power as correlated with histopathology in prostate cancer patients. The three-dimensional (3D) displacement field was obtained using a fractionally encoded gradient echo sequence using a custom-made transducer. The repeatability of the method was assessed based on three repeat studies and by changing the driving frequency by 3% in studies on a phantom and six healthy volunteers. Subsequently, 11 patients were examined with MRE prior to radical prostatectomy. The areas under the receiver operating characteristic curves were calculated using a windowed voxel-to-voxel approach by comparing the 2D registered slides, masked with the Gleason score. For the repeatability study, the average intraclass correlation coefficient for elasticity images was 99% for repeat phantom studies, 98% for ±6 Hz phantom studies, 95% for volunteer repeat studies with 2 min acquisition time, 82% for ±2 Hz volunteer studies with 2 min acquisition time and 73% for repeat volunteer studies with 8 min acquisition time. For the patient study, the average elasticity was 8.2 ± 1.7 kPa in the prostate capsule, 7.5 ± 1.9 kPa in the peripheral zone (PZ), 9.7 ± 3.0 kPa in the central gland (CG) and 9.0 ± 3.4 kPa in the transition zone. In the patient study, cancerous tissue with Gleason score at least 3 + 3 was significantly (p < 0.05) different from normal tissue in 10 out of 11 cases with tumors in the PZ, and 6 out of 9 cases with tumors in the CG. However, the overall case-averaged area under the curve was 0.72 in the PZ and 0.67 in the CG. Cancerous tissue was not always stiffer than normal tissue. The inversion algorithm was sensitive to (i) vibration amplitude and displacement nodes and (ii) misalignment of the 3D wave field due to subject movement.

Prostate MR elastography with transperineal electromagnetic actuation and a fast fractionally encoded steady-state gradient echo sequence.

Our aim is to develop a clinically viable, fast-acquisition, prostate MR elastography (MRE) system with transperineal excitation. We developed a new actively shielded electromagnetic transducer, designed to enable quick deployment and positioning within the scanner. The shielding of the transducer was optimized using simulations. We also employed a new rapid pulse sequence that encodes the three-dimensional displacement field in the prostate gland using a fractionally encoded steady-state gradient echo sequence, thereby shortening the acquisition time to a clinically acceptable 8-10 min. The methods were tested in two phantoms and seven human subjects (six volunteers and one patient with prostate cancer). The MRE acquisition time for 24 slices, with an isotropic resolution of 2 mm and eight phase offsets, was 8 min, and the total scan, including positioning and set-up, was performed in 15-20 min. The phantom study demonstrated that the transducer does not interfere with the acquisition process and that it generates displacement amplitudes that exceed 100 µm even at frequencies as high as 300 Hz. In the in vivo human study, average wave amplitudes of 30 µm (46 µm at the apex) were routinely achieved within the prostate gland at 70 Hz. No pain or discomfort was reported. Results in a single patient suggest that MRE can identify cancer tumors, although this result is preliminary. The proposed methods allow the integration of prostate MRE with other multiparametric MRI methods. The results of this study clearly motivate the clinical evaluation of transperineal MRE in patients.

Head motion-corrected eye gaze tracking with the da Vinci surgical system.

PURPOSE: To facilitate the integration of point of gaze (POG) as an input modality for robot-assisted surgery, we introduce a robust head movement compensation gaze tracking system for the da Vinci Surgical System. Previous surgical eye gaze trackers require multiple recalibrations and suffer from accuracy loss when users move from the calibrated position. We investigate whether eye corner detection can reduce gaze estimation error in a robotic surgery context. METHODS: A polynomial regressor is first used to estimate POG after an 8-point calibration, and then, using another regressor, the POG error from head movement is estimated from the shift in 2D eye corner location. Eye corners are computed by first detecting regions of interest using the You Only Look Once (YOLO) object detector trained on 1600 annotated eye images (open dataset included). Contours are then extracted from the bounding boxes and a derivative-based curvature detector refines the eye corner. RESULTS: Through a user study (n = 24), our corner-contingent head compensation algorithm showed an error reduction in degrees of visual angle of 1.20 ∘ (p = 0.037) for the left eye and 1.26 ∘ (p = 0.079) for the right compared to the previous gold-standard POG error correction method. In addition, the eye corner pipeline showed a root-mean-squared error of 3.57 (SD = 1.92) pixels in detecting eye corners over 201 annotated frames. CONCLUSION: We introduce an effective method of using eye corners to correct for eye gaze estimation, enabling the practical acquisition of POG in robotic surgery.

Tracking and mapping in medical computer vision: A review.

As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.

Multi-scale relational graph convolutional network for multiple instance learning in histopathology images.

Graph convolutional neural networks have shown significant potential in natural and histopathology images. However, their use has only been studied in a single magnification or multi-magnification with either homogeneous graphs or only different node types. In order to leverage the multi-magnification information and improve message passing with graph convolutional networks, we handle different embedding spaces at each magnification by introducing the Multi-Scale Relational Graph Convolutional Network (MS-RGCN) as a multiple instance learning method. We model histopathology image patches and their relation with neighboring patches and patches at other scales (i.e., magnifications) as a graph. We define separate message-passing neural networks based on node and edge types to pass the information between different magnification embedding spaces. We experiment on prostate cancer histopathology images to predict the grade groups based on the extracted features from patches. We also compare our MS-RGCN with multiple state-of-the-art methods with evaluations on several source and held-out datasets. Our method outperforms the state-of-the-art on all of the datasets and image types consisting of tissue microarrays, whole-mount slide regions, and whole-slide images. Through an ablation study, we test and show the value of the pertinent design features of the MS-RGCN.

Enabling extracorporeal ultrasound imaging with the da Vinci robot for transoral robotic surgery: a feasibility study.

PURPOSE: Transoral robotic surgery (TORS) is a challenging procedure due to its small workspace and complex anatomy. Ultrasound (US) image guidance has the potential to improve surgical outcomes, but an appropriate method for US probe manipulation has not been defined. This study evaluates using an additional robotic (4th) arm on the da Vinci Surgical System to perform extracorporeal US scanning for image guidance in TORS. METHODS: A stereoscopic imaging system and da Vinci-compatible US probe attachment were developed to enable control of the extracorporeal US probe from the surgeon console. The prototype was compared to freehand US by nine operators in three tasks on a healthy volunteer: (1) identification of the common carotid artery, (2) carotid artery scanning, and (3) identification of the submandibular gland. Operator workload and user experience were evaluated using a questionnaire. RESULTS: The robotic US tasks took longer than freehand US tasks (2.09x longer; p = 0.001 ) and had higher operator workload (2.12x higher; p = 0.004 ). However, operator-rated performance was closer (avg robotic/avg freehand = 0.66; p = 0.017 ), and scanning performance measured by MRI-US average Hausdorff distance provided no statistically significant difference. CONCLUSION: Extracorporeal US scanning for intraoperative US image guidance is a convenient approach for providing the surgeon direct control over the US image plane during TORS, with little modification to the existing operating room workflow. Although more time-consuming and higher operator workload, several methods have been identified to address these limitations.

Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping.

Quantifying performance of methods for tracking and mapping tissue in endoscopic environments is essential for enabling image guidance and automation of medical interventions and surgery. Datasets developed so far either use rigid environments, visible markers, or require annotators to label salient points in videos after collection. These are respectively: not general, visible to algorithms, or costly and error-prone. We introduce a novel labeling methodology along with a dataset that uses said methodology, Surgical Tattoos in Infrared (STIR). STIR has labels that are persistent but invisible to visible spectrum algorithms. This is done by labelling tissue points with IR-fluorescent dye, indocyanine green (ICG), and then collecting visible light video clips. STIR comprises hundreds of stereo video clips in both in vivo and ex vivo scenes with start and end points labelled in the IR spectrum. With over 3,000 labelled points, STIR will help to quantify and enable better analysis of tracking and mapping methods. After introducing STIR, we analyze multiple different frame-based tracking methods on STIR using both 3D and 2D endpoint error and accuracy metrics. STIR is available at https://dx.doi.org/10.21227/w8g4-g548.

Arc-to-line frame registration method for ultrasound and photoacoustic image-guided intraoperative robot-assisted laparoscopic prostatectomy.

PURPOSE: To achieve effective robot-assisted laparoscopic prostatectomy, the integration of transrectal ultrasound (TRUS) imaging system which is the most widely used imaging modality in prostate imaging is essential. However, manual manipulation of the ultrasound transducer during the procedure will significantly interfere with the surgery. Therefore, we propose an image co-registration algorithm based on a photoacoustic marker (PM) method, where the ultrasound/photoacoustic (US/PA) images can be registered to the endoscopic camera images to ultimately enable the TRUS transducer to automatically track the surgical instrument. METHODS: An optimization-based algorithm is proposed to co-register the images from the two different imaging modalities. The principle of light propagation and an uncertainty in PM detection were assumed in this algorithm to improve the stability and accuracy of the algorithm. The algorithm is validated using the previously developed US/PA image-guided system with a da Vinci surgical robot. RESULTS: The target-registration-error (TRE) is measured to evaluate the proposed algorithm. In both simulation and experimental demonstration, the proposed algorithm achieved a sub-centimeter accuracy which is acceptable in practical clinics (i.e., 1.15 ± 0.29 mm from the experimental evaluation). The result is also comparable with our previous approach (i.e., 1.05 ± 0.37 mm), and the proposed method can be implemented with a normal white light stereo camera and does not require highly accurate localization of the PM. CONCLUSION: The proposed frame registration algorithm enabled a simple yet efficient integration of commercial US/PA imaging system into laparoscopic surgical setting by leveraging the characteristic properties of acoustic wave propagation and laser excitation, contributing to automated US/PA image-guided surgical intervention applications.

Prostate Cancer Risk Stratification by Digital Histopathology and Deep Learning.

PURPOSE: Prostate cancer (PCa) represents a highly heterogeneous disease that requires tools to assess oncologic risk and guide patient management and treatment planning. Current models are based on various clinical and pathologic parameters including Gleason grading, which suffers from a high interobserver variability. In this study, we determine whether objective machine learning (ML)-driven histopathology image analysis would aid us in better risk stratification of PCa. MATERIALS AND METHODS: We propose a deep learning, histopathology image-based risk stratification model that combines clinicopathologic data along with hematoxylin and eosin- and Ki-67-stained histopathology images. We train and test our model, using a five-fold cross-validation strategy, on a data set from 502 treatment-naïve PCa patients who underwent radical prostatectomy (RP) between 2000 and 2012. RESULTS: We used the concordance index as a measure to evaluate the performance of various risk stratification models. Our risk stratification model on the basis of convolutional neural networks demonstrated superior performance compared with Gleason grading and the Cancer of the Prostate Risk Assessment Post-Surgical risk stratification models. Using our model, 3.9% of the low-risk patients were correctly reclassified to be high-risk and 21.3% of the high-risk patients were correctly reclassified as low-risk. CONCLUSION: These findings highlight the importance of ML as an objective tool for histopathology image assessment and patient risk stratification. With further validation on large cohorts, the digital pathology risk classification we propose may be helpful in guiding administration of adjuvant therapy including radiotherapy after RP.

Transperineal prostate MR elastography: initial in vivo results.

This article presents a new approach to magnetic resonance elastography of the prostate using transperineal mechanical excitation. This approach is validated using a prostate elasticity phantom and in vivo studies of healthy volunteers. It is demonstrated that the transperineal approach can generate shear wave amplitudes on the order of 6-30 µm in the mid-gland region. The driver was implemented using an electromagnetic actuator with a hydraulic transmission system. The magnetic resonance elastography acquisition time has been reduced significantly by using a "second harmonic" approach. Displacement fields are processed using the established three-dimensional local frequency estimation algorithm. The three-dimensional curl-based direct inversion was used to calculate the local wavelength. The traveling wave expansion algorithm was used to reconstruct the wave damping image for one case. Using the proposed method, it was possible to resolve lesions of 0.5 cc in the phantom study. Repeatability experiments were performed and analyzed. The results from this study indicate that transperineal magnetic resonance elastography--without an endorectal coil--is a suitable candidate for a patient study involving multiparametric magnetic resonance imaging of prostate cancer, where magnetic resonance elastography may provide additional information for improved diagnosis and image-based surveillance.

Rapid acquisition of multifrequency, multislice and multidirectional MR elastography data with a fractionally encoded gradient echo sequence.

In MR elastography (MRE), periodic tissue motion is phase encoded using motion-encoding gradients synchronized to an externally applied periodic mechanical excitation. Conventional methods result in extended scan time for quality phase images, thus limiting the broad application of MRE in the clinic. For practical scan times, researchers have been relying on one-dimensional or two-dimensional motion-encoding, low-phase sampling and a limited number of slices, and artifact-prone, single-shot, echo planar imaging (EPI) readout. Here, we introduce a rapid multislice pulse sequence capable of three-dimensional motion encoding that is also suitable for simultaneously encoding motion with multiple frequency components. This sequence is based on a gradient-recalled echo (GRE) sequence and exploits the principles of fractional encoding. This GRE MRE pulse sequence was validated as capable of acquiring full three-dimensional motion encoding of isotropic voxels in a large volume within less than a minute. This sequence is suitable for monofrequency and multifrequency MRE experiments. In homogeneous paraffin phantoms, the eXpresso sequence yielded similar storage modulus values as those obtained with conventional methods, although with markedly reduced variances (7.11 ± 0.26 kPa for GRE MRE versus 7.16 ± 1.33 kPa for the conventional spin-echo EPI sequence). The GRE MRE sequence obtained better phase-to-noise ratios than the equivalent spin-echo EPI sequence (matched for identical acquisition time) in both paraffin phantoms and in vivo data in the liver (59.62 ± 11.89 versus 27.86 ± 3.81, 61.49 ± 14.16 versus 24.78 ± 2.48 and 58.23 ± 10.39 versus 23.48 ± 2.91 in the X, Y and Z components, respectively, in the case of liver experiments). Phase-to-noise ratios were similar between GRE MRE used in monofrequency or multifrequency experiments (75.39 ± 14.93 versus 86.13 ± 18.25 at 28 Hz, 71.52 ± 24.74 versus 86.96 ± 30.53 at 56 Hz and 95.60 ± 36.96 versus 61.35 ± 26.25 at 84Hz, respectively).

Human-as-a-robot performance in mixed reality teleultrasound.

PURPOSE: In "human teleoperation" (HT), mixed reality (MR) and haptics are used to tightly couple an expert leader to a human follower [1]. To determine the feasibility of HT for teleultrasound, we quantify the ability of humans to track a position and/or force trajectory via MR cues. The human response time, precision, frequency response, and step response were characterized, and several rendering methods were compared. METHODS: Volunteers (n=11) performed a series of tasks as the follower in our HT system. The tasks involved tracking pre-recorded series of motions and forces while pose and force were recorded. The volunteers then performed frequency response tests and filled out a questionnaire. RESULTS: Following force and pose simultaneously was more difficult but did not lead to significant performance degradation versus following one at a time. On average, subjects tracked positions, orientations, and forces with RMS tracking errors of [Formula: see text] mm, [Formula: see text], [Formula: see text] N, steady-state errors of [Formula: see text] mm, [Formula: see text] N, and lags of [Formula: see text] ms, respectively. Performance decreased with input frequency, depending on the input amplitude. CONCLUSION: Teleoperating a person through MR is a novel concept with many possible applications. However, it is unknown what performance is achievable and which approaches work best. This paper thus characterizes human tracking ability in MR HT for teleultrasound, which is important for designing future tightly coupled guidance and training systems using MR.

Robotic ultrasound imaging: State-of-the-art and future perspectives.

Ultrasound (US) is one of the most widely used modalities for clinical intervention and diagnosis due to the merits of providing non-invasive, radiation-free, and real-time images. However, free-hand US examinations are highly operator-dependent. Robotic US System (RUSS) aims at overcoming this shortcoming by offering reproducibility, while also aiming at improving dexterity, and intelligent anatomy and disease-aware imaging. In addition to enhancing diagnostic outcomes, RUSS also holds the potential to provide medical interventions for populations suffering from the shortage of experienced sonographers. In this paper, we categorize RUSS as teleoperated or autonomous. Regarding teleoperated RUSS, we summarize their technical developments, and clinical evaluations, respectively. This survey then focuses on the review of recent work on autonomous robotic US imaging. We demonstrate that machine learning and artificial intelligence present the key techniques, which enable intelligent patient and process-specific, motion and deformation-aware robotic image acquisition. We also show that the research on artificial intelligence for autonomous RUSS has directed the research community toward understanding and modeling expert sonographers' semantic reasoning and action. Here, we call this process, the recovery of the "language of sonography". This side result of research on autonomous robotic US acquisitions could be considered as valuable and essential as the progress made in the robotic US examination itself. This article will provide both engineers and clinicians with a comprehensive understanding of RUSS by surveying underlying techniques. Additionally, we present the challenges that the scientific community needs to face in the coming years in order to achieve its ultimate goal of developing intelligent robotic sonographer colleagues. These colleagues are expected to be capable of collaborating with human sonographers in dynamic environments to enhance both diagnostic and intraoperative imaging.

Robotically controlled three-dimensional micro-ultrasound for prostate biopsy guidance.

PURPOSE: Prostate imaging to guide biopsy remains unsatisfactory, with current solutions suffering from high complexity and poor accuracy and reliability. One novel entrant into this field is micro-ultrasound (microUS), which uses a high-frequency imaging probe to achieve very high spatial resolution, and achieves prostate cancer detection rates equivalent to multiparametric magnetic resonance imaging (mpMRI). However, the ExactVu transrectal microUS probe has a unique geometry that makes it challenging to acquire controlled, repeatable three-dimensional (3D) transrectal ultrasound (TRUS) volumes. We describe the design, fabrication, and validation of a 3D acquisition system that allows for the accurate use of the ExactVu microUS device for volumetric prostate imaging. METHODS: The design uses a motorized, computer-controlled brachytherapy stepper to rotate the ExactVu transducer about its axis. We perform geometric validation using a phantom with known dimensions and compare performance with magnetic resonance imaging (MRI) using a commercial quality assurance anthropomorphic prostate phantom. RESULTS: Our geometric validation shows accuracy of 1 mm or less in all three directions, and images of an anthropomorphic phantom qualitatively match those acquired using MRI and show good agreement quantitatively. CONCLUSION: We describe the first system to acquire robotically controlled 3D microUS images using the ExactVu microUS system. The reconstructed 3D microUS images are accurate, which will allow for future applications of the ExactVu microUS system in prostate specimen and in vivo imaging.