Fast and automatic bone segmentation and registration of 3D ultrasound to CT for the full pelvic anatomy: a comparative study.

PURPOSE: Ultrasound (US) is a safer alternative to X-rays for bone imaging, and its popularity for orthopedic surgical navigation is growing. Routine use of intraoperative US for navigation requires fast, accurate and automatic alignment of tracked US to preoperative computed tomography (CT) patient models. Our group previously investigated image segmentation and registration to align untracked US to CT of only the partial pelvic anatomy. In this paper, we extend this to study the performance of these previously published techniques over the full pelvis in a tracked framework, to characterize their suitability in more realistic scenarios, along with an additional simplified segmentation method and similarity metric for registration. METHOD: We evaluated phase symmetry segmentation, and Gaussian mixture model (GMM) and coherent point drift (CPD) registration methods on a pelvic phantom augmented with human soft tissue images. Additionally, we proposed and evaluated a simplified 3D bone segmentation algorithm we call Shadow-Peak (SP), which uses acoustic shadowing and peak intensities to detect bone surfaces. We paired this with a registration pipeline that optimizes the normalized cross-correlation (NCC) between distance maps of the segmented US-CT images. RESULTS: SP segmentation combined with the proposed NCC registration successfully aligned tracked US volumes to the preoperative CT model in all trials, in contrast to the other techniques. SP with NCC achieved a median target registration error (TRE) of 2.44 mm (maximum 4.06 mm), when imaging all three anterior pelvic structures, and a mean runtime of 27.3 s. SP segmentation with CPD registration was the next most accurate combination: median TRE of 3.19 mm (maximum 6.07 mm), though a much faster runtime of 4.2 s. CONCLUSION: We demonstrate an accurate, automatic image processing pipeline for intraoperative alignment of US-CT over the full pelvis and compare its performance with the state-of-the-art methods. The proposed methods are amenable to clinical implementation due to their high accuracy on realistic data and acceptably low runtimes.

A Systematic Review and Meta-analysis on the Reproducibility of Ultrasound-based Metrics for Assessing Developmental Dysplasia of the Hip.

BACKGROUND: The purposes of this study were to (1) perform a systematic review of articles that reported agreement or reproducibility in repeated diagnosis of developmental dysplasia of the hip (DDH) using ultrasound imaging, (2) estimate the reproducibility in the available dysplasia metrics, and (3) compare reproducibility of the available dysplasia metrics. METHODS: A systematic review of the Medline and Embase databases was performed by using a search strategy formulated from our research question: "For infants at risk of DDH, are US imaging-based diagnoses reproducible?" Two reviewers independently identified articles for inclusion in the systematic review, and then assessed the quality of the included studies using the Guidelines for Reporting Reliability and Agreement Studies guideline. Variability and agreement-related statistics in the included studies were extracted and included in a meta-analysis for summarizing the available statistics. The reproducibility of the available dysplasia metrics was compared, with a Bonferroni correction made to adjust for multiple comparisons. RESULTS: Twenty eight studies were included in the systematic review. Overall, the quality of the included studies was moderate (average, 10.7/15; range, 6 to 12). Graf's alpha angle had the lowest interexamination variability of the metrics assessed, followed by Graf's beta angle (the variability of the alpha angle was 10% lower than the variability of the beta angle, P<0.05). However, despite Graf's angles having lower variability compared with other dysplasia metrics, their actual variability was still problematically high. This finding was supported by the low intraclass correlation and Kappa coefficient values reported in the included studies. There was also evidence to suggest that the reproducibility in DDH diagnosis has potentially worsened over time. CONCLUSIONS: Overall, we found high variability and low agreement in all reported dysplasia metrics. Furthermore, in the last 3 decades, the repeatability of dysplasia metrics has not markedly improved and may even have declined, indicating a genuine need for improving repeatability and reliability of ultrasound-based DDH diagnosis. LEVEL OF EVIDENCE: Level III-systematic review of level III studies.

Modelling and extraction of pulsatile radial distension and compression motion for automatic vessel segmentation from video.

Identification of vascular structures from medical images is integral to many clinical procedures. Most vessel segmentation techniques ignore the characteristic pulsatile motion of vessels in their formulation. In a recent effort to automatically segment vessels that are hidden under fat, we motivated the use of the magnitude of local pulsatile motion extracted from surgical endoscopic video. In this article we propose a new approach that leverages the local orientation, in addition to magnitude of motion, and demonstrate that the extended computation and utilization of motion vectors can improve the segmentation of vascular structures. We implement our approach using four alternatives to magnitude-only motion estimation by using traditional optical flow and by exploiting the monogenic signal for fast flow estimation. Our evaluations are conducted on both synthetic phantoms as well as two real ultrasound datasets showing improved segmentation results with negligible change in computational performance compared to the previous magnitude only approach.

Variability in muscle activation of simple speech motions: A biomechanical modeling approach.

Biomechanical models of the oropharynx facilitate the study of speech function by providing information that cannot be directly derived from imaging data, such as internal muscle forces and muscle activation patterns. Such models, when constructed and simulated based on anatomy and motion captured from individual speakers, enable the exploration of inter-subject variability of speech biomechanics. These models also allow one to answer questions, such as whether speakers produce similar sounds using essentially the same motor patterns with subtle differences, or vastly different motor equivalent patterns. Following this direction, this study uses speaker-specific modeling tools to investigate the muscle activation variability in two simple speech tasks that move the tongue forward (/ə-ɡis/) vs backward (/ə-suk/). Three dimensional tagged magnetic resonance imaging data were used to inversely drive the biomechanical models in four English speakers. Results show that the genioglossus is the workhorse muscle of the tongue, with activity levels of 10% in different subdivisions at different times. Jaw and hyoid positioners (inferior pterygoid and digastric) also show high activation during specific phonemes. Other muscles may be more involved in fine tuning the shapes. For example, slightly more activation of the anterior portion of the transverse is found during apical than laminal /s/, which would protrude the tongue tip to a greater extent for the apical /s/.

Automatic Evaluation of Scan Adequacy and Dysplasia Metrics in 2-D Ultrasound Images of the Neonatal Hip.

Ultrasound (US) imaging of an infant's hip joint is widely used for early detection of developmental dysplasia of the hip. In current US-based diagnosis of developmental dysplasia of the hip, trained clinicians acquire US images and, if they judge them to be adequate (i.e., to contain relevant hip joint structures), analyze them manually to extract clinically useful dysplasia metrics. However, both the scan adequacy classification and dysplasia metrics extraction steps exhibit significant variability within and between both clinicians and institutions, which can result in significant over- and undertreatment rates. To reduce the subjectivity resulting from this variability, we propose a computational image analysis technique that automatically identifies adequate images and subsequently extracts dysplasia metrics from these 2-D US images. Our automatic method uses local phase symmetry-based image measures to robustly identify intensity-invariant geometric features of bone/cartilage boundaries from the US images. Using the extracted geometric features, we trained a random forest classifier to classify images as adequate or inadequate, and in the adequate images we used a subset of the geometric features to calculate key dysplasia metrics. We validated our method on a data set of 693 US scans collected from 35 infants. Our approach produces excellent agreement with clinician adequacy classifications (area under the receiver operating characteristic curve = 0.985) and in reducing variability in the measured developmental dysplasia of the hip metrics (p < 0.05). The automatically computed dysplasia metrics appear to be slightly biased toward higher Graf categories than the manually estimated metrics, which could potentially reduce missed early diagnoses.

Strain-Initialized Robust Bone Surface Detection in 3-D Ultrasound.

Three-dimensional ultrasound has been increasingly considered as a safe radiation-free alternative to radiation-based fluoroscopic imaging for surgical guidance during computer-assisted orthopedic interventions, but because ultrasound images contain significant artifacts, it is challenging to automatically extract bone surfaces from these images. We propose an effective way to extract 3-D bone surfaces using a surface growing approach that is seeded from 2-D bone contours. The initial 2-D bone contours are estimated from a combination of ultrasound strain images and envelope power images. Novel features of the proposed method include: (i) improvement of a previously reported 2-D strain imaging-based bone segmentation method by incorporation of a depth-dependent cumulative power of the envelope into the elastographic data; (ii) incorporation of an echo decorrelation measure-based weight to fuse the strain and envelope maps; (iii) use of local statistics of the bone surface candidate points to detect the presence of any bone discontinuity; and (iv) an extension of our 2-D bone contour into a 3-D bone surface by use of an effective surface growing approach. Our new method produced average improvements in the mean absolute error of 18% and 23%, respectively, on 2-D and 3-D experimental phantom data, compared with those of two state-of-the-art bone segmentation methods. Validation on 2-D and 3-D clinical in vivo data also reveals, respectively, an average improvement in the mean absolute fitting error of 55% and an 18-fold improvement in the computation time.

Endoscopic scene labelling and augmentation using intraoperative pulsatile motion and colour appearance cues with preoperative anatomical priors.

PURPOSE: Despite great advances in medical image segmentation, the accurate and automatic segmentation of endoscopic scenes remains a challenging problem. Two important aspects have to be considered in segmenting an endoscopic scene: (1) noise and clutter due to light reflection and smoke from cutting tissue, and (2) structure occlusion (e.g. vessels occluded by fat, or endophytic tumours occluded by healthy kidney tissue). METHODS: In this paper, we propose a variational technique to augment a surgeon's endoscopic view by segmenting visible as well as occluded structures in the intraoperative endoscopic view. Our method estimates the 3D pose and deformation of anatomical structures segmented from 3D preoperative data in order to align to and segment corresponding structures in 2D intraoperative endoscopic views. Our preoperative to intraoperative alignment is driven by, first, spatio-temporal, signal processing based vessel pulsation cues and, second, machine learning based analysis of colour and textural visual cues. To our knowledge, this is the first work that utilizes vascular pulsation cues for guiding preoperative to intraoperative registration. In addition, we incorporate a tissue-specific (i.e. heterogeneous) physically based deformation model into our framework to cope with the non-rigid deformation of structures that occurs during the intervention. RESULTS: We validated the utility of our technique on fifteen challenging clinical cases with 45 % improvements in accuracy compared to the state-of-the-art method. CONCLUSIONS: A new technique for localizing both visible and occluded structures in an endoscopic view was proposed and tested. This method leverages both preoperative data, as a source of patient-specific prior knowledge, as well as vasculature pulsation and endoscopic visual cues in order to accurately segment the highly noisy and cluttered environment of an endoscopic video. Our results on in vivo clinical cases of partial nephrectomy illustrate the potential of the proposed framework for augmented reality applications in minimally invasive surgeries.

Simultaneous Multi-Structure Segmentation and 3D Nonrigid Pose Estimation in Image-Guided Robotic Surgery.

In image-guided robotic surgery, segmenting the endoscopic video stream into meaningful parts provides important contextual information that surgeons can exploit to enhance their perception of the surgical scene. This information provides surgeons with real-time decision-making guidance before initiating critical tasks such as tissue cutting. Segmenting endoscopic video is a challenging problem due to a variety of complications including significant noise attributed to bleeding and smoke from cutting, poor appearance contrast between different tissue types, occluding surgical tools, and limited visibility of the objects' geometries on the projected camera views. In this paper, we propose a multi-modal approach to segmentation where preoperative 3D computed tomography scans and intraoperative stereo-endoscopic video data are jointly analyzed. The idea is to segment multiple poorly visible structures in the stereo/multichannel endoscopic videos by fusing reliable prior knowledge captured from the preoperative 3D scans. More specifically, we estimate and track the pose of the preoperative models in 3D and consider the models' non-rigid deformations to match with corresponding visual cues in multi-channel endoscopic video and segment the objects of interest. Further, contrary to most augmented reality frameworks in endoscopic surgery that assume known camera parameters, an assumption that is often violated during surgery due to non-optimal camera calibration and changes in camera focus/zoom, our method embeds these parameters into the optimization hence correcting the calibration parameters within the segmentation process. We evaluate our technique on synthetic data, ex vivo lamb kidney datasets, and in vivo clinical partial nephrectomy surgery with results demonstrating high accuracy and robustness.

Stable Overlapping Replicator Dynamics for Brain Community Detection.

A fundamental means for understanding the brain's organizational structure is to group its spatially disparate regions into functional subnetworks based on their interactions. Most community detection techniques are designed for generating partitions, but certain brain regions are known to interact with multiple subnetworks. Thus, the brain's underlying subnetworks necessarily overlap. In this paper, we propose a technique for identifying overlapping subnetworks from weighted graphs with statistical control over false node inclusion. Our technique improves upon the replicator dynamics formulation by incorporating a graph augmentation strategy to enable subnetwork overlaps, and a graph incrementation scheme for merging subnetworks that might be falsely split by replicator dynamics due to its stringent mutual similarity criterion in defining subnetworks. To statistically control for inclusion of false nodes into the detected subnetworks, we further present a procedure for integrating stability selection into our subnetwork identification technique. We refer to the resulting technique as stable overlapping replicator dynamics (SORD). Our experiments on synthetic data show significantly higher accuracy in subnetwork identification with SORD than several state-of-the-art techniques. We also demonstrate higher test-retest reliability in multiple network measures on the Human Connectome Project data. Further, we illustrate that SORD enables identification of neuroanatomically-meaningful subnetworks and network hubs.

Fast and Accurate Data Extraction for Near Real-Time Registration of 3-D Ultrasound and Computed Tomography in Orthopedic Surgery.

Automatic, accurate and real-time registration is an important step in providing effective guidance and successful anatomic restoration in ultrasound (US)-based computer assisted orthopedic surgery. We propose a method in which local phase-based bone surfaces, extracted from intra-operative US data, are registered to pre-operatively segmented computed tomography data. Extracted bone surfaces are downsampled and reinforced with high curvature features. A novel hierarchical simplification algorithm is used to further optimize the point clouds. The final point clouds are represented as Gaussian mixture models and iteratively matched by minimizing the dissimilarity between them using an L2 metric. For 44 clinical data sets from 25 pelvic fracture patients and 49 phantom data sets, we report mean surface registration accuracies of 0.31 and 0.77 mm, respectively, with an average registration time of 1.41 s. Our results suggest the viability and potential of the chosen method for real-time intra-operative registration in orthopedic surgery.

Coupled Stable Overlapping Replicator Dynamics for Multimodal Brain Subnetwork Identification.

Combining imaging modalities to synthesize their inherent strengths provides a promising means for improving brain subnetwork identification. We propose a multimodal integration technique based on a sex-differentiated formulation of replicator dynamics for identifying subnetworks of brain regions that exhibit high inter-connectivity both functionally and structurally. Our method has a number of desired properties, namely, it can operate on weighted graphs derived from functional magnetic resonance imaging (tMRI) and diffusion MRI (dMRI) data, allows for subnetwork overlaps, has an intrinsic criterion for setting the number of subnetworks, and provides statistical control on false node inclusion in the identified subnetworks via the incorporation of stability selection. We thus refer to our technique as coupled stable overlapping replicator dynamics (CSORD). On synthetic data, We demonstrate that CSORD achieves significantly higher subnetwork identification accuracy than state-of-the-art techniques. On real. data from the Human Connectome Project (HCP), we show that CSORD attains improved test-retest reliability on multiple network measures and superior task classification accuracy.

Automatic segmentation of occluded vasculature via pulsatile motion analysis in endoscopic robot-assisted partial nephrectomy video.

Hilar dissection is an important and delicate stage in partial nephrectomy, during which surgeons remove connective tissue surrounding renal vasculature. Serious complications arise when the occluded blood vessels, concealed by fat, are missed in the endoscopic view and as a result are not appropriately clamped. Such complications may include catastrophic blood loss from internal bleeding and associated occlusion of the surgical view during the excision of the cancerous mass (due to heavy bleeding), both of which may compromise the visibility of surgical margins or even result in a conversion from a minimally invasive to an open intervention. To aid in vessel discovery, we propose a novel automatic method to segment occluded vasculature from labeling minute pulsatile motion that is otherwise imperceptible with the naked eye. Our segmentation technique extracts subtle tissue motions using a technique adapted from phase-based video magnification, in which we measure motion from periodic changes in local phase information albeit for labeling rather than magnification. Based on measuring local phase through spatial decomposition of each frame of the endoscopic video using complex wavelet pairs, our approach assigns segmentation labels by detecting regions exhibiting temporal local phase changes matching the heart rate. We demonstrate how our technique is a practical solution for time-critical surgical applications by presenting quantitative and qualitative performance evaluations of our vessel detection algorithms with a retrospective study of fifteen clinical robot-assisted partial nephrectomies.

Automatic extraction of bone surfaces from 3D ultrasound images in orthopaedic trauma cases.

PURPOSE: 3D ultrasound (US) imaging has the potential to become a powerful alternative imaging modality in orthopaedic surgery as it is radiation-free and can produce 3D images (in contrast to fluoroscopy) in near-real time. Conventional B-mode US images, however, are characterized by high levels of noise and reverberation artifacts, image quality is user-dependent, and bone surfaces are blurred, which makes it difficult to both interpret images and to use them as a basis for navigated interventions. 3D US has great potential to assist orthopaedic care, possibly assisting during surgery if the anatomical structures of interest could be localized and visualized with sufficient accuracy and clarity and in a highly automated rapid manner. METHODS: In this paper, we present clinical results for a novel 3D US segmentation technique we have recently developed based on multi-resolution analysis to localize bone surfaces in 3D US volumes. Our method is validated on scans obtained from 29 trauma patients with distal radius and pelvic ring fractures. RESULTS: Qualitative and quantitative results demonstrate remarkably clear segmentations of bone surfaces with an average surface fitting error of 0.62 mm (standard deviation (SD) of 0.42 mm) for pelvic patients and 0.21 mm (SD 0.14 mm) for distal radius patients. CONCLUSIONS: These results suggest that our technique is sufficiently accurate for potential use in orthopaedic trauma applications.

Automatic labelling of tumourous frames in free-hand laparoscopic ultrasound video.

Laparoscopic ultrasound (US) is often used during partial nephrectomy surgeries to identify tumour boundaries within the kidney. However, visual identification is challenging as tumour appearance varies across patients and US images exhibit significant noise levels. To address these challenges, we present the first fully automatic method for detecting the presence of kidney tumour in free-hand laparoscopic ultrasound sequences in near real-time. Our novel approach predicts the probability that a frame contains tumourous tissue using random forests and encodes this probability combined with a regularization term within a graph. Using Dijkstra's algorithm we find a globally optimal labelling (tumour vs. non-tumour) of each frame. We validate our method on a challenging clinical dataset composed of five patients, with a total of 2025 2D ultrasound frames, and demonstrate the ability to detect the presence of kidney tumour with a sensitivity and specificity of 0.774 and 0.916, respectively.

Efficient multi-organ segmentation in multi-view endoscopic videos using pre-operative priors.

Synergistic fusion of pre-operative (pre-op) and intraoperative (intra-op) imaging data provides surgeons with invaluable insightful information that can improve their decision-making during minimally invasive robotic surgery. In this paper, we propose an efficient technique to segment multiple objects in intra-op multi-view endoscopic videos based on priors captured from pre-op data. Our approach leverages information from 3D pre-op data into the analysis of visual cues in the 2D intra-op data by formulating the problem as one of finding the 3D pose and non-rigid deformations of tissue models driven by features from 2D images. We present a closed-form solution for our formulation and demonstrate how it allows for the inclusion of laparoscopic camera motion model. Our efficient method runs in real-time on a single core CPU making it practical even for robotic surgery systems with limited computational resources. We validate the utility of our technique on ex vivo data as well as in vivo clinical data from laparoscopic partial nephrectomy surgery and demonstrate its robustness in segmenting stereo endoscopic videos.

Robust bone detection in ultrasound using combined strain imaging and envelope signal power detection.

Bone localization in ultrasound (US) remains challenging despite encouraging advances. Current methods, e.g. local image phase-based feature analysis, showed promising results but remain reliant on delicate parameter selection processes and prone to errors at confounding soft tissue interfaces of similar appearance to bone interfaces. We propose a different approach combining US strain imaging and envelope power detection at each radio-frequency (RF) sample. After initial estimation of strain and envelope power maps, we modify their dynamic ranges into a modified strain map (MSM) and a modified envelope map (MEM) that we subsequently fuse into a single combined map that we show corresponds robustly to actual bone boundaries. Our quantitative results demonstrate a marked reduction in false positive responses at soft tissue interfaces and an increase in bone delineation accuracy. Comparisons to the state-of-the-art on a finite-element-modelling (FEM) phantom and fiducial-based experimental phantom show an average improvement in mean absolute error (MAE) between actual and estimated bone boundaries of 32% and 14%, respectively. We also demonstrate an average reduction in false bone responses of 87% and 56%, respectively. Finally, we qualitatively validate on clinical in vivo data of the human radius and ulna bones, and demonstrate similar improvements to those observed on phantoms.

Auto localization and segmentation of occluded vessels in robot-assisted partial nephrectomy.

Hilar dissection is an important and delicate stage in partial nephrectomy during which surgeons remove connective tissue surrounding renal vasculature. Potentially serious complications arise when vessels occluded by fat are missed in the endoscopic view and are not appropriately clamped. To aid in vessel discovery, we propose an automatic method to localize and label occluded vasculature. Our segmentation technique is adapted from phase-based video magnification, in which we measure subtle motion from periodic changes in local phase information albeit for labeling rather than magnification. We measure local phase through spatial decomposition of each frame of the endoscopic video using complex wavelet pairs. We then assign segmentation labels based on identifying responses of regions exhibiting temporal local phase changes matching the heart rate frequency. Our method is evaluated with a retrospective study of eight real robot-assisted partial nephrectomies demonstrating utility for surgical guidance that could potentially reduce operation times and complication rates.

Volume-specific parameter optimization of 3D local phase features for improved extraction of bone surfaces in ultrasound.

BACKGROUND: Accurate localization of bone surfaces remains a challenge hampering adoption of ultrasound guidance in computer-assisted orthopaedic surgery. Local phase image features have recently been proven efficacious for segmenting bone surfaces from ultrasound images, but the quality of the processing depends on numerous filter parameters that are currently set through a trial and error process that is tedious, unintuitive and subject to large inter-user variability. METHODS: A method is presented for automatically selecting parameters of Log-Gabor filters used to extract bone surfaces from 3D ultrasound volumes that is based on properties estimated directly from the specific image. RESULTS: A 15% and 69% average improvement in bone surface localization accuracy on phantom and clinical data, respectively, is demonstrated compared with empirically-set parameters. CONCLUSIONS: These findings imply that Log-Gabor filter parameter optimization is necessary for accurate extraction of bone surfaces from ultrasound data.

Overlapping replicator dynamics for functional subnetwork identification.

Functional magnetic resonance imaging (fMRI) has been widely used for inferring brain regions that tend to work in tandem and grouping them into subnetworks. Despite that certain brain regions are known to interact with multiple subnetworks, few existing techniques support identification of subnetworks with overlaps. To address this limitation, we propose a novel approach based on replicator dynamics that facilitates detection of sparse overlapping subnetworks. We refer to our approach as overlapping replicator dynamics (RDOL). On synthetic data, we show that RDOL achieves higher accuracy in subnetwork identification than state-of-the-art methods. On real data, we demonstrate that RDOL is able to identify major functional hubs that are known to serve as communication channels between brain regions, in addition to detecting commonly observed functional subnetworks. Moreover, we illustrate that knowing the subnetwork overlaps enables inference of functional pathways, e.g. from primary sensory areas to the integration hubs.

Fiber connectivity integrated brain activation detection.

Inference of brain activation through the analysis of functional magnetic resonance imaging (fMRI) data is seriously confounded by the high level f noise in the observations. To mitigate the effects of noise, we propose incorporating anatomical connectivity into brain activation detection as motivated by how the functional integration of distinct brain areas is facilitated via neural fiber pathways. In this work, we formulate activation detection as a probabilistic graph-based segmentation problem with fiber networks estimated from diffusion MRI (dMRI) data serving as a prior. Our approach is reinforced with a data-driven scheme for refining the connectivity prior to reflect the fact that not all fibers are necessarily deployed during a given cognitive task as well as to account for false fiber tracts arising from limitations of dMRI tractography. Validating on real clinical data collected from 7 schizophrenia patients and 13 matched healthy controls, we show that incorporating anatomical connectivity significantly increases sensitivity in detecting task activation in controls compared to existing univariate techniques. Further, we illustrate how our model enables the detection of significant group activation differences between controls and patients that are missed with standard methods.