Multi-Task Learning for Motion Analysis and Segmentation in 3D Echocardiography.

Characterizing left ventricular deformation and strain using 3D+time echocardiography provides useful insights into cardiac function and can be used to detect and localize myocardial injury. To achieve this, it is imperative to obtain accurate motion estimates of the left ventricle. In many strain analysis pipelines, this step is often accompanied by a separate segmentation step; however, recent works have shown both tasks to be highly related and can be complementary when optimized jointly. In this work, we present a multi-task learning network that can simultaneously segment the left ventricle and track its motion between multiple time frames. Two task-specific networks are trained using a composite loss function. Cross-stitch units combine the activations of these networks by learning shared representations between the tasks at different levels. We also propose a novel shape-consistency unit that encourages motion propagated segmentations to match directly predicted segmentations. Using a combined synthetic and in-vivo 3D echocardiography dataset, we demonstrate that our proposed model can achieve excellent estimates of left ventricular motion displacement and myocardial segmentation. Additionally, we observe strong correlation of our image-based strain measurements with crystal-based strain measurements as well as good correspondence with SPECT perfusion mappings. Finally, we demonstrate the clinical utility of the segmentation masks in estimating ejection fraction and sphericity indices that correspond well with benchmark measurements.

Co-attention spatial transformer network for unsupervised motion tracking and cardiac strain analysis in 3D echocardiography.

Myocardial ischemia/infarction causes wall-motion abnormalities in the left ventricle. Therefore, reliable motion estimation and strain analysis using 3D+time echocardiography for localization and characterization of myocardial injury is valuable for early detection and targeted interventions. Previous unsupervised cardiac motion tracking methods rely on heavily-weighted regularization functions to smooth out the noisy displacement fields in echocardiography. In this work, we present a Co-Attention Spatial Transformer Network (STN) for improved motion tracking and strain analysis in 3D echocardiography. Co-Attention STN aims to extract inter-frame dependent features between frames to improve the motion tracking in otherwise noisy 3D echocardiography images. We also propose a novel temporal constraint to further regularize the motion field to produce smooth and realistic cardiac displacement paths over time without prior assumptions on cardiac motion. Our experimental results on both synthetic and in vivo 3D echocardiography datasets demonstrate that our Co-Attention STN provides superior performance compared to existing methods. Strain analysis from Co-Attention STNs also correspond well with the matched SPECT perfusion maps, demonstrating the clinical utility for using 3D echocardiography for infarct localization.

Multiparameter analysis of timelapse imaging reveals kinetics of megakaryocytic erythroid progenitor clonal expansion and differentiation.

Single-cell assays have enriched our understanding of hematopoiesis and, more generally, stem and progenitor cell biology. However, these single-end-point approaches provide only a static snapshot of the state of a cell. To observe and measure dynamic changes that may instruct cell fate, we developed an approach for examining hematopoietic progenitor fate specification using long-term (> 7-day) single-cell time-lapse imaging for up to 13 generations with in situ fluorescence staining of primary human hematopoietic progenitors followed by algorithm-assisted lineage tracing. We analyzed progenitor cell dynamics, including the division rate, velocity, viability, and probability of lineage commitment at the single-cell level over time. We applied a Markov probabilistic model to predict progenitor division outcome over each generation in culture. We demonstrated the utility of this methodological pipeline by evaluating the effects of the cytokines thrombopoietin and erythropoietin on the dynamics of self-renewal and lineage specification in primary human bipotent megakaryocytic-erythroid progenitors (MEPs). Our data support the hypothesis that thrombopoietin and erythropoietin support the viability and self-renewal of MEPs, but do not affect fate specification. Thus, single-cell tracking of time-lapse imaged colony-forming unit assays provides a robust method for assessing the dynamics of progenitor self-renewal and lineage commitment.

Simultaneous Segmentation and Motion Estimation of Left Ventricular Myocardium in 3D Echocardiography Using Multi-task Learning.

Motion estimation and segmentation are both critical steps in identifying and assessing myocardial dysfunction, but are traditionally treated as unique tasks and solved as separate steps. However, many motion estimation techniques rely on accurate segmentations. It has been demonstrated in the computer vision and medical image analysis literature that both these tasks may be mutually beneficial when solved simultaneously. In this work, we propose a multi-task learning network that can concurrently predict volumetric segmentations of the left ventricle and estimate motion between 3D echocardiographic image pairs. The model exploits complementary latent features between the two tasks using a shared feature encoder with task-specific decoding branches. Anatomically inspired constraints are incorporated to enforce realistic motion patterns. We evaluate our proposed model on an in vivo 3D echocardiographic canine dataset. Results suggest that coupling these two tasks in a learning framework performs favorably when compared against single task learning and other alternative methods.

Multi-frame Attention Network for Left Ventricle Segmentation in 3D Echocardiography.

Echocardiography is one of the main imaging modalities used to assess the cardiovascular health of patients. Among the many analyses performed on echocardiography, segmentation of left ventricle is crucial to quantify the clinical measurements like ejection fraction. However, segmentation of left ventricle in 3D echocardiography remains a challenging and tedious task. In this paper, we propose a multi-frame attention network to improve the performance of segmentation of left ventricle in 3D echocardiography. The multi-frame attention mechanism allows highly correlated spatiotemporal features in a sequence of images that come after a target image to be used to augment the performance of segmentation. Experimental results shown on 51 in vivo porcine 3D+time echocardiography images show that utilizing correlated spatiotemporal features significantly improves the performance of left ventricle segmentation when compared to other standard deep learning-based medical image segmentation models.

SHAPE-REGULARIZED UNSUPERVISED LEFT VENTRICULAR MOTION NETWORK WITH SEGMENTATION CAPABILITY IN 3D+TIME ECHOCARDIOGRAPHY.

Accurate motion estimation and segmentation of the left ventricle from medical images are important tasks for quantitative evaluation of cardiovascular health. Echocardiography offers a cost-efficient and non-invasive modality for examining the heart, but provides additional challenges for automated analyses due to the low signal-to-noise ratio inherent in ultrasound imaging. In this work, we propose a shape regularized convolutional neural network for estimating dense displacement fields between sequential 3D B-mode echocardiography images with the capability of also predicting left ventricular segmentation masks. Manually traced segmentations are used as a guide to assist in the unsupervised estimation of displacement between a source and a target image while also serving as labels to train the network to additionally predict segmentations. To enforce realistic cardiac motion patterns, a flow incompressibility term is also incorporated to penalize divergence. Our proposed network is evaluated on an in vivo canine 3D+t B-mode echocardiographic dataset. It is shown that the shape regularizer improves the motion estimation performance of the network and our overall model performs favorably against competing methods.

Learning-Based Regularization for Cardiac Strain Analysis via Domain Adaptation.

Reliable motion estimation and strain analysis using 3D+ time echocardiography (4DE) for localization and characterization of myocardial injury is valuable for early detection and targeted interventions. However, motion estimation is difficult due to the low-SNR that stems from the inherent image properties of 4DE, and intelligent regularization is critical for producing reliable motion estimates. In this work, we incorporated the notion of domain adaptation into a supervised neural network regularization framework. We first propose a semi-supervised Multi-Layered Perceptron (MLP) network with biomechanical constraints for learning a latent representation that is shown to have more physiologically plausible displacements. We extended this framework to include a supervised loss term on synthetic data and showed the effects of biomechanical constraints on the network's ability for domain adaptation. We validated the semi-supervised regularization method on in vivo data with implanted sonomicrometers. Finally, we showed the ability of our semi-supervised learning regularization approach to identify infarct regions using estimated regional strain maps with good agreement to manually traced infarct regions from postmortem excised hearts.

A SEMI-SUPERVISED JOINT LEARNING APPROACH TO LEFT VENTRICULAR SEGMENTATION AND MOTION TRACKING IN ECHOCARDIOGRAPHY.

Accurate interpretation and analysis of echocardiography is important in assessing cardiovascular health. However, motion tracking often relies on accurate segmentation of the myocardium, which can be difficult to obtain due to inherent ultrasound properties. In order to address this limitation, we propose a semi-supervised joint learning network that exploits overlapping features in motion tracking and segmentation. The network simultaneously trains two branches: one for motion tracking and one for segmentation. Each branch learns to extract features relevant to their respective tasks and shares them with the other. Learned motion estimations propagate a manually segmented mask through time, which is used to guide future segmentation predictions. Physiological constraints are introduced to enforce realistic cardiac behavior. Experimental results on synthetic and in vivo canine 2D+t echocardiographic sequences outperform some competing methods in both tasks.

A Semi-supervised Joint Network for Simultaneous Left Ventricular Motion Tracking and Segmentation in 4D Echocardiography.

This work presents a novel deep learning method to combine segmentation and motion tracking in 4D echocardiography. The network iteratively trains a motion branch and a segmentation branch. The motion branch is initially trained entirely unsupervised and learns to roughly map the displacements between a source and a target frame. The estimated displacement maps are then used to generate pseudo-ground truth labels to train the segmentation branch. The labels predicted by the trained segmentation branch are fed back into the motion branch and act as landmarks to help retrain the branch to produce smoother displacement estimations. These smoothed out displacements are then used to obtain smoother pseudo-labels to retrain the segmentation branch. Additionally, a biomechanically-inspired incompressibility constraint is implemented in order to encourage more realistic cardiac motion. The proposed method is evaluated against other approaches using synthetic and in-vivo canine studies. Both the segmentation and motion tracking results of our model perform favorably against competing methods.

Unsupervised Motion Tracking of Left Ventricle in Echocardiography.

Accurate motion tracking of the left ventricle is critical in detecting wall motion abnormalities in the heart after an injury such as a myocardial infarction. We propose an unsupervised motion tracking framework with physiological constraints to learn dense displacement fields between sequential pairs of 2-D B-mode echocardiography images. Current deep-learning motion-tracking algorithms require large amounts of data to provide ground-truth, which is difficult to obtain for in vivo datasets (such as patient data and animal studies), or are unsuccessful in tracking motion between echocardiographic images due to inherent ultrasound properties (such as low signal-to-noise ratio and various image artifacts). We design a U-Net inspired convolutional neural network that uses manually traced segmentations as a guide to learn displacement estimations between a source and target image without ground-truth displacement fields by minimizing the difference between a transformed source frame and the original target frame. We then penalize divergence in the displacement field in order to enforce incompressibility within the left ventricle. We demonstrate the performance of our model on synthetic and in vivo canine 2-D echocardiography datasets by comparing it against a non-rigid registration algorithm and a shape-tracking algorithm. Our results show favorable performance of our model against both methods.

Flow network tracking for spatiotemporal and periodic point matching: Applied to cardiac motion analysis.

The accurate quantification of left ventricular (LV) deformation/strain shows significant promise for quantitatively assessing cardiac function for use in diagnosis and therapy planning. However, accurate estimation of the displacement of myocardial tissue and hence LV strain has been challenging due to a variety of issues, including those related to deriving tracking tokens from images and following tissue locations over the entire cardiac cycle. In this work, we propose a point matching scheme where correspondences are modeled as flow through a graphical network. Myocardial surface points are set up as nodes in the network and edges define neighborhood relationships temporally. The novelty lies in the constraints that are imposed on the matching scheme, which render the correspondences one-to-one through the entire cardiac cycle, and not just two consecutive frames. The constraints also encourage motion to be cyclic, which an important characteristic of LV motion. We validate our method by applying it to the estimation of quantitative LV displacement and strain estimation using 8 synthetic and 8 open-chested canine 4D echocardiographic image sequences, the latter with sonomicrometric crystals implanted on the LV wall. We were able to achieve excellent tracking accuracy on the synthetic dataset and observed a good correlation with crystal-based strains on the in-vivo data.

Efficient Two-Pass 3-D Speckle Tracking for Ultrasound Imaging.

Speckle tracking based on block matching is the most common method for multi-dimensional motion estimation in ultrasound elasticity imaging. Extension of two-dimensional (2-D) methods to three dimensions (3-D) has been problematic because of the large computational load of 3-D tracking, as well as performance issues related to the low frame (volume) rates of 3-D images. To address both of these problems, we have developed an efficient two-pass tracking method suited to cardiac elasticity imaging. PatchMatch, originally developed for image editing, has been adapted for ultrasound to provide first-pass displacement estimates. Second-pass estimation uses conventional block matching within a much smaller search region. 3-D displacements are then obtained using correlation filtering previously shown to be effective against speckle decorrelation. Both simulated and in vivo canine cardiac results demonstrate that the proposed two-pass method reduces computational cost compared to conventional 3-D exhaustive search by a factor of 10. Moreover, it outperforms one-pass tracking by a factor of about 3 in terms of root-mean-square error relative to available ground-truth displacements.