Automatic detection of pulmonary embolism on computed tomography pulmonary angiogram scan using a three-dimensional convolutional neural network.

OBJECTIVE: To propose a convolutional neural network (EmbNet) for automatic pulmonary embolism detection on computed tomography pulmonary angiogram (CTPA) scans and to assess its diagnostic performance. METHODS: 305 consecutive CTPA scans between January 2019 and December 2021 were enrolled in this study (142 for training, 163 for internal validation), and 250 CTPA scans from a public dataset were used for external validation. The framework comprised a preprocessing step to segment the pulmonary vessels and the EmbNet to detect emboli. Emboli were divided into three location-based subgroups for detailed evaluation: central arteries, lobar branches, and peripheral regions. Ground truth was established by three radiologists. RESULTS: The EmbNet's per-scan level sensitivity, specificity, positive predictive value (PPV), and negative predictive value were 90.9%, 75.4%, 48.4%, and 97.0% (internal validation) and 88.0%, 70.5%, 42.7%, and 95.9% (external validation). At the per-embolus level, the overall sensitivity and PPV of the EmbNet were 86.0% and 61.3% (internal validation), and 83.5% and 57.5% (external validation). The sensitivity and PPV of central emboli were 89.7% and 52.0% (internal validation), and 94.4% and 43.0% (external validation); of lobar emboli were 95.2% and 76.9% (internal validation), and 93.5% and 72.5% (external validation); and of peripheral emboli were 82.6% and 61.7% (internal validation), and 80.2% and 59.4% (external validation). The average false positive rate was 0.45 false emboli per scan (internal validation) and 0.69 false emboli per scan (external validation). CONCLUSION: The EmbNet provides high sensitivity across embolus locations, suggesting its potential utility for initial screening in clinical practice.

AVDNet: Joint coronary artery and vein segmentation with topological consistency.

Coronary CT angiography (CCTA) is an effective and non-invasive method for coronary artery disease diagnosis. Extracting an accurate coronary artery tree from CCTA image is essential for centerline extraction, plaque detection, and stenosis quantification. In practice, data quality varies. Sometimes, the arteries and veins have similar intensities and locate closely, which may confuse segmentation algorithms, even deep learning based ones, to obtain accurate arteries. However, it is not always feasible to re-scan the patient for better image quality. In this paper, we propose an artery and vein disentanglement network (AVDNet) for robust and accurate segmentation by incorporating the coronary vein into the segmentation task. This is the first work to segment coronary artery and vein at the same time. The AVDNet consists of an image based vessel recognition network (IVRN) and a topology based vessel refinement network (TVRN). IVRN learns to segment the arteries and veins, while TVRN learns to correct the segmentation errors based on topology consistency. We also design a novel inverse distance weighted dice (IDD) loss function to recover more thin vessel branches and preserve the vascular boundaries. Extensive experiments are conducted on a multi-center dataset of 700 patients. Quantitative and qualitative results demonstrate the effectiveness of the proposed method by comparing it with state-of-the-art methods and different variants. Prediction results of the AVDNet on the Automated Segmentation of Coronary Artery Challenge dataset are avaliabel at https://github.com/WennyJJ/Coronary-Artery-Vein-Segmentation for follow-up research.

Mining multi-center heterogeneous medical data with distributed synthetic learning.

Overcoming barriers on the use of multi-center data for medical analytics is challenging due to privacy protection and data heterogeneity in the healthcare system. In this study, we propose the Distributed Synthetic Learning (DSL) architecture to learn across multiple medical centers and ensure the protection of sensitive personal information. DSL enables the building of a homogeneous dataset with entirely synthetic medical images via a form of GAN-based synthetic learning. The proposed DSL architecture has the following key functionalities: multi-modality learning, missing modality completion learning, and continual learning. We systematically evaluate the performance of DSL on different medical applications using cardiac computed tomography angiography (CTA), brain tumor MRI, and histopathology nuclei datasets. Extensive experiments demonstrate the superior performance of DSL as a high-quality synthetic medical image provider by the use of an ideal synthetic quality metric called Dist-FID. We show that DSL can be adapted to heterogeneous data and remarkably outperforms the real misaligned modalities segmentation model by 55% and the temporal datasets segmentation model by 8%.

Root canal treatment planning by automatic tooth and root canal segmentation in dental CBCT with deep multi-task feature learning.

Accurate and automatic segmentation of individual tooth and root canal from cone-beam computed tomography (CBCT) images is an essential but challenging step for dental surgical planning. In this paper, we propose a novel framework, which consists of two neural networks, DentalNet and PulpNet, for efficient, precise, and fully automatic tooth instance segmentation and root canal segmentation from CBCT images. We first use the proposed DentalNet to achieve tooth instance segmentation and identification. Then, the region of interest (ROI) of the affected tooth is extracted and fed into the PulpNet to obtain precise segmentation of the pulp chamber and the root canal space. These two networks are trained by multi-task feature learning and evaluated on two clinical datasets respectively and achieve superior performances to several comparing methods. In addition, we incorporate our method into an efficient clinical workflow to improve the surgical planning process. In two clinical case studies, our workflow took only 2 min instead of 6 h to obtain the 3D model of tooth and root canal effectively for the surgical planning, resulting in satisfying outcomes in difficult root canal treatments.

Modality Bank: Learn multi-modality images across data centers without sharing medical data.

Multi-modality images have been widely used and provide comprehensive information for medical image analysis. However, acquiring all modalities among all institutes is costly and often impossible in clinical settings. To leverage more comprehensive multi-modality information, we propose privacy secured decentralized multi-modality adaptive learning architecture named ModalityBank. Our method could learn a set of effective domain-specific modulation parameters plugged into a common domain-agnostic network. We demonstrate by switching different sets of configurations, the generator could output high-quality images for a specific modality. Our method could also complete the missing modalities across all data centers, thus could be used for modality completion purposes. The downstream task trained from the synthesized multi-modality samples could achieve higher performance than learning from one real data center and achieve close- to- real performance compare with all real images.

Genetic mutation and biological pathway prediction based on whole slide images in breast carcinoma using deep learning.

Breast carcinoma is the most common cancer among women worldwide that consists of a heterogeneous group of subtype diseases. The whole-slide images (WSIs) can capture the cell-level heterogeneity, and are routinely used for cancer diagnosis by pathologists. However, key driver genetic mutations related to targeted therapies are identified by genomic analysis like high-throughput molecular profiling. In this study, we develop a deep-learning model to predict the genetic mutations and biological pathway activities directly from WSIs. Our study offers unique insights into WSI visual interactions between mutation and its related pathway, enabling a head-to-head comparison to reinforce our major findings. Using the histopathology images from the Genomic Data Commons Database, our model can predict the point mutations of six important genes (AUC 0.68-0.85) and copy number alteration of another six genes (AUC 0.69-0.79). Additionally, the trained models can predict the activities of three out of ten canonical pathways (AUC 0.65-0.79). Next, we visualized the weight maps of tumor tiles in WSI to understand the decision-making process of deep-learning models via a self-attention mechanism. We further validated our models on liver and lung cancers that are related to metastatic breast cancer. Our results provide insights into the association between pathological image features, molecular outcomes, and targeted therapies for breast cancer patients.

Surgical planning of pelvic tumor using multi-view CNN with relation-context representation learning.

Limb salvage surgery of malignant pelvic tumors is the most challenging procedure in musculoskeletal oncology due to the complex anatomy of the pelvic bones and soft tissues. It is crucial to accurately resect the pelvic tumors with appropriate margins in this procedure. However, there is still a lack of efficient and repetitive image planning methods for tumor identification and segmentation in many hospitals. In this paper, we present a novel deep learning-based method to accurately segment pelvic bone tumors in MRI. Our method uses a multi-view fusion network to extract pseudo-3D information from two scans in different directions and improves the feature representation by learning a relational context. In this way, it can fully utilize spatial information in thick MRI scans and reduce over-fitting when learning from a small dataset. Our proposed method was evaluated on two independent datasets collected from 90 and 15 patients, respectively. The segmentation accuracy of our method was superior to several comparing methods and comparable to the expert annotation, while the average time consumed decreased about 100 times from 1820.3 seconds to 19.2 seconds. In addition, we incorporate our method into an efficient workflow to improve the surgical planning process. Our workflow took only 15 minutes to complete surgical planning in a phantom study, which is a dramatic acceleration compared with the 2-day time span in a traditional workflow.

Few-Shot Learning by a Cascaded Framework With Shape-Constrained Pseudo Label Assessment for Whole Heart Segmentation.

Automatic and accurate 3D cardiac image segmentation plays a crucial role in cardiac disease diagnosis and treatment. Even though CNN based techniques have achieved great success in medical image segmentation, the expensive annotation, large memory consumption, and insufficient generalization ability still pose challenges to their application in clinical practice, especially in the case of 3D segmentation from high-resolution and large-dimension volumetric imaging. In this paper, we propose a few-shot learning framework by combining ideas of semi-supervised learning and self-training for whole heart segmentation and achieve promising accuracy with a Dice score of 0.890 and a Hausdorff distance of 18.539 mm with only four labeled data for training. When more labeled data provided, the model can generalize better across institutions. The key to success lies in the selection and evolution of high-quality pseudo labels in cascaded learning. A shape-constrained network is built to assess the quality of pseudo labels, and the self-training stages with alternative global-local perspectives are employed to improve the pseudo labels. We evaluate our method on the CTA dataset of the MM-WHS 2017 Challenge and a larger multi-center dataset. In the experiments, our method outperforms the state-of-the-art methods significantly and has great generalization ability on the unseen data. We also demonstrate, by a study of two 4D (3D+T) CTA data, the potential of our method to be applied in clinical practice.

Weakly Supervised Deep Nuclei Segmentation Using Partial Points Annotation in Histopathology Images.

Nuclei segmentation is a fundamental task in histopathology image analysis. Typically, such segmentation tasks require significant effort to manually generate accurate pixel-wise annotations for fully supervised training. To alleviate such tedious and manual effort, in this paper we propose a novel weakly supervised segmentation framework based on partial points annotation, i.e., only a small portion of nuclei locations in each image are labeled. The framework consists of two learning stages. In the first stage, we design a semi-supervised strategy to learn a detection model from partially labeled nuclei locations. Specifically, an extended Gaussian mask is designed to train an initial model with partially labeled data. Then, self-training with background propagation is proposed to make use of the unlabeled regions to boost nuclei detection and suppress false positives. In the second stage, a segmentation model is trained from the detected nuclei locations in a weakly-supervised fashion. Two types of coarse labels with complementary information are derived from the detected points and are then utilized to train a deep neural network. The fully-connected conditional random field loss is utilized in training to further refine the model without introducing extra computational complexity during inference. The proposed method is extensively evaluated on two nuclei segmentation datasets. The experimental results demonstrate that our method can achieve competitive performance compared to the fully supervised counterpart and the state-of-the-art methods while requiring significantly less annotation effort.

Protective effects of choline against hypoxia-induced injuries of vessels and endothelial cells.

The current study aimed to lay a theoretical foundation for further development of choline as an anti-hypoxia damage drug. Wild-type, 3- to 5-month-old male Sprague-Dawley rats, weighing 180-220 g, were used in this study. The rats were randomly divided into a normoxic control group (n=16) and a chronic intermittent hypoxia (CIH) group (n=16). The effects of CIH on acetylcholine (ACh)-mediated endothelium-dependent vasodilatation in the rat cerebral basilar arterioles and mesenteric arterioles, as well as the protective effects of choline on the arterioles damaged by hypoxia were observed. Moreover, the effects of choline on endothelial cell proliferation during hypoxia were observed, and choline's functional mechanism further explored. The ACh-mediated vasodilatation of rat cerebral basilar and mesenteric arterioles significantly reduced during hypoxia (P<0.01). Choline significantly increased dilation in the rat cerebral basilar (P<0.01) and mesenteric arterioles (P<0.05) damaged by CIH compared with those in the control group. In addition, under hypoxic conditions, choline significantly promoted the proliferation of rat aortic endothelial cells (P<0.05) and significantly reduced lactate dehydrogenase activity in the cell culture supernatant in vitro (P<0.05). Furthermore, the effect of choline could be related to its ability to significantly increase the secretion of vascular endothelial growth factor (P<0.01) and activation of α7 non-neuronal nicotinic acetylcholine receptors under hypoxia (P<0.01). This study demonstrated that choline could have protective effects against hypoxic injuries.

Automated compromised right lung segmentation method using a robust atlas-based active volume model with sparse shape composition prior in CT.

To resolve challenges in image segmentation in oncologic patients with severely compromised lung, we propose an automated right lung segmentation framework that uses a robust, atlas-based active volume model with a sparse shape composition prior. The robust atlas is achieved by combining the atlas with the output of sparse shape composition. Thoracic computed tomography images (n=38) from patients with lung tumors were collected. The right lung in each scan was manually segmented to build a reference training dataset against which the performance of the automated segmentation method was assessed. The quantitative results of this proposed segmentation method with sparse shape composition achieved mean Dice similarity coefficient (DSC) of (0.72, 0.81) with 95% CI, mean accuracy (ACC) of (0.97, 0.98) with 95% CI, and mean relative error (RE) of (0.46, 0.74) with 95% CI. Both qualitative and quantitative comparisons suggest that this proposed method can achieve better segmentation accuracy with less variance than other atlas-based segmentation methods in the compromised lung segmentation.

Bodypart Recognition Using Multi-stage Deep Learning.

Automatic medical image analysis systems often start from identifying the human body part contained in the image; Specifically, given a transversal slice, it is important to know which body part it comes from, namely "slice-based bodypart recognition". This problem has its unique characteristic--the body part of a slice is usually identified by local discriminative regions instead of global image context, e.g., a cardiac slice is differentiated from an aorta arch slice by the mediastinum region. To leverage this characteristic, we design a multi-stage deep learning framework that aims at: (1) discover the local regions that are discriminative to the bodypart recognition, and (2) learn a bodypart identifier based on these local regions. These two tasks are achieved by the two stages of our learning scheme, respectively. In the pre-train stage, a convolutional neural network (CNN) is learned in a multi-instance learning fashion to extract the most discriminative local patches from the training slices. In the boosting stage, the learned CNN is further boosted by these local patches for bodypart recognition. By exploiting the discriminative local appearances, the learned CNN becomes more accurate than global image context-based approaches. As a key hallmark, our method does not require manual annotations of the discriminative local patches. Instead, it automatically discovers them through multi-instance deep learning. We validate our method on a synthetic dataset and a large scale CT dataset (7000+ slices from wholebody CT scans). Our method achieves better performances than state-of-the-art approaches, including the standard CNN.

Is interactional dissynchrony a clue to deception? Insights from automated analysis of nonverbal visual cues.

Detecting deception in interpersonal dialog is challenging since deceivers take advantage of the give-and-take of interaction to adapt to any sign of skepticism in an interlocutor's verbal and nonverbal feedback. Human detection accuracy is poor, often with no better than chance performance. In this investigation, we consider whether automated methods can produce better results and if emphasizing the possible disruption in interactional synchrony can signal whether an interactant is truthful or deceptive. We propose a data-driven and unobtrusive framework using visual cues that consists of face tracking, head movement detection, facial expression recognition, and interactional synchrony estimation. Analysis were conducted on 242 video samples from an experiment in which deceivers and truth-tellers interacted with professional interviewers either face-to-face or through computer mediation. Results revealed that the framework is able to automatically track head movements and expressions of both interlocutors to extract normalized meaningful synchrony features and to learn classification models for deception recognition. Further experiments show that these features reliably capture interactional synchrony and efficiently discriminate deception from truth.

Atlas-based liver segmentation and hepatic fat-fraction assessment for clinical trials.

Automated assessment of hepatic fat-fraction is clinically important. A robust and precise segmentation would enable accurate, objective and consistent measurement of hepatic fat-fraction for disease quantification, therapy monitoring and drug development. However, segmenting the liver in clinical trials is a challenging task due to the variability of liver anatomy as well as the diverse sources the images were acquired from. In this paper, we propose an automated and robust framework for liver segmentation and assessment. It uses single statistical atlas registration to initialize a robust deformable model to obtain fine segmentation. Fat-fraction map is computed by using chemical shift based method in the delineated region of liver. This proposed method is validated on 14 abdominal magnetic resonance (MR) volumetric scans. The qualitative and quantitative comparisons show that our proposed method can achieve better segmentation accuracy with less variance comparing with two other atlas-based methods. Experimental results demonstrate the promises of our assessment framework.

Accurate Whole-Brain Segmentation for Alzheimer's Disease Combining an Adaptive Statistical Atlas and Multi-atlas.

Accurate segmentation of whole brain MR images including the cortex, white matter and subcortical structures is challenging due to inter-subject variability and the complex geometry of brain anatomy. However a precise solution would enable accurate, objective measurement of structure volumes for disease quantification. Our contribution is three-fold. First we construct an adaptive statistical atlas that combines structure specific relaxation and spatially varying adaptivity. Second we integrate an isotropic pairwise class-specific MRF model of label connectivity. Together these permit precise control over adaptivity, allowing many structures to be segmented simultaneously with superior accuracy. Third, we develop a framework combining the improved adaptive statistical atlas with a multi-atlas method which achieves simultaneous accurate segmentation of the cortex, ventricles, and sub-cortical structures in severely diseased brains, a feat not attained in [18]. We test the proposed method on 46 brains including 28 diseased brain with Alzheimer's and 18 healthy brains. Our proposed method yields higher accuracy than state-of-the-art approaches on both healthy and diseased brains.

Accurate segmentation of brain images into 34 structures combining a non-stationary adaptive statistical atlas and a multi-atlas with applications to Alzheimer's disease.

Accurate segmentation of the 30+ subcortical structures in MR images of whole diseased brains is challenging due to inter-subject variability and complex geometry of brain anatomy. However a clinically viable solution yielding precise segmentation of the structures would enable: 1) accurate, objective measurement of structure volumes many of which are associated with diseases such as Alzheimer's, 2) therapy monitoring and 3) drug development. Our contributions are two-fold. First we construct an extended adaptive statistical atlas method (EASA) to use a non-stationary relaxation factor rather than a global one. This permits finer control over adaptivity allowing 34 structures to be simultaneously segmented rather than just 4 as in [13]. Second we use the output of a weighted majority voting (WMV) label fusion multi-atlas method as the input to EASA in a hybrid WMV-EASA approach. We assess our proposed approaches on 18 healthy subjects in the public IBSR database and on 9 subjects with Alzheimer's disease in the AIBL database. EASA is shown to produce state-of-the-art accuracy on healthy brains in a fraction of the time of comparable methods, while our hybrid WMV-EASA visibly improves segmentation accuracy for structures throughout the diseased brains.

Simulation of organ deformation using boundary element method and meshless shape matching.

In this paper, a novel approach to perform real-time simulation of the deformation of anatomical organs is proposed for virtual surgery study. The method, which is both physics and interactivity motivated, is composed of two algorithms: boundary element method and meshless shape matching. We employ boundary element method to simulate a precise global deformation, and use meshless shape matching method to achieve low latency. In addition, a state machine is applied to control the computation patterns of deformation. The initial experiment reveals that the proposed approach can simulate organ deformations both efficiently and accurately.

Soft tissue deformation simulation in virtual surgery using nonlinear finite element method.

Simulation for soft tissue's realistic deformation is an important part in Virtual Surgery. For large global deformation of soft tissue, linear elastic models are inappropriate, such as Mass-Spring and linear Finite Element Method (FEM). In this paper we present a simulation for 3D soft tissue using nonlinear strain computation. To get a finer mesh for FEM, we consider meshing algorithm based on Improved Delaunay criterion. Besides, we would present Spatial Hashing Collision Detection method and some improvement for real-time computation.