Lumbar Bone Mineral Density Estimation From Chest X-Ray Images: Anatomy-Aware Attentive Multi-ROI Modeling.

Osteoporosis is a common chronic metabolic bone disease often under-diagnosed and under-treated due to the limited access to bone mineral density (BMD) examinations, e.g., via Dual-energy X-ray Absorptiometry (DXA). This paper proposes a method to predict BMD from Chest X-ray (CXR), one of the most commonly accessible and low-cost medical imaging examinations. The proposed method first automatically detects Regions of Interest (ROIs) of local CXR bone structures. Then a multi-ROI deep model with transformer encoder is developed to exploit both local and global information in the chest X-ray image for accurate BMD estimation. The proposed method is evaluated on 13719 CXR patient cases with ground truth BMD measured by the gold standard DXA. The model predicted BMD has a strong correlation with the ground truth (Pearson correlation coefficient 0.894 on lumbar 1). When applied in osteoporosis screening, it achieves a high classification performance (average AUC of 0.968). As the first effort of using CXR scans to predict the BMD, the proposed algorithm holds strong potential to promote early osteoporosis screening and public health.

RNA-Sequencing Analysis and Expression of Lymphocyte-Specific Protein Tyrosine Kinase, Linker for Activation of T Cells, and 70-kDa T-Cell Receptor Zeta-Chain Associated Protein Kinase in Rat Liver Transplant Rejection.

OBJECTIVES: The prevention and treatment of liver transplant rejection remain challenging. We investigated the pathophysiological mechanisms of liver transplant rejection in rats and screened candidate genes to determine their degree of rejection response for possible development of potential therapeutic targets. MATERIALS AND METHODS: Brown Norway-Brown Norway transplant tolerant models and Lewis-Brown Norway transplant rejection models were established. We collected liver tissue and venous blood at 7 days posttransplant for hematoxylin and eosin staining and RNA sequencing analysis, respectively. We conducted differential expression gene analysis, KEGG and GO enrichment analysis. We performed immunohistochemistry to detect highly expressed immunerelated proteins, including lymphocyte-specific protein tyrosine kinase, linker for activation of T cells, and 70-kDa T-cell receptor zeta-chain-associated protein kinase. RESULTS: Significant differences were found in liver function and Banff scores between rejection and tolerant groups, indicating the successful establishment of liver transplant models. RNA-sequencing screened 7521 differentially expressed genes, with 3355 upregulated and 3058 downregulated. KEGG analysis of upregulated genes showed that 8 of the top 20 enrichment pathways were associated with immune system processes and 5 were related to immune system diseases. Among these immune pathways, 289 genes were upregulated; of these, 147 genes were removed after comparison with the IMMPORT database, of which 97 genes were significantly changed. Our GO analysis showed upregulated genes mainly participating in immune response processes, with downregulated genes mainly participating in metabolic processes. Real-time polymerase chain reaction and immunohistochemistry verified expression of the immune-related proteins, consistent with RNAsequencing results, which were mainly expressed in inflammatory cells in sinus and portal vein. CONCLUSIONS: Immune-related genes were found to be associated with liver transplant rejection. The 3 immune-related genes that we analyzed may play a role in liver transplant rejection and can possibly serve as candidate markers for monitoring the degree of liver transplant rejection.

SAM: Self-Supervised Learning of Pixel-Wise Anatomical Embeddings in Radiological Images.

Radiological images such as computed tomography (CT) and X-rays render anatomy with intrinsic structures. Being able to reliably locate the same anatomical structure across varying images is a fundamental task in medical image analysis. In principle it is possible to use landmark detection or semantic segmentation for this task, but to work well these require large numbers of labeled data for each anatomical structure and sub-structure of interest. A more universal approach would learn the intrinsic structure from unlabeled images. We introduce such an approach, called Self-supervised Anatomical eMbedding (SAM). SAM generates semantic embeddings for each image pixel that describes its anatomical location or body part. To produce such embeddings, we propose a pixel-level contrastive learning framework. A coarse-to-fine strategy ensures both global and local anatomical information are encoded. Negative sample selection strategies are designed to enhance the embedding's discriminability. Using SAM, one can label any point of interest on a template image and then locate the same body part in other images by simple nearest neighbor searching. We demonstrate the effectiveness of SAM in multiple tasks with 2D and 3D image modalities. On a chest CT dataset with 19 landmarks, SAM outperforms widely-used registration algorithms while only taking 0.23 seconds for inference. On two X-ray datasets, SAM, with only one labeled template image, surpasses supervised methods trained on 50 labeled images. We also apply SAM on whole-body follow-up lesion matching in CT and obtain an accuracy of 91%. SAM can also be applied for improving image registration and initializing CNN weights.

Automated bone mineral density prediction and fracture risk assessment using plain radiographs via deep learning.

Dual-energy X-ray absorptiometry (DXA) is underutilized to measure bone mineral density (BMD) and evaluate fracture risk. We present an automated tool to identify fractures, predict BMD, and evaluate fracture risk using plain radiographs. The tool performance is evaluated on 5164 and 18175 patients with pelvis/lumbar spine radiographs and Hologic DXA. The model is well calibrated with minimal bias in the hip (slope = 0.982, calibration-in-the-large = -0.003) and the lumbar spine BMD (slope = 0.978, calibration-in-the-large = 0.003). The area under the precision-recall curve and accuracy are 0.89 and 91.7% for hip osteoporosis, 0.89 and 86.2% for spine osteoporosis, 0.83 and 95.0% for high 10-year major fracture risk, and 0.96 and 90.0% for high hip fracture risk. The tool classifies 5206 (84.8%) patients with 95% positive or negative predictive value for osteoporosis, compared to 3008 DXA conducted at the same study period. This automated tool may help identify high-risk patients for osteoporosis.

A scalable physician-level deep learning algorithm detects universal trauma on pelvic radiographs.

Pelvic radiograph (PXR) is essential for detecting proximal femur and pelvis injuries in trauma patients, which is also the key component for trauma survey. None of the currently available algorithms can accurately detect all kinds of trauma-related radiographic findings on PXRs. Here, we show a universal algorithm can detect most types of trauma-related radiographic findings on PXRs. We develop a multiscale deep learning algorithm called PelviXNet trained with 5204 PXRs with weakly supervised point annotation. PelviXNet yields an area under the receiver operating characteristic curve (AUROC) of 0.973 (95% CI, 0.960-0.983) and an area under the precision-recall curve (AUPRC) of 0.963 (95% CI, 0.948-0.974) in the clinical population test set of 1888 PXRs. The accuracy, sensitivity, and specificity at the cutoff value are 0.924 (95% CI, 0.912-0.936), 0.908 (95% CI, 0.885-0.908), and 0.932 (95% CI, 0.919-0.946), respectively. PelviXNet demonstrates comparable performance with radiologists and orthopedics in detecting pelvic and hip fractures.

Contour Transformer Network for One-Shot Segmentation of Anatomical Structures.

Accurate segmentation of anatomical structures is vital for medical image analysis. The state-of-the-art accuracy is typically achieved by supervised learning methods, where gathering the requisite expert-labeled image annotations in a scalable manner remains a main obstacle. Therefore, annotation-efficient methods that permit to produce accurate anatomical structure segmentation are highly desirable. In this work, we present Contour Transformer Network (CTN), a one-shot anatomy segmentation method with a naturally built-in human-in-the-loop mechanism. We formulate anatomy segmentation as a contour evolution process and model the evolution behavior by graph convolutional networks (GCNs). Training the CTN model requires only one labeled image exemplar and leverages additional unlabeled data through newly introduced loss functions that measure the global shape and appearance consistency of contours. On segmentation tasks of four different anatomies, we demonstrate that our one-shot learning method significantly outperforms non-learning-based methods and performs competitively to the state-of-the-art fully supervised deep learning methods. With minimal human-in-the-loop editing feedback, the segmentation performance can be further improved to surpass the fully supervised methods.

Deep hiearchical multi-label classification applied to chest X-ray abnormality taxonomies.

Chest X-rays (CXRs) are a crucial and extraordinarily common diagnostic tool, leading to heavy research for computer-aided diagnosis (CAD) solutions. However, both high classification accuracy and meaningful model predictions that respect and incorporate clinical taxonomies are crucial for CAD usability. To this end, we present a deep hierarchical multi-label classification (HMLC) approach for CXR CAD. Different than other hierarchical systems, we show that first training the network to model conditional probability directly and then refining it with unconditional probabilities is key in boosting performance. In addition, we also formulate a numerically stable cross-entropy loss function for unconditional probabilities that provides concrete performance improvements. Finally, we demonstrate that HMLC can be an effective means to manage missing or incomplete labels. To the best of our knowledge, we are the first to apply HMLC to medical imaging CAD. We extensively evaluate our approach on detecting abnormality labels from the CXR arm of the Prostate, Lung, Colorectal and Ovarian (PLCO) dataset, which comprises over 198,000 manually annotated CXRs. When using complete labels, we report a mean area under the curve (AUC) of 0.887, the highest yet reported for this dataset. These results are supported by ancillary experiments on the PadChest dataset, where we also report significant improvements, 1.2% and 4.1% in AUC and average precision, respectively over strong "flat" classifiers. Finally, we demonstrate that our HMLC approach can much better handle incompletely labelled data. These performance improvements, combined with the inherent usefulness of taxonomic predictions, indicate that our approach represents a useful step forward for CXR CAD.

Unsupervised X-ray image segmentation with task driven generative adversarial networks.

Semantic parsing of anatomical structures in X-ray images is a critical task in many clinical applications. Modern methods leverage deep convolutional networks, and generally require a large amount of labeled data for model training. However, obtaining accurate pixel-wise labels on X-ray images is very challenging due to the appearance of anatomy overlaps and complex texture patterns. In comparison, labeled CT data are more accessible since organs in 3D CT scans preserve clearer structures and thus can be easily delineated. In this paper, we propose a model framework for learning automatic X-ray image parsing from labeled 3D CT scans. Specifically, a Deep Image-to-Image network (DI2I) for multi-organ segmentation is first trained on X-ray like Digitally Reconstructed Radiographs (DRRs) rendered from 3D CT volumes. Then we build a Task Driven Generative Adversarial Network (TD-GAN) to achieve simultaneous synthesis and parsing for unseen real X-ray images. The entire model pipeline does not require any annotations from the X-ray image domain. In the numerical experiments, we validate the proposed model on over 800 DRRs and 300 topograms. While the vanilla DI2I trained on DRRs without any adaptation fails completely on segmenting the topograms, the proposed model does not require any topogram labels and is able to provide a promising average dice of 86% which achieves the same level of accuracy as results from supervised training (89%). Furthermore, we also demonstrate the generality of TD-GAN through quantatitive and qualitative study on widely used public dataset.

3D/2D model-to-image registration by imitation learning for cardiac procedures.

PURPOSE: In cardiac interventions, such as cardiac resynchronization therapy (CRT), image guidance can be enhanced by involving preoperative models. Multimodality 3D/2D registration for image guidance, however, remains a significant research challenge for fundamentally different image data, i.e., MR to X-ray. Registration methods must account for differences in intensity, contrast levels, resolution, dimensionality, field of view. Furthermore, same anatomical structures may not be visible in both modalities. Current approaches have focused on developing modality-specific solutions for individual clinical use cases, by introducing constraints, or identifying cross-modality information manually. Machine learning approaches have the potential to create more general registration platforms. However, training image to image methods would require large multimodal datasets and ground truth for each target application. METHODS: This paper proposes a model-to-image registration approach instead, because it is common in image-guided interventions to create anatomical models for diagnosis, planning or guidance prior to procedures. An imitation learning-based method, trained on 702 datasets, is used to register preoperative models to intraoperative X-ray images. RESULTS: Accuracy is demonstrated on cardiac models and artificial X-rays generated from CTs. The registration error was [Formula: see text] on 1000 test cases, superior to that of manual ([Formula: see text]) and gradient-based ([Formula: see text]) registration. High robustness is shown in 19 clinical CRT cases. CONCLUSION: Besides the proposed methods feasibility in a clinical environment, evaluation has shown good accuracy and high robustness indicating that it could be applied in image-guided interventions.

Pairwise domain adaptation module for CNN-based 2-D/3-D registration.

Accurate two-dimensional to three-dimensional (2-D/3-D) registration of preoperative 3-D data and intraoperative 2-D x-ray images is a key enabler for image-guided therapy. Recent advances in 2-D/3-D registration formulate the problem as a learning-based approach and exploit the modeling power of convolutional neural networks (CNN) to significantly improve the accuracy and efficiency of 2-D/3-D registration. However, for surgery-related applications, collecting a large clinical dataset with accurate annotations for training can be very challenging or impractical. Therefore, deep learning-based 2-D/3-D registration methods are often trained with synthetically generated data, and a performance gap is often observed when testing the trained model on clinical data. We propose a pairwise domain adaptation (PDA) module to adapt the model trained on source domain (i.e., synthetic data) to target domain (i.e., clinical data) by learning domain invariant features with only a few paired real and synthetic data. The PDA module is designed to be flexible for different deep learning-based 2-D/3-D registration frameworks, and it can be plugged into any pretrained CNN model such as a simple Batch-Norm layer. The proposed PDA module has been quantitatively evaluated on two clinical applications using different frameworks of deep networks, demonstrating its significant advantages of generalizability and flexibility for 2-D/3-D medical image registration when a small number of paired real-synthetic data can be obtained.

Automatic and efficient contrast-based 2-D/3-D fusion for trans-catheter aortic valve implantation (TAVI).

Trans-catheter aortic valve implantation (TAVI) is a new breakthrough in the field of minimally invasive surgery applied on high-risk patients with aortic valve defects. 2-D X-ray angiographic and fluoroscopic images are typically used to guide TAVI procedures, for which contrast agent needs to be injected from time to time in order to make the anatomy of the aortic root visible under X-ray. Advanced visualization and guidance technology involving patient-specific 3-D models of the aorta can greatly facilitate the relatively complex TAVI procedures by providing a more realistic anatomy of the aortic root and more accurate C-Arm angulation. In this paper, a fully automatic and efficient system for contrast-based 2-D/3-D fusion for TAVI is presented. Contrast agent injection into the aortic root is automatically detected based on histogram analysis and a likelihood ratio test on the X-ray images. A hybrid method is then applied for contrast-based 2-D/3-D registration between the 3-D model and the detected angiographic frame. By integrating the information of aorta segmentation and aortic landmark detection into intensity-based registration, the proposed method combines the merits of intensity-based registration and feature/landmark-based registration. Experiments on 34 clinical data sets from TAVI patients achieve 100% correct detection on the contrast-enhanced frame, and a mean registration error of 0.66±0.47mm for 2-D/3-D registration. The proposed method is furthermore highly efficient with an average processing time of 2.5s after the most contrast-enhanced frame is available, demonstrating the efficacy of the proposed method to be adopted in a clinical setup.

System and method for 3-D/3-D registration between non-contrast-enhanced CBCT and contrast-enhanced CT for abdominal aortic aneurysm stenting.

In this paper, we present an image guidance system for abdominal aortic aneurysm stenting, which brings pre-operative 3-D computed tomography (CT) into the operating room by registering it against intra-operative non-contrast-enhanced cone-beam CT (CBCT). Registration between CT and CBCT volumes is a challenging task due to two factors: the relatively low signal-to-noise ratio of the abdominal aorta in CBCT without contrast enhancement, and the drastically different field of view between the two image modalities. The proposed automatic registration method handles the first issue through a fast quasi-global search utilizing surrogate 2-D images, and solves the second problem by relying on neighboring dominant structures of the abdominal aorta (i.e. the spine) for initial coarse alignment, and using a confined and image-processed volume of interest around the abdominal aorta for fine registration. The proposed method is validated offline using 17 clinical datasets, and achieves 1.48 mm target registration error and 100% success rate in 2.83 s. The prototype system has been installed in hospitals for clinical trial and applied in around 30 clinical cases, with 100% success rate reported qualitatively.