Recent advancements in artificial intelligence for breast cancer: Image augmentation, segmentation, diagnosis, and prognosis approaches.

Breast cancer is a significant global health burden, with increasing morbidity and mortality worldwide. Early screening and accurate diagnosis are crucial for improving prognosis. Radiographic imaging modalities such as digital mammography (DM), digital breast tomosynthesis (DBT), magnetic resonance imaging (MRI), ultrasound (US), and nuclear medicine techniques, are commonly used for breast cancer assessment. And histopathology (HP) serves as the gold standard for confirming malignancy. Artificial intelligence (AI) technologies show great potential for quantitative representation of medical images to effectively assist in segmentation, diagnosis, and prognosis of breast cancer. In this review, we overview the recent advancements of AI technologies for breast cancer, including 1) improving image quality by data augmentation, 2) fast detection and segmentation of breast lesions and diagnosis of malignancy, 3) biological characterization of the cancer such as staging and subtyping by AI-based classification technologies, 4) prediction of clinical outcomes such as metastasis, treatment response, and survival by integrating multi-omics data. Then, we then summarize large-scale databases available to help train robust, generalizable, and reproducible deep learning models. Furthermore, we conclude the challenges faced by AI in real-world applications, including data curating, model interpretability, and practice regulations. Besides, we expect that clinical implementation of AI will provide important guidance for the patient-tailored management.

DTR-Net: Dual-Space 3D Tooth Model Reconstruction From Panoramic X-Ray Images.

In digital dentistry, cone-beam computed tomography (CBCT) can provide complete 3D tooth models, yet suffers from a long concern of requiring excessive radiation dose and higher expense. Therefore, 3D tooth model reconstruction from 2D panoramic X-ray image is more cost-effective, and has attracted great interest in clinical applications. In this paper, we propose a novel dual-space framework, namely DTR-Net, to reconstruct 3D tooth model from 2D panoramic X-ray images in both image and geometric spaces. Specifically, in the image space, we apply a 2D-to-3D generative model to recover intensities of CBCT image, guided by a task-oriented tooth segmentation network in a collaborative training manner. Meanwhile, in the geometric space, we benefit from an implicit function network in the continuous space, learning using points to capture complicated tooth shapes with geometric properties. Experimental results demonstrate that our proposed DTR-Net achieves state-of-the-art performance both quantitatively and qualitatively in 3D tooth model reconstruction, indicating its potential application in dental practice.

uRP: An integrated research platform for one-stop analysis of medical images.

Introduction: Medical image analysis is of tremendous importance in serving clinical diagnosis, treatment planning, as well as prognosis assessment. However, the image analysis process usually involves multiple modality-specific software and relies on rigorous manual operations, which is time-consuming and potentially low reproducible. Methods: We present an integrated platform - uAI Research Portal (uRP), to achieve one-stop analyses of multimodal images such as CT, MRI, and PET for clinical research applications. The proposed uRP adopts a modularized architecture to be multifunctional, extensible, and customizable. Results and Discussion: The uRP shows 3 advantages, as it 1) spans a wealth of algorithms for image processing including semi-automatic delineation, automatic segmentation, registration, classification, quantitative analysis, and image visualization, to realize a one-stop analytic pipeline, 2) integrates a variety of functional modules, which can be directly applied, combined, or customized for specific application domains, such as brain, pneumonia, and knee joint analyses, 3) enables full-stack analysis of one disease, including diagnosis, treatment planning, and prognosis assessment, as well as full-spectrum coverage for multiple disease applications. With the continuous development and inclusion of advanced algorithms, we expect this platform to largely simplify the clinical scientific research process and promote more and better discoveries.

Fast and low-dose medical imaging generation empowered by hybrid deep-learning and iterative reconstruction.

Fast and low-dose reconstructions of medical images are highly desired in clinical routines. We propose a hybrid deep-learning and iterative reconstruction (hybrid DL-IR) framework and apply it for fast magnetic resonance imaging (MRI), fast positron emission tomography (PET), and low-dose computed tomography (CT) image generation tasks. First, in a retrospective MRI study (6,066 cases), we demonstrate its capability of handling 3- to 10-fold under-sampled MR data, enabling organ-level coverage with only 10- to 100-s scan time; second, a low-dose CT study (142 cases) shows that our framework can successfully alleviate the noise and streak artifacts in scans performed with only 10% radiation dose (0.61 mGy); and last, a fast whole-body PET study (131 cases) allows us to faithfully reconstruct tumor-induced lesions, including small ones (<4 mm), from 2- to 4-fold-accelerated PET acquisition (30-60 s/bp). This study offers a promising avenue for accurate and high-quality image reconstruction with broad clinical value.

Deep learning empowered volume delineation of whole-body organs-at-risk for accelerated radiotherapy.

In radiotherapy for cancer patients, an indispensable process is to delineate organs-at-risk (OARs) and tumors. However, it is the most time-consuming step as manual delineation is always required from radiation oncologists. Herein, we propose a lightweight deep learning framework for radiotherapy treatment planning (RTP), named RTP-Net, to promote an automatic, rapid, and precise initialization of whole-body OARs and tumors. Briefly, the framework implements a cascade coarse-to-fine segmentation, with adaptive module for both small and large organs, and attention mechanisms for organs and boundaries. Our experiments show three merits: 1) Extensively evaluates on 67 delineation tasks on a large-scale dataset of 28,581 cases; 2) Demonstrates comparable or superior accuracy with an average Dice of 0.95; 3) Achieves near real-time delineation in most tasks with <2 s. This framework could be utilized to accelerate the contouring process in the All-in-One radiotherapy scheme, and thus greatly shorten the turnaround time of patients.

Learning Hierarchical Attention for Weakly-Supervised Chest X-Ray Abnormality Localization and Diagnosis.

We consider the problem of abnormality localization for clinical applications. While deep learning has driven much recent progress in medical imaging, many clinical challenges are not fully addressed, limiting its broader usage. While recent methods report high diagnostic accuracies, physicians have concerns trusting these algorithm results for diagnostic decision-making purposes because of a general lack of algorithm decision reasoning and interpretability. One potential way to address this problem is to further train these models to localize abnormalities in addition to just classifying them. However, doing this accurately will require a large amount of disease localization annotations by clinical experts, a task that is prohibitively expensive to accomplish for most applications. In this work, we take a step towards addressing these issues by means of a new attention-driven weakly supervised algorithm comprising a hierarchical attention mining framework that unifies activation- and gradient-based visual attention in a holistic manner. Our key algorithmic innovations include the design of explicit ordinal attention constraints, enabling principled model training in a weakly-supervised fashion, while also facilitating the generation of visual-attention-driven model explanations by means of localization cues. On two large-scale chest X-ray datasets (NIH ChestX-ray14 and CheXpert), we demonstrate significant localization performance improvements over the current state of the art while also achieving competitive classification performance.

Modeling essential connections in obsessive-compulsive disorder patients using functional MRI.

OBJECT: Obsessive-compulsive disorder (OCD) is a mental disease in which people experience uncontrollable and repetitive thoughts or behaviors. Clinical diagnosis of OCD is achieved by using neuropsychological assessment metrics, which are often subjectively affected by psychologists and patients. In this study, we propose a classification model for OCD diagnosis using functional MR images. METHODS: Using functional connectivity (FC) matrices calculated from brain region of interest (ROI) pairs, a novel Riemann Kernel principal component analysis (PCA) model is employed for feature extraction, which preserves the topological information in the FC matrices. Hierarchical features are then fed into an ensemble classifier based on the XGBoost algorithm. Finally, decisive features extracted during classification are used to investigate the brain FC variations between patients with OCD and healthy controls. RESULTS: The proposed algorithm yielded a classification accuracy of 91.8%. Additionally, the well-known cortico-striatal-thalamic-cortical (CSTC) circuit and cerebellum were found as highly related regions with OCD. To further analyze the cerebellar-related function in OCD, we demarcated cerebellum into three subregions according to their anatomical and functional property. Using these three functional cerebellum regions as seeds for brain connectivity computation, statistical results showed that patients with OCD have decreased posterior cerebellar connections. CONCLUSIONS: This study provides a new and efficient method to characterize patients with OCD using resting-state functional MRI. We also provide a new perspective to analyze disease-related features. Despite of CSTC circuit, our model-driven feature analysis reported cerebellum as an OCD-related region. This paper may provide novel insight to the understanding of genetic etiology of OCD.

Total variation models for variable lighting face recognition.

In this paper, we present the logarithmic total variation (LTV) model for face recognition under varying illumination, including natural lighting conditions, where we rarely know the strength, direction, or number of light sources. The proposed LTV model has the ability to factorize a single face image and obtain the illumination invariant facial structure, which is then used for face recognition. Our model is inspired by the SQI model but has better edge-preserving ability and simpler parameter selection. The merit of this model is that neither does it require any lighting assumption nor does it need any training. The LTV model reaches very high recognition rates in the tests using both Yale and CMU PIE face databases as well as a face database containing 765 subjects under outdoor lighting conditions.

Health-e-child: an integrated biomedical platform for grid-based paediatric applications.

There is a compelling demand for the integration and exploitation of heterogeneous biomedical information for improved clinical practice, medical research, and personalised healthcare across the EU. The Health-e-Child project aims at developing an integrated healthcare platform for European Paediatrics, providing seamless integration of traditional and emerging sources of biomedical information. The long-term goal of the project is to provide uninhibited access to universal biomedical knowledge repositories for personalised and preventive healthcare, large-scale information-based biomedical research and training, and informed policy making. The project focus will be on individualized disease prevention, screening, early diagnosis, therapy and follow-up of paediatric heart diseases, inflammatory diseases, and brain tumours. The project will build a Grid-enabled European network of leading clinical centres that will share and annotate biomedical data, validate systems clinically, and diffuse clinical excellence across Europe by setting up new technologies, clinical workflows, and standards. This paper outlines the design approach being adopted in Health-e-Child to enable the delivery of an integrated biomedical information platform.

Automatic Lumbar Spondylolisthesis Measurement in CT Images.

Lumbar spondylolisthesis is one of the most common spinal diseases. It is caused by the anterior shift of a lumbar vertebrae relative to subjacent vertebrae. In current clinical practices, staging of spondylolisthesis is often conducted in a qualitative way. Although meyerding grading opens the door to stage spondylolisthesis in a more quantitative way, it relies on the manual measurement, which is time consuming and irreproducible. Thus, an automatic measurement algorithm becomes desirable for spondylolisthesis diagnosis and staging. However, there are two challenges. 1) Accurate detection of the most anterior and posterior points on the superior and inferior surfaces of each lumbar vertebrae. Due to the small size of the vertebrae, slight errors of detection may lead to significant measurement errors, hence, wrong disease stages. 2) Automatic localize and label each lumbar vertebrae is required to provide the semantic meaning of the measurement. It is difficult since different lumbar vertebraes have high similarity of both shape and image appearance. To resolve these challenges, a new auto measurement framework is proposed with two major contributions: First, a learning based spine labeling method that integrates both the image appearance and spine geometry information is designed to detect lumbar vertebrae. Second, a hierarchical method using both the population information from atlases and domain-specific information in the target image is proposed for most anterior and posterior points positioning. Validated on 258 CT spondylolisthesis patients, our method shows very similar results to manual measurements by radiologists and significantly increases the measurement efficiency.

An information fusion framework for robust shape tracking.

Abstract-Existing methods for incorporating subspace model constraints in shape tracking use only partial information from the measurements and model distribution. We propose a unified framework for robust shape tracking, optimally fusing heteroscedastic uncertainties or noise from measurement, system dynamics, and a subspace model. The resulting nonorthogonal subspace projection and fusion are natural extensions of the traditional model constraint using orthogonal projection. We present two motion measurement algorithms and introduce alternative solutions for measurement uncertainty estimation. We build shape models offline from training data and exploit information from the ground truth initialization online through a strong model adaptation. Our framework is applied for tracking in echocardiograms where the motion estimation errors are heteroscedastic in nature, each heart has a distinct shape, and the relative motions of epicardial and endocardial borders reveal crucial diagnostic features. The proposed method significantly outperforms the existing shape-space-constrained tracking algorithm. Due to the complete treatment of heteroscedastic uncertainties, the strong model adaptation, and the coupled tracking of double-contours, robust performance is observed even on the most challenging cases.

Bodypart Recognition Using Multi-stage Deep Learning.

Automatic medical image analysis systems often start from identifying the human body part contained in the image; Specifically, given a transversal slice, it is important to know which body part it comes from, namely "slice-based bodypart recognition". This problem has its unique characteristic--the body part of a slice is usually identified by local discriminative regions instead of global image context, e.g., a cardiac slice is differentiated from an aorta arch slice by the mediastinum region. To leverage this characteristic, we design a multi-stage deep learning framework that aims at: (1) discover the local regions that are discriminative to the bodypart recognition, and (2) learn a bodypart identifier based on these local regions. These two tasks are achieved by the two stages of our learning scheme, respectively. In the pre-train stage, a convolutional neural network (CNN) is learned in a multi-instance learning fashion to extract the most discriminative local patches from the training slices. In the boosting stage, the learned CNN is further boosted by these local patches for bodypart recognition. By exploiting the discriminative local appearances, the learned CNN becomes more accurate than global image context-based approaches. As a key hallmark, our method does not require manual annotations of the discriminative local patches. Instead, it automatically discovers them through multi-instance deep learning. We validate our method on a synthetic dataset and a large scale CT dataset (7000+ slices from wholebody CT scans). Our method achieves better performances than state-of-the-art approaches, including the standard CNN.

Robust real-time myocardial border tracking for echocardiography: an information fusion approach.

Ultrasound is a main noninvasive modality for the assessment of the heart function. Wall tracking from ultrasound data is, however, inherently difficult due to weak echoes, clutter, poor signal-to-noise ratio, and signal dropouts. To cope with these artifacts, pretrained shape models can be applied to constrain the tracking. However, existing methods for incorporating subspace shape constraints in myocardial border tracking use only partial information from the model distribution, and do not exploit spatially varying uncertainties from feature tracking. In this paper, we propose a complete fusion formulation in the information space for robust shape tracking, optimally resolving uncertainties from the system dynamics, heteroscedastic measurement noise, and subspace shape model. We also exploit information from the ground truth initialization where this is available. The new framework is applied for tracking of myocardial borders in very noisy echocardiography sequences. Numerous myocardium tracking experiments validate the theory and show the potential of very accurate wall motion measurements. The proposed framework outperforms the traditional shape-space-constrained tracking algorithm by a significant margin. Due to the optimal fusion of different sources of uncertainties, robust performance is observed even for the most challenging cases.

Robust MR spine detection using hierarchical learning and local articulated model.

A clinically acceptable auto-spine detection system, i.e., localization and labeling of vertebrae and inter-vertebral discs, is required to have high robustness, in particular to severe diseases (e.g., scoliosis) and imaging artifacts (e.g. metal artifacts in MR). Our method aims to achieve this goal with two novel components. First, instead of treating vertebrae/discs as either repetitive components or completely independent entities, we emulate a radiologist and use a hierarchial strategy to learn detectors dedicated to anchor (distinctive) vertebrae, bundle (non-distinctive) vertebrae and inter-vertebral discs, respectively. At run-time, anchor vertebrae are detected concurrently to provide redundant and distributed appearance cues robust to local imaging artifacts. Bundle vertebrae detectors provide candidates of vertebrae with subtle appearance differences, whose labels are mutually determined by anchor vertebrae to gain additional robustness. Disc locations are derived from a cloud of responses from disc detectors, which is robust to sporadic voxel-level errors. Second, owing to the non-rigidness of spine anatomies, we employ a local articulated model to effectively model the spatial relations across vertebrae and discs. The local articulated model fuses appearance cues from different detectors in a way that is robust to abnormal spine geometry resulting from severe diseases. Our method is validated by 300 MR spine scout scans and exhibits robust performance, especially to cases with severe diseases and imaging artifacts.

Towards robust and effective shape modeling: sparse shape composition.

Organ shape plays an important role in various clinical practices, e.g., diagnosis, surgical planning and treatment evaluation. It is usually derived from low level appearance cues in medical images. However, due to diseases and imaging artifacts, low level appearance cues might be weak or misleading. In this situation, shape priors become critical to infer and refine the shape derived by image appearances. Effective modeling of shape priors is challenging because: (1) shape variation is complex and cannot always be modeled by a parametric probability distribution; (2) a shape instance derived from image appearance cues (input shape) may have gross errors; and (3) local details of the input shape are difficult to preserve if they are not statistically significant in the training data. In this paper we propose a novel Sparse Shape Composition model (SSC) to deal with these three challenges in a unified framework. In our method, a sparse set of shapes in the shape repository is selected and composed together to infer/refine an input shape. The a priori information is thus implicitly incorporated on-the-fly. Our model leverages two sparsity observations of the input shape instance: (1) the input shape can be approximately represented by a sparse linear combination of shapes in the shape repository; (2) parts of the input shape may contain gross errors but such errors are sparse. Our model is formulated as a sparse learning problem. Using L1 norm relaxation, it can be solved by an efficient expectation-maximization (EM) type of framework. Our method is extensively validated on two medical applications, 2D lung localization in X-ray images and 3D liver segmentation in low-dose CT scans. Compared to state-of-the-art methods, our model exhibits better performance in both studies.

Robust learning-based parsing and annotation of medical radiographs.

In this paper, we propose a learning-based algorithm for automatic medical image annotation based on robust aggregation of learned local appearance cues, achieving high accuracy and robustness against severe diseases, imaging artifacts, occlusion, or missing data. The algorithm starts with a number of landmark detectors to collect local appearance cues throughout the image, which are subsequently verified by a group of learned sparse spatial configuration models. In most cases, a decision could already be made at this stage by simply aggregating the verified detections. For the remaining cases, an additional global appearance filtering step is employed to provide complementary information to make the final decision. This approach is evaluated on a large-scale chest radiograph view identification task, demonstrating a very high accuracy ( > 99.9%) for a posteroanterior/anteroposterior (PA-AP) and lateral view position identification task, compared with the recently reported large-scale result of only 98.2% (Luo, , 2006). Our approach also achieved the best accuracies for a three-class and a multiclass radiograph annotation task, when compared with other state of the art algorithms. Our algorithm was used to enhance advanced image visualization workflows by enabling content-sensitive hanging-protocols and auto-invocation of a computer aided detection algorithm for identified PA-AP chest images. Finally, we show that the same methodology could be utilized for several image parsing applications including anatomy/organ region of interest prediction and optimized image visualization.

Auto-alignment of knee MR scout scans through redundant, adaptive and hierarchical anatomy detection.

3D knee magnetic resonance (MR) scout scan is an emerging imaging sequence that facilitates technicians in aligning the imaging planes of diagnostic high resolution MR scans. In this paper, we propose a method to automate this process with the goal of improving the accuracy, robustness and speed of the workflow. To tackle the various challenges coming from MR knee scout scans, our auto-alignment method is built upon a redundant, adaptive and hierarchical anatomy detection system. More specifically, we learn 1) a hierarchical redudant set of anatomy detectors, and 2) ensemble of group-wise spatial configurations across different anatomies, from training data. These learned statistics are integrated into a comprehensive objective function optimized using an expectation-maximization (EM) framework. The optimization provides a new framework for hierarchical detection and adaptive selection of anatomy primitives to derive optimal alignment. Being extensively validated on 744 clinical datasets, our method achieves high accuracy (sub-voxel alignment error), robustness (to severe diseases or imaging artifacts) and fast speed ( 5 sees for 10 alignments).

Robust automatic knee MR slice positioning through redundant and hierarchical anatomy detection.

Diagnostic magnetic resonance (MR) image quality is highly dependent on the position and orientation of the slice groups, due to the intrinsic high in-slice and low through-slice resolutions of MR imaging. Hence, the higher speed, accuracy, and reproducibility of automatic slice positioning, make it highly desirable over manual slice positioning. However, imaging artifacts, diseases, joint articulation, variations across ages and demographics as well as the extremely high performance requirements prevent state-of-the-art methods, such as volumetric registration, to be an off-the-shelf solution. In this paper, we address all these issues through an automatic slice positioning framework based on redundant and hierarchical learning. Our method has two hallmarks that are specifically designed to achieve high robustness and accuracy. 1) A redundant set of anatomy detectors are learned to provide local appearance cues. These detections are pruned and assembled according to a distributed anatomy model, which captures group-wise spatial configurations among anatomy primitives. This strategy brings about a high level of robustness and works even if a large portion of the target is distorted, missing, or occluded. 2) The detectors are learned and invoked in a hierarchical fashion, with each local detection scheduled and iterated according to its intrinsic invariance property. This iterative alignment process is shown to dramatically improve alignment accuracy. The proposed system is extensively validated on a large dataset including 744 clinical MR scans. Compared to state-of-the-art methods, our method exhibits superior performance in terms of robustness, accuracy, and reproducibility. The methodology is general and can be applied to other anatomies and other imaging modalities.

Deformable segmentation via sparse shape representation.

Appearance and shape are two key elements exploited in medical image segmentation. However, in some medical image analysis tasks, appearance cues are weak/misleading due to disease/artifacts and often lead to erroneous segmentation. In this paper, a novel deformable model is proposed for robust segmentation in the presence of weak/misleading appearance cues. Owing to the less trustable appearance information, this method focuses on the effective shape modeling with two contributions. First, a shape composition method is designed to incorporate shape prior on-the-fly. Based on two sparsity observations, this method is robust to false appearance information and adaptive to statistically insignificant shape modes. Second, shape priors are modeled and used in a hierarchical fashion. More specifically, by using affinity propagation method, our deformable surface is divided into multiple partitions, on which local shape models are built independently. This scheme facilitates a more compact shape prior modeling and hence a more robust and efficient segmentation. Our deformable model is applied on two very diverse segmentation problems, liver segmentation in PET-CT images and rodent brain segmentation in MR images. Compared to state-of-art methods, our method achieves better performance in both studies.

Cross modality deformable segmentation using hierarchical clustering and learning.

Segmentation of anatomical objects is always a fundamental task for various clinical applications. Although many automatic segmentation methods have been designed to segment specific anatomical objects in a given imaging modality, a more generic solution that is directly applicable to different imaging modalities and different deformable surfaces is desired, if attainable. In this paper, we propose such a framework, which learns from examples the spatially adaptive appearance and shape of a 3D surface (either open or closed). The application to a new object/surface in a new modality requires only the annotation of training examples. Key contributions of our method include: (1) an automatic clustering and learning algorithm to capture the spatial distribution of appearance similarities/variations on the 3D surface. More specifically, the model vertices are hierarchically clustered into a set of anatomical primitives (sub-surfaces) using both geometric and appearance features. The appearance characteristics of each learned anatomical primitive are then captured through a cascaded boosting learning method. (2) To effectively incorporate non-Gaussian shape priors, we cluster the training shapes in order to build multiple statistical shape models. (3) To our best knowledge, this is the first time the same segmentation algorithm has been directly employed in two very diverse applications: (a) Liver segmentation (closed surface) in PET-CT, in which CT has very low-resolution and low-contrast; (b) Distal femur (condyle) surface (open surface) segmentation in MRI.