Semi-supervised training using cooperative labeling of weakly annotated data for nodule detection in chest CT.

PURPOSE: Machine learning algorithms are best trained with large quantities of accurately annotated samples. While natural scene images can often be labeled relatively cheaply and at large scale, obtaining accurate annotations for medical images is both time consuming and expensive. In this study, we propose a cooperative labeling method that allows us to make use of weakly annotated medical imaging data for the training of a machine learning algorithm. As most clinically produced data are weakly-annotated - produced for use by humans rather than machines and lacking information machine learning depends upon - this approach allows us to incorporate a wider range of clinical data and thereby increase the training set size. METHODS: Our pseudo-labeling method consists of multiple stages. In the first stage, a previously established network is trained using a limited number of samples with high-quality expert-produced annotations. This network is used to generate annotations for a separate larger dataset that contains only weakly annotated scans. In the second stage, by cross-checking the two types of annotations against each other, we obtain higher-fidelity annotations. In the third stage, we extract training data from the weakly annotated scans, and combine it with the fully annotated data, producing a larger training dataset. We use this larger dataset to develop a computer-aided detection (CADe) system for nodule detection in chest CT. RESULTS: We evaluated the proposed approach by presenting the network with different numbers of expert-annotated scans in training and then testing the CADe using an independent expert-annotated dataset. We demonstrate that when availability of expert annotations is severely limited, the inclusion of weakly-labeled data leads to a 5% improvement in the competitive performance metric (CPM), defined as the average of sensitivities at different false-positive rates. CONCLUSIONS: Our proposed approach can effectively merge a weakly-annotated dataset with a small, well-annotated dataset for algorithm training. This approach can help enlarge limited training data by leveraging the large amount of weakly labeled data typically generated in clinical image interpretation.

Automatic lung nodule detection in thoracic CT scans using dilated slice-wise convolutions.

PURPOSE: Most state-of-the-art automated medical image analysis methods for volumetric data rely on adaptations of two-dimensional (2D) and three-dimensional (3D) convolutional neural networks (CNNs). In this paper, we develop a novel unified CNN-based model that combines the benefits of 2D and 3D networks for analyzing volumetric medical images. METHODS: In our proposed framework, multiscale contextual information is first extracted from 2D slices inside a volume of interest (VOI). This is followed by dilated 1D convolutions across slices to aggregate in-plane features in a slice-wise manner and encode the information in the entire volume. Moreover, we formalize a curriculum learning strategy for a two-stage system (i.e., a system that consists of screening and false positive reduction), where the training samples are presented to the network in a meaningful order to further improve the performance. RESULTS: We evaluated the proposed approach by developing a computer-aided detection (CADe) system for lung nodules. Our results on 888 CT exams demonstrate that the proposed approach can effectively analyze volumetric data by achieving a sensitivity of > 0.99 in the screening stage and a sensitivity of > 0.96 at eight false positives per case in the false positive reduction stage. CONCLUSION: Our experimental results show that the proposed method provides competitive results compared to state-of-the-art 3D frameworks. In addition, we illustrate the benefits of curriculum learning strategies in two-stage systems that are of common use in medical imaging applications.

Recurrent attention network for false positive reduction in the detection of pulmonary nodules in thoracic CT scans.

PURPOSE: Multiview two-dimensional (2D) convolutional neural networks (CNNs) and three-dimensional (3D) CNNs have been successfully used for analyzing volumetric data in many state-of-the-art medical imaging applications. We propose an alternative modular framework that analyzes volumetric data with an approach that is analogous to radiologists' interpretation, and apply the framework to reduce false positives that are generated in computer-aided detection (CADe) systems for pulmonary nodules in thoracic computed tomography (CT) scans. METHODS: In our approach, a deep network consisting of 2D CNNs first processes slices individually. The features extracted in this stage are then passed to a recurrent neural network (RNN), thereby modeling consecutive slices as a sequence of temporal data and capturing the contextual information across all three dimensions in the volume of interest. Outputs of the RNN layer are weighed before the final fully connected layer, enabling the network to scale the importance of different slices within a volume of interest in an end-to-end training framework. RESULTS: We validated the proposed architecture on the false positive reduction track of the lung nodule analysis (LUNA) challenge for pulmonary nodule detection in chest CT scans, and obtained competitive results compared to 3D CNNs. Our results show that the proposed approach can encode the 3D information in volumetric data effectively by achieving a sensitivity >0.8 with just 1/8 false positives per scan. CONCLUSIONS: Our experimental results demonstrate the effectiveness of temporal analysis of volumetric images for the application of false positive reduction in chest CT scans and show that state-of-the-art 2D architectures from the literature can be directly applied to analyzing volumetric medical data. As newer and better 2D architectures are being developed at a much faster rate compared to 3D architectures, our approach makes it easy to obtain state-of-the-art performance on volumetric data using new 2D architectures.

Evaluation of data augmentation via synthetic images for improved breast mass detection on mammograms using deep learning.

We evaluated whether using synthetic mammograms for training data augmentation may reduce the effects of overfitting and increase the performance of a deep learning algorithm for breast mass detection. Synthetic mammograms were generated using in silico procedural analytic breast and breast mass modeling algorithms followed by simulated x-ray projections of the breast models into mammographic images. In silico breast phantoms containing masses were modeled across the four BI-RADS breast density categories, and the masses were modeled with different sizes, shapes, and margins. A Monte Carlo-based x-ray transport simulation code, MC-GPU, was used to project the three-dimensional phantoms into realistic synthetic mammograms. 2000 mammograms with 2522 masses were generated to augment a real data set during training. From the Curated Breast Imaging Subset of the Digital Database for Screening Mammography (CBIS-DDSM) data set, we used 1111 mammograms (1198 masses) for training, 120 mammograms (120 masses) for validation, and 361 mammograms (378 masses) for testing. We used faster R-CNN for our deep learning network with pretraining from ImageNet using the Resnet-101 architecture. We compared the detection performance when the network was trained using different percentages of the real CBIS-DDSM training set (100%, 50%, and 25%), and when these subsets of the training set were augmented with 250, 500, 1000, and 2000 synthetic mammograms. Free-response receiver operating characteristic (FROC) analysis was performed to compare performance with and without the synthetic mammograms. We generally observed an improved test FROC curve when training with the synthetic images compared to training without them, and the amount of improvement depended on the number of real and synthetic images used in training. Our study shows that enlarging the training data with synthetic samples can increase the performance of deep learning systems.

3-D Convolutional Neural Networks for Automatic Detection of Pulmonary Nodules in Chest CT.

Deep two-dimensional (2-D) convolutional neural networks (CNNs) have been remarkably successful in producing record-breaking results in a variety of computer vision tasks. It is possible to extend CNNs to three dimensions using 3-D kernels to make them suitable for volumetric medical imaging data such as CT or MRI, but this increases the processing time as well as the required number of training samples (due to the higher number of parameters that need to be learned). In this paper, we address both of these issues for a 3-D CNN implementation through the development of a two-stage computer-aided detection system for automatic detection of pulmonary nodules. The first stage consists of a 3-D fully convolutional network for fast screening and generation of candidate suspicious regions. The second stage consists of an ensemble of 3-D CNNs trained using extensive transformations applied to both the positive and negative patches to augment the training set. To enable the second stage classifiers to learn differently, they are trained on false positive patches obtained from the screening model using different thresholds on their associated scores as well as different augmentation types. The networks in the second stage are averaged together to produce the final classification score for each candidate patch. Using this procedure, our overall nodule detection system called DeepMed is fast and can achieve 91% sensitivity at 2 false positives per scan on cases from the LIDC dataset.

Deep learning in medical imaging and radiation therapy.

The goals of this review paper on deep learning (DL) in medical imaging and radiation therapy are to (a) summarize what has been achieved to date; (b) identify common and unique challenges, and strategies that researchers have taken to address these challenges; and (c) identify some of the promising avenues for the future both in terms of applications as well as technical innovations. We introduce the general principles of DL and convolutional neural networks, survey five major areas of application of DL in medical imaging and radiation therapy, identify common themes, discuss methods for dataset expansion, and conclude by summarizing lessons learned, remaining challenges, and future directions.

Evaluation of Simulated Lesions as Surrogates to Clinical Lesions for Thoracic CT Volumetry: The Results of an International Challenge.

RATIONALE AND OBJECTIVES: To evaluate a new approach to establish compliance of segmentation tools with the computed tomography volumetry profile of the Quantitative Imaging Biomarker Alliance (QIBA); and determine the statistical exchangeability between real and simulated lesions through an international challenge. MATERIALS AND METHODS: The study used an anthropomorphic phantom with 16 embedded physical lesions and 30 patient cases from the Reference Image Database to Evaluate Therapy Response with pathologically confirmed malignancies. Hybrid datasets were generated by virtually inserting simulated lesions corresponding to physical lesions into the phantom datasets using one projection-domain-based method (Method 1), two image-domain insertion methods (Methods 2 and 3), and simulated lesions corresponding to real lesions into the Reference Image Database to Evaluate Therapy Response dataset (using Method 2). The volumes of the real and simulated lesions were compared based on bias (measured mean volume differences between physical and virtually inserted lesions in phantoms as quantified by segmentation algorithms), repeatability, reproducibility, equivalence (phantom phase), and overall QIBA compliance (phantom and clinical phase). RESULTS: For phantom phase, three of eight groups were fully QIBA compliant, and one was marginally compliant. For compliant groups, the estimated biases were -1.8 ± 1.4%, -2.5 ± 1.1%, -3 ± 1%, -1.8 ± 1.5% (±95% confidence interval). No virtual insertion method showed statistical equivalence to physical insertion in bias equivalence testing using Schuirmann's two one-sided test (±5% equivalence margin). Differences in repeatability and reproducibility across physical and simulated lesions were largely comparable (0.1%-16% and 7%-18% differences, respectively). For clinical phase, 7 of 16 groups were QIBA compliant. CONCLUSION: Hybrid datasets yielded conclusions similar to real computed tomography datasets where phantom QIBA compliant was also compliant for hybrid datasets. Some groups deemed compliant for simulated methods, not for physical lesion measurements. The magnitude of this difference was small (<5.4%). While technical performance is not equivalent, they correlate, such that, volumetrically simulated lesions could potentially serve as practical proxies.

Computational insertion of microcalcification clusters on mammograms: reader differentiation from native clusters and computer-aided detection comparison.

Mammographic computer-aided detection (CADe) devices are typically first developed and assessed for a specific "original" acquisition system. When developers are ready to apply their CADe device to a mammographic acquisition system, they typically assess the device with images acquired using the system. Collecting large repositories of clinical images containing verified lesion locations acquired by a system is costly and time consuming. We previously developed an image blending technique that allows users to seamlessly insert regions of interest (ROIs) from one medical image into another image. Our goal is to assess the performance of this technique for inserting microcalcification clusters from one mammogram into another, with the idea that when fully developed, our technique may be useful for reducing the clinical data burden in the assessment of a CADe device for use with an image acquisition system. We first perform a reader study to assess whether experienced observers can distinguish between computationally inserted and native clusters. For this purpose, we apply our insertion technique to 55 clinical cases. ROIs containing microcalcification clusters from one breast of a patient are inserted into the contralateral breast of the same patient. The analysis of the reader ratings using receiver operating characteristic (ROC) methodology indicates that inserted clusters cannot be reliably distinguished from native clusters (area under the ROC curve = 0.58 ± 0.04 ). Furthermore, CADe sensitivity is evaluated on mammograms of 68 clinical cases with native and inserted microcalcification clusters using a commercial CADe system. The average by-case sensitivities for native and inserted clusters are equal, 85.3% (58/68). The average by-image sensitivities for native and inserted clusters are 72.3% and 67.6%, respectively, with a difference of 4.7% and a 95% confidence interval of [ - 2.1 11.6]. These results demonstrate the potential for using the inserted microcalcification clusters for assessing mammographic CADe devices.

Calibration of medical diagnostic classifier scores to the probability of disease.

Scores produced by statistical classifiers in many clinical decision support systems and other medical diagnostic devices are generally on an arbitrary scale, so the clinical meaning of these scores is unclear. Calibration of classifier scores to a meaningful scale such as the probability of disease is potentially useful when such scores are used by a physician. In this work, we investigated three methods (parametric, semi-parametric, and non-parametric) for calibrating classifier scores to the probability of disease scale and developed uncertainty estimation techniques for these methods. We showed that classifier scores on arbitrary scales can be calibrated to the probability of disease scale without affecting their discrimination performance. With a finite dataset to train the calibration function, it is important to accompany the probability estimate with its confidence interval. Our simulations indicate that, when a dataset used for finding the transformation for calibration is also used for estimating the performance of calibration, the resubstitution bias exists for a performance metric involving the truth states in evaluating the calibration performance. However, the bias is small for the parametric and semi-parametric methods when the sample size is moderate to large (>100 per class).

Seamless Lesion Insertion for Data Augmentation in CAD Training.

The performance of a classifier is largely dependent on the size and representativeness of data used for its training. In circumstances where accumulation and/or labeling of training samples is difficult or expensive, such as medical applications, data augmentation can potentially be used to alleviate the limitations of small datasets. We have previously developed an image blending tool that allows users to modify or supplement an existing CT or mammography dataset by seamlessly inserting a lesion extracted from a source image into a target image. This tool also provides the option to apply various types of transformations to different properties of the lesion prior to its insertion into a new location. In this study, we used this tool to create synthetic samples that appear realistic in chest CT. We then augmented different size training sets with these artificial samples, and investigated the effect of the augmentation on training various classifiers for the detection of lung nodules. Our results indicate that the proposed lesion insertion method can improve classifier performance for small training datasets, and thereby help reduce the need to acquire and label actual patient data.

3D Convolutional Neural Network for Automatic Detection of Lung Nodules in Chest CT.

Deep convolutional neural networks (CNNs) form the backbone of many state-of-the-art computer vision systems for classification and segmentation of 2D images. The same principles and architectures can be extended to three dimensions to obtain 3D CNNs that are suitable for volumetric data such as CT scans. In this work, we train a 3D CNN for automatic detection of pulmonary nodules in chest CT images using volumes of interest extracted from the LIDC dataset. We then convert the 3D CNN which has a fixed field of view to a 3D fully convolutional network (FCN) which can generate the score map for the entire volume efficiently in a single pass. Compared to the sliding window approach for applying a CNN across the entire input volume, the FCN leads to a nearly 800-fold speed-up, and thereby fast generation of output scores for a single case. This screening FCN is used to generate difficult negative examples that are used to train a new discriminant CNN. The overall system consists of the screening FCN for fast generation of candidate regions of interest, followed by the discrimination CNN.

Techniques for virtual lung nodule insertion: volumetric and morphometric comparison of projection-based and image-based methods for quantitative CT.

Virtual nodule insertion paves the way towards the development of standardized databases of hybrid CT images with known lesions. The purpose of this study was to assess three methods (an established and two newly developed techniques) for inserting virtual lung nodules into CT images. Assessment was done by comparing virtual nodule volume and shape to the CT-derived volume and shape of synthetic nodules. 24 synthetic nodules (three sizes, four morphologies, two repeats) were physically inserted into the lung cavity of an anthropomorphic chest phantom (KYOTO KAGAKU). The phantom was imaged with and without nodules on a commercial CT scanner (SOMATOM Definition Flash, Siemens) using a standard thoracic CT protocol at two dose levels (1.4 and 22 mGy CTDIvol). Raw projection data were saved and reconstructed with filtered back-projection and sinogram affirmed iterative reconstruction (SAFIRE, strength 5) at 0.6 mm slice thickness. Corresponding 3D idealized, virtual nodule models were co-registered with the CT images to determine each nodule's location and orientation. Virtual nodules were voxelized, partial volume corrected, and inserted into nodule-free CT data (accounting for system imaging physics) using two methods: projection-based Technique A, and image-based Technique B. Also a third Technique C based on cropping a region of interest from the acquired image of the real nodule and blending it into the nodule-free image was tested. Nodule volumes were measured using a commercial segmentation tool (iNtuition, TeraRecon, Inc.) and deformation was assessed using the Hausdorff distance. Nodule volumes and deformations were compared between the idealized, CT-derived and virtual nodules using a linear mixed effects regression model which utilized the mean, standard deviation, and coefficient of variation ([Formula: see text], [Formula: see text] and [Formula: see text] of the regional Hausdorff distance. Overall, there was a close concordance between the volumes of the CT-derived and virtual nodules. Percent differences between them were less than 3% for all insertion techniques and were not statistically significant in most cases. Correlation coefficient values were greater than 0.97. The deformation according to the Hausdorff distance was also similar between the CT-derived and virtual nodules with minimal statistical significance in the ([Formula: see text]) for Techniques A, B, and C. This study shows that both projection-based and image-based nodule insertion techniques yield realistic nodule renderings with statistical similarity to the synthetic nodules with respect to nodule volume and deformation. These techniques could be used to create a database of hybrid CT images containing nodules of known size, location and morphology.

Seamless Insertion of Pulmonary Nodules in Chest CT Images.

The availability of large medical image datasets is critical in many applications, such as training and testing of computer-aided diagnosis systems, evaluation of segmentation algorithms, and conducting perceptual studies. However, collection of data and establishment of ground truth for medical images are both costly and difficult. To address this problem, we are developing an image blending tool that allows users to modify or supplement existing datasets by seamlessly inserting a lesion extracted from a source image into a target image. In this study, we focus on the application of this tool to pulmonary nodules in chest CT exams. We minimize the impact of user skill on the perceived quality of the composite image by limiting user involvement to two simple steps: the user first draws a casual boundary around a nodule in the source, and, then, selects the center of desired insertion area in the target. We demonstrate the performance of our system on clinical samples, and report the results of a reader study evaluating the realism of inserted nodules compared to clinical nodules. We further evaluate our image blending techniques using phantoms simulated under different noise levels and reconstruction filters. Specifically, we compute the area under the ROC curve of the Hotelling observer (HO) and noise power spectrum of regions of interest enclosing native and inserted nodules, and compare the detectability, noise texture, and noise magnitude of inserted and native nodules. Our results indicate the viability of our approach for insertion of pulmonary nodules in clinical CT images.