Large-scale pancreatic cancer detection via non-contrast CT and deep learning.

Pancreatic ductal adenocarcinoma (PDAC), the most deadly solid malignancy, is typically detected late and at an inoperable stage. Early or incidental detection is associated with prolonged survival, but screening asymptomatic individuals for PDAC using a single test remains unfeasible due to the low prevalence and potential harms of false positives. Non-contrast computed tomography (CT), routinely performed for clinical indications, offers the potential for large-scale screening, however, identification of PDAC using non-contrast CT has long been considered impossible. Here, we develop a deep learning approach, pancreatic cancer detection with artificial intelligence (PANDA), that can detect and classify pancreatic lesions with high accuracy via non-contrast CT. PANDA is trained on a dataset of 3,208 patients from a single center. PANDA achieves an area under the receiver operating characteristic curve (AUC) of 0.986-0.996 for lesion detection in a multicenter validation involving 6,239 patients across 10 centers, outperforms the mean radiologist performance by 34.1% in sensitivity and 6.3% in specificity for PDAC identification, and achieves a sensitivity of 92.9% and specificity of 99.9% for lesion detection in a real-world multi-scenario validation consisting of 20,530 consecutive patients. Notably, PANDA utilized with non-contrast CT shows non-inferiority to radiology reports (using contrast-enhanced CT) in the differentiation of common pancreatic lesion subtypes. PANDA could potentially serve as a new tool for large-scale pancreatic cancer screening.

Two Computational Approaches to Visual Analogy: Task-Specific Models Versus Domain-General Mapping.

Advances in artificial intelligence have raised a basic question about human intelligence: Is human reasoning best emulated by applying task-specific knowledge acquired from a wealth of prior experience, or is it based on the domain-general manipulation and comparison of mental representations? We address this question for the case of visual analogical reasoning. Using realistic images of familiar three-dimensional objects (cars and their parts), we systematically manipulated viewpoints, part relations, and entity properties in visual analogy problems. We compared human performance to that of two recent deep learning models (Siamese Network and Relation Network) that were directly trained to solve these problems and to apply their task-specific knowledge to analogical reasoning. We also developed a new model using part-based comparison (PCM) by applying a domain-general mapping procedure to learned representations of cars and their component parts. Across four-term analogies (Experiment 1) and open-ended analogies (Experiment 2), the domain-general PCM model, but not the task-specific deep learning models, generated performance similar in key aspects to that of human reasoners. These findings provide evidence that human-like analogical reasoning is unlikely to be achieved by applying deep learning with big data to a specific type of analogy problem. Rather, humans do (and machines might) achieve analogical reasoning by learning representations that encode structural information useful for multiple tasks, coupled with efficient computation of relational similarity.

External Attention Assisted Multi-Phase Splenic Vascular Injury Segmentation With Limited Data.

The spleen is one of the most commonly injured solid organs in blunt abdominal trauma. The development of automatic segmentation systems from multi-phase CT for splenic vascular injury can augment severity grading for improving clinical decision support and outcome prediction. However, accurate segmentation of splenic vascular injury is challenging for the following reasons: 1) Splenic vascular injury can be highly variant in shape, texture, size, and overall appearance; and 2) Data acquisition is a complex and expensive procedure that requires intensive efforts from both data scientists and radiologists, which makes large-scale well-annotated datasets hard to acquire in general. In light of these challenges, we hereby design a novel framework for multi-phase splenic vascular injury segmentation, especially with limited data. On the one hand, we propose to leverage external data to mine pseudo splenic masks as the spatial attention, dubbed external attention, for guiding the segmentation of splenic vascular injury. On the other hand, we develop a synthetic phase augmentation module, which builds upon generative adversarial networks, for populating the internal data by fully leveraging the relation between different phases. By jointly enforcing external attention and populating internal data representation during training, our proposed method outperforms other competing methods and substantially improves the popular DeepLab-v3+ baseline by more than 7% in terms of average DSC, which confirms its effectiveness.

CT Radiomics-Based Preoperative Survival Prediction in Patients With Pancreatic Ductal Adenocarcinoma.

OBJECTIVE. Pancreatic ductal adenocarcinoma (PDAC) is often a lethal malignancy with limited preoperative predictors of long-term survival. The purpose of this study was to evaluate the prognostic utility of preoperative CT radiomics features in predicting postoperative survival of patients with PDAC. MATERIALS AND METHODS. A total of 153 patients with surgically resected PDAC who underwent preoperative CT between 2011 and 2017 were retrospectively identified. Demographic, clinical, and survival information was collected from the medical records. Survival time after the surgical resection was used to stratify patients into a low-risk group (survival time > 3 years) and a high-risk group (survival time < 1 year). The 3D volume of the whole pancreatic tumor and background pancreas were manually segmented. A total of 478 radiomics features were extracted from tumors and 11 extra features were computed from pancreas boundaries. The 10 most relevant features were selected by feature reduction. Survival analysis was performed on the basis of clinical parameters both with and without the addition of the selected features. Survival status and time were estimated by a random survival forest algorithm. Concordance index (C-index) was used to evaluate performance of the survival prediction model. RESULTS. The mean age of patients with PDAC was 67 ± 11 (SD) years. The mean tumor size was 3.31 ± 2.55 cm. The 10 most relevant radiomics features showed 82.2% accuracy in the classification of high-risk versus low-risk groups. The C-index of survival prediction with clinical parameters alone was 0.6785. The addition of CT radiomics features improved the C-index to 0.7414. CONCLUSION. Addition of CT radiomics features to standard clinical factors improves survival prediction in patients with PDAC.

Current Status of Radiomics and Deep Learning in Liver Imaging.

ABSTRACT: Artificial intelligence is poised to revolutionize medical image. It takes advantage of the high-dimensional quantitative features present in medical images that may not be fully appreciated by humans. Artificial intelligence has the potential to facilitate automatic organ segmentation, disease detection and characterization, and prediction of disease recurrence. This article reviews the current status of artificial intelligence in liver imaging and reviews the opportunities and challenges in clinical implementation.

Learning Inductive Attention Guidance for Partially Supervised Pancreatic Ductal Adenocarcinoma Prediction.

Pancreatic ductal adenocarcinoma (PDAC) is the third most common cause of cancer death in the United States. Predicting tumors like PDACs (including both classification and segmentation) from medical images by deep learning is becoming a growing trend, but usually a large number of annotated data are required for training, which is very labor-intensive and time-consuming. In this paper, we consider a partially supervised setting, where cheap image-level annotations are provided for all the training data, and the costly per-voxel annotations are only available for a subset of them. We propose an Inductive Attention Guidance Network (IAG-Net) to jointly learn a global image-level classifier for normal/PDAC classification and a local voxel-level classifier for semi-supervised PDAC segmentation. We instantiate both the global and the local classifiers by multiple instance learning (MIL), where the attention guidance, indicating roughly where the PDAC regions are, is the key to bridging them: For global MIL based normal/PDAC classification, attention serves as a weight for each instance (voxel) during MIL pooling, which eliminates the distraction from the background; For local MIL based semi-supervised PDAC segmentation, the attention guidance is inductive, which not only provides bag-level pseudo-labels to training data without per-voxel annotations for MIL training, but also acts as a proxy of an instance-level classifier. Experimental results show that our IAG-Net boosts PDAC segmentation accuracy by more than 5% compared with the state-of-the-arts.

Added value of deep learning-based liver parenchymal CT volumetry for predicting major arterial injury after blunt hepatic trauma: a decision tree analysis.

PURPOSE: In patients presenting with blunt hepatic injury (BHI), the utility of CT for triage to hepatic angiography remains uncertain since simple binary assessment of contrast extravasation (CE) as being present or absent has only modest accuracy for major arterial injury on digital subtraction angiography (DSA). American Association for the Surgery of Trauma (AAST) liver injury grading is coarse and subjective, with limited diagnostic utility in this setting. Volumetric measurements of hepatic injury burden could improve prediction. We hypothesized that in a cohort of patients that underwent catheter-directed hepatic angiography following admission trauma CT, a deep learning quantitative visualization method that calculates % liver parenchymal disruption (the LPD index, or LPDI) would add value to CE assessment for prediction of major hepatic arterial injury (MHAI). METHODS: This retrospective study included adult patients with BHI between 1/1/2008 and 5/1/2017 from two institutions that underwent admission trauma CT prior to hepatic angiography (n = 73). Presence (n = 41) or absence (n = 32) of MHAI (pseudoaneurysm, AVF, or active contrast extravasation on DSA) served as the outcome. Voxelwise measurements of liver laceration were derived using an existing multiscale deep learning algorithm trained on manually labeled data using cross-validation with a 75-25% split in four unseen folds. Liver volume was derived using a pre-trained whole liver segmentation algorithm. LPDI was automatically calculated for each patient by determining the percentage of liver involved by laceration. Classification and regression tree (CART) analyses were performed using a combination of automated LPDI measurements and either manually segmented CE volumes, or CE as a binary sign. Performance metrics for the decision rules were compared for significant differences with binary CE alone (the current standard of care for predicting MHAI), and the AAST grade. RESULTS: 36% of patients (n = 26) had contrast extravasation on CT. Median [Q1-Q3] automated LPDI was 4.0% [1.0-12.1%]. 41/73 (56%) of patients had MHAI. A decision tree based on auto-LPDI and volumetric CE measurements (CEvol) had the highest accuracy (0.84, 95% CI 0.73-0.91) with significant improvement over binary CE assessment (0.68, 95% CI 0.57-0.79; p = 0.01). AAST grades at different cut-offs performed poorly for predicting MHAI, with accuracies ranging from 0.44-0.63. Decision tree analysis suggests an auto-LPDI cut-off of ≥ 12% for minimizing false negative CT exams when CE is absent or diminutive. CONCLUSION: Current CT imaging paradigms are coarse, subjective, and limited for predicting which BHIs are most likely to benefit from AE. LPDI, automated using deep learning methods, may improve objective personalized triage of BHI patients to angiography at the point of care.

Pancreatic Cancer Imaging: A New Look at an Old Problem.

Computed tomography is the most commonly used imaging modality to detect and stage pancreatic cancer. Previous advances in pancreatic cancer imaging have focused on optimizing image acquisition parameters and reporting standards. However, current state-of-the-art imaging approaches still misdiagnose some potentially curable pancreatic cancers and do not provide prognostic information or inform optimal management strategies beyond stage. Several recent developments in pancreatic cancer imaging, including artificial intelligence and advanced visualization techniques, are rapidly changing the field. The purpose of this article is to review how these recent advances have the potential to revolutionize pancreatic cancer imaging.

A Multiscale Deep Learning Method for Quantitative Visualization of Traumatic Hemoperitoneum at CT: Assessment of Feasibility and Comparison with Subjective Categorical Estimation.

PURPOSE: To evaluate the feasibility of a multiscale deep learning algorithm for quantitative visualization and measurement of traumatic hemoperitoneum and to compare diagnostic performance for relevant outcomes with categorical estimation. MATERIALS AND METHODS: This retrospective, single-institution study included 130 patients (mean age, 38 years; interquartile range, 25-50 years; 79 men) with traumatic hemoperitoneum who underwent CT of the abdomen and pelvis at trauma admission between January 2016 and April 2019. Labeled cases were separated into five combinations of training (80%) and test (20%) sets, and fivefold cross-validation was performed. Dice similarity coefficients (DSCs) were compared with those from a three-dimensional (3D) U-Net and a coarse-to-fine deep learning method. Areas under the receiver operating characteristic curve (AUCs) for a composite outcome, including hemostatic intervention, transfusion, and in-hospital mortality, were compared with consensus categorical assessment by two radiologists. An optimal cutoff was derived by using a radial basis function-based support vector machine. RESULTS: Mean DSC for the multiscale algorithm was 0.61 ± 0.15 (standard deviation) compared with 0.32 ± 0.16 for the 3D U-Net method and 0.52 ± 0.17 for the coarse-to-fine method (P < .0001). Correlation and agreement between automated and manual volumes were excellent (Pearson r = 0.97, intraclass correlation coefficient = 0.93). The algorithm produced intuitive and explainable visual results. AUCs for automated volume measurement and categorical estimation were 0.86 and 0.77, respectively (P = .004). An optimal cutoff of 278.9 mL yielded accuracy of 84%, sensitivity of 82%, specificity of 93%, positive predictive value of 86%, and negative predictive value of 83%. CONCLUSION: A multiscale deep learning method for traumatic hemoperitoneum quantitative visualization had improved diagnostic performance for predicting hemorrhage-control interventions and mortality compared with subjective volume estimation. Supplemental material is available for this article. © RSNA, 2020.

Diagnostic performance of commercially available vs. in-house radiomics software in classification of CT images from patients with pancreatic ductal adenocarcinoma vs. healthy controls.

PURPOSE: The purpose of this study is to evaluate diagnostic performance of a commercially available radiomics research prototype vs. an in-house radiomics software in the binary classification of CT images from patients with pancreatic ductal adenocarcinoma (PDAC) vs. healthy controls. MATERIALS AND METHODS: In this retrospective case-control study, 190 patients with PDAC (97 men, 93 women; 66 ± 9 years) from 2012 to 2017 and 190 healthy potential renal donors (96 men, 94 women; 52 ± 8 years) without known pancreatic disease from 2005 to 2009 were identified from radiology and pathology databases. 3D volume of the pancreas was manually segmented from preoperative CT scans. Four hundred and seventy-eight radiomics features were extracted using in-house radiomics software. Eight hundred and fifty-four radiomics features were extracted using a commercially available research prototype. Random forest classifier was used for binary classification of PDAC vs. normal pancreas. Accuracy, sensitivity, and specificity of commercially available radiomics software were compared to in-house software. RESULTS: When 40 radiomics features were used in the random forest classification, in-house software achieved superior sensitivity (1.00) and accuracy (0.992) compared to the commercially available research prototype (sensitivity = 0.950, accuracy = 0.968). When the number of features was reduced to five features, diagnostic performance of the in-house software decreased to sensitivity (0.950), specificity (0.923), and accuracy (0.936). Diagnostic performance of the commercially available research prototype was unchanged. CONCLUSION: Commercially available and in-house radiomics software achieve similar diagnostic performance, which may lower the barrier of entry for radiomics research and allow more clinician-scientists to perform radiomics research.

Deep learning-based quantitative visualization and measurement of extraperitoneal hematoma volumes in patients with pelvic fractures: Potential role in personalized forecasting and decision support.

INTRODUCTION: Admission computed tomography (CT) is a widely used diagnostic tool for patients with pelvic fractures. In this pilot study, we hypothesized that pelvic hematoma volumes derived using a rapid automated deep learning-based quantitative visualization and measurement algorithm predict interventions and outcomes including (a) need for angioembolization (AE), pelvic packing (PP), or massive transfusion (MT), and (b) in-hospital mortality. METHODS: We performed a single-institution retrospective analysis of 253 patients with bleeding pelvic fractures who underwent admission abdominopelvic trauma CT between 2008 and 2017. Included patients had hematoma volumes of 30 mL or greater, were 18 years and older, and underwent contrast-enhanced CT before surgical or angiographic intervention. Automated pelvic hematoma volume measurements were previously derived using a deep-learning quantitative visualization and measurement algorithm through cross-validation. A composite dependent variable of need for MT, AE, or PP was used as the primary endpoint. The added utility of hematoma volume was assessed by comparing the performance of multivariable models with and without hematoma volume as a predictor. Areas under the receiver operating characteristic curve (AUCs) and sensitivities, specificities, and predictive values were determined at clinically relevant thresholds. Adjusted odds ratios of automated pelvic hematoma volumes at 200 mL increments were derived. RESULTS: Median age was 47 years (interquartile range, 29-61), and 70% of patients were male. Median Injury Severity Score was 22 (14-36). Ninety-four percent of patients had injuries in other body regions, and 73% had polytrauma (Injury Severity Score, ≥16). Thirty-three percent had Tile/Orthopedic Trauma Association type B, and 24% had type C pelvic fractures. A total of 109 patients underwent AE, 22 underwent PP, and 53 received MT. A total of 123 patients received all 3 interventions. Sixteen patients died during hospitalization from causes other than untreatable (abbreviated injury scale, 6) head injury. Variables incorporated into multivariable models included age, sex, Tile/Orthopedic Trauma Association grade, admission lactate, heart rate (HR), and systolic blood pressure (SBP). Addition of hematoma volume resulted in a significant improvement in model performance, with AUC for the composite outcome (AE, PP, or MT) increasing from 0.74 to 0.83 (p < 0.001). Adjusted unit odds more than doubled for every additional 200 mL of hematoma volume. Increase in model AUC for mortality with incorporation of hematoma volume was not statistically significant (0.85 vs. 0.90, p = 0.12). CONCLUSION: Hematoma volumes measured using a rapid automated deep learning algorithm improved prediction of need for AE, PP, or MT. Simultaneous automated measurement of multiple sources of bleeding at CT could augment outcome prediction in trauma patients. LEVEL OF EVIDENCE: Diagnostic, level IV.

Recurrent Saliency Transformation Network for Tiny Target Segmentation in Abdominal CT Scans.

We aim at segmenting a wide variety of organs, including tiny targets (e.g., adrenal gland), and neoplasms (e.g., pancreatic cyst), from abdominal CT scans. This is a challenging task in two aspects. First, some organs (e.g., the pancreas), are highly variable in both anatomy and geometry, and thus very difficult to depict. Second, the neoplasms often vary a lot in its size, shape, as well as its location within the organ. Third, the targets (organs and neoplasms) can be considerably small compared to the human body, and so standard deep networks for segmentation are often less sensitive to these targets and thus predict less accurately especially around their boundaries. In this paper, we present an end-to-end framework named recurrent saliency transformation network (RSTN) for segmenting tiny and/or variable targets. The RSTN is a coarse-to-fine approach that uses prediction from the first (coarse) stage to shrink the input region for the second (fine) stage. A saliency transformation module is inserted between these two stages so that 1) the coarse-scaled segmentation mask can be transferred as spatial weights and applied to the fine stage and 2) the gradients can be back-propagated from the loss layer to the entire network so that the two stages are optimized in a joint manner. In the testing stage, we perform segmentation iteratively to improve accuracy. In this extended journal paper, we allow a gradual optimization to improve the stability of the RSTN, and introduce a hierarchical version named H-RSTN to segment tiny and variable neoplasms such as pancreatic cysts. Experiments are performed on several CT datasets including a public pancreas segmentation dataset, our own multi-organ dataset, and a cystic pancreas dataset. In all these cases, the RSTN outperforms the baseline (a stage-wise coarse-to-fine approach) significantly. Confirmed by the radiologists in our team, these promising segmentation results can help early diagnosis of pancreatic cancer. The code and pre-trained models of our project were made available at https://github.com/198808xc/OrganSegRSTN.

Performance of a Deep Learning Algorithm for Automated Segmentation and Quantification of Traumatic Pelvic Hematomas on CT.

The volume of pelvic hematoma at CT has been shown to be the strongest independent predictor of major arterial injury requiring angioembolization in trauma victims with pelvic fractures, and also correlates with transfusion requirement and mortality. Measurement of pelvic hematomas (unopacified extraperitoneal blood accumulated from time of injury) using semi-automated seeded region growing is time-consuming and requires trained experts, precluding routine measurement at the point of care. Pelvic hematomas are markedly variable in shape and location, have irregular ill-defined margins, have low contrast with respect to viscera and muscle, and reside within anatomically distorted pelvises. Furthermore, pelvic hematomas occupy a small proportion of the entire volume of a chest, abdomen, and pelvis (C/A/P) trauma CT. The challenges are many, and no automated methods for segmentation and volumetric analysis have been described to date. Traditional approaches using fully convolutional networks result in coarse segmentations and class imbalance with suboptimal convergence. In this study, we implement a modified coarse-to-fine deep learning approach-the Recurrent Saliency Transformation Network (RSTN) for pelvic hematoma volume segmentation. RSTN previously yielded excellent results in pancreas segmentation, where low contrast with adjacent structures, small target volume, variable location, and fine contours are also problematic. We have curated a unique single-institution corpus of 253 C/A/P admission trauma CT studies in patients with bleeding pelvic fractures with manually labeled pelvic hematomas. We hypothesized that RSTN would result in sufficiently high Dice similarity coefficients to facilitate accurate and objective volumetric measurements for outcome prediction (arterial injury requiring angioembolization). Cases were separated into five combinations of training and test sets in an 80/20 split and fivefold cross-validation was performed. Dice scores in the test set were 0.71 (SD ± 0.10) using RSTN, compared to 0.49 (SD ± 0.16) using a baseline Deep Learning Tool Kit (DLTK) reference 3D U-Net architecture. Mean inference segmentation time for RSTN was 0.90 min (± 0.26). Pearson correlation between predicted and manual labels was 0.95 with p < 0.0001. Measurement bias was within 10 mL. AUC of hematoma volumes for predicting need for angioembolization was 0.81 (predicted) versus 0.80 (manual). Qualitatively, predicted labels closely followed hematoma contours and avoided muscle and displaced viscera. Further work will involve validation using a federated dataset and incorporation into a predictive model using multiple segmented features.

Abdominal multi-organ segmentation with organ-attention networks and statistical fusion.

Accurate and robust segmentation of abdominal organs on CT is essential for many clinical applications such as computer-aided diagnosis and computer-aided surgery. But this task is challenging due to the weak boundaries of organs, the complexity of the background, and the variable sizes of different organs. To address these challenges, we introduce a novel framework for multi-organ segmentation of abdominal regions by using organ-attention networks with reverse connections (OAN-RCs) which are applied to 2D views, of the 3D CT volume, and output estimates which are combined by statistical fusion exploiting structural similarity. More specifically, OAN is a two-stage deep convolutional network, where deep network features from the first stage are combined with the original image, in a second stage, to reduce the complex background and enhance the discriminative information for the target organs. Intuitively, OAN reduces the effect of the complex background by focusing attention so that each organ only needs to be discriminated from its local background. RCs are added to the first stage to give the lower layers more semantic information thereby enabling them to adapt to the sizes of different organs. Our networks are trained on 2D views (slices) enabling us to use holistic information and allowing efficient computation (compared to using 3D patches). To compensate for the limited cross-sectional information of the original 3D volumetric CT, e.g., the connectivity between neighbor slices, multi-sectional images are reconstructed from the three different 2D view directions. Then we combine the segmentation results from the different views using statistical fusion, with a novel term relating the structural similarity of the 2D views to the original 3D structure. To train the network and evaluate results, 13 structures were manually annotated by four human raters and confirmed by a senior expert on 236 normal cases. We tested our algorithm by 4-fold cross-validation and computed Dice-Sørensen similarity coefficients (DSC) and surface distances for evaluating our estimates of the 13 structures. Our experiments show that the proposed approach gives strong results and outperforms 2D- and 3D-patch based state-of-the-art methods in terms of DSC and mean surface distances.

Utility of CT Radiomics Features in Differentiation of Pancreatic Ductal Adenocarcinoma From Normal Pancreatic Tissue.

OBJECTIVE. The objective of our study was to determine the utility of radiomics features in differentiating CT cases of pancreatic ductal adenocarcinoma (PDAC) from normal pancreas. MATERIALS AND METHODS. In this retrospective case-control study, 190 patients with PDAC (97 men, 93 women; mean age ± SD, 66 ± 9 years) from 2012 to 2017 and 190 healthy potential renal donors (96 men, 94 women; mean age ± SD, 52 ± 8 years) without known pancreatic disease from 2005 to 2009 were identified from radiology and pathology databases. The 3D volume of the pancreas was manually segmented from the preoperative CT scans by four trained researchers and verified by three abdominal radiologists. Four hundred seventy-eight radiomics features were extracted to express the phenotype of the pancreas. Forty features were selected for analysis because of redundancy of computed features. The dataset was divided into 255 training cases (125 normal control cases and 130 PDAC cases) and 125 validation cases (65 normal control cases and 60 PDAC cases). A random forest classifier was used for binary classification of PDAC versus normal pancreas of control cases. Accuracy, sensitivity, and specificity were calculated. RESULTS. Mean tumor size was 4.1 ± 1.7 (SD) cm. The overall accuracy of the random forest binary classification was 99.2% (124/125), and AUC was 99.9%. All PDAC cases (60/60) were correctly classified. One case from a renal donor was misclassified as PDAC (1/65). The sensitivity was 100%, and specificity was 98.5%. CONCLUSION. Radiomics features extracted from whole pancreas can be used to differentiate between CT cases from patients with PDAC and healthy control subjects with normal pancreas.

Robust 3D Human Pose Estimation from Single Images or Video Sequences.

We propose a method for estimating 3D human poses from single images or video sequences. The task is challenging because: (a) many 3D poses can have similar 2D pose projections which makes the lifting ambiguous, and (b) current 2D joint detectors are not accurate which can cause big errors in 3D estimates. We represent 3D poses by a sparse combination of bases which encode structural pose priors to reduce the lifting ambiguity. This prior is strengthened by adding limb length constraints. We estimate the 3D pose by minimizing an L1 norm measurement error between the 2D pose and the 3D pose because it is less sensitive to inaccurate 2D poses. We modify our algorithm to output K 3D pose candidates for an image, and for videos, we impose a temporal smoothness constraint to select the best sequence of 3D poses from the candidates. We demonstrate good results on 3D pose estimation from static images and improved performance by selecting the best 3D pose from the K proposals. Our results on video sequences also show improvements (over static images) of roughly 15%.

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs.

In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First, we highlight convolution with upsampled filters, or 'atrous convolution', as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second, we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third, we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed "DeepLab" system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7 percent mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online.

Semi-Supervised Sparse Representation Based Classification for Face Recognition With Insufficient Labeled Samples.

This paper addresses the problem of face recognition when there is only few, or even only a single, labeled examples of the face that we wish to recognize. Moreover, these examples are typically corrupted by nuisance variables, both linear (i.e., additive nuisance variables, such as bad lighting and wearing of glasses) and non-linear (i.e., non-additive pixel-wise nuisance variables, such as expression changes). The small number of labeled examples means that it is hard to remove these nuisance variables between the training and testing faces to obtain good recognition performance. To address the problem, we propose a method called semi-supervised sparse representation-based classification. This is based on recent work on sparsity, where faces are represented in terms of two dictionaries: a gallery dictionary consisting of one or more examples of each person, and a variation dictionary representing linear nuisance variables (e.g., different lighting conditions and different glasses). The main idea is that: 1) we use the variation dictionary to characterize the linear nuisance variables via the sparsity framework and 2) prototype face images are estimated as a gallery dictionary via a Gaussian mixture model, with mixed labeled and unlabeled samples in a semi-supervised manner, to deal with the non-linear nuisance variations between labeled and unlabeled samples. We have done experiments with insufficient labeled samples, even when there is only a single labeled sample per person. Our results on the AR, Multi-PIE, CAS-PEAL, and LFW databases demonstrate that the proposed method is able to deliver significantly improved performance over existing methods.

Empirical Minimum Bayes Risk Prediction.

When building vision systems that predict structured objects such as image segmentations or human poses, a crucial concern is performance under task-specific evaluation measures (e.g., Jaccard Index or Average Precision). An ongoing research challenge is to optimize predictions so as to maximize performance on such complex measures. In this work, we present a simple meta-algorithm that is surprisingly effective - Empirical Min Bayes Risk. EMBR takes as input a pre-trained model that would normally be the final product and learns three additional parameters so as to optimize performance on the complex instance-level high-order task-specific measure. We demonstrate EMBR in several domains, taking existing state-of-the-art algorithms and improving performance up to 8 percent, simply by learning three extra parameters. Our code is publicly available and the results presented in this paper can be replicated from our code-release.