TransUNet: Rethinking the U-Net architecture design for medical image segmentation through the lens of transformers.

Medical image segmentation is crucial for healthcare, yet convolution-based methods like U-Net face limitations in modeling long-range dependencies. To address this, Transformers designed for sequence-to-sequence predictions have been integrated into medical image segmentation. However, a comprehensive understanding of Transformers' self-attention in U-Net components is lacking. TransUNet, first introduced in 2021, is widely recognized as one of the first models to integrate Transformer into medical image analysis. In this study, we present the versatile framework of TransUNet that encapsulates Transformers' self-attention into two key modules: (1) a Transformer encoder tokenizing image patches from a convolution neural network (CNN) feature map, facilitating global context extraction, and (2) a Transformer decoder refining candidate regions through cross-attention between proposals and U-Net features. These modules can be flexibly inserted into the U-Net backbone, resulting in three configurations: Encoder-only, Decoder-only, and Encoder+Decoder. TransUNet provides a library encompassing both 2D and 3D implementations, enabling users to easily tailor the chosen architecture. Our findings highlight the encoder's efficacy in modeling interactions among multiple abdominal organs and the decoder's strength in handling small targets like tumors. It excels in diverse medical applications, such as multi-organ segmentation, pancreatic tumor segmentation, and hepatic vessel segmentation. Notably, our TransUNet achieves a significant average Dice improvement of 1.06% and 4.30% for multi-organ segmentation and pancreatic tumor segmentation, respectively, when compared to the highly competitive nn-UNet, and surpasses the top-1 solution in the BrasTS2021 challenge. 2D/3D Code and models are available at https://github.com/Beckschen/TransUNet and https://github.com/Beckschen/TransUNet-3D, respectively.

Universal and extensible language-vision models for organ segmentation and tumor detection from abdominal computed tomography.

The advancement of artificial intelligence (AI) for organ segmentation and tumor detection is propelled by the growing availability of computed tomography (CT) datasets with detailed, per-voxel annotations. However, these AI models often struggle with flexibility for partially annotated datasets and extensibility for new classes due to limitations in the one-hot encoding, architectural design, and learning scheme. To overcome these limitations, we propose a universal, extensible framework enabling a single model, termed Universal Model, to deal with multiple public datasets and adapt to new classes (e.g., organs/tumors). Firstly, we introduce a novel language-driven parameter generator that leverages language embeddings from large language models, enriching semantic encoding compared with one-hot encoding. Secondly, the conventional output layers are replaced with lightweight, class-specific heads, allowing Universal Model to simultaneously segment 25 organs and six types of tumors and ease the addition of new classes. We train our Universal Model on 3410 CT volumes assembled from 14 publicly available datasets and then test it on 6173 CT volumes from four external datasets. Universal Model achieves first place on six CT tasks in the Medical Segmentation Decathlon (MSD) public leaderboard and leading performance on the Beyond The Cranial Vault (BTCV) dataset. In summary, Universal Model exhibits remarkable computational efficiency (6× faster than other dataset-specific models), demonstrates strong generalization across different hospitals, transfers well to numerous downstream tasks, and more importantly, facilitates the extensibility to new classes while alleviating the catastrophic forgetting of previously learned classes. Codes, models, and datasets are available at https://github.com/ljwztc/CLIP-Driven-Universal-Model.

3D genomic mapping reveals multifocality of human pancreatic precancers.

Pancreatic intraepithelial neoplasias (PanINs) are the most common precursors of pancreatic cancer, but their small size and inaccessibility in humans make them challenging to study1. Critically, the number, dimensions and connectivity of human PanINs remain largely unknown, precluding important insights into early cancer development. Here, we provide a microanatomical survey of human PanINs by analysing 46 large samples of grossly normal human pancreas with a machine-learning pipeline for quantitative 3D histological reconstruction at single-cell resolution. To elucidate genetic relationships between and within PanINs, we developed a workflow in which 3D modelling guides multi-region microdissection and targeted and whole-exome sequencing. From these samples, we calculated a mean burden of 13 PanINs per cm3 and extrapolated that the normal intact adult pancreas harbours hundreds of PanINs, almost all with oncogenic KRAS hotspot mutations. We found that most PanINs originate as independent clones with distinct somatic mutation profiles. Some spatially continuous PanINs were found to contain multiple KRAS mutations; computational and in situ analyses demonstrated that different KRAS mutations localize to distinct cell subpopulations within these neoplasms, indicating their polyclonal origins. The extensive multifocality and genetic heterogeneity of PanINs raises important questions about mechanisms that drive precancer initiation and confer differential progression risk in the human pancreas. This detailed 3D genomic mapping of molecular alterations in human PanINs provides an empirical foundation for early detection and rational interception of pancreatic cancer.

Large-scale pancreatic cancer detection via non-contrast CT and deep learning.

Pancreatic ductal adenocarcinoma (PDAC), the most deadly solid malignancy, is typically detected late and at an inoperable stage. Early or incidental detection is associated with prolonged survival, but screening asymptomatic individuals for PDAC using a single test remains unfeasible due to the low prevalence and potential harms of false positives. Non-contrast computed tomography (CT), routinely performed for clinical indications, offers the potential for large-scale screening, however, identification of PDAC using non-contrast CT has long been considered impossible. Here, we develop a deep learning approach, pancreatic cancer detection with artificial intelligence (PANDA), that can detect and classify pancreatic lesions with high accuracy via non-contrast CT. PANDA is trained on a dataset of 3,208 patients from a single center. PANDA achieves an area under the receiver operating characteristic curve (AUC) of 0.986-0.996 for lesion detection in a multicenter validation involving 6,239 patients across 10 centers, outperforms the mean radiologist performance by 34.1% in sensitivity and 6.3% in specificity for PDAC identification, and achieves a sensitivity of 92.9% and specificity of 99.9% for lesion detection in a real-world multi-scenario validation consisting of 20,530 consecutive patients. Notably, PANDA utilized with non-contrast CT shows non-inferiority to radiology reports (using contrast-enhanced CT) in the differentiation of common pancreatic lesion subtypes. PANDA could potentially serve as a new tool for large-scale pancreatic cancer screening.

Three-dimensional genomic mapping of human pancreatic tissue reveals striking multifocality and genetic heterogeneity in precancerous lesions.

Pancreatic intraepithelial neoplasia (PanIN) is a precursor to pancreatic cancer and represents a critical opportunity for cancer interception. However, the number, size, shape, and connectivity of PanINs in human pancreatic tissue samples are largely unknown. In this study, we quantitatively assessed human PanINs using CODA, a novel machine-learning pipeline for 3D image analysis that generates quantifiable models of large pieces of human pancreas with single-cell resolution. Using a cohort of 38 large slabs of grossly normal human pancreas from surgical resection specimens, we identified striking multifocality of PanINs, with a mean burden of 13 spatially separate PanINs per cm3 of sampled tissue. Extrapolating this burden to the entire pancreas suggested a median of approximately 1000 PanINs in an entire pancreas. In order to better understand the clonal relationships within and between PanINs, we developed a pipeline for CODA-guided multi-region genomic analysis of PanINs, including targeted and whole exome sequencing. Multi-region assessment of 37 PanINs from eight additional human pancreatic tissue slabs revealed that almost all PanINs contained hotspot mutations in the oncogene KRAS, but no gene other than KRAS was altered in more than 20% of the analyzed PanINs. PanINs contained a mean of 13 somatic mutations per region when analyzed by whole exome sequencing. The majority of analyzed PanINs originated from independent clonal events, with distinct somatic mutation profiles between PanINs in the same tissue slab. A subset of the analyzed PanINs contained multiple KRAS mutations, suggesting a polyclonal origin even in PanINs that are contiguous by rigorous 3D assessment. This study leverages a novel 3D genomic mapping approach to describe, for the first time, the spatial and genetic multifocality of human PanINs, providing important insights into the initiation and progression of pancreatic neoplasia.

CT Radiomics-Based Preoperative Survival Prediction in Patients With Pancreatic Ductal Adenocarcinoma.

OBJECTIVE. Pancreatic ductal adenocarcinoma (PDAC) is often a lethal malignancy with limited preoperative predictors of long-term survival. The purpose of this study was to evaluate the prognostic utility of preoperative CT radiomics features in predicting postoperative survival of patients with PDAC. MATERIALS AND METHODS. A total of 153 patients with surgically resected PDAC who underwent preoperative CT between 2011 and 2017 were retrospectively identified. Demographic, clinical, and survival information was collected from the medical records. Survival time after the surgical resection was used to stratify patients into a low-risk group (survival time > 3 years) and a high-risk group (survival time < 1 year). The 3D volume of the whole pancreatic tumor and background pancreas were manually segmented. A total of 478 radiomics features were extracted from tumors and 11 extra features were computed from pancreas boundaries. The 10 most relevant features were selected by feature reduction. Survival analysis was performed on the basis of clinical parameters both with and without the addition of the selected features. Survival status and time were estimated by a random survival forest algorithm. Concordance index (C-index) was used to evaluate performance of the survival prediction model. RESULTS. The mean age of patients with PDAC was 67 ± 11 (SD) years. The mean tumor size was 3.31 ± 2.55 cm. The 10 most relevant radiomics features showed 82.2% accuracy in the classification of high-risk versus low-risk groups. The C-index of survival prediction with clinical parameters alone was 0.6785. The addition of CT radiomics features improved the C-index to 0.7414. CONCLUSION. Addition of CT radiomics features to standard clinical factors improves survival prediction in patients with PDAC.

Current Status of Radiomics and Deep Learning in Liver Imaging.

ABSTRACT: Artificial intelligence is poised to revolutionize medical image. It takes advantage of the high-dimensional quantitative features present in medical images that may not be fully appreciated by humans. Artificial intelligence has the potential to facilitate automatic organ segmentation, disease detection and characterization, and prediction of disease recurrence. This article reviews the current status of artificial intelligence in liver imaging and reviews the opportunities and challenges in clinical implementation.

Learning Inductive Attention Guidance for Partially Supervised Pancreatic Ductal Adenocarcinoma Prediction.

Pancreatic ductal adenocarcinoma (PDAC) is the third most common cause of cancer death in the United States. Predicting tumors like PDACs (including both classification and segmentation) from medical images by deep learning is becoming a growing trend, but usually a large number of annotated data are required for training, which is very labor-intensive and time-consuming. In this paper, we consider a partially supervised setting, where cheap image-level annotations are provided for all the training data, and the costly per-voxel annotations are only available for a subset of them. We propose an Inductive Attention Guidance Network (IAG-Net) to jointly learn a global image-level classifier for normal/PDAC classification and a local voxel-level classifier for semi-supervised PDAC segmentation. We instantiate both the global and the local classifiers by multiple instance learning (MIL), where the attention guidance, indicating roughly where the PDAC regions are, is the key to bridging them: For global MIL based normal/PDAC classification, attention serves as a weight for each instance (voxel) during MIL pooling, which eliminates the distraction from the background; For local MIL based semi-supervised PDAC segmentation, the attention guidance is inductive, which not only provides bag-level pseudo-labels to training data without per-voxel annotations for MIL training, but also acts as a proxy of an instance-level classifier. Experimental results show that our IAG-Net boosts PDAC segmentation accuracy by more than 5% compared with the state-of-the-arts.

Added value of deep learning-based liver parenchymal CT volumetry for predicting major arterial injury after blunt hepatic trauma: a decision tree analysis.

PURPOSE: In patients presenting with blunt hepatic injury (BHI), the utility of CT for triage to hepatic angiography remains uncertain since simple binary assessment of contrast extravasation (CE) as being present or absent has only modest accuracy for major arterial injury on digital subtraction angiography (DSA). American Association for the Surgery of Trauma (AAST) liver injury grading is coarse and subjective, with limited diagnostic utility in this setting. Volumetric measurements of hepatic injury burden could improve prediction. We hypothesized that in a cohort of patients that underwent catheter-directed hepatic angiography following admission trauma CT, a deep learning quantitative visualization method that calculates % liver parenchymal disruption (the LPD index, or LPDI) would add value to CE assessment for prediction of major hepatic arterial injury (MHAI). METHODS: This retrospective study included adult patients with BHI between 1/1/2008 and 5/1/2017 from two institutions that underwent admission trauma CT prior to hepatic angiography (n = 73). Presence (n = 41) or absence (n = 32) of MHAI (pseudoaneurysm, AVF, or active contrast extravasation on DSA) served as the outcome. Voxelwise measurements of liver laceration were derived using an existing multiscale deep learning algorithm trained on manually labeled data using cross-validation with a 75-25% split in four unseen folds. Liver volume was derived using a pre-trained whole liver segmentation algorithm. LPDI was automatically calculated for each patient by determining the percentage of liver involved by laceration. Classification and regression tree (CART) analyses were performed using a combination of automated LPDI measurements and either manually segmented CE volumes, or CE as a binary sign. Performance metrics for the decision rules were compared for significant differences with binary CE alone (the current standard of care for predicting MHAI), and the AAST grade. RESULTS: 36% of patients (n = 26) had contrast extravasation on CT. Median [Q1-Q3] automated LPDI was 4.0% [1.0-12.1%]. 41/73 (56%) of patients had MHAI. A decision tree based on auto-LPDI and volumetric CE measurements (CEvol) had the highest accuracy (0.84, 95% CI 0.73-0.91) with significant improvement over binary CE assessment (0.68, 95% CI 0.57-0.79; p = 0.01). AAST grades at different cut-offs performed poorly for predicting MHAI, with accuracies ranging from 0.44-0.63. Decision tree analysis suggests an auto-LPDI cut-off of ≥ 12% for minimizing false negative CT exams when CE is absent or diminutive. CONCLUSION: Current CT imaging paradigms are coarse, subjective, and limited for predicting which BHIs are most likely to benefit from AE. LPDI, automated using deep learning methods, may improve objective personalized triage of BHI patients to angiography at the point of care.

Pancreatic Cancer Imaging: A New Look at an Old Problem.

Computed tomography is the most commonly used imaging modality to detect and stage pancreatic cancer. Previous advances in pancreatic cancer imaging have focused on optimizing image acquisition parameters and reporting standards. However, current state-of-the-art imaging approaches still misdiagnose some potentially curable pancreatic cancers and do not provide prognostic information or inform optimal management strategies beyond stage. Several recent developments in pancreatic cancer imaging, including artificial intelligence and advanced visualization techniques, are rapidly changing the field. The purpose of this article is to review how these recent advances have the potential to revolutionize pancreatic cancer imaging.

Diagnostic performance of commercially available vs. in-house radiomics software in classification of CT images from patients with pancreatic ductal adenocarcinoma vs. healthy controls.

PURPOSE: The purpose of this study is to evaluate diagnostic performance of a commercially available radiomics research prototype vs. an in-house radiomics software in the binary classification of CT images from patients with pancreatic ductal adenocarcinoma (PDAC) vs. healthy controls. MATERIALS AND METHODS: In this retrospective case-control study, 190 patients with PDAC (97 men, 93 women; 66 ± 9 years) from 2012 to 2017 and 190 healthy potential renal donors (96 men, 94 women; 52 ± 8 years) without known pancreatic disease from 2005 to 2009 were identified from radiology and pathology databases. 3D volume of the pancreas was manually segmented from preoperative CT scans. Four hundred and seventy-eight radiomics features were extracted using in-house radiomics software. Eight hundred and fifty-four radiomics features were extracted using a commercially available research prototype. Random forest classifier was used for binary classification of PDAC vs. normal pancreas. Accuracy, sensitivity, and specificity of commercially available radiomics software were compared to in-house software. RESULTS: When 40 radiomics features were used in the random forest classification, in-house software achieved superior sensitivity (1.00) and accuracy (0.992) compared to the commercially available research prototype (sensitivity = 0.950, accuracy = 0.968). When the number of features was reduced to five features, diagnostic performance of the in-house software decreased to sensitivity (0.950), specificity (0.923), and accuracy (0.936). Diagnostic performance of the commercially available research prototype was unchanged. CONCLUSION: Commercially available and in-house radiomics software achieve similar diagnostic performance, which may lower the barrier of entry for radiomics research and allow more clinician-scientists to perform radiomics research.

Deep learning-based quantitative visualization and measurement of extraperitoneal hematoma volumes in patients with pelvic fractures: Potential role in personalized forecasting and decision support.

INTRODUCTION: Admission computed tomography (CT) is a widely used diagnostic tool for patients with pelvic fractures. In this pilot study, we hypothesized that pelvic hematoma volumes derived using a rapid automated deep learning-based quantitative visualization and measurement algorithm predict interventions and outcomes including (a) need for angioembolization (AE), pelvic packing (PP), or massive transfusion (MT), and (b) in-hospital mortality. METHODS: We performed a single-institution retrospective analysis of 253 patients with bleeding pelvic fractures who underwent admission abdominopelvic trauma CT between 2008 and 2017. Included patients had hematoma volumes of 30 mL or greater, were 18 years and older, and underwent contrast-enhanced CT before surgical or angiographic intervention. Automated pelvic hematoma volume measurements were previously derived using a deep-learning quantitative visualization and measurement algorithm through cross-validation. A composite dependent variable of need for MT, AE, or PP was used as the primary endpoint. The added utility of hematoma volume was assessed by comparing the performance of multivariable models with and without hematoma volume as a predictor. Areas under the receiver operating characteristic curve (AUCs) and sensitivities, specificities, and predictive values were determined at clinically relevant thresholds. Adjusted odds ratios of automated pelvic hematoma volumes at 200 mL increments were derived. RESULTS: Median age was 47 years (interquartile range, 29-61), and 70% of patients were male. Median Injury Severity Score was 22 (14-36). Ninety-four percent of patients had injuries in other body regions, and 73% had polytrauma (Injury Severity Score, ≥16). Thirty-three percent had Tile/Orthopedic Trauma Association type B, and 24% had type C pelvic fractures. A total of 109 patients underwent AE, 22 underwent PP, and 53 received MT. A total of 123 patients received all 3 interventions. Sixteen patients died during hospitalization from causes other than untreatable (abbreviated injury scale, 6) head injury. Variables incorporated into multivariable models included age, sex, Tile/Orthopedic Trauma Association grade, admission lactate, heart rate (HR), and systolic blood pressure (SBP). Addition of hematoma volume resulted in a significant improvement in model performance, with AUC for the composite outcome (AE, PP, or MT) increasing from 0.74 to 0.83 (p < 0.001). Adjusted unit odds more than doubled for every additional 200 mL of hematoma volume. Increase in model AUC for mortality with incorporation of hematoma volume was not statistically significant (0.85 vs. 0.90, p = 0.12). CONCLUSION: Hematoma volumes measured using a rapid automated deep learning algorithm improved prediction of need for AE, PP, or MT. Simultaneous automated measurement of multiple sources of bleeding at CT could augment outcome prediction in trauma patients. LEVEL OF EVIDENCE: Diagnostic, level IV.

Recurrent Saliency Transformation Network for Tiny Target Segmentation in Abdominal CT Scans.

We aim at segmenting a wide variety of organs, including tiny targets (e.g., adrenal gland), and neoplasms (e.g., pancreatic cyst), from abdominal CT scans. This is a challenging task in two aspects. First, some organs (e.g., the pancreas), are highly variable in both anatomy and geometry, and thus very difficult to depict. Second, the neoplasms often vary a lot in its size, shape, as well as its location within the organ. Third, the targets (organs and neoplasms) can be considerably small compared to the human body, and so standard deep networks for segmentation are often less sensitive to these targets and thus predict less accurately especially around their boundaries. In this paper, we present an end-to-end framework named recurrent saliency transformation network (RSTN) for segmenting tiny and/or variable targets. The RSTN is a coarse-to-fine approach that uses prediction from the first (coarse) stage to shrink the input region for the second (fine) stage. A saliency transformation module is inserted between these two stages so that 1) the coarse-scaled segmentation mask can be transferred as spatial weights and applied to the fine stage and 2) the gradients can be back-propagated from the loss layer to the entire network so that the two stages are optimized in a joint manner. In the testing stage, we perform segmentation iteratively to improve accuracy. In this extended journal paper, we allow a gradual optimization to improve the stability of the RSTN, and introduce a hierarchical version named H-RSTN to segment tiny and variable neoplasms such as pancreatic cysts. Experiments are performed on several CT datasets including a public pancreas segmentation dataset, our own multi-organ dataset, and a cystic pancreas dataset. In all these cases, the RSTN outperforms the baseline (a stage-wise coarse-to-fine approach) significantly. Confirmed by the radiologists in our team, these promising segmentation results can help early diagnosis of pancreatic cancer. The code and pre-trained models of our project were made available at https://github.com/198808xc/OrganSegRSTN.

Performance of a Deep Learning Algorithm for Automated Segmentation and Quantification of Traumatic Pelvic Hematomas on CT.

The volume of pelvic hematoma at CT has been shown to be the strongest independent predictor of major arterial injury requiring angioembolization in trauma victims with pelvic fractures, and also correlates with transfusion requirement and mortality. Measurement of pelvic hematomas (unopacified extraperitoneal blood accumulated from time of injury) using semi-automated seeded region growing is time-consuming and requires trained experts, precluding routine measurement at the point of care. Pelvic hematomas are markedly variable in shape and location, have irregular ill-defined margins, have low contrast with respect to viscera and muscle, and reside within anatomically distorted pelvises. Furthermore, pelvic hematomas occupy a small proportion of the entire volume of a chest, abdomen, and pelvis (C/A/P) trauma CT. The challenges are many, and no automated methods for segmentation and volumetric analysis have been described to date. Traditional approaches using fully convolutional networks result in coarse segmentations and class imbalance with suboptimal convergence. In this study, we implement a modified coarse-to-fine deep learning approach-the Recurrent Saliency Transformation Network (RSTN) for pelvic hematoma volume segmentation. RSTN previously yielded excellent results in pancreas segmentation, where low contrast with adjacent structures, small target volume, variable location, and fine contours are also problematic. We have curated a unique single-institution corpus of 253 C/A/P admission trauma CT studies in patients with bleeding pelvic fractures with manually labeled pelvic hematomas. We hypothesized that RSTN would result in sufficiently high Dice similarity coefficients to facilitate accurate and objective volumetric measurements for outcome prediction (arterial injury requiring angioembolization). Cases were separated into five combinations of training and test sets in an 80/20 split and fivefold cross-validation was performed. Dice scores in the test set were 0.71 (SD ± 0.10) using RSTN, compared to 0.49 (SD ± 0.16) using a baseline Deep Learning Tool Kit (DLTK) reference 3D U-Net architecture. Mean inference segmentation time for RSTN was 0.90 min (± 0.26). Pearson correlation between predicted and manual labels was 0.95 with p < 0.0001. Measurement bias was within 10 mL. AUC of hematoma volumes for predicting need for angioembolization was 0.81 (predicted) versus 0.80 (manual). Qualitatively, predicted labels closely followed hematoma contours and avoided muscle and displaced viscera. Further work will involve validation using a federated dataset and incorporation into a predictive model using multiple segmented features.

Utility of CT Radiomics Features in Differentiation of Pancreatic Ductal Adenocarcinoma From Normal Pancreatic Tissue.

OBJECTIVE. The objective of our study was to determine the utility of radiomics features in differentiating CT cases of pancreatic ductal adenocarcinoma (PDAC) from normal pancreas. MATERIALS AND METHODS. In this retrospective case-control study, 190 patients with PDAC (97 men, 93 women; mean age ± SD, 66 ± 9 years) from 2012 to 2017 and 190 healthy potential renal donors (96 men, 94 women; mean age ± SD, 52 ± 8 years) without known pancreatic disease from 2005 to 2009 were identified from radiology and pathology databases. The 3D volume of the pancreas was manually segmented from the preoperative CT scans by four trained researchers and verified by three abdominal radiologists. Four hundred seventy-eight radiomics features were extracted to express the phenotype of the pancreas. Forty features were selected for analysis because of redundancy of computed features. The dataset was divided into 255 training cases (125 normal control cases and 130 PDAC cases) and 125 validation cases (65 normal control cases and 60 PDAC cases). A random forest classifier was used for binary classification of PDAC versus normal pancreas of control cases. Accuracy, sensitivity, and specificity were calculated. RESULTS. Mean tumor size was 4.1 ± 1.7 (SD) cm. The overall accuracy of the random forest binary classification was 99.2% (124/125), and AUC was 99.9%. All PDAC cases (60/60) were correctly classified. One case from a renal donor was misclassified as PDAC (1/65). The sensitivity was 100%, and specificity was 98.5%. CONCLUSION. Radiomics features extracted from whole pancreas can be used to differentiate between CT cases from patients with PDAC and healthy control subjects with normal pancreas.

Abdominal multi-organ segmentation with organ-attention networks and statistical fusion.

Accurate and robust segmentation of abdominal organs on CT is essential for many clinical applications such as computer-aided diagnosis and computer-aided surgery. But this task is challenging due to the weak boundaries of organs, the complexity of the background, and the variable sizes of different organs. To address these challenges, we introduce a novel framework for multi-organ segmentation of abdominal regions by using organ-attention networks with reverse connections (OAN-RCs) which are applied to 2D views, of the 3D CT volume, and output estimates which are combined by statistical fusion exploiting structural similarity. More specifically, OAN is a two-stage deep convolutional network, where deep network features from the first stage are combined with the original image, in a second stage, to reduce the complex background and enhance the discriminative information for the target organs. Intuitively, OAN reduces the effect of the complex background by focusing attention so that each organ only needs to be discriminated from its local background. RCs are added to the first stage to give the lower layers more semantic information thereby enabling them to adapt to the sizes of different organs. Our networks are trained on 2D views (slices) enabling us to use holistic information and allowing efficient computation (compared to using 3D patches). To compensate for the limited cross-sectional information of the original 3D volumetric CT, e.g., the connectivity between neighbor slices, multi-sectional images are reconstructed from the three different 2D view directions. Then we combine the segmentation results from the different views using statistical fusion, with a novel term relating the structural similarity of the 2D views to the original 3D structure. To train the network and evaluate results, 13 structures were manually annotated by four human raters and confirmed by a senior expert on 236 normal cases. We tested our algorithm by 4-fold cross-validation and computed Dice-Sørensen similarity coefficients (DSC) and surface distances for evaluating our estimates of the 13 structures. Our experiments show that the proposed approach gives strong results and outperforms 2D- and 3D-patch based state-of-the-art methods in terms of DSC and mean surface distances.

Detection and segmentation of pathological structures by the extended graph-shifts algorithm.

We propose an extended graph-shifts algorithm for image segmentation and labeling. This algorithm performs energy minimization by manipulating a dynamic hierarchical representation of the image. It consists of a set of moves occurring at different levels of the hierarchy where the types of move, and the level of the hierarchy, are chosen automatically so as to maximally decrease the energy. Extended graph-shifts can be applied to a broad range of problems in medical imaging. In this paper, we apply extended graph-shifts to the detection of pathological brain structures: (i) segmentation of brain tumors, and (ii) detection of multiple sclerosis lesions. The energy terms in these tasks are learned from training data by statistical learning algorithms. We demonstrate accurate results, precision and recall in the order of 93%, and also show that the algorithm is computationally efficient, segmenting a full 3D volume in about one minute.

Multilevel segmentation and integrated bayesian model classification with an application to brain tumor segmentation.

We present a new method for automatic segmentation of heterogeneous image data, which is very common in medical image analysis. The main contribution of the paper is a mathematical formulation for incorporating soft model assignments into the calculation of affinities, which are traditionally model free. We integrate the resulting model-aware affinities into the multilevel segmentation by weighted aggregation algorithm. We apply the technique to the task of detecting and segmenting brain tumor and edema in multimodal MR volumes. Our results indicate the benefit of incorporating model-aware affinities into the segmentation process for the difficult case of brain tumor.