Expert-centered Evaluation of Deep Learning Algorithms for Brain Tumor Segmentation.

Purpose To present results from a literature survey on practices in deep learning segmentation algorithm evaluation and perform a study on expert quality perception of brain tumor segmentation. Materials and Methods A total of 180 articles reporting on brain tumor segmentation algorithms were surveyed for the reported quality evaluation. Additionally, ratings of segmentation quality on a four-point scale were collected from medical professionals for 60 brain tumor segmentation cases. Results Of the surveyed articles, Dice score, sensitivity, and Hausdorff distance were the most popular metrics to report segmentation performance. Notably, only 2.8% of the articles included clinical experts' evaluation of segmentation quality. The experimental results revealed a low interrater agreement (Krippendorff α, 0.34) in experts' segmentation quality perception. Furthermore, the correlations between the ratings and commonly used quantitative quality metrics were low (Kendall tau between Dice score and mean rating, 0.23; Kendall tau between Hausdorff distance and mean rating, 0.51), with large variability among the experts. Conclusion The results demonstrate that quality ratings are prone to variability due to the ambiguity of tumor boundaries and individual perceptual differences, and existing metrics do not capture the clinical perception of segmentation quality. Keywords: Brain Tumor Segmentation, Deep Learning Algorithms, Glioblastoma, Cancer, Machine Learning Clinical trial registration nos. NCT00756106 and NCT00662506 Supplemental material is available for this article. © RSNA, 2023.

Not without Context-A Multiple Methods Study on Evaluation and Correction of Automated Brain Tumor Segmentations by Experts.

RATIONALE AND OBJECTIVES: Brain tumor segmentations are integral to the clinical management of patients with glioblastoma, the deadliest primary brain tumor in adults. The manual delineation of tumors is time-consuming and highly provider-dependent. These two problems must be addressed by introducing automated, deep-learning-based segmentation tools. This study aimed to identify criteria experts use to evaluate the quality of automatically generated segmentations and their thought processes as they correct them. MATERIALS AND METHODS: Multiple methods were used to develop a detailed understanding of the complex factors that shape experts' perception of segmentation quality and their thought processes in correcting proposed segmentations. Data from a questionnaire and semistructured interview with neuro-oncologists and neuroradiologists were collected between August and December 2021 and analyzed using a combined deductive and inductive approach. RESULTS: Brain tumors are highly complex and ambiguous segmentation targets. Therefore, physicians rely heavily on the given context related to the patient and clinical context in evaluating the quality and need to correct brain tumor segmentation. Most importantly, the intended clinical application determines the segmentation quality criteria and editing decisions. Physicians' personal beliefs and preferences about the capabilities of AI algorithms and whether questionable areas should not be included are additional criteria influencing the perception of segmentation quality and appearance of an edited segmentation. CONCLUSION: Our findings on experts' perceptions of segmentation quality will allow the design of improved frameworks for expert-centered evaluation of brain tumor segmentation models. In particular, the knowledge presented here can inspire the development of brain tumor-specific metrics for segmentation model training and evaluation.

Abnormal vascular structure and function within brain metastases is linked to pembrolizumab resistance.

BACKGROUND: We recently conducted a phase 2 trial (NCT028865685) evaluating intracranial efficacy of pembrolizumab for brain metastases (BM) of diverse histologies. Our study met its primary efficacy endpoint and illustrates that pembrolizumab exerts promising activity in a select group of patients with BM. Given the importance of aberrant vasculature in mediating immunosuppression, we explored the relationship between checkpoint inhibitor (ICI) efficacy and vascular architecture in the hopes of identifying potential mechanisms of intracranial ICI response or resistance for BM. METHODS: Using Vessel Architectural Imaging (VAI), a histologically validated quantitative metric for in vivo tumor vascular physiology, we analyzed dual echo DSC/DCE MRI for 44 patients on trial. Tumor and peri-tumor cerebral blood volume/flow, vessel size, arterial- and venous-dominance, and vascular permeability were measured before and after treatment with pembrolizumab. RESULTS: BM that progressed on ICI were characterized by a highly aberrant vasculature dominated by large-caliber vessels. In contrast, ICI-responsive BM possessed a more structurally balanced vasculature consisting of both small and large vessels, and there was a trend towards a decrease in under-perfused tissue, suggesting a reversal of the negative effects of hypoxia. In the peri-tumor region, development of smaller blood vessels, consistent with neo-angiogenesis, was associated with tumor growth before radiographic evidence of contrast enhancement on anatomical MRI. CONCLUSIONS: This study, one of the largest functional imaging studies for BM, suggests that vascular architecture is linked with ICI efficacy. Studies identifying modulators of vascular architecture, and effects on immune activity, are warranted and may inform future combination treatments.

Structural and functional vascular dysfunction within brain metastases is linked to pembrolizumab inefficacy.

Structurally and functionally aberrant vasculature is a hallmark of tumor angiogenesis and treatment resistance. Given the synergistic link between aberrant tumor vasculature and immunosuppression, we analyzed perfusion MRI for 44 patients with brain metastases (BM) undergoing treatment with pembrolizumab. To date, vascular-immune communication, or the relationship between immune checkpoint inhibitor (ICI) efficacy and vascular architecture, has not been well-characterized in human imaging studies. We found that ICI-responsive BM possessed a structurally balanced vascular makeup, which was linked to improved vascular efficiency and an immune-stimulatory microenvironment. In contrast, ICI-resistant BM were characterized by a lack of immune cell infiltration and a highly aberrant vasculature dominated by large-caliber vessels. Peri-tumor region analysis revealed early functional changes predictive of ICI resistance before radiographic evidence on conventional MRI. This study was one of the largest functional imaging studies for BM and establishes a foundation for functional studies that illuminate the mechanisms linking patterns of vascular architecture with immunosuppression, as targeting these aspects of cancer biology may serve as the basis for future combination treatments.

FDU-Net: Deep Learning-Based Three-Dimensional Diffuse Optical Image Reconstruction.

Near-infrared diffuse optical tomography (DOT) is a promising functional modality for breast cancer imaging; however, the clinical translation of DOT is hampered by technical limitations. Specifically, conventional finite element method (FEM)-based optical image reconstruction approaches are time-consuming and ineffective in recovering full lesion contrast. To address this, we developed a deep learning-based reconstruction model (FDU-Net) comprised of a Fully connected subnet, followed by a convolutional encoder-Decoder subnet, and a U-Net for fast, end-to-end 3D DOT image reconstruction. The FDU-Net was trained on digital phantoms that include randomly located singular spherical inclusions of various sizes and contrasts. Reconstruction performance was evaluated in 400 simulated cases with realistic noise profiles for the FDU-Net and conventional FEM approaches. Our results show that the overall quality of images reconstructed by FDU-Net is significantly improved compared to FEM-based methods and a previously proposed deep-learning network. Importantly, once trained, FDU-Net demonstrates substantially better capability to recover true inclusion contrast and location without using any inclusion information during reconstruction. The model was also generalizable to multi-focal and irregularly shaped inclusions unseen during training. Finally, FDU-Net, trained on simulated data, could successfully reconstruct a breast tumor from a real patient measurement. Overall, our deep learning-based approach demonstrates marked superiority over the conventional DOT image reconstruction methods while also offering over four orders of magnitude acceleration in computational time. Once adapted to the clinical breast imaging workflow, FDU-Net has the potential to provide real-time accurate lesion characterization by DOT to assist the clinical diagnosis and management of breast cancer.

Improving the repeatability of deep learning models with Monte Carlo dropout.

The integration of artificial intelligence into clinical workflows requires reliable and robust models. Repeatability is a key attribute of model robustness. Ideal repeatable models output predictions without variation during independent tests carried out under similar conditions. However, slight variations, though not ideal, may be unavoidable and acceptable in practice. During model development and evaluation, much attention is given to classification performance while model repeatability is rarely assessed, leading to the development of models that are unusable in clinical practice. In this work, we evaluate the repeatability of four model types (binary classification, multi-class classification, ordinal classification, and regression) on images that were acquired from the same patient during the same visit. We study the each model's performance on four medical image classification tasks from public and private datasets: knee osteoarthritis, cervical cancer screening, breast density estimation, and retinopathy of prematurity. Repeatability is measured and compared on ResNet and DenseNet architectures. Moreover, we assess the impact of sampling Monte Carlo dropout predictions at test time on classification performance and repeatability. Leveraging Monte Carlo predictions significantly increases repeatability, in particular at the class boundaries, for all tasks on the binary, multi-class, and ordinal models leading to an average reduction of the 95% limits of agreement by 16% points and of the class disagreement rate by 7% points. The classification accuracy improves in most settings along with the repeatability. Our results suggest that beyond about 20 Monte Carlo iterations, there is no further gain in repeatability. In addition to the higher test-retest agreement, Monte Carlo predictions are better calibrated which leads to output probabilities reflecting more accurately the true likelihood of being correctly classified.

QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results.

Deep learning (DL) models have provided state-of-the-art performance in various medical imaging benchmarking challenges, including the Brain Tumor Segmentation (BraTS) challenges. However, the task of focal pathology multi-compartment segmentation (e.g., tumor and lesion sub-regions) is particularly challenging, and potential errors hinder translating DL models into clinical workflows. Quantifying the reliability of DL model predictions in the form of uncertainties could enable clinical review of the most uncertain regions, thereby building trust and paving the way toward clinical translation. Several uncertainty estimation methods have recently been introduced for DL medical image segmentation tasks. Developing scores to evaluate and compare the performance of uncertainty measures will assist the end-user in making more informed decisions. In this study, we explore and evaluate a score developed during the BraTS 2019 and BraTS 2020 task on uncertainty quantification (QU-BraTS) and designed to assess and rank uncertainty estimates for brain tumor multi-compartment segmentation. This score (1) rewards uncertainty estimates that produce high confidence in correct assertions and those that assign low confidence levels at incorrect assertions, and (2) penalizes uncertainty measures that lead to a higher percentage of under-confident correct assertions. We further benchmark the segmentation uncertainties generated by 14 independent participating teams of QU-BraTS 2020, all of which also participated in the main BraTS segmentation task. Overall, our findings confirm the importance and complementary value that uncertainty estimates provide to segmentation algorithms, highlighting the need for uncertainty quantification in medical image analyses. Finally, in favor of transparency and reproducibility, our evaluation code is made publicly available at https://github.com/RagMeh11/QU-BraTS.

Radiomics Repeatability Pitfalls in a Scan-Rescan MRI Study of Glioblastoma.

PURPOSE: To determine the influence of preprocessing on the repeatability and redundancy of radiomics features extracted using a popular open-source radiomics software package in a scan-rescan glioblastoma MRI study. MATERIALS AND METHODS: In this study, a secondary analysis of T2-weighted fluid-attenuated inversion recovery (FLAIR) and T1-weighted postcontrast images from 48 patients (mean age, 56 years [range, 22-77 years]) diagnosed with glioblastoma were included from two prospective studies (ClinicalTrials.gov NCT00662506 [2009-2011] and NCT00756106 [2008-2011]). All patients underwent two baseline scans 2-6 days apart using identical imaging protocols on 3-T MRI systems. No treatment occurred between scan and rescan, and tumors were essentially unchanged visually. Radiomic features were extracted by using PyRadiomics (https://pyradiomics.readthedocs.io/) under varying conditions, including normalization strategies and intensity quantization. Subsequently, intraclass correlation coefficients were determined between feature values of the scan and rescan. RESULTS: Shape features showed a higher repeatability than intensity (adjusted P < .001) and texture features (adjusted P < .001) for both T2-weighted FLAIR and T1-weighted postcontrast images. Normalization improved the overlap between the region of interest intensity histograms of scan and rescan (adjusted P < .001 for both T2-weighted FLAIR and T1-weighted postcontrast images), except in scans where brain extraction fails. As such, normalization significantly improves the repeatability of intensity features from T2-weighted FLAIR scans (adjusted P = .003 [z score normalization] and adjusted P = .002 [histogram matching]). The use of a relative intensity binning strategy as opposed to default absolute intensity binning reduces correlation between gray-level co-occurrence matrix features after normalization. CONCLUSION: Both normalization and intensity quantization have an effect on the level of repeatability and redundancy of features, emphasizing the importance of both accurate reporting of methodology in radiomics articles and understanding the limitations of choices made in pipeline design. Supplemental material is available for this article. © RSNA, 2020See also the commentary by Tiwari and Verma in this issue.

Multi-Institutional Assessment and Crowdsourcing Evaluation of Deep Learning for Automated Classification of Breast Density.

OBJECTIVE: We developed deep learning algorithms to automatically assess BI-RADS breast density. METHODS: Using a large multi-institution patient cohort of 108,230 digital screening mammograms from the Digital Mammographic Imaging Screening Trial, we investigated the effect of data, model, and training parameters on overall model performance and provided crowdsourcing evaluation from the attendees of the ACR 2019 Annual Meeting. RESULTS: Our best-performing algorithm achieved good agreement with radiologists who were qualified interpreters of mammograms, with a four-class κ of 0.667. When training was performed with randomly sampled images from the data set versus sampling equal number of images from each density category, the model predictions were biased away from the low-prevalence categories such as extremely dense breasts. The net result was an increase in sensitivity and a decrease in specificity for predicting dense breasts for equal class compared with random sampling. We also found that the performance of the model degrades when we evaluate on digital mammography data formats that differ from the one that we trained on, emphasizing the importance of multi-institutional training sets. Lastly, we showed that crowdsourced annotations, including those from attendees who routinely read mammograms, had higher agreement with our algorithm than with the original interpreting radiologists. CONCLUSION: We demonstrated the possible parameters that can influence the performance of the model and how crowdsourcing can be used for evaluation. This study was performed in tandem with the development of the ACR AI-LAB, a platform for democratizing artificial intelligence.

Machine Learning Models can Detect Aneurysm Rupture and Identify Clinical Features Associated with Rupture.

BACKGROUND: Machine learning (ML) has been increasingly used in medicine and neurosurgery. We sought to determine whether ML models can distinguish ruptured from unruptured aneurysms and identify features associated with rupture. METHODS: We performed a retrospective review of patients with intracranial aneurysms detected on vascular imaging at our institution between 2002 and 2018. The dataset was used to train 3 ML models (random forest, linear support vector machine [SVM], and radial basis function kernel SVM). Relative contributions of individual predictors were derived from the linear SVM model. RESULTS: Complete data were available for 845 aneurysms in 615 patients. Ruptured aneurysms (n = 309, 37%) were larger (mean 6.51 mm vs. 5.73 mm; P = 0.02) and more likely to be in the posterior circulation (20% vs. 11%; P < 0.001) than unruptured aneurysms. Area under the receiver operating curve was 0.77 for the linear SVM, 0.78 for the radial basis function kernel SVM models, and 0.81 for the random forest model. Aneurysm location and size were the 2 features that contributed most significantly to the model. Posterior communicating artery, anterior communicating artery, and posterior inferior cerebellar artery locations were most highly associated with rupture, whereas paraclinoid and middle cerebral artery locations had the strongest association with unruptured status. CONCLUSIONS: ML models are capable of accurately distinguishing ruptured from unruptured aneurysms and identifying features associated with rupture. Consistent with prior studies, location and size show the strongest association with aneurysm rupture.

Balloon catheter-based radiofrequency ablation monitoring in porcine esophagus using optical coherence tomography.

We present a microscopic image guidance platform for radiofrequency ablation (RFA) using a clinical balloon-catheter-based optical coherence tomography (OCT) system, currently used in the surveillance of Barrett's esophagus patients. Our integrated thermal therapy delivery and monitoring platform consists of a flexible, customized bipolar RFA electrode array designed for use with a clinical balloon OCT catheter and a processing algorithm to accurately map the thermal coagulation process. Non-uniform rotation distortion was corrected using a feature tracking-based technique, which enables robust, frame-to-frame analysis of the temporal fluctuation of the complex OCT signal. With proper noise calibration, precise delineation of the thermal therapy zone was demonstrated using cumulative complex differential variance in porcine esophagus ex vivo with the integrated OCT-RFA system, as validated by nitroblue tetrazolium chloride (NBTC) histology. The ability to directly and accurately visualize the thermal coagulation process at high resolution is critical to the precise delivery of thermal energy to a wide range of epithelial lesions.