Pseudoprospective Paraclinical Interaction of Radiology Residents With a Deep Learning System for Prostate Cancer Detection: Experience, Performance, and Identification of the Need for Intermittent Recalibration.

OBJECTIVES: The aim of this study was to estimate the prospective utility of a previously retrospectively validated convolutional neural network (CNN) for prostate cancer (PC) detection on prostate magnetic resonance imaging (MRI). MATERIALS AND METHODS: The biparametric (T2-weighted and diffusion-weighted) portion of clinical multiparametric prostate MRI from consecutive men included between November 2019 and September 2020 was fully automatically and individually analyzed by a CNN briefly after image acquisition (pseudoprospective design). Radiology residents performed 2 research Prostate Imaging Reporting and Data System (PI-RADS) assessments of the multiparametric dataset independent from clinical reporting (paraclinical design) before and after review of the CNN results and completed a survey. Presence of clinically significant PC was determined by the presence of an International Society of Urological Pathology grade 2 or higher PC on combined targeted and extended systematic transperineal MRI/transrectal ultrasound fusion biopsy. Sensitivities and specificities on a patient and prostate sextant basis were compared using the McNemar test and compared with the receiver operating characteristic (ROC) curve of CNN. Survey results were summarized as absolute counts and percentages. RESULTS: A total of 201 men were included. The CNN achieved an ROC area under the curve of 0.77 on a patient basis. Using PI-RADS ≥3-emulating probability threshold (c3), CNN had a patient-based sensitivity of 81.8% and specificity of 54.8%, not statistically different from the current clinical routine PI-RADS ≥4 assessment at 90.9% and 54.8%, respectively ( P = 0.30/ P = 1.0). In general, residents achieved similar sensitivity and specificity before and after CNN review. On a prostate sextant basis, clinical assessment possessed the highest ROC area under the curve of 0.82, higher than CNN (AUC = 0.76, P = 0.21) and significantly higher than resident performance before and after CNN review (AUC = 0.76 / 0.76, P ≤ 0.03). The resident survey indicated CNN to be helpful and clinically useful. CONCLUSIONS: Pseudoprospective paraclinical integration of fully automated CNN-based detection of suspicious lesions on prostate multiparametric MRI was demonstrated and showed good acceptance among residents, whereas no significant improvement in resident performance was found. General CNN performance was preserved despite an observed shift in CNN calibration, identifying the requirement for continuous quality control and recalibration.

Improvement of PI-RADS-dependent prostate cancer classification by quantitative image assessment using radiomics or mean ADC.

Background Currently, interpretation of prostate MRI is performed qualitatively. Quantitative assessment of the mean apparent diffusion coefficient (mADC) is promising to improve diagnostic accuracy while radiomic machine learning (RML) allows to probe complex parameter spaces to identify the most promising multi-parametric models. We have previously developed quantitative RML and ADC classifiers for prediction of clinically significant prostate cancer (sPC) from prostate MRI, however these have not been combined with radiologist PI-RADS assessment. Purpose To propose and evaluate diagnostic algorithms combining quantitative ADC or RML and qualitative PI-RADS assessment for prediction of sPC. Methods and population The previously published quantitative models (RML and mADC) were utilized to construct four algorithms: 1) Down(ADC) and 2) Down(RML): clinically detected PI-RADS positive prostate lesions (defined as either PI-RADS≥3 or ≥4) were downgraded to MRI negative upon negative quantitative assessment; and 3) Up(ADC) and 4) Up(RML): MRI-negative lesions were upgraded to MRI-positive upon positive assessment of quantitative parameters. Analyses were performed at the individual lesion level and the patient level in 133 consecutive patients with suspicion for clinically significant prostate cancer (sPC, International Society of Urological Pathology (ISUP) grade group≥2), the test set subcohort of a previously published patient population. McNemar test was used to compare differences in sensitivity, specificity and accuracy. Differences between lesions of different prostate zones were assessed using ANOVA. Reduction in false positive assessments was assessed as ratios. Results Compared to clinical assessment at the PI-RADS≥4 cut-off alone, algorithms Down(ADC/RML) improved specificity from 43% to 65% (p = 0.001)/62% (p = 0.003), while sensitivity did not change significantly at 89% compared to 87% (p = 1.0)/89% (unchanged) on the patient level. Reduction of false positive lesions was 50% [26/52] in the PZ and 53% [15/28] in the TZ. Algorithms Up(ADC/RML) led, on a patient basis, to an unfavorable loss of specificity from 43% to 30% (p = 0.039)/32% (p = 0.106), with insignificant increase of sensitivity from 89% to 96%/96% (both p = 1.0). Compared to clinical assessment at the PI-RADS≥3 cut-off alone, similar results were observed for Down(ADC) with significantly increased specificity from 2% to 23% (p < 0.001) and unchanged sensitivity on the lesion level; patient level specificity increased only non-significantly. Conclusion Downgrading PI-RADS≥3 and ≥ 4 lesions based on quantitative mADC measurements or RML classifiers can increase diagnostic accuracy by enhancing specificity and preserving sensitivity for detection of sPC and reduce false positives.

Fully Automatic Deep Learning in Bi-institutional Prostate Magnetic Resonance Imaging: Effects of Cohort Size and Heterogeneity.

BACKGROUND: The potential of deep learning to support radiologist prostate magnetic resonance imaging (MRI) interpretation has been demonstrated. PURPOSE: The aim of this study was to evaluate the effects of increased and diversified training data (TD) on deep learning performance for detection and segmentation of clinically significant prostate cancer-suspicious lesions. MATERIALS AND METHODS: In this retrospective study, biparametric (T2-weighted and diffusion-weighted) prostate MRI acquired with multiple 1.5-T and 3.0-T MRI scanners in consecutive men was used for training and testing of prostate segmentation and lesion detection networks. Ground truth was the combination of targeted and extended systematic MRI-transrectal ultrasound fusion biopsies, with significant prostate cancer defined as International Society of Urological Pathology grade group greater than or equal to 2. U-Nets were internally validated on full, reduced, and PROSTATEx-enhanced training sets and subsequently externally validated on the institutional test set and the PROSTATEx test set. U-Net segmentation was calibrated to clinically desired levels in cross-validation, and test performance was subsequently compared using sensitivities, specificities, predictive values, and Dice coefficient. RESULTS: One thousand four hundred eighty-eight institutional examinations (median age, 64 years; interquartile range, 58-70 years) were temporally split into training (2014-2017, 806 examinations, supplemented by 204 PROSTATEx examinations) and test (2018-2020, 682 examinations) sets. In the test set, Prostate Imaging-Reporting and Data System (PI-RADS) cutoffs greater than or equal to 3 and greater than or equal to 4 on a per-patient basis had sensitivity of 97% (241/249) and 90% (223/249) at specificity of 19% (82/433) and 56% (242/433), respectively. The full U-Net had corresponding sensitivity of 97% (241/249) and 88% (219/249) with specificity of 20% (86/433) and 59% (254/433), not statistically different from PI-RADS (P > 0.3 for all comparisons). U-Net trained using a reduced set of 171 consecutive examinations achieved inferior performance (P < 0.001). PROSTATEx training enhancement did not improve performance. Dice coefficients were 0.90 for prostate and 0.42/0.53 for MRI lesion segmentation at PI-RADS category 3/4 equivalents. CONCLUSIONS: In a large institutional test set, U-Net confirms similar performance to clinical PI-RADS assessment and benefits from more TD, with neither institutional nor PROSTATEx performance improved by adding multiscanner or bi-institutional TD.

Comparison of Prostate MRI Lesion Segmentation Agreement Between Multiple Radiologists and a Fully Automatic Deep Learning System.

PURPOSE: A recently developed deep learning model (U-Net) approximated the clinical performance of radiologists in the prediction of clinically significant prostate cancer (sPC) from prostate MRI. Here, we compare the agreement between lesion segmentations by U-Net with manual lesion segmentations performed by different radiologists. MATERIALS AND METHODS: 165 patients with suspicion for sPC underwent targeted and systematic fusion biopsy following 3 Tesla multiparametric MRI (mpMRI). Five sets of segmentations were generated retrospectively: segmentations of clinical lesions, independent segmentations by three radiologists, and fully automated bi-parametric U-Net segmentations. Per-lesion agreement was calculated for each rater by averaging Dice coefficients with all overlapping lesions from other raters. Agreement was compared using descriptive statistics and linear mixed models. RESULTS: The mean Dice coefficient for manual segmentations showed only moderate agreement at 0.48-0.52, reflecting the difficult visual task of determining the outline of otherwise jointly detected lesions. U-net segmentations were significantly smaller than manual segmentations (p < 0.0001) and exhibited a lower mean Dice coefficient of 0.22, which was significantly lower compared to manual segmentations (all p < 0.0001). These differences remained after correction for lesion size and were unaffected between sPC and non-sPC lesions and between peripheral and transition zone lesions. CONCLUSION: Knowledge of the order of agreement of manual segmentations of different radiologists is important to set the expectation value for artificial intelligence (AI) systems in the task of prostate MRI lesion segmentation. Perfect agreement (Dice coefficient of one) should not be expected for AI. Lower Dice coefficients of U-Net compared to manual segmentations are only partially explained by smaller segmentation sizes and may result from a focus on the lesion core and a small relative lesion center shift. Although it is primarily important that AI detects sPC correctly, the Dice coefficient for overlapping lesions from multiple raters can be used as a secondary measure for segmentation quality in future studies. KEY POINTS: · Intermediate human Dice coefficients reflect the difficulty of outlining jointly detected lesions.. · Lower Dice coefficients of deep learning motivate further research to approximate human perception.. · Comparable predictive performance of deep learning appears independent of Dice agreement.. · Dice agreement independent of significant cancer presence indicates indistinguishability of some benign imaging findings.. · Improving DWI to T2 registration may improve the observed U-Net Dice coefficients.. CITATION FORMAT: · Schelb P, Tavakoli AA, Tubtawee T et al. Comparison of Prostate MRI Lesion Segmentation Agreement Between Multiple Radiologists and a Fully Automatic Deep Learning System. Fortschr Röntgenstr 2021; 193: 559 - 573.

Simulated clinical deployment of fully automatic deep learning for clinical prostate MRI assessment.

OBJECTIVES: To simulate clinical deployment, evaluate performance, and establish quality assurance of a deep learning algorithm (U-Net) for detection, localization, and segmentation of clinically significant prostate cancer (sPC), ISUP grade group ≥ 2, using bi-parametric MRI. METHODS: In 2017, 284 consecutive men in active surveillance, biopsy-naïve or pre-biopsied, received targeted and extended systematic MRI/transrectal US-fusion biopsy, after examination on a single MRI scanner (3 T). A prospective adjustment scheme was evaluated comparing the performance of the Prostate Imaging Reporting and Data System (PI-RADS) and U-Net using sensitivity, specificity, predictive values, and the Dice coefficient. RESULTS: In the 259 eligible men (median 64 [IQR 61-72] years), PI-RADS had a sensitivity of 98% [106/108]/84% [91/108] with a specificity of 17% [25/151]/58% [88/151], for thresholds at ≥ 3/≥ 4 respectively. U-Net using dynamic threshold adjustment had a sensitivity of 99% [107/108]/83% [90/108] (p > 0.99/> 0.99) with a specificity of 24% [36/151]/55% [83/151] (p > 0.99/> 0.99) for probability thresholds d3 and d4 emulating PI-RADS ≥ 3 and ≥ 4 decisions respectively, not statistically different from PI-RADS. Co-occurrence of a radiological PI-RADS ≥ 4 examination and U-Net ≥ d3 assessment significantly improved the positive predictive value from 59 to 63% (p = 0.03), on a per-patient basis. CONCLUSIONS: U-Net has similar performance to PI-RADS in simulated continued clinical use. Regular quality assurance should be implemented to ensure desired performance. KEY POINTS: • U-Net maintained similar diagnostic performance compared to radiological assessment of PI-RADS ≥ 4 when applied in a simulated clinical deployment. • Application of our proposed prospective dynamic calibration method successfully adjusted U-Net performance within acceptable limits of the PI-RADS reference over time, while not being limited to PI-RADS as a reference. • Simultaneous detection by U-Net and radiological assessment significantly improved the positive predictive value on a per-patient and per-lesion basis, while the negative predictive value remained unchanged.

Classification of Cancer at Prostate MRI: Deep Learning versus Clinical PI-RADS Assessment.

Background Men suspected of having clinically significant prostate cancer (sPC) increasingly undergo prostate MRI. The potential of deep learning to provide diagnostic support for human interpretation requires further evaluation. Purpose To compare the performance of clinical assessment to a deep learning system optimized for segmentation trained with T2-weighted and diffusion MRI in the task of detection and segmentation of lesions suspicious for sPC. Materials and Methods In this retrospective study, T2-weighted and diffusion prostate MRI sequences from consecutive men examined with a single 3.0-T MRI system between 2015 and 2016 were manually segmented. Ground truth was provided by combined targeted and extended systematic MRI-transrectal US fusion biopsy, with sPC defined as International Society of Urological Pathology Gleason grade group greater than or equal to 2. By using split-sample validation, U-Net was internally validated on the training set (80% of the data) through cross validation and subsequently externally validated on the test set (20% of the data). U-Net-derived sPC probability maps were calibrated by matching sextant-based cross-validation performance to clinical performance of Prostate Imaging Reporting and Data System (PI-RADS). Performance of PI-RADS and U-Net were compared by using sensitivities, specificities, predictive values, and Dice coefficient. Results A total of 312 men (median age, 64 years; interquartile range [IQR], 58-71 years) were evaluated. The training set consisted of 250 men (median age, 64 years; IQR, 58-71 years) and the test set of 62 men (median age, 64 years; IQR, 60-69 years). In the test set, PI-RADS cutoffs greater than or equal to 3 versus cutoffs greater than or equal to 4 on a per-patient basis had sensitivity of 96% (25 of 26) versus 88% (23 of 26) at specificity of 22% (eight of 36) versus 50% (18 of 36). U-Net at probability thresholds of greater than or equal to 0.22 versus greater than or equal to 0.33 had sensitivity of 96% (25 of 26) versus 92% (24 of 26) (both P > .99) with specificity of 31% (11 of 36) versus 47% (17 of 36) (both P > .99), not statistically different from PI-RADS. Dice coefficients were 0.89 for prostate and 0.35 for MRI lesion segmentation. In the test set, coincidence of PI-RADS greater than or equal to 4 with U-Net lesions improved the positive predictive value from 48% (28 of 58) to 67% (24 of 36) for U-Net probability thresholds greater than or equal to 0.33 (P = .01), while the negative predictive value remained unchanged (83% [25 of 30] vs 83% [43 of 52]; P > .99). Conclusion U-Net trained with T2-weighted and diffusion MRI achieves similar performance to clinical Prostate Imaging Reporting and Data System assessment. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Padhani and Turkbey in this issue.

Histopathological to multiparametric MRI spatial mapping of extended systematic sextant and MR/TRUS-fusion-targeted biopsy of the prostate.

PURPOSE: MRI has limited ability to detect multifocal disease or the full extent of prostate involvement with clinically significant prostate cancer (sPC). We compare the spatial co-localization at sextant resolution of MRI lesions and histopathological mapping by combined targeted and extended systematic biopsies. MATERIALS AND METHODS: Sextants were mapped for sPC (ISUP group ≥ 2) by 24-core transperineal systematic biopsies in 316 patients with suspicion for sPC and by MR lesions of PI-RADS score of ≥ 3. The gold standard is combined systematic (median 23 cores) and targeted biopsies. RESULTS: Of 316 men, 121 (38%) harbored sPC. Of these 121 patients, 4 (3%) had a negative MRI. MRI correctly identified 117/121 (97%) patients with sPC. In these patients, mpMRI missed no additional sPC in 96 (82%), while MRI-negative sPC lesions were present in 21 patients (18%). Of 1896 sextants, 379 (20%) harbored sPC. MR-positive sextants contained sPC in 26% (337/1275), compared to 7% (42/621) in MR-negative sextants. On a patient basis, sensitivity was 0.97, specificity 0.22, positive predictive value 0.43, and negative predictive value 0.91. On a sextant basis, sensitivity was 0.73, specificity 0.38, positive predictive value 0.26, and negative predictive value 0.93. CONCLUSION: MpMRI mapping agreed well with histopathology with, at the observed sPC prevalence and on a patient basis, excellent sensitivity and negative predictive value, and acceptable specificity and positive predictive value for sPC. However, 18% of sPC was outside the mpMRI mapped region, quantifying limitations of MRI for complete localization of disease extent. KEY POINTS: • Currently, exclusive MRI mapping of the prostate for focal treatment planning cannot be recommended, as significant prostate cancer may remain untreated in a substantial number of cases. • At the observed sPC prevalence and on a patient basis, mpMRI has excellent sensitivity and NPV, and acceptable specificity and PPV for detection of prostate cancer, supporting its use to detect suspicious lesions before biopsy. • Despite the excellent global performance, 18% of sPC was outside the mpMRI mapped region even when a security margin of 10 mm was considered, indicating that prostate MRI has limited ability to completely map all cancer foci within the prostate.

Radiomic Machine Learning for Characterization of Prostate Lesions with MRI: Comparison to ADC Values.

Purpose To compare biparametric contrast-free radiomic machine learning (RML), mean apparent diffusion coefficient (ADC), and radiologist assessment for characterization of prostate lesions detected during prospective MRI interpretation. Materials and Methods This single-institution study included 316 men (mean age ± standard deviation, 64.0 years ± 7.8) with an indication for MRI-transrectal US fusion biopsy between May 2015 and September 2016 (training cohort, 183 patients; test cohort, 133 patients). Lesions identified by prospective clinical readings were manually segmented for mean ADC and radiomics analysis. Global and zone-specific random forest RML and mean ADC models for classification of clinically significant prostate cancer (Gleason grade group ≥ 2) were developed on the training set and the fixed models tested on an independent test set. Clinical readings, mean ADC, and radiomics were compared by using the McNemar test and receiver operating characteristic (ROC) analysis. Results In the test set, radiologist interpretation had a per-lesion sensitivity of 88% (53 of 60) and specificity of 50% (79 of 159). Quantitative measurement of the mean ADC (cut-off 732 mm2/sec) significantly reduced false-positive (FP) lesions from 80 to 60 (specificity 62% [99 of 159]) and false-negative (FN) lesions from seven to six (sensitivity 90% [54 of 60]) (P = .048). Radiologist interpretation had a per-patient sensitivity of 89% (40 of 45) and specificity of 43% (38 of 88). Quantitative measurement of the mean ADC reduced the number of patients with FP lesions from 50 to 43 (specificity 51% [45 of 88]) and the number of patients with FN lesions from five to three (sensitivity 93% [42 of 45]) (P = .496). Comparison of the area under the ROC curve (AUC) for the mean ADC (AUCglobal = 0.84; AUCzone-specific ≤ 0.87) vs the RML (AUCglobal = 0.88, P = .176; AUCzone-specific ≤ 0.89, P ≥ .493) showed no significantly different performance. Conclusion Quantitative measurement of the mean apparent diffusion coefficient (ADC) improved differentiation of benign versus malignant prostate lesions, compared with clinical assessment. Radiomic machine learning had comparable but not better performance than mean ADC assessment. © RSNA, 2018 Online supplemental material is available for this article.