Stress testing reveals gaps in clinic readiness of image-based diagnostic artificial intelligence models.

Artificial intelligence models match or exceed dermatologists in melanoma image classification. Less is known about their robustness against real-world variations, and clinicians may incorrectly assume that a model with an acceptable area under the receiver operating characteristic curve or related performance metric is ready for clinical use. Here, we systematically assessed the performance of dermatologist-level convolutional neural networks (CNNs) on real-world non-curated images by applying computational "stress tests". Our goal was to create a proxy environment in which to comprehensively test the generalizability of off-the-shelf CNNs developed without training or evaluation protocols specific to individual clinics. We found inconsistent predictions on images captured repeatedly in the same setting or subjected to simple transformations (e.g., rotation). Such transformations resulted in false positive or negative predictions for 6.5-22% of skin lesions across test datasets. Our findings indicate that models meeting conventionally reported metrics need further validation with computational stress tests to assess clinic readiness.

Molecular risk prediction in cutaneous melanoma: A meta-analysis of the 31-gene expression profile prognostic test in 1,479 patients.

BACKGROUND: Multiple studies have reported on the accuracy of the prognostic 31-gene expression profile test for cutaneous melanoma. Consistency of the test results across studies has not been systematically evaluated. OBJECTIVE: To assess the robustness of the prognostic value of the 31-gene expression profile. METHODS: Raw data were obtained from studies identified from systematic review. A meta-analysis was performed to determine overall effect of the 31-gene expression profile. Clinical outcome metrics for the 31-gene expression profile were compared with American Joint Committee on Cancer staging. RESULTS: Three studies met inclusion criteria; data from a novel cohort of 211 patients were included (n = 1,479). Five-year recurrence-free and distant metastasis-free survival rates were 91.4% and 94.1% for Class 1A patients and 43.6% and 55.5% for Class 2B patients (P < .0001). Meta-analysis results showed that Class 2 was significantly associated with recurrence (hazard ratio 2.90; P < .0001) and distant metastasis (hazard ratio 2.75; P < .0001). The 31-gene expression profile identified American Joint Committee on Cancer stage I to III patient subsets with high likelihood for recurrence and distant metastasis. Sensitivity was 76% (95% confidence interval 71%-80%) and 76% (95% confidence interval 70%-82%) for each end point, respectively. When 31-gene expression profile and sentinel lymph node biopsy results were considered together, sensitivity and negative predictive value for distant metastasis-free survival were both improved. CONCLUSION: The 31-gene expression profile test consistently and accurately identifies melanoma patients at increased risk of metastasis, is independent of other clinicopathologic covariates, and augments current risk stratification by reclassifying patients for heightened surveillance who were previously designated as being at low risk.