METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII.

PURPOSE: To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies. METHODS: We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated. RESULT: In total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community. CONCLUSION: In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers. CRITICAL RELEVANCE STATEMENT: A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning. KEY POINTS: • A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol. • The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time. • METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines. • A web application has been developed to help with the calculation of the METRICS score ( https://metricsscore.github.io/metrics/METRICS.html ) and a repository created to collect feedback from the radiomics community ( https://github.com/metricsscore/metrics ).

Artificial Intelligence-based Quantification of Pleural Plaque Volume and Association With Lung Function in Asbestos-exposed Patients.

PURPOSE: Pleural plaques (PPs) are morphologic manifestations of long-term asbestos exposure. The relationship between PP and lung function is not well understood, whereas the time-consuming nature of PP delineation to obtain volume impedes research. To automate the laborious task of delineation, we aimed to develop automatic artificial intelligence (AI)-driven segmentation of PP. Moreover, we aimed to explore the relationship between pleural plaque volume (PPV) and pulmonary function tests. MATERIALS AND METHODS: Radiologists manually delineated PPs retrospectively in computed tomography (CT) images of patients with occupational exposure to asbestos (May 2014 to November 2019). We trained an AI model with a no-new-UNet architecture. The Dice Similarity Coefficient quantified the overlap between AI and radiologists. The Spearman correlation coefficient ( r ) was used for the correlation between PPV and pulmonary function test metrics. When recorded, these were vital capacity (VC), forced vital capacity (FVC), and diffusing capacity for carbon monoxide (DLCO). RESULTS: We trained the AI system on 422 CT scans in 5 folds, each time with a different fold (n = 84 to 85) as a test set. On these independent test sets combined, the correlation between the predicted volumes and the ground truth was r = 0.90, and the median overlap was 0.71 Dice Similarity Coefficient. We found weak to moderate correlations with PPV for VC (n = 80, r = -0.40) and FVC (n = 82, r = -0.38), but no correlation for DLCO (n = 84, r = -0.09). When the cohort was split on the median PPV, we observed statistically significantly lower VC ( P = 0.001) and FVC ( P = 0.04) values for the higher PPV patients, but not for DLCO ( P = 0.19). CONCLUSION: We successfully developed an AI algorithm to automatically segment PP in CT images to enable fast volume extraction. Moreover, we have observed that PPV is associated with loss in VC and FVC.

How Does Target Lesion Selection Affect RECIST? A Computer Simulation Study.

OBJECTIVES: Response Evaluation Criteria in Solid Tumors (RECIST) is grounded on the assumption that target lesion selection is objective and representative of the change in total tumor burden (TTB) during therapy. A computer simulation model was designed to challenge this assumption, focusing on a particular aspect of subjectivity: target lesion selection. MATERIALS AND METHODS: Disagreement among readers and the disagreement between individual reader measurements and TTB were analyzed as a function of the total number of lesions, affected organs, and lesion growth. RESULTS: Disagreement rises when the number of lesions increases, when lesions are concentrated on a few organs, and when lesion growth borders the thresholds of progressive disease and partial response. There is an intrinsic methodological error in the estimation of TTB via RECIST 1.1, which depends on the number of lesions and their distributions. For example, for a fixed number of lesions at 5 and 15, distributed over a maximum of 4 organs, the error rates are observed to be 7.8% and 17.3%, respectively. CONCLUSIONS: Our results demonstrate that RECIST can deliver an accurate estimate of TTB in localized disease, but fails in cases of distal metastases and multiple organ involvement. This is worsened by the "selection of the largest lesions," which introduces a bias that makes it hardly possible to perform an accurate estimate of the TTB. Including more (if not all) lesions in the quantitative analysis of tumor burden is desirable.

Radiomic signatures from T2W and DWI MRI are predictive of tumour hypoxia in colorectal liver metastases.

BACKGROUND: Tumour hypoxia is a negative predictive and prognostic biomarker in colorectal cancer typically assessed by invasive sampling methods, which suffer from many shortcomings. This retrospective proof-of-principle study explores the potential of MRI-derived imaging markers in predicting tumour hypoxia non-invasively in patients with colorectal liver metastases (CLM). METHODS: A single-centre cohort of 146 CLMs from 112 patients were segmented on preoperative T2-weighted (T2W) images and diffusion-weighted imaging (DWI). HIF-1 alpha immunohistochemical staining index (high/low) was used as a reference standard. Radiomic features were extracted, and machine learning approaches were implemented to predict the degree of histopathological tumour hypoxia. RESULTS: Radiomic signatures from DWI b200 (AUC = 0.79, 95% CI 0.61-0.93, p = 0.002) and ADC (AUC = 0.72, 95% CI 0.50-0.90, p = 0.019) were significantly predictive of tumour hypoxia. Morphological T2W TE75 (AUC = 0.64, 95% CI 0.42-0.82, p = 0.092) and functional DWI b0 (AUC = 0.66, 95% CI 0.46-0.84, p = 0.069) and b800 (AUC = 0.64, 95% CI 0.44-0.82, p = 0.071) images also provided predictive information. T2W TE300 (AUC = 0.57, 95% CI 0.33-0.78, p = 0.312) and b = 10 (AUC = 0.53, 95% CI 0.33-0.74, p = 0.415) images were not predictive of tumour hypoxia. CONCLUSIONS: T2W and DWI sequences encode information predictive of tumour hypoxia. Prospective multicentre studies could help develop and validate robust non-invasive hypoxia-detection algorithms. CRITICAL RELEVANCE STATEMENT: Hypoxia is a negative prognostic biomarker in colorectal cancer. Hypoxia is usually assessed by invasive sampling methods. This proof-of-principle retrospective study explores the role of AI-based MRI-derived imaging biomarkers in non-invasively predicting tumour hypoxia in patients with colorectal liver metastases (CLM).

Is the generalizability of a developed artificial intelligence algorithm for COVID-19 on chest CT sufficient for clinical use? Results from the International Consortium for COVID-19 Imaging AI (ICOVAI).

OBJECTIVES: Only few published artificial intelligence (AI) studies for COVID-19 imaging have been externally validated. Assessing the generalizability of developed models is essential, especially when considering clinical implementation. We report the development of the International Consortium for COVID-19 Imaging AI (ICOVAI) model and perform independent external validation. METHODS: The ICOVAI model was developed using multicenter data (n = 1286 CT scans) to quantify disease extent and assess COVID-19 likelihood using the COVID-19 Reporting and Data System (CO-RADS). A ResUNet model was modified to automatically delineate lung contours and infectious lung opacities on CT scans, after which a random forest predicted the CO-RADS score. After internal testing, the model was externally validated on a multicenter dataset (n = 400) by independent researchers. CO-RADS classification performance was calculated using linearly weighted Cohen's kappa and segmentation performance using Dice Similarity Coefficient (DSC). RESULTS: Regarding internal versus external testing, segmentation performance of lung contours was equally excellent (DSC = 0.97 vs. DSC = 0.97, p = 0.97). Lung opacities segmentation performance was adequate internally (DSC = 0.76), but significantly worse on external validation (DSC = 0.59, p < 0.0001). For CO-RADS classification, agreement with radiologists on the internal set was substantial (kappa = 0.78), but significantly lower on the external set (kappa = 0.62, p < 0.0001). CONCLUSION: In this multicenter study, a model developed for CO-RADS score prediction and quantification of COVID-19 disease extent was found to have a significant reduction in performance on independent external validation versus internal testing. The limited reproducibility of the model restricted its potential for clinical use. The study demonstrates the importance of independent external validation of AI models. KEY POINTS: • The ICOVAI model for prediction of CO-RADS and quantification of disease extent on chest CT of COVID-19 patients was developed using a large sample of multicenter data. • There was substantial performance on internal testing; however, performance was significantly reduced on external validation, performed by independent researchers. The limited generalizability of the model restricts its potential for clinical use. • Results of AI models for COVID-19 imaging on internal tests may not generalize well to external data, demonstrating the importance of independent external validation.

Artificial intelligence-based diagnosis of asbestosis: analysis of a database with applicants for asbestosis state aid.

OBJECTIVES: In many countries, workers who developed asbestosis due to their occupation are eligible for government support. Based on the results of clinical examination, a team of pulmonologists determine the eligibility of patients to these programs. In this Dutch cohort study, we aim to demonstrate the potential role of an artificial intelligence (AI)-based system for automated, standardized, and cost-effective evaluation of applications for asbestosis patients. METHODS: A dataset of n = 523 suspected asbestosis cases/applications from across the Netherlands was retrospectively collected. Each case/application was reviewed, and based on the criteria, a panel of three pulmonologists would determine eligibility for government support. An AI system is proposed, which uses thoracic CT images as input, and predicts the assessment of the clinical panel. Alongside imaging, we evaluated the added value of lung function parameters. RESULTS: The proposed AI algorithm reached an AUC of 0.87 (p < 0.001) in the prediction of accepted versus rejected applications. Diffusion capacity (DLCO) also showed comparable predictive value (AUC = 0.85, p < 0.001), with little correlation between the two parameters (r-squared = 0.22, p < 0.001). The combination of the imaging AI score and DLCO achieved superior performance (AUC = 0.95, p < 0.001). Interobserver variability between pulmonologists on the panel was estimated at alpha = 0.65 (Krippendorff's alpha). CONCLUSION: We developed an AI system to support the clinical decision-making process for the application to the government support for asbestosis. A multicenter prospective validation study is currently ongoing to examine the added value and reliability of this system alongside the clinic panel. KEY POINTS: • Artificial intelligence can detect imaging patterns of asbestosis in CT scans in a cohort of patients applying for state aid. • Combining the AI prediction with the diffusing lung function parameter reaches the highest diagnostic performance. • Specific cases with fibrosis but no asbestosis were correctly classified, suggesting robustness of the AI system, which is currently under prospective validation.