Reproducible Radiomics Features from Multi-MRI-Scanner Test-Retest-Study: Influence on Performance and Generalizability of Models.

BACKGROUND: Radiomics models trained on data from one center typically show a decline of performance when applied to data from external centers, hindering their introduction into large-scale clinical practice. Current expert recommendations suggest to use only reproducible radiomics features isolated by multiscanner test-retest experiments, which might help to overcome the problem of limited generalizability to external data. PURPOSE: To evaluate the influence of using only a subset of robust radiomics features, defined in a prior in vivo multi-MRI-scanner test-retest-study, on the performance and generalizability of radiomics models. STUDY TYPE: Retrospective. POPULATION: Patients with monoclonal plasma cell disorders. Training set (117 MRIs from center 1); internal test set (42 MRIs from center 1); external test set (143 MRIs from center 2-8). FIELD STRENGTH/SEQUENCE: 1.5T and 3.0T; T1-weighted turbo spin echo. ASSESSMENT: The task for the radiomics models was to predict plasma cell infiltration, determined by bone marrow biopsy, noninvasively from MRI. Radiomics machine learning models, including linear regressor, support vector regressor (SVR), and random forest regressor (RFR), were trained on data from center 1, using either all radiomics features, or using only reproducible radiomics features. Models were tested on an internal (center 1) and a multicentric external data set (center 2-8). STATISTICAL TESTS: Pearson correlation coefficient r and mean absolute error (MAE) between predicted and actual plasma cell infiltration. Fisher's z-transformation, Wilcoxon signed-rank test, Wilcoxon rank-sum test; significance level P < 0.05. RESULTS: When using only reproducible features compared with all features, the performance of the SVR on the external test set significantly improved (r = 0.43 vs. r = 0.18 and MAE = 22.6 vs. MAE = 28.2). For the RFR, the performance on the external test set deteriorated when using only reproducible instead of all radiomics features (r = 0.33 vs. r = 0.44, P = 0.29 and MAE = 21.9 vs. MAE = 20.5, P = 0.10). CONCLUSION: Using only reproducible radiomics features improves the external performance of some, but not all machine learning models, and did not automatically lead to an improvement of the external performance of the overall best radiomics model. TECHNICAL EFFICACY: Stage 2.

(Smoldering) multiple myeloma: mismatch between tumor load estimated from bone marrow biopsy at iliac crest and tumor load shown by MRI.

In multiple myeloma and its precursor stages, precise quantification of tumor load is of high importance for diagnosis, risk assessment, and therapy response evaluation. Both whole-body MRI, which allows to investigate the complete bone marrow of a patient, and bone marrow biopsy, which is commonly used to assess the histologic and genetic status, are relevant methods for tumor load assessment in multiple myeloma. We report on a series of striking mismatches between the plasma cell infiltration estimating the tumor load from unguided biopsies of the bone marrow at the posterior iliac crest and the tumor load assessment from whole-body MRI.

Prediction of Bone Marrow Biopsy Results From MRI in Multiple Myeloma Patients Using Deep Learning and Radiomics.

OBJECTIVES: In multiple myeloma and its precursor stages, plasma cell infiltration (PCI) and cytogenetic aberrations are important for staging, risk stratification, and response assessment. However, invasive bone marrow (BM) biopsies cannot be performed frequently and multifocally to assess the spatially heterogenous tumor tissue. Therefore, the goal of this study was to establish an automated framework to predict local BM biopsy results from magnetic resonance imaging (MRI). MATERIALS AND METHODS: This retrospective multicentric study used data from center 1 for algorithm training and internal testing, and data from center 2 to 8 for external testing. An nnU-Net was trained for automated segmentation of pelvic BM from T1-weighted whole-body MRI. Radiomics features were extracted from these segmentations, and random forest models were trained to predict PCI and the presence or absence of cytogenetic aberrations. Pearson correlation coefficient and the area under the receiver operating characteristic were used to evaluate the prediction performance for PCI and cytogenetic aberrations, respectively. RESULTS: A total of 672 MRIs from 512 patients (median age, 61 years; interquartile range, 53-67 years; 307 men) from 8 centers and 370 corresponding BM biopsies were included. The predicted PCI from the best model was significantly correlated ( P ≤ 0.01) to the actual PCI from biopsy in all internal and external test sets (internal test set: r = 0.71 [0.51, 0.83]; center 2, high-quality test set: r = 0.45 [0.12, 0.69]; center 2, other test set: r = 0.30 [0.07, 0.49]; multicenter test set: r = 0.57 [0.30, 0.76]). The areas under the receiver operating characteristic of the prediction models for the different cytogenetic aberrations ranged from 0.57 to 0.76 for the internal test set, but no model generalized well to all 3 external test sets. CONCLUSIONS: The automated image analysis framework established in this study allows for noninvasive prediction of a surrogate parameter for PCI, which is significantly correlated to the actual PCI from BM biopsy.

Test-retest, inter- and intra-rater reproducibility of size measurements of focal bone marrow lesions in MRI in patients with multiple myeloma.

OBJECTIVE: To investigate the reproducibility of size measurements of focal bone marrow lesions (FL) in MRI in patients with monoclonal plasma cell disorders under variation of patient positioning and observer. METHODS: A data set from a prospective test-retest study was used, in which 37 patients with a total of 140 FL had undergone 2 MRI scans with identical parameters after patient repositioning. Two readers measured long and short axis diameter on the initial scan in T1 weighted, T2 weighted short tau inversion recovery and diffusion-weighted imaging sequences. The first reader additionally measured FL on the retest-scan. The Bland-Altman method was used to assess limits of agreement (LoA), and the frequencies of absolute size changes were calculated. RESULTS: In the simple test-retest experiment with one identical reader, a deviation of ≥1 mm / ≥2 mm / ≥3 mm for the long axis diameter in T1 weighted images was observed in 66% / 25% / 8% of cases. When comparing measurements of one reader on the first scan to the measurement of the other reader on the retest scan, a change of ≥1 mm / ≥3 mm / ≥5 mm for the long axis diameter in T1 weighted images was observed in 78% / 21% / 5% of cases. CONCLUSION: Small deviations in FL size are common and probably due to variation in patient positioning or inter-rater variability alone, without any actual biological change of the FL. Knowledge of the uncertainty associated with size measurements of FLs is critical for radiologists and oncologists when interpreting changes in FL size in clinical practice and in clinical trials. ADVANCES IN KNOWLEDGE: According to the MY-RADs criteria, size measurements of focal lesions in MRI are now of relevance for response assessment in patients with monoclonal plasma cell disorders.Size changes of 1 or 2 mm are frequently observed due to uncertainty of the measurement only, while the actual focal lesion has not undergone any biological change.Size changes of at least 6 mm or more in T1 weighted or T2 weighted short tau inversion recovery sequences occur in only 5% or less of cases when the focal lesion has not undergone any biological change.

Deep Learning for Automatic Bone Marrow Apparent Diffusion Coefficient Measurements From Whole-Body Magnetic Resonance Imaging in Patients With Multiple Myeloma: A Retrospective Multicenter Study.

OBJECTIVES: Diffusion-weighted magnetic resonance imaging (MRI) is increasingly important in patients with multiple myeloma (MM). The objective of this study was to train and test an algorithm for automatic pelvic bone marrow analysis from whole-body apparent diffusion coefficient (ADC) maps in patients with MM, which automatically segments pelvic bones and subsequently extracts objective, representative ADC measurements from each bone. MATERIALS AND METHODS: In this retrospective multicentric study, 180 MRIs from 54 patients were annotated (semi)manually and used to train an nnU-Net for automatic, individual segmentation of the right hip bone, the left hip bone, and the sacral bone. The quality of the automatic segmentation was evaluated on 15 manually segmented whole-body MRIs from 3 centers using the dice score. In 3 independent test sets from 3 centers, which comprised a total of 312 whole-body MRIs, agreement between automatically extracted mean ADC values from the nnU-Net segmentation and manual ADC measurements from 2 independent radiologists was evaluated. Bland-Altman plots were constructed, and absolute bias, relative bias to mean, limits of agreement, and coefficients of variation were calculated. In 56 patients with newly diagnosed MM who had undergone bone marrow biopsy, ADC measurements were correlated with biopsy results using Spearman correlation. RESULTS: The ADC-nnU-Net achieved automatic segmentations with mean dice scores of 0.92, 0.93, and 0.85 for the right pelvis, the left pelvis, and the sacral bone, whereas the interrater experiment gave mean dice scores of 0.86, 0.86, and 0.77, respectively. The agreement between radiologists' manual ADC measurements and automatic ADC measurements was as follows: the bias between the first reader and the automatic approach was 49 × 10 -6 mm 2 /s, 7 × 10 -6 mm 2 /s, and -58 × 10 -6 mm 2 /s, and the bias between the second reader and the automatic approach was 12 × 10 -6 mm 2 /s, 2 × 10 -6 mm 2 /s, and -66 × 10 -6 mm 2 /s for the right pelvis, the left pelvis, and the sacral bone, respectively. The bias between reader 1 and reader 2 was 40 × 10 -6 mm 2 /s, 8 × 10 -6 mm 2 /s, and 7 × 10 -6 mm 2 /s, and the mean absolute difference between manual readers was 84 × 10 -6 mm 2 /s, 65 × 10 -6 mm 2 /s, and 75 × 10 -6 mm 2 /s. Automatically extracted ADC values significantly correlated with bone marrow plasma cell infiltration ( R = 0.36, P = 0.007). CONCLUSIONS: In this study, a nnU-Net was trained that can automatically segment pelvic bone marrow from whole-body ADC maps in multicentric data sets with a quality comparable to manual segmentations. This approach allows automatic, objective bone marrow ADC measurements, which agree well with manual ADC measurements and can help to overcome interrater variability or nonrepresentative measurements. Automatically extracted ADC values significantly correlate with bone marrow plasma cell infiltration and might be of value for automatic staging, risk stratification, or therapy response assessment.

In Vivo Repeatability and Multiscanner Reproducibility of MRI Radiomics Features in Patients With Monoclonal Plasma Cell Disorders: A Prospective Bi-institutional Study.

OBJECTIVES: Despite the extensive number of publications in the field of radiomics, radiomics algorithms barely enter large-scale clinical application. Supposedly, the low external generalizability of radiomics models is one of the main reasons, which hinders the translation from research to clinical application. The objectives of this study were to investigate reproducibility of radiomics features (RFs) in vivo under variation of patient positioning, magnetic resonance imaging (MRI) sequence, and MRI scanners, and to identify a subgroup of RFs that shows acceptable reproducibility across all different acquisition scenarios. MATERIALS AND METHODS: Between November 30, 2020 and February 16, 2021, 55 patients with monoclonal plasma cell disorders were included in this prospective, bi-institutional, single-vendor study. Participants underwent one reference scan at a 1.5 T MRI scanner and several retest scans: once after simple repositioning, once with a second MRI protocol, once at another 1.5 T scanner, and once at a 3 T scanner. Radiomics feature from the bone marrow of the left hip bone were extracted, both from original scans and after different image normalizations. Intraclass correlation coefficient (ICC) was used to assess RF repeatability and reproducibility. RESULTS: Fifty-five participants (mean age, 59 ± 7 years; 36 men) were enrolled. For T1-weighted images after muscle normalization, in the simple test-retest experiment, 110 (37%) of 295 RFs showed an ICC ≥0.8: 54 (61%) of 89 first-order features (FOFs), 35 (95%) of 37 volume and shape features, and 21 (12%) of 169 texture features (TFs). When the retest was performed with different technical settings, even after muscle normalization, the number of FOF/TF with an ICC ≥0.8 declined to 58/13 for the second protocol, 29/7 for the second 1.5 T scanner, and 49/7 for the 3 T scanner, respectively. Twenty-five (28%) of the 89 FOFs and 6 (4%) of the 169 TFs from muscle-normalized T1-weighted images showed an ICC ≥0.8 throughout all repeatability and reproducibility experiments. CONCLUSIONS: In vivo, only few RFs are reproducible with different MRI sequences or different MRI scanners, even after application of a simple image normalization. Radiomics features selected by a repeatability experiment only are not necessarily suited to build radiomics models for multicenter clinical application. This study isolated a subset of RFs, which are robust to variations in MRI acquisition observed in scanners from 1 vendor, and therefore are candidates to build reproducible radiomics models for monoclonal plasma cell disorders for multicentric applications, at least when centers are equipped with scanners from this vendor.

Combining Deep Learning and Radiomics for Automated, Objective, Comprehensive Bone Marrow Characterization From Whole-Body MRI: A Multicentric Feasibility Study.

OBJECTIVES: Disseminated bone marrow (BM) involvement is frequent in multiple myeloma (MM). Whole-body magnetic resonance imaging (wb-MRI) enables to evaluate the whole BM. Reading of such whole-body scans is time-consuming, and yet radiologists can transfer only a small fraction of the information of the imaging data set to the report. This limits the influence that imaging can have on clinical decision-making and in research toward precision oncology. The objective of this feasibility study was to implement a concept for automatic, comprehensive characterization of the BM from wb-MRI, by automatic BM segmentation and subsequent radiomics analysis of 30 different BM spaces (BMS). MATERIALS AND METHODS: This retrospective multicentric pilot study used a total of 106 wb-MRI from 102 patients with (smoldering) MM from 8 centers. Fifty wb-MRI from center 1 were used for training of segmentation algorithms (nnU-Nets) and radiomics algorithms. Fifty-six wb-MRI from 8 centers, acquired with a variety of different MRI scanners and protocols, were used for independent testing. Manual segmentations of 2700 BMS from 90 wb-MRI were performed for training and testing of the segmentation algorithms. For each BMS, 296 radiomics features were calculated individually. Dice score was used to assess similarity between automatic segmentations and manual reference segmentations. RESULTS: The "multilabel nnU-Net" segmentation algorithm, which performs segmentation of 30 BMS and labels them individually, reached mean dice scores of 0.88 ± 0.06/0.87 ± 0.06/0.83 ± 0.11 in independent test sets from center 1/center 2/center 3-8 (interrater variability between radiologists, 0.88 ± 0.01). The subset from the multicenter, multivendor test set (center 3-8) that was of high imaging quality was segmented with high precision (mean dice score, 0.87), comparable to the internal test data from center 1. The radiomic BM phenotype consisting of 8880 descriptive parameters per patient, which result from calculation of 296 radiomics features for each of the 30 BMS, was calculated for all patients. Exemplary cases demonstrated connections between typical BM patterns in MM and radiomic signatures of the respective BMS. In plausibility tests, predicted size and weight based on radiomics models of the radiomic BM phenotype significantly correlated with patients' actual size and weight ( P = 0.002 and P = 0.003, respectively). CONCLUSIONS: This pilot study demonstrates the feasibility of automatic, objective, comprehensive BM characterization from wb-MRI in multicentric data sets. This concept allows the extraction of high-dimensional phenotypes to capture the complexity of disseminated BM disorders from imaging. Further studies need to assess the clinical potential of this method for automatic staging, therapy response assessment, or prediction of biopsy results.

Repeatability and Reproducibility of ADC Measurements and MRI Signal Intensity Measurements of Bone Marrow in Monoclonal Plasma Cell Disorders: A Prospective Bi-institutional Multiscanner, Multiprotocol Study.

BACKGROUND/OBJECTIVES: Apparent diffusion coefficient (ADC) and signal intensity (SI) measurements play an increasing role in magnetic resonance imaging (MRI) of monoclonal plasma cell disorders. The purpose of this study was to assess interrater variability, repeatability, and reproducibility of ADC and SI measurements from bone marrow (BM) under variation of MRI protocols and scanners. PATIENTS AND METHODS: Fifty-five patients with suspected or confirmed monoclonal plasma cell disorder were prospectively included in this institutional review board-approved study and underwent several measurements after the standard clinical whole-body MR scan, including repeated scan after repositioning, scan with a second MRI protocol, scan at a second 1.5 T scanner with a harmonized MRI protocol, and scan at a 3 T scanner. For T1-weighted, T2-weighted STIR, B800 images, and ADC maps, regions of interest were placed in the BM of the iliac crest and sacral bone, and in muscle tissue for image normalization. Bland-Altman plots were constructed, and absolute bias, relative bias to mean, limits of agreement, and coefficients of variation were calculated. RESULTS: Interrater variability and repeatability experiments showed a maximal relative bias of -0.077 and a maximal coefficient of variation of 16.2% for all sequences. Although the deviations at the second 1.5 T scanner with harmonized MRI protocol to the first 1.5 T scanner showed a maximal relative bias of 0.124 for all sequences, the variation of the MRI protocol and scan at the 3 T scanner led to large relative biases of up to -0.357 and -0.526, respectively. When comparing the 3 T scanner to the 1.5 T scanner, normalization to muscle reduced the bias of T1-weighted and T2-weighted sequences, but not of ADC maps. CONCLUSIONS: The MRI scanners with identical field strength and harmonized MRI protocols can provide relatively stable quantitative measurements of BM ADC and SI. Deviations in MRI field strength and MRI protocol should be avoided when applying ADC cutoff values, which were established at other scanners or when performing multicentric imaging trials.