Use of Response Permutation to Measure an Imaging Dataset's Susceptibility to Overfitting by Selected Standard Analysis Pipelines.

RATIONALE AND OBJECTIVES: This study demonstrates a method for quantifying the impact of overfitting on the receiving operator characteristic curve (AUC) when using standard analysis pipelines to develop imaging biomarkers. We illustrate the approach using two publicly available repositories of radiology and pathology images for breast cancer diagnosis. MATERIALS AND METHODS: For each dataset, we permuted the outcome (cancer diagnosis) values to eliminate any true association between imaging features and outcome. Seven types of classification models (logistic regression, linear discriminant analysis, Naïve Bayes, linear support vector machines, nonlinear support vector machine, random forest, and multi-layer perceptron) were fitted to each scrambled dataset and evaluated by each of four techniques (all data, hold-out, 10-fold cross-validation, and bootstrapping). After repeating this process for a total of 50 outcome permutations, we averaged the resulting AUCs. Any increase over a null AUC of 0.5 can be attributed to overfitting. RESULTS: Applying this approach and varying sample size and the number of imaging features, we found that failing to control for overfitting could result in near-perfect prediction (AUC near 1.0). Cross-validation offered greater protection against overfitting than the other evaluation techniques, and for most classification algorithms a sample size of at least 200 was required to assess as few as 10 features with less than 0.05 AUC inflation attributable to overfitting. CONCLUSION: This approach could be applied to any curated dataset to suggest the number of features and analysis approaches to limit overfitting.

RADCURE: An open-source head and neck cancer CT dataset for clinical radiation therapy insights.

PURPOSE: This manuscript presents RADCURE, one of the most extensive head and neck cancer (HNC) imaging datasets accessible to the public. Initially collected for clinical radiation therapy (RT) treatment planning, this dataset has been retrospectively reconstructed for use in imaging research. ACQUISITION AND VALIDATION METHODS: RADCURE encompasses data from 3346 patients, featuring computed tomography (CT) RT simulation images with corresponding target and organ-at-risk contours. These CT scans were collected using systems from three different manufacturers. Standard clinical imaging protocols were followed, and contours were manually generated and reviewed at weekly RT quality assurance rounds. RADCURE imaging and structure set data was extracted from our institution's radiation treatment planning and oncology information systems using a custom-built data mining and processing system. Furthermore, images were linked to our clinical anthology of outcomes data for each patient and includes demographic, clinical and treatment information based on the 7th edition TNM staging system (Tumor-Node-Metastasis Classification System of Malignant Tumors). The median patient age is 63, with the final dataset including 80% males. Half of the cohort is diagnosed with oropharyngeal cancer, while laryngeal, nasopharyngeal, and hypopharyngeal cancers account for 25%, 12%, and 5% of cases, respectively. The median duration of follow-up is five years, with 60% of the cohort surviving until the last follow-up point. DATA FORMAT AND USAGE NOTES: The dataset provides images and contours in DICOM CT and RT-STRUCT formats, respectively. We have standardized the nomenclature for individual contours-such as the gross primary tumor, gross nodal volumes, and 19 organs-at-risk-to enhance the RT-STRUCT files' utility. Accompanying demographic, clinical, and treatment data are supplied in a comma-separated values (CSV) file format. This comprehensive dataset is publicly accessible via The Cancer Imaging Archive. POTENTIAL APPLICATIONS: RADCURE's amalgamation of imaging, clinical, demographic, and treatment data renders it an invaluable resource for a broad spectrum of radiomics image analysis research endeavors. Researchers can utilize this dataset to advance routine clinical procedures using machine learning or artificial intelligence, to identify new non-invasive biomarkers, or to forge prognostic models.

Radiomic Analysis: Study Design, Statistical Analysis, and Other Bias Mitigation Strategies.

Rapid advances in automated methods for extracting large numbers of quantitative features from medical images have led to tremendous growth of publications reporting on radiomic analyses. Translation of these research studies into clinical practice can be hindered by biases introduced during the design, analysis, or reporting of the studies. Herein, the authors review biases, sources of variability, and pitfalls that frequently arise in radiomic research, with an emphasis on study design and statistical analysis considerations. Drawing on existing work in the statistical, radiologic, and machine learning literature, approaches for avoiding these pitfalls are described.

Automated detection of dental artifacts for large-scale radiomic analysis in radiation oncology.

BACKGROUND AND PURPOSE: Computed tomography (CT) is one of the most common medical imaging modalities in radiation oncology and radiomics research, the computational voxel-level analysis of medical images. Radiomics is vulnerable to the effects of dental artifacts (DA) caused by metal implants or fillings and can hamper future reproducibility on new datasets. In this study we seek to better understand the robustness of quantitative radiomic features to DAs. Furthermore, we propose a novel method of detecting DAs in order to safeguard radiomic studies and improve reproducibility. MATERIALS AND METHODS: We analyzed the correlations between radiomic features and the location of dental artifacts in a new dataset containing 3D CT scans from 3211 patients. We then combined conventional image processing techniques with a pre-trained convolutional neural network to create a three-class patient-level DA classifier and slice-level DA locator. Finally, we demonstrated its utility in reducing the correlations between the location of DAs and certain radiomic features. RESULTS: We found that when strong DAs were present, the proximity of the tumour to the mouth was highly correlated with 36 radiomic features. We predicted the correct DA magnitude yielding a Matthews correlation coefficient of 0.73 and location of DAs achieving the same level of agreement as human labellers. CONCLUSIONS: Removing radiomic features or CT slices containing DAs could reduce the unwanted correlations between the location of DAs and radiomic features. Automated DA detection can be used to improve the reproducibility of radiomic studies; an important step towards creating effective radiomic models for use in clinical radiation oncology.

User-controlled pipelines for feature integration and head and neck radiation therapy outcome predictions.

PURPOSE: Precision cancer medicine is dependent on accurate prediction of disease and treatment outcome, requiring integration of clinical, imaging and interventional knowledge. User controlled pipelines are capable of feature integration with varied levels of human interaction. In this work we present two pipelines designed to combine clinical, radiomic (quantified imaging), and RTx-omic (quantified radiation therapy (RT) plan) information for prediction of locoregional failure (LRF) in head and neck cancer (H&N). METHODS: Pipelines were designed to extract information and model patient outcomes based on clinical features, computed tomography (CT) imaging, and planned RT dose volumes. We predict H&N LRF using: 1) a highly user-driven pipeline that leverages modular design and machine learning for feature extraction and model development; and 2) a pipeline with minimal user input that utilizes deep learning convolutional neural networks to extract and combine CT imaging, RT dose and clinical features for model development. RESULTS: Clinical features with logistic regression in our highly user-driven pipeline had the highest precision recall area under the curve (PR-AUC) of 0.66 (0.33-0.93), where a PR-AUC = 0.11 is considered random. CONCLUSIONS: Our work demonstrates the potential to aggregate features from multiple specialties for conditional-outcome predictions using pipelines with varied levels of human interaction. Most importantly, our results provide insights into the importance of data curation and quality, as well as user, data and methodology bias awareness as it pertains to result interpretation in user controlled pipelines.

External validation and transfer learning of convolutional neural networks for computed tomography dental artifact classification.

Quality assurance of data prior to use in automated pipelines and image analysis would assist in safeguarding against biases and incorrect interpretation of results. Automation of quality assurance steps would further improve robustness and efficiency of these methods, motivating widespread adoption of techniques. Previous work by our group demonstrated the ability of convolutional neural networks (CNN) to efficiently classify head and neck (H&N) computed-tomography (CT) images for the presence of dental artifacts (DA) that obscure visualization of structures and the accuracy of Hounsfield units. In this work we demonstrate the generalizability of our previous methodology by validating CNNs on six external datasets, and the potential benefits of transfer learning with fine-tuning on CNN performance. 2112 H&N CT images from seven institutions were scored as DA positive or negative. 1538 images from a single institution were used to train three CNNs with resampling grid sizes of 643, 1283 and 2563. The remaining six external datasets were used in five-fold cross-validation with a data split of 20% training/fine-tuning and 80% validation. The three pre-trained models were each validated using the five-folds of the six external datasets. The pre-trained models also underwent transfer learning with fine-tuning using the 20% training/fine-tuning data, and validated using the corresponding validation datasets. The highest micro-averaged AUC for our pre-trained models across all external datasets occurred with a resampling grid of 2563 (AUC = 0.91 ± 0.01). Transfer learning with fine-tuning improved generalizability when utilizing a resampling grid of 2563 to a micro-averaged AUC of 0.92 ± 0.01. Despite these promising results, transfer learning did not improve AUC when utilizing small resampling grids or small datasets. Our work demonstrates the potential of our previously developed automated quality assurance methods to generalize to external datasets. Additionally, we showed that transfer learning with fine-tuning using small portions of external datasets can be used to fine-tune models for improved performance when large variations in images are present.

Sensitivity of radiomic features to inter-observer variability and image pre-processing in Apparent Diffusion Coefficient (ADC) maps of cervix cancer patients.

PURPOSE: The aims of this study are to evaluate the stability of radiomic features from Apparent Diffusion Coefficient (ADC) maps of cervical cancer with respect to: (1) reproducibility in inter-observer delineation, and (2) image pre-processing (normalization/quantization) prior to feature extraction. MATERIALS AND METHODS: Two observers manually delineated the tumor on ADC maps derived from pre-treatment diffusion-weighted Magnetic Resonance imaging of 81 patients with FIGO stage IB-IVA cervical cancer. First-order, shape, and texture features were extracted from the original and filtered images considering 5 different normalizations (four taken from the available literature, and one based on urine ADC) and two different quantization techniques (fixed-bin widths from 0.05 to 25, and fixed-bin count). Stability of radiomic features was assessed using intraclass correlation coefficient (ICC): poor (ICC < 0.75); good (0.75 ≤ ICC ≤ 0.89), and excellent (ICC ≥ 0.90). Dependencies of the features with tumor volume were assessed using Spearman's correlation coefficient (ρ). RESULTS: The approach using urine-normalized values together with a smaller bin width (0.05) was the most reproducible (428/552, 78% features with ICC ≥ 0.75); the fixed-bin count approach was the least (215/552, 39% with ICC ≥ 0.75). Without normalization, using a fixed bin width of 25, 348/552 (63%) of features had an ICC ≥ 0.75. Overall, 26% (range 25-30%) of the features were volume-dependent (ρ ≥ 0.6). None of the volume-independent shape features were found to be reproducible. CONCLUSION: Applying normalization prior to features extraction increases the reproducibility of ADC-based radiomics features. When normalization is applied, a fixed-bin width approach with smaller widths is suggested.

Automatic classification of dental artifact status for efficient image veracity checks: effects of image resolution and convolutional neural network depth.

Enabling automated pipelines, image analysis and big data methodology in cancer clinics requires thorough understanding of the data. Automated quality assurance steps could improve the efficiency and robustness of these methods by verifying possible data biases. In particular, in head and neck (H&N) computed-tomography (CT) images, dental artifacts (DA) obscure visualization of structures and the accuracy of Hounsfield units; a challenge for image analysis tasks, including radiomics, where poor image quality can lead to systemic biases. In this work we analyze the performance of three-dimensional convolutional neural networks (CNN) trained to classify DA statuses. 1538 patient images were scored by a single observer as DA positive or negative. Stratified five-fold cross validation was performed to train and test CNNs using various isotropic resampling grids (643, 1283 and 2563), with CNN depths designed to produce 323, 163, and 83 machine generated features. These parameters were selected to determine if more computationally efficient CNNs could be utilized to achieve the same performance. The area under the precision recall curve (PR-AUC) was used to assess CNN performance. The highest PR-AUC (0.92 ± 0.03) was achieved with a CNN depth = 5, resampling grid = 256. The CNN performance with 2563 resampling grid size is not significantly better than 643 and 1283 after 20 epochs, which had PR-AUC = 0.89 ± 0.03 (p -value = 0.28) and 0.91 ± 0.02 (p -value = 0.93) at depths of 3 and 4, respectively. Our experiments demonstrate the potential to automate specific quality assurance tasks required for unbiased and robust automated pipeline and image analysis research. Additionally, we determined that there is an opportunity to simplify CNNs with smaller resampling grids to make the process more amenable to very large datasets that will be available in the future.

Repeatability and reproducibility of MRI-based radiomic features in cervical cancer.

PURPOSE: The aims of this study are to evaluate the stability of radiomic features from T2-weighted MRI of cervical cancer in three ways: (1) repeatability via test-retest; (2) reproducibility between diagnostic MRI and simulation MRI; (3) reproducibility in inter-observer setting. MATERIALS AND METHODS: This retrospective cohort study included FIGO stage IB-IVA cervical cancer patients treated with chemoradiation between 2005 and 2014. There were three cohorts of women corresponding to each aim of the study: (1) 8 women who underwent test-retest MRI; (2) 20 women who underwent MRI on different scanners (diagnostic and simulation MRI); (3) 34 women whose diagnostic MRIs were contoured by three observers. Radiomic features based on first-order statistics, shape features and texture features were extracted from the original, Laplacian of Gaussian (LoG)-filtered and wavelet-filtered images, for a total of 1761 features. Stability of radiomic features was assessed using intraclass correlation coefficient (ICC). RESULTS: The inter-observer cohort had the most reproducible features (95.2% with ICC ≥0.75) whereas the diagnostic-simulation cohort had the fewest (14.1% with ICC ≥0.75). Overall, 229 features had ICC ≥0.75 in all three tests. Shape features emerged as the most stable features in all cohorts. CONCLUSION: The diagnostic-simulation test resulted in the fewest reproducible features. Further research in MRI-based radiomics is required to validate the use of reproducible features in prognostic models.

Vulnerabilities of radiomic signature development: The need for safeguards.

PURPOSE: Refinement of radiomic results and methodologies is required to ensure progression of the field. In this work, we establish a set of safeguards designed to improve and support current radiomic methodologies through detailed analysis of a radiomic signature. METHODS: A radiomic model (MW2018) was fitted and externally validated using features extracted from previously reported lung and head and neck (H&N) cancer datasets using gross-tumour-volume contours, as well as from images with randomly permuted voxel index values; i.e. images without meaningful texture. To determine MW2018's added benefit, the prognostic accuracy of tumour volume alone was calculated as a baseline. RESULTS: MW2018 had an external validation concordance index (c-index) of 0.64. However, a similar performance was achieved using features extracted from images with randomized signal intensities (c-index = 0.64 and 0.60 for H&N and lung, respectively). Tumour volume had a c-index = 0.64 and correlated strongly with three of the four model features. It was determined that the signature was a surrogate for tumour volume and that intensity and texture values were not pertinent for prognostication. CONCLUSION: Our experiments reveal vulnerabilities in radiomic signature development processes and suggest safeguards that can be used to refine methodologies, and ensure productive radiomic development using objective and independent features.

The correction of time and temperature effects in MR-based 3D Fricke xylenol orange dosimetry.

Previously developed MR-based three-dimensional (3D) Fricke-xylenol orange (FXG) dosimeters can provide end-to-end quality assurance and validation protocols for pre-clinical radiation platforms. FXG dosimeters quantify ionizing irradiation induced oxidation of Fe2+ ions using pre- and post-irradiation MR imaging methods that detect changes in spin-lattice relaxation rates (R 1 = [Formula: see text]) caused by irradiation induced oxidation of Fe2+. Chemical changes in MR-based FXG dosimeters that occur over time and with changes in temperature can decrease dosimetric accuracy if they are not properly characterized and corrected. This paper describes the characterization, development and utilization of an empirical model-based correction algorithm for time and temperature effects in the context of a pre-clinical irradiator and a 7 T pre-clinical MR imaging system. Time and temperature dependent changes of R 1 values were characterized using variable TR spin-echo imaging. R 1-time and R 1-temperature dependencies were fit using non-linear least squares fitting methods. Models were validated using leave-one-out cross-validation and resampling. Subsequently, a correction algorithm was developed that employed the previously fit empirical models to predict and reduce baseline R 1 shifts that occurred in the presence of time and temperature changes. The correction algorithm was tested on R 1-dose response curves and 3D dose distributions delivered using a small animal irradiator at 225 kVp. The correction algorithm reduced baseline R 1 shifts from -2.8 × 10-2 s-1 to 1.5 × 10-3 s-1. In terms of absolute dosimetric performance as assessed with traceable standards, the correction algorithm reduced dose discrepancies from approximately 3% to approximately 0.5% (2.90 ± 2.08% to 0.20 ± 0.07%, and 2.68 ± 1.84% to 0.46 ± 0.37% for the 10 × 10 and 8 × 12 mm2 fields, respectively). Chemical changes in MR-based FXG dosimeters produce time and temperature dependent R 1 values for the time intervals and temperature changes found in a typical small animal imaging and irradiation laboratory setting. These changes cause baseline R 1 shifts that negatively affect dosimeter accuracy. Characterization, modeling and correction of these effects improved in-field reported dose accuracy to less than 1% when compared to standardized ion chamber measurements.