Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition.

BACKGROUND AND PURPOSE: Artificial intelligence (AI) models in radiology are frequently developed and validated using datasets from a single institution and are rarely tested on independent, external datasets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multi-center AI competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency. MATERIALS AND METHODS: In total, 1201 anonymized, full-head NCCT clinical scans from five institutions were pooled to form the dataset. The dataset encompassed normal studies as well as pathologies including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these-task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2: age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3: evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed each model's performance. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each AI model, along with the area under the ROC curve (AUROC). RESULTS: 1177 studies were processed by four teams. The median age of patients was 62, with an interquartile range of 33. 19 teams from various academic institutions registered for the competition. Of these, four teams submitted their final results. No commercial entities participated in the competition. For task 1, AUROCs ranged from 0.49 to 0.59. For task 2, two teams completed the task with AUROC values of 0.57 and 0.52. For task 3, teams had little to no agreement with the ground truth. CONCLUSIONS: To assess the performance of AI models in real-world clinical scenarios, we analyzed their performance in the ASFNR AI Competition. The first ASFNR Competition underscored the gap between expectation and reality; the models largely fell short in their assessments. As the integration of AI tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.ABBREVIATIONS: AI = artificial intelligence; ASFNR = American Society of Functional Neuroradiology; AUROC = area under the receiver operating characteristic curve; DICOM = Digital Imaging and Communications in Medicine; GEE = generalized estimation equation; IQR = interquartile range; NPV = negative predictive value; PPV = positive predictive value; ROC = receiver operating characteristic; TBI = traumatic brain injury.

RANO 2.0: Update to the Response Assessment in Neuro-Oncology Criteria for High- and Low-Grade Gliomas in Adults.

PURPOSE: The Response Assessment in Neuro-Oncology (RANO) criteria for high-grade gliomas (RANO-HGG) and low-grade gliomas (RANO-LGG) were developed to improve reliability of response assessment in glioma trials. Over time, some limitations of these criteria were identified, and challenges emerged regarding integrating features of the modified RANO (mRANO) or the immunotherapy RANO (iRANO) criteria. METHODS: Informed by data from studies evaluating the different criteria, updates to the RANO criteria are proposed (RANO 2.0). RESULTS: We recommend a standard set of criteria for both high- and low-grade gliomas, to be used for all trials regardless of the treatment modalities being evaluated. In the newly diagnosed setting, the postradiotherapy magnetic resonance imaging (MRI), rather than the postsurgical MRI, will be used as the baseline for comparison with subsequent scans. Since the incidence of pseudoprogression is high in the 12 weeks after radiotherapy, continuation of treatment and confirmation of progression during this period with a repeat MRI, or histopathologic evidence of unequivocal recurrent tumor, are required to define tumor progression. However, confirmation scans are not mandatory after this period nor for the evaluation of treatment for recurrent tumors. For treatments with a high likelihood of pseudoprogression, mandatory confirmation of progression with a repeat MRI is highly recommended. The primary measurement remains the maximum cross-sectional area of tumor (two-dimensional) but volumetric measurements are an option. For IDH wild-type glioblastoma, the nonenhancing disease will no longer be evaluated except when assessing response to antiangiogenic agents. In IDH-mutated tumors with a significant nonenhancing component, clinical trials may require evaluating both the enhancing and nonenhancing tumor components for response assessment. CONCLUSION: The revised RANO 2.0 criteria refine response assessment in gliomas.

Early post-bevacizumab change in rCBV from DSC-MRI identifies pseudoresponse in recurrent glioblastoma: Results from ACRIN 6677/RTOG 0625.

Background: Progressive enhancement predicted poor survival in ACRIN 6677/RTOG 0625, a multi-center trial of bevacizumab with irinotecan or temozolomide in recurrent glioblastoma, but pseudoresponse likely limited enhancement-based survival prognostication in T1 non-progressors. We aimed to determine whether early change in cerebral blood volume from baseline (ΔCBV) could further stratify the T1 non-progressors according to overall (OS) and progression-free (PFS) survival. Methods: 37/123 enrolled patients had DSC-MRI, including 13, 15, and 8 patients without 2D-T1 progression at 2, 8, and 16 weeks post-treatment initiation, respectively. Mean CBV normalized to white matter (nRCBV) and mean standardized CBV (sRCBV) were extracted from enhancing tumor. ROC curves were derived for ΔCBV using six-month PFS and one-year OS as reference standards. Kaplan-Meier survival estimates and log-rank test compared PFS and OS for both ΔCBV (increase vs. decrease) and T1 response status (stable vs. decreasing enhancement). Results: PFS and OS were significantly worse for increasing CBV at 2 weeks (p=0.003 and p=0.002 for nRCBV, and p=0.03 and p=0.03 for sRCBV, respectively), but not for 2D-T1 patients with stable vs. decreasing enhancement (p=0.44 and p=0.86, respectively). ΔCBV at week 2 was also a good prognostic marker for OS-1 and PFS-6 using ROC analysis. By contrast, 2D-T1 response status at weeks 2, 8, and 16 was not associated with PFS-6. ΔCBV at 16 weeks (p=0.008 for sRCBV) but not 8 weeks (p=0.74 for nRCBV and p=0.56 for sRCBV) was associated with significant difference in median survival, but no difference in survival was observed for 2D-T1 patients with stable vs. decreasing enhancement at 8 weeks (p=0.69) or 16 weeks (p=0.21). At 16 weeks, OS did not differ significantly between 2D-T1 progressors and 2D-T1 non-progressors with increasing CBV (median survival 3.3 months post week 16 scan vs. 9.2 months, respectively; p=0.13), suggesting that 2D-T1 non-progressors with increasing CBV may have a prognosis like that of 2D-T1 progressors. Conclusion: After 2 weeks of anti-angiogenic therapy, ΔCBV in 2D-T1 non-progressors significantly prognosticated PFS and OS, whereas 2D-T1 response status did not, identifying a subpopulation that benefits from bevacizumab. Combining 2D-T1 progression and ΔCBV may yield a response assessment paradigm with 3-tiered OS stratification.

Value of dynamic contrast perfusion MRI to predict early response to bevacizumab in newly diagnosed glioblastoma: results from ACRIN 6686 multicenter trial.

BACKGROUND: In Radiation Therapy Oncology Group (RTOG) 0825, a phase III trial of standard therapy with bevacizumab or without (placebo) in newly diagnosed glioblastoma, 44 patients underwent dynamic contrast enhanced (DCE) and/or dynamic susceptibility contrast (DSC) MRI in the American College of Radiology Imaging Network (ACRIN) trial 6686. The association between early changes in relative cerebral blood volume (rCBV) and volume transfer constant (Ktrans) with overall survival (OS) was evaluated. METHODS: MRI was performed at postop baseline (S0), immediately before (S1), 1 day after (S2), and 7 weeks after (S3) bevacizumab or placebo initiation. Mean normalized and standardized rCBV (nRCBV, sRCBV) and Ktrans were measured within contrast-enhancing lesion. Wilcoxon rank sum tests compared parameter changes from S1-S2 and S1-S3. Association with OS and progression-free survival (PFS) were determined using Kaplan-Meier and log-rank tests. Treatment response for groups stratified by pretreatment nRCBV (S0, S1) was explored. The intraclass correlation coefficient and repeatability coefficient for the placebo arm (S1-S2) were used to assess repeatability. RESULTS: Evaluable were 27-36 datasets per time point. Significant differences between treatment arms were found for changes in nRCBV and sRCBV from S1-S2 and S1-S3, and in Ktrans for S1-S3. Improved PFS (P = 0.05) but not OS (P = 0.46) was observed. High pretreatment rCBV predicted improved OS for bevacizumab-treated patients. Based on the intraclass correlation coefficient, sRCBV (0.92) was more repeatable than nRCBV (0.71) and Ktrans (0.75), consistent with repeatability coefficient values. CONCLUSIONS: Bevacizumab significantly changes rCBV but not Ktrans as early as 1 day posttreatment in newly diagnosed glioblastoma unrelated to outcomes. Improvements in clinical trial design to maximize rCBV benefit are indicated.

Consensus recommendations for a dynamic susceptibility contrast MRI protocol for use in high-grade gliomas.

Despite the widespread clinical use of dynamic susceptibility contrast (DSC) MRI, DSC-MRI methodology has not been standardized, hindering its utilization for response assessment in multicenter trials. Recently, the DSC-MRI Standardization Subcommittee of the Jumpstarting Brain Tumor Drug Development Coalition issued an updated consensus DSC-MRI protocol compatible with the standardized brain tumor imaging protocol (BTIP) for high-grade gliomas that is increasingly used in the clinical setting and is the default MRI protocol for the National Clinical Trials Network. After reviewing the basis for controversy over DSC-MRI protocols, this paper provides evidence-based best practices for clinical DSC-MRI as determined by the Committee, including pulse sequence (gradient echo vs spin echo), BTIP-compliant contrast agent dosing (preload and bolus), flip angle (FA), echo time (TE), and post-processing leakage correction. In summary, full-dose preload, full-dose bolus dosing using intermediate (60°) FA and field strength-dependent TE (40-50 ms at 1.5 T, 20-35 ms at 3 T) provides overall best accuracy and precision for cerebral blood volume estimates. When single-dose contrast agent usage is desired, no-preload, full-dose bolus dosing using low FA (30°) and field strength-dependent TE provides excellent performance, with reduced contrast agent usage and elimination of potential systematic errors introduced by variations in preload dose and incubation time.

Consensus recommendations for a standardized brain tumor imaging protocol for clinical trials in brain metastases.

A recent meeting was held on March 22, 2019, among the FDA, clinical scientists, pharmaceutical and biotech companies, clinical trials cooperative groups, and patient advocacy groups to discuss challenges and potential solutions for increasing development of therapeutics for central nervous system metastases. A key issue identified at this meeting was the need for consistent tumor measurement for reliable tumor response assessment, including the first step of standardized image acquisition with an MRI protocol that could be implemented in multicenter studies aimed at testing new therapeutics. This document builds upon previous consensus recommendations for a standardized brain tumor imaging protocol (BTIP) in high-grade gliomas and defines a protocol for brain metastases (BTIP-BM) that addresses unique challenges associated with assessment of CNS metastases. The "minimum standard" recommended pulse sequences include: (i) parameter matched pre- and post-contrast inversion recovery (IR)-prepared, isotropic 3D T1-weighted gradient echo (IR-GRE); (ii) axial 2D T2-weighted turbo spin echo acquired after injection of gadolinium-based contrast agent and before post-contrast 3D T1-weighted images; (iii) axial 2D or 3D T2-weighted fluid attenuated inversion recovery; (iv) axial 2D, 3-directional diffusion-weighted images; and (v) post-contrast 2D T1-weighted spin echo images for increased lesion conspicuity. Recommended sequence parameters are provided for both 1.5T and 3T MR systems. An "ideal" protocol is also provided, which replaces IR-GRE with 3D TSE T1-weighted imaging pre- and post-gadolinium, and is best performed at 3T, for which dynamic susceptibility contrast perfusion is included. Recommended perfusion parameters are given.

Interreader Variability of Dynamic Contrast-enhanced MRI of Recurrent Glioblastoma: The Multicenter ACRIN 6677/RTOG 0625 Study.

Purpose To evaluate factors contributing to interreader variation (IRV) in parameters measured at dynamic contrast material-enhanced (DCE) MRI in patients with glioblastoma who were participating in a multicenter trial. Materials and Methods A total of 18 patients (mean age, 57 years ± 13 [standard deviation]; 10 men) who volunteered for the advanced imaging arm of ACRIN 6677, a substudy of the RTOG 0625 clinical trial for recurrent glioblastoma treatment, underwent analyzable DCE MRI at one of four centers. The 78 imaging studies were analyzed centrally to derive the volume transfer constant (Ktrans) for gadolinium between blood plasma and tissue extravascular extracellular space, fractional volume of the extracellular extravascular space (ve), and initial area under the gadolinium concentration curve (IAUGC). Two independently trained teams consisting of a neuroradiologist and a technologist segmented the enhancing tumor on three-dimensional spoiled gradient-recalled acquisition in the steady-state images. Mean and median parameter values in the enhancing tumor were extracted after registering segmentations to parameter maps. The effect of imaging time relative to treatment, map quality, imager magnet and sequence, average tumor volume, and reader variability in tumor volume on IRV was studied by using intraclass correlation coefficients (ICCs) and linear mixed models. Results Mean interreader variations (± standard deviation) (difference as a percentage of the mean) for mean and median IAUGC, mean and median Ktrans, and median ve were 18% ± 24, 17% ± 23, 27% ± 34, 16% ± 27, and 27% ± 34, respectively. ICCs for these metrics ranged from 0.90 to 1.0 for baseline and from 0.48 to 0.76 for posttreatment examinations. Variability in reader-derived tumor volume was significantly related to IRV for all parameters. Conclusion Differences in reader tumor segmentations are a significant source of interreader variation for all dynamic contrast-enhanced MRI parameters. © RSNA, 2018 Online supplemental material is available for this article. See also the editorial by Wolf in this issue.

Prognostic value of contrast enhancement and FLAIR for survival in newly diagnosed glioblastoma treated with and without bevacizumab: results from ACRIN 6686.

Background: ACRIN 6686/RTOG 0825 was a phase III trial of conventional chemoradiation plus adjuvant temozolomide with bevacizumab or without (placebo) in newly diagnosed glioblastoma. This study investigated whether changes in contrast-enhancing and fluid attenuated inversion recovery (FLAIR)-hyperintense tumor assessed by central reading prognosticate overall survival (OS). Methods: Two hundred eighty-four patients (171 men; median age 57 y, range 19-79; 159 on bevacizumab) had MRI at post-op (baseline) and pre-cycle 4 of adjuvant temozolomide (22 wk post chemoradiation initiation). Four central readers measured bidimensional lesion enhancement (2D-T1) and FLAIR hyperintensity at both time points. Changes from baseline to pre-cycle 4 for both markers were dichotomized (increasing vs non-increasing). Cox proportional hazards model and Kaplan-Meier survival estimates were used for inference. Results: Adjusting for treatment, increasing 2D-T1 (n = 262, hazard ratio [HR] = 2.07, 95% CI: 1.48-2.91, P < 0.0001) and FLAIR (n = 273, HR = 1.75, 95% CI: 1.26-2.41, P = 0.0008) significantly predicted worse OS. Median OS (days) was significantly shorter for patients with increasing versus non-increasing 2D-T1 for both bevacizumab (443 vs 535, P = 0.004) and placebo (526 vs 887, P = 0.001). Median OS was significantly shorter for patients with increasing versus non-increasing FLAIR for placebo (595 vs 872, P = 0.001), and trended similarly for bevacizumab (499 vs 535, P = 0.0935). Adjusting for 2D-T1 and treatment, increasing FLAIR represented significantly higher risk for death (HR = 1.59 [1.11-2.26], P = 0.01). Conclusion: Increased 2D-T1 significantly predicts worse OS in both treatment groups, implying absence of a substantial proportion of pseudoprogression 22 weeks after initiation of standard therapy. FLAIR adds value beyond 2D-T1 in predicting OS, potentially addressing the pseudoresponse effect by substratifying bevacizumab-treated patients with non-increasing 2D-T1.

ACRIN 6684: Assessment of Tumor Hypoxia in Newly Diagnosed Glioblastoma Using 18F-FMISO PET and MRI.

PURPOSE: Structural and functional alterations in tumor vasculature are thought to contribute to tumor hypoxia which is a primary driver of malignancy through its negative impact on the efficacy of radiation, immune surveillance, apoptosis, genomic stability, and accelerated angiogenesis. We performed a prospective, multicenter study to test the hypothesis that abnormal tumor vasculature and hypoxia, as measured with MRI and PET, will negatively impact survival in patients with newly diagnosed glioblastoma. EXPERIMENTAL DESIGN: Prior to the start of chemoradiation, patients with glioblastoma underwent MRI scans that included dynamic contrast enhanced and dynamic susceptibility contrast perfusion sequences to quantitate tumor cerebral blood volume/flow (CBV/CBF) and vascular permeability (ktrans) as well as 18F-Fluoromisonidazole (18F-FMISO) PET to quantitate tumor hypoxia. ROC analysis and Cox regression models were used to determine the association of imaging variables with progression-free and overall survival. RESULTS: Fifty patients were enrolled of which 42 had evaluable imaging data. Higher pretreatment 18F-FMISO SUVpeak (P = 0.048), mean ktrans (P = 0.024), and median ktrans (P = 0.045) were significantly associated with shorter overall survival. Higher pretreatment median ktrans (P = 0.021), normalized RCBV (P = 0.0096), and nCBF (P = 0.038) were significantly associated with shorter progression-free survival. SUVpeak [AUC = 0.75; 95% confidence interval (CI), 0.59-0.91], nRCBV (AUC = 0.72; 95% CI, 0.56-0.89), and nCBF (AUC = 0.72; 95% CI, 0.56-0.89) were predictive of survival at 1 year. CONCLUSIONS: Increased tumor perfusion, vascular volume, vascular permeability, and hypoxia are negative prognostic markers in newly diagnosed patients with gioblastoma, and these important physiologic markers can be measured safely and reliably using MRI and 18F-FMISO PET. Clin Cancer Res; 22(20); 5079-86. ©2016 AACR.

Statistical Issues in Testing Conformance with the Quantitative Imaging Biomarker Alliance (QIBA) Profile Claims.

A major initiative of the Quantitative Imaging Biomarker Alliance is to develop standards-based documents called "Profiles," which describe one or more technical performance claims for a given imaging modality. The term "actor" denotes any entity (device, software, or person) whose performance must meet certain specifications for the claim to be met. The objective of this paper is to present the statistical issues in testing actors' conformance with the specifications. In particular, we present the general rationale and interpretation of the claims, the minimum requirements for testing whether an actor achieves the performance requirements, the study designs used for testing conformity, and the statistical analysis plan. We use three examples to illustrate the process: apparent diffusion coefficient in solid tumors measured by MRI, change in Perc 15 as a biomarker for the progression of emphysema, and percent change in solid tumor volume by computed tomography as a biomarker for lung cancer progression.

Metrology Standards for Quantitative Imaging Biomarkers.

Although investigators in the imaging community have been active in developing and evaluating quantitative imaging biomarkers (QIBs), the development and implementation of QIBs have been hampered by the inconsistent or incorrect use of terminology or methods for technical performance and statistical concepts. Technical performance is an assessment of how a test performs in reference objects or subjects under controlled conditions. In this article, some of the relevant statistical concepts are reviewed, methods that can be used for evaluating and comparing QIBs are described, and some of the technical performance issues related to imaging biomarkers are discussed. More consistent and correct use of terminology and study design principles will improve clinical research, advance regulatory science, and foster better care for patients who undergo imaging studies.

Response assessment criteria for brain metastases: proposal from the RANO group.

CNS metastases are the most common cause of malignant brain tumours in adults. Historically, patients with brain metastases have been excluded from most clinical trials, but their inclusion is now becoming more common. The medical literature is difficult to interpret because of substantial variation in the response and progression criteria used across clinical trials. The Response Assessment in Neuro-Oncology Brain Metastases (RANO-BM) working group is an international, multidisciplinary effort to develop standard response and progression criteria for use in clinical trials of treatment for brain metastases. Previous efforts have focused on aspects of trial design, such as patient population, variations in existing response and progression criteria, and challenges when incorporating neurological, neuro-cognitive, and quality-of-life endpoints into trials of patients with brain metastases. Here, we present our recommendations for standard response and progression criteria for the assessment of brain metastases in clinical trials. The proposed criteria will hopefully facilitate the development of novel approaches to this difficult problem by providing more uniformity in the assessment of CNS metastases across trials.

Dynamic susceptibility contrast MRI measures of relative cerebral blood volume as a prognostic marker for overall survival in recurrent glioblastoma: results from the ACRIN 6677/RTOG 0625 multicenter trial.

BACKGROUND: The study goal was to determine whether changes in relative cerebral blood volume (rCBV) derived from dynamic susceptibility contrast (DSC) MRI are predictive of overall survival (OS) in patients with recurrent glioblastoma multiforme (GBM) when measured 2, 8, and 16 weeks after treatment initiation. METHODS: Patients with recurrent GBM (37/123) enrolled in ACRIN 6677/RTOG 0625, a multicenter, randomized, phase II trial of bevacizumab with irinotecan or temozolomide, consented to DSC-MRI plus conventional MRI, 21 with DSC-MRI at baseline and at least 1 postbaseline scan. Contrast-enhancing regions of interest were determined semi-automatically using pre- and postcontrast T1-weighted images. Mean tumor rCBV normalized to white matter (nRCBV) and standardized rCBV (sRCBV) were determined for these regions of interest. The OS rates for patients with positive versus negative changes from baseline in nRCBV and sRCBV were compared using Wilcoxon rank-sum and Kaplan-Meier survival estimates with log-rank tests. RESULTS: Patients surviving at least 1 year (OS-1) had significantly larger decreases in nRCBV at week 2 (P = .0451) and sRCBV at week 16 (P = .014). Receiver operating characteristic analysis found the percent changes of nRCBV and sRCBV at week 2 and sRCBV at week 16, but not rCBV data at week 8, to be good prognostic markers for OS-1. Patients with positive change from baseline rCBV had significantly shorter OS than those with negative change at both week 2 and week 16 (P = .0015 and P = .0067 for nRCBV and P = .0251 and P = .0004 for sRCBV, respectively). CONCLUSIONS: Early decreases in rCBV are predictive of improved survival in patients with recurrent GBM treated with bevacizumab.

Diffusion MRI quality control and functional diffusion map results in ACRIN 6677/RTOG 0625: a multicenter, randomized, phase II trial of bevacizumab and chemotherapy in recurrent glioblastoma.

Functional diffusion mapping (fDM) is a cancer imaging technique that quantifies voxelwise changes in apparent diffusion coefficient (ADC). Previous studies have shown value of fDMs in bevacizumab therapy for recurrent glioblastoma multiforme (GBM). The aim of the present study was to implement explicit criteria for diffusion MRI quality control and independently evaluate fDM performance in a multicenter clinical trial (RTOG 0625/ACRIN 6677). A total of 123 patients were enrolled in the current multicenter trial and signed institutional review board-approved informed consent at their respective institutions. MRI was acquired prior to and 8 weeks following therapy. A 5-point QC scoring system was used to evaluate DWI quality. fDM performance was evaluated according to the correlation of these metrics with PFS and OS at the first follow-up time-point. Results showed ADC variability of 7.3% in NAWM and 10.5% in CSF. A total of 68% of patients had usable DWI data and 47% of patients had high quality DWI data when also excluding patients that progressed before the first follow-up. fDM performance was improved by using only the highest quality DWI. High pre-treatment contrast enhancing tumor volume was associated with shorter PFS and OS. A high volume fraction of increasing ADC after therapy was associated with shorter PFS, while a high volume fraction of decreasing ADC was associated with shorter OS. In summary, DWI in multicenter trials are currently of limited value due to image quality. Improvements in consistency of image quality in multicenter trials are necessary for further advancement of DWI biomarkers.

Quantitative imaging biomarkers: a review of statistical methods for computer algorithm comparisons.

Quantitative biomarkers from medical images are becoming important tools for clinical diagnosis, staging, monitoring, treatment planning, and development of new therapies. While there is a rich history of the development of quantitative imaging biomarker (QIB) techniques, little attention has been paid to the validation and comparison of the computer algorithms that implement the QIB measurements. In this paper we provide a framework for QIB algorithm comparisons. We first review and compare various study designs, including designs with the true value (e.g. phantoms, digital reference images, and zero-change studies), designs with a reference standard (e.g. studies testing equivalence with a reference standard), and designs without a reference standard (e.g. agreement studies and studies of algorithm precision). The statistical methods for comparing QIB algorithms are then presented for various study types using both aggregate and disaggregate approaches. We propose a series of steps for establishing the performance of a QIB algorithm, identify limitations in the current statistical literature, and suggest future directions for research.

A fully automatic extraction of magnetic resonance image features in glioblastoma patients.

PURPOSE: Glioblastoma is the most common malignant brain tumor. It is characterized by low median survival time and high survival variability. Survival prognosis for glioblastoma is very important for optimized treatment planning. Imaging features observed in magnetic resonance (MR) images were shown to be a good predictor of survival. However, manual assessment of MR features is time-consuming and can be associated with a high inter-reader variability as well as inaccuracies in the assessment. In response to this limitation, the authors proposed and evaluated a computer algorithm that extracts important MR image features in a fully automatic manner. METHODS: The algorithm first automatically segmented the available volumes into a background region and four tumor regions. Then, it extracted ten features from the segmented MR imaging volumes, some of which were previously indicated as predictive of clinical outcomes. To evaluate the algorithm, the authors compared the extracted features for 73 glioblastoma patients to the reference standard established by manual segmentation of the tumors. RESULTS: The experiments showed that their algorithm was able to extract most of the image features with moderate to high accuracy. High correlation coefficients between the automatically extracted value and reference standard were observed for the tumor location, minor and major axis length as well as tumor volume. Moderately high correlation coefficients were also observed for proportion of enhancing tumor, proportion of necrosis, and thickness of enhancing margin. The correlation coefficients for all these features were statistically significant (p < 0.0001). CONCLUSIONS: The authors proposed and evaluated an algorithm that, given a set of MR volumes of a glioblastoma patient, is able to extract MR image features that correlate well with their reference standard. Future studies will evaluate how well the computer-extracted features predict survival.

Early post-bevacizumab progression on contrast-enhanced MRI as a prognostic marker for overall survival in recurrent glioblastoma: results from the ACRIN 6677/RTOG 0625 Central Reader Study.

BACKGROUND: RTOG 0625/ACRIN 6677 is a multicenter, randomized, phase II trial of bevacizumab with irinotecan or temozolomide in recurrent glioblastoma (GBM). This study investigated whether early posttreatment progression on FLAIR or postcontrast MRI assessed by central reading predicts overall survival (OS). METHODS: Of 123 enrolled patients, 107 had baseline and at least 1 posttreatment MRI. Two central neuroradiologists serially measured bidimensional (2D) and volumetric (3D) enhancement on postcontrast T1-weighted images and volume of FLAIR hyperintensity. Progression status on all posttreatment MRIs was determined using Macdonald and RANO imaging threshold criteria, with a third neuroradiologist adjudicating discrepancies of both progression occurrence and timing. For each MRI pulse sequence, Kaplan-Meier survival estimates and log-rank test were used to compare OS between cases with or without radiologic progression. RESULTS: Radiologic progression occurred after 2 chemotherapy cycles (8 weeks) in 9 of 97 (9%), 9 of 73 (12%), and 11 of 98 (11%) 2D-T1, 3D-T1, and FLAIR cases, respectively, and 34 of 80 (43%), 21 of 58 (36%), and 37 of 79 (47%) corresponding cases after 4 cycles (16 weeks). Median OS among patients progressing at 8 or 16 weeks was significantly less than that among nonprogressors, as determined on 2D-T1 (114 vs 278 days and 214 vs 426 days, respectively; P < .0001 for both) and 3D-T1 (117 vs 306 days [P < .0001] and 223 vs 448 days [P = .0003], respectively) but not on FLAIR (201 vs 276 days [P = .38] and 303 vs 321 days [P = .13], respectively). CONCLUSION: Early progression on 2D-T1 and 3D-T1, but not FLAIR MRI, after 8 and 16 weeks of anti-vascular endothelial growth factor therapy has highly significant prognostic value for OS in recurrent GBM.