A practical method to quantify knowledge-based DVH prediction accuracy and uncertainty with reference cohorts.

The adoption of knowledge-based dose-volume histogram (DVH) prediction models for assessing organ-at-risk (OAR) sparing in radiotherapy necessitates quantification of prediction accuracy and uncertainty. Moreover, DVH prediction error bands should be readily interpretable as confidence intervals in which to find a percentage of clinically acceptable DVHs. In the event such DVH error bands are not available, we present an independent error quantification methodology using a local reference cohort of high-quality treatment plans, and apply it to two DVH prediction models, ORBIT-RT and RapidPlan, trained on the same set of 90 volumetric modulated arc therapy (VMAT) plans. Organ-at-risk DVH predictions from each model were then generated for a separate set of 45 prostate VMAT plans. Dose-volume histogram predictions were then compared to their analogous clinical DVHs to define prediction errors V c l i n , i - V p r e d , i (ith plan), from which prediction bias µ, prediction error variation σ, and root-mean-square error R M S E pred ≡ 1 N ∑ i V c l i n , i - V p r e d , i 2 ≅ σ 2 + µ 2 could be calculated for the cohort. The empirical R M S E pred was then contrasted to the model-provided DVH error estimates. For all prostate OARs, above 50% Rx dose, ORBIT-RT µ and σ were comparable to or less than those of RapidPlan. Above 80% Rx dose, µ < 1% and σ < 3-4% for both models. As a result, above 50% Rx dose, ORBIT-RT R M S E pred was below that of RapidPlan, indicating slightly improved accuracy in this cohort. Because µ ≈ 0, R M S E pred is readily interpretable as a canonical standard deviation σ, whose error band is expected to correctly predict 68% of normally distributed clinical DVHs. By contrast, RapidPlan's provided error band, although described in literature as a standard deviation range, was slightly less predictive than R M S E pred (55-70% success), while the provided ORBIT-RT error band was confirmed to resemble an interquartile range (40-65% success) as described. Clinicians can apply this methodology using their own institutions' reference cohorts to (a) independently assess a knowledge-based model's predictive accuracy of local treatment plans, and (b) interpret from any error band whether further OAR dose sparing is likely attainable.

ORBIT-RT: A Real-Time, Open Platform for Knowledge-Based Quality Control of Radiotherapy Treatment Planning.

PURPOSE: Access to knowledge-based treatment plan quality control has been hindered by the complexity of developing models and integration with different treatment planning systems (TPS). Online Real-time Benchmarking Information Technology for RadioTherapy (ORBIT-RT) provides a free, web-based platform for knowledge-based dose estimation that can be used by clinicians worldwide to benchmark the quality of their radiotherapy plans. MATERIALS AND METHODS: The ORBIT-RT platform was developed to satisfy four primary design criteria: web-based access, TPS independence, Health Insurance Portability and Accountability Act compliance, and autonomous operation. ORBIT-RT uses a cloud-based server to automatically anonymize a user's Digital Imaging and Communications in Medicine for RadioTherapy (DICOM-RT) file before upload and processing of the case. From there, ORBIT-RT uses established knowledge-based dose-volume histogram (DVH) estimation methods to autonomously create DVH estimations for the uploaded DICOM-RT. ORBIT-RT performance was evaluated with an independent validation set of 45 volumetric modulated arc therapy prostate plans with two key metrics: (i) accuracy of the DVH estimations, as quantified by their error, DVHclinical - DVHprediction and (ii) time to process and display the DVH estimations on the ORBIT-RT platform. RESULTS: ORBIT-RT organ DVH predictions show < 1% bias and 3% error uncertainty at doses > 80% of prescription for the prostate validation set. The ORBIT-RT extensions require 3.0 seconds per organ to analyze. The DICOM upload, data transfer, and DVH output display extend the entire system workflow to 2.5-3 minutes. CONCLUSION: ORBIT-RT demonstrated fast and fully autonomous knowledge-based feedback on a web-based platform that takes only anonymized DICOM-RT as input. The ORBIT-RT system can be used for real-time quality control feedback that provides users with objective comparisons for final plan DVHs.