OpenKBP: The open-access knowledge-based planning grand challenge and dataset.

PURPOSE: To advance fair and consistent comparisons of dose prediction methods for knowledge-based planning (KBP) in radiation therapy research. METHODS: We hosted OpenKBP, a 2020 AAPM Grand Challenge, and challenged participants to develop the best method for predicting the dose of contoured computed tomography (CT) images. The models were evaluated according to two separate scores: (a) dose score, which evaluates the full three-dimensional (3D) dose distributions, and (b) dose-volume histogram (DVH) score, which evaluates a set DVH metrics. We used these scores to quantify the quality of the models based on their out-of-sample predictions. To develop and test their models, participants were given the data of 340 patients who were treated for head-and-neck cancer with radiation therapy. The data were partitioned into training ( n = 200 ), validation ( n = 40 ), and testing ( n = 100 ) datasets. All participants performed training and validation with the corresponding datasets during the first (validation) phase of the Challenge. In the second (testing) phase, the participants used their model on the testing data to quantify the out-of-sample performance, which was hidden from participants and used to determine the final competition ranking. Participants also responded to a survey to summarize their models. RESULTS: The Challenge attracted 195 participants from 28 countries, and 73 of those participants formed 44 teams in the validation phase, which received a total of 1750 submissions. The testing phase garnered submissions from 28 of those teams, which represents 28 unique prediction methods. On average, over the course of the validation phase, participants improved the dose and DVH scores of their models by a factor of 2.7 and 5.7, respectively. In the testing phase one model achieved the best dose score (2.429) and DVH score (1.478), which were both significantly better than the dose score (2.564) and the DVH score (1.529) that was achieved by the runner-up models. Lastly, many of the top performing teams reported that they used generalizable techniques (e.g., ensembles) to achieve higher performance than their competition. CONCLUSION: OpenKBP is the first competition for knowledge-based planning research. The Challenge helped launch the first platform that enables researchers to compare KBP prediction methods fairly and consistently using a large open-source dataset and standardized metrics. OpenKBP has also democratized KBP research by making it accessible to everyone, which should help accelerate the progress of KBP research. The OpenKBP datasets are available publicly to help benchmark future KBP research.

The importance of evaluating the complete automated knowledge-based planning pipeline.

We determine how prediction methods combine with optimization methods in two-stage knowledge-based planning (KBP) pipelines to produce radiation therapy treatment plans. We trained two dose prediction methods, a generative adversarial network (GAN) and a random forest (RF) with the same 130 treatment plans. The models were applied to 87 out-of-sample patients to create two sets of predicted dose distributions that were used as input to two optimization models. The first optimization model, inverse planning (IP), estimates weights for dose-objectives from a predicted dose distribution and generates new plans using conventional inverse planning. The second optimization model, dose mimicking (DM), minimizes the sum of one-sided quadratic penalties between the predictions and the generated plans using several dose-objectives. Altogether, four KBP pipelines (GAN-IP, GAN-DM, RF-IP, and RF-DM) were constructed and benchmarked against the corresponding clinical plans using clinical criteria; the error of both prediction methods was also evaluated. The best performing plans were GAN-IP plans, which satisfied the same criteria as their corresponding clinical plans (78%) more often than any other KBP pipeline. However, GAN did not necessarily provide the best prediction for the second-stage optimization models. Specifically, both the RF-IP and RF-DM plans satisfied the same criteria as the clinical plans 25% and 15% more often than GAN-DM plans (the worst performing plans), respectively. GAN predictions also had a higher mean absolute error (3.9 Gy) than those from RF (3.6 Gy). We find that state-of-the-art prediction methods when paired with different optimization algorithms, produce treatment plans with considerable variation in quality.

Knowledge-based automated planning with three-dimensional generative adversarial networks.

PURPOSE: To develop a knowledge-based automated planning pipeline that generates treatment plans without feature engineering, using deep neural network architectures for predicting three-dimensional (3D) dose. METHODS: Our knowledge-based automated planning (KBAP) pipeline consisted of a knowledge-based planning (KBP) method that predicts dose for a contoured computed tomography (CT) image followed by two optimization models that learn objective function weights and generate fluence-based plans, respectively. We developed a novel generative adversarial network (GAN)-based KBP approach, a 3D GAN model, which predicts dose for the full 3D CT image at once and accounts for correlations between adjacent CT slices. Baseline comparisons were made against two state-of-the-art deep learning-based KBP methods from the literature. We also developed an additional benchmark, a two-dimensional (2D) GAN model which predicts dose to each axial slice independently. For all models, we investigated the impact of multiplicatively scaling the predictions before optimization, such that the predicted dose distributions achieved all target clinical criteria. Each KBP model was trained on 130 previously delivered oropharyngeal treatment plans. Performance was tested on 87 out-of-sample previously delivered treatment plans. All KBAP plans were evaluated using clinical planning criteria and compared to their corresponding clinical plans. KBP prediction quality was assessed using dose-volume histogram (DVH) differences from the corresponding clinical plans. RESULTS: The best performing KBAP plans were generated using predictions from the 3D GAN model that were multiplicatively scaled. These plans satisfied 77% of all clinical criteria, compared to the clinical plans, which satisfied 67% of all criteria. In general, multiplicatively scaling predictions prior to optimization increased the fraction of clinical criteria satisfaction by 11% relative to the plans generated with nonscaled predictions. Additionally, these KBAP plans satisfied the same criteria as the clinical plans 84% and 8% more frequently as compared to the two benchmark methods, respectively. CONCLUSIONS: We developed the first knowledge-based automated planning framework using a 3D generative adversarial network for prediction. Our results, based on 217 oropharyngeal cancer treatment plans, demonstrated superior performance in satisfying clinical criteria and generated more realistic plans as compared to the previous state-of-the-art approaches.

Knowledge-based automated planning for oropharyngeal cancer.

PURPOSE: The purpose of this study was to automatically generate radiation therapy plans for oropharynx patients by combining knowledge-based planning (KBP) predictions with an inverse optimization (IO) pipeline. METHODS: We developed two KBP approaches, the bagging query (BQ) method and the generalized principal component analysis-based (gPCA) method, to predict achievable dose-volume histograms (DVHs). These approaches generalize existing methods by predicting physically feasible organ-at-risk (OAR) and target DVHs in sites with multiple targets. Using leave-one-out cross validation, we applied both models to a large dataset of 217 oropharynx patients. The predicted DVHs were input into an IO pipeline that generated treatment plans (BQ and gPCA plans) via an intermediate step that estimated objective function weights for an inverse planning model. The KBP predictions were compared to the clinical DVHs for benchmarking. To assess the complete pipeline, we compared the BQ and gPCA plans to both the predictions and clinical plans. To isolate the effect of the KBP predictions, we put clinical DVHs through the IO pipeline to produce clinical inverse optimized (CIO) plans. This approach also allowed us to estimate the complexity of the clinical plans. The BQ and gPCA plans were benchmarked against the CIO plans using DVH differences and clinical planning criteria. Iso-complexity plans (relative to CIO) were also generated and evaluated. RESULTS: The BQ method tended to predict that less dose is delivered than what was observed in the clinical plans while the gPCA predictions were more similar to clinical DVHs. Both populations of KBP predictions were reproduced with inverse plans to within a median DVH difference of 3 Gy. Clinical planning criteria for OARs were satisfied most frequently by the BQ plans (74.4%), by 6.3% points more than the clinical plans. Meanwhile, target criteria were satisfied most frequently by the gPCA plans (90.2%), and by 21.2% points more than clinical plans. However, once the complexity of the plans was constrained to that of the CIO plans, the performance of the BQ plans degraded significantly. In contrast, the gPCA plans still satisfied more clinical criteria than both the clinical and CIO plans, with the most notable improvement being in target criteria. CONCLUSION: Our automated pipeline can successfully use DVH predictions to generate high-quality plans without human intervention. Between the two KBP methods, gPCA plans tend to achieve comparable performance as clinical plans, even when controlling for plan complexity, whereas BQ plans tended to underperform.

Inverse optimization of objective function weights for treatment planning using clinical dose-volume histograms.

We developed and evaluated a novel inverse optimization (IO) model to estimate objective function weights from clinical dose-volume histograms (DVHs). These weights were used to solve a treatment planning problem to generate 'inverse plans' that had similar DVHs to the original clinical DVHs. Our methodology was applied to 217 clinical head and neck cancer treatment plans that were previously delivered at Princess Margaret Cancer Centre in Canada. Inverse plan DVHs were compared to the clinical DVHs using objective function values, dose-volume differences, and frequency of clinical planning criteria satisfaction. Median differences between the clinical and inverse DVHs were within 1.1 Gy. For most structures, the difference in clinical planning criteria satisfaction between the clinical and inverse plans was at most 1.4%. For structures where the two plans differed by more than 1.4% in planning criteria satisfaction, the difference in average criterion violation was less than 0.5 Gy. Overall, the inverse plans were very similar to the clinical plans. Compared with a previous inverse optimization method from the literature, our new inverse plans typically satisfied the same or more clinical criteria, and had consistently lower fluence heterogeneity. Overall, this paper demonstrates that DVHs, which are essentially summary statistics, provide sufficient information to estimate objective function weights that result in high quality treatment plans. However, as with any summary statistic that compresses three-dimensional dose information, care must be taken to avoid generating plans with undesirable features such as hotspots; our computational results suggest that such undesirable spatial features were uncommon. Our IO-based approach can be integrated into the current clinical planning paradigm to better initialize the planning process and improve planning efficiency. It could also be embedded in a knowledge-based planning or adaptive radiation therapy framework to automatically generate a new plan given a predicted or updated target DVH, respectively.

A new metric for assessing IMRT modulation complexity and plan deliverability.

PURPOSE: To evaluate the utility of a new complexity metric, the modulation complexity score (MCS), in the treatment planning and quality assurance processes and to evaluate the relationship of the metric with deliverability. METHODS: A multisite (breast, rectum, prostate, prostate bed, lung, and head and neck) and site-specific (lung) dosimetric evaluation has been completed. The MCS was calculated for each beam and the overall treatment plan. A 2D diode array (MapCHECK, Sun Nuclear, Melbourne, FL) was used to acquire measurements for each beam. The measured and planned dose (PINNACLE3, Phillips, Madison, WI) was evaluated using different percent differences and distance to agreement (DTA) criteria (3%/ 3 mm and 2%/ 1 mm) and the relationship between the dosimetric results and complexity (as measured by the MCS or simple beam parameters) assessed. RESULTS: For the multisite analysis (243 plans total), the mean MCS scores for each treatment site were breast (0.92), rectum (0.858), prostate (0.837), prostate bed (0.652), lung (0.631), and head and neck (0.356). The MCS allowed for compilation of treatment site-specific statistics, which is useful for comparing different techniques, as well as for comparison of individual treatment plans with the typical complexity levels. For the six plans selected for dosimetry, the average diode percent pass rate was 98.7% (minimum of 96%) for 3%/3 mm evaluation criteria. The average difference in absolute dose measurement between the planned and measured dose was 1.7 cGy. The detailed lung analysis also showed excellent agreement between the measured and planned dose, as all beams had a diode percentage pass rate for 3%/3 mm criteria of greater than 95.9%, with an average pass rate of 99.0%. The average absolute maximum dose difference for the lung plans was 0.7 cGy. There was no direct correlation between the MCS and simple beam parameters which could be used as a surrogate for complexity level (i.e., number of segments or MU). An evaluation criterion of 2%/ 1 mm reliably allowed for the identification of beams that are dosimetrically robust. In this study we defined a robust beam or plan as one that maintained a diode percentage pass rate greater than 90% at 2%/ 1 mm, indicating delivery that was deemed accurate when compared to the planned dose, even under stricter evaluation criterion. MCS and MU threshold criteria were determined by defining a required specificity of 1.0. A MCS threshold of 0.8 allowed for identification of robust deliverability with a sensitivity of 0.36. In contrast, MU had a lower sensitivity of 0.23 for a threshold of 50 MU. CONCLUSIONS: The MCS allows for a quantitative assessment of plan complexity, on a fixed scale, that can be applied to all treatment sites and can provide more information related to dose delivery than simple beam parameters. This could prove useful throughout the entire treatment planning and QA process.

Ionization chamber volume determination and quality assurance using micro-CT imaging.

Ion chamber dosimetry requires a high degree of precision, at all steps within the dosimetric process, in order to ensure accurate dose measurements. This work presents a novel technique for ion chamber volume determination and quality assurance, using micro-computed tomography (micro-CT). Four nominally identical Exradin A1SL chambers (0.056 cm(3)) (Standard Imaging, WI, USA) were imaged using a micro-CT system (GE Locus, GE Healthcare, London, Ontario) and irradiated in a 6 MV x-ray reference field. Air volumes were calculated from the CT datasets using 3D analysis software (Microview 2.1.1, General Electric Healthcare, London, Ontario). Differences in the volumes of each chamber determined using micro-CT images agreed with differences in the ionization response within 1% for each chamber. Calibration coefficients were then compared through cross-calibration with a calibrated ion chamber and from the CT-measured volumes. The average ratio of these values was found to be 0.958 +/- 0.009 indicating good correlation. The results demonstrate the promise of using micro-CT imaging for the absolute volumetric characterization of ion chambers. The images have the potential to be an important clinical tool for quality assurance of ion chamber construction and integrity after routine clinical usage.

The response of prototype plane-parallel ionization chambers in small megavoltage x-ray fields.

Accurate small-field dosimetry has become important with the use of multiple small fields in modern radiotherapy treatments such as IMRT and stereotactic radiosurgery. In this study, we investigate the response of a set of prototype plane-parallel ionization chambers, based upon the Exradin T11 chamber, with active volume diameters of 2, 4, 10, and 20 mm, exposed to 6 MV stereotactic radiotherapy x-ray fields. Our goal was to assess their usefulness for accurate small x-ray field dose measurements. The relative ionization response was measured in circular fields (0.5 to 4 cm diameter) as compared to a 10 x 10 cm2 reference field. A large discrepancy (approximately 40%) was found between the relative response in the smallest plane-parallel chamber and other small volume dosimeters (radiochromic film, micro-metal-oxide-semiconductor field-effect transistor and diode) used for comparison. Monte Carlo BEAMnrc simulations were used to simulate the experimental setup in order to investigate the cause of the under-response and to calculate appropriate correction factors that could be applied to experimental measurements. It was found that in small fields, the air cavity of these custom-made research chambers perturbed the secondary electron fluence profile significantly, resulting in decreased fluence within the active volume, which in turn produces a chamber under-response. It is demonstrated that a large correction to the p(fl) correction factor would be required to improve dosimetric accuracy in small fields, and that these factors could be derived using Monte Carlo simulations.