Evaluating automatically generated normal tissue contours for safe use in head and neck and cervical cancer treatment planning.

PURPOSE: Volumetric-modulated arc therapy (VMAT) is a widely accepted treatment method for head and neck (HN) and cervical cancers; however, creating contours and plan optimization for VMAT plans is a time-consuming process. Our group has created an automated treatment planning tool, the Radiation Planning Assistant (RPA), that uses deep learning models to generate organs at risk (OARs), planning structures and automates plan optimization. This study quantitatively evaluates the quality of contours generated by the RPA tool. METHODS: For patients with HN (54) and cervical (39) cancers, we retrospectively generated autoplans using the RPA. Autoplans were generated using deep-learning and RapidPlan models developed in-house. The autoplans were, then, applied to the original, physician-drawn contours, which were used as a ground truth (GT) to compare with the autocontours (RPA). Using a "two one-sided tests" (TOST) procedure, we evaluated whether the autocontour normal tissue dose was equivalent to that of the ground truth by a margin, δ, that we determined based on clinical judgement. We also calculated the number of plans that met established clinically accepted dosimetric criteria. RESULTS: For HN plans, 91.8% and 91.7% of structures met dosimetric criteria for automatic and manual contours, respectively; for cervical plans, 95.6% and 95.7% of structures met dosimetric criteria for automatic and manual contours, respectively. Autocontours were equivalent to the ground truth for 71% and 75% of common DVH metrics for the HN and cervix, respectively. CONCLUSIONS: This study shows that dosimetrically equivalent normal tissue contours can be created for HN and cervical cancers using deep learning techniques. In general, differences between the contours did not affect the passing or failing of clinical dose tolerances.

Hazard testing to reduce risk in the development of automated planning tools.

PURPOSE: Hazard scenarios were created to assess and reduce the risk of planning errors in automated planning processes. This was accomplished through iterative testing and improvement of examined user interfaces. METHODS: Automated planning requires three user inputs: a computed tomography (CT), a prescription document, known as the service request, and contours. We investigated the ability of users to catch errors that were intentionally introduced into each of these three stages, according to an FMEA analysis. Five radiation therapists each reviewed 15 patient CTs, containing three errors: inappropriate field of view, incorrect superior border, and incorrect identification of isocenter. Four radiation oncology residents reviewed 10 service requests, containing two errors: incorrect prescription and treatment site. Four physicists reviewed 10 contour sets, containing two errors: missing contour slices and inaccurate target contour. Reviewers underwent video training prior to reviewing and providing feedback for various mock plans. RESULTS: Initially, 75% of hazard scenarios were detected in the service request approval. The visual display of prescription information was then updated to improve the detectability of errors based on user feedback. The change was then validated with five new radiation oncology residents who detected 100% of errors present. 83% of the hazard scenarios were detected in the CT approval portion of the workflow. For the contour approval portion of the workflow none of the errors were detected by physicists, indicating this step will not be used for quality assurance of contours. To mitigate the risk from errors that could occur at this step, radiation oncologists must perform a thorough review of contour quality prior to final plan approval. CONCLUSIONS: Hazard testing was used to pinpoint the weaknesses of an automated planning tool and as a result, subsequent improvements were made. This study identified that not all workflow steps should be used for quality assurance and demonstrated the importance of performing hazard testing to identify points of risk in automated planning tools.

Customizable landmark-based field aperture design for automated whole-brain radiotherapy treatment planning.

PURPOSE: To develop and evaluate an automated whole-brain radiotherapy (WBRT) treatment planning pipeline with a deep learning-based auto-contouring and customizable landmark-based field aperture design. METHODS: The pipeline consisted of the following steps: (1) Auto-contour normal structures on computed tomography scans and digitally reconstructed radiographs using deep learning techniques, (2) locate the landmark structures using the beam's-eye-view, (3) generate field apertures based on eight different landmark rules addressing different clinical purposes and physician preferences. Two parallel approaches for generating field apertures were developed for quality control. The performance of the generated field shapes and dose distributions were compared with the original clinical plans. The clinical acceptability of the plans was assessed by five radiation oncologists from four hospitals. RESULTS: The performance of the generated field apertures was evaluated by the Hausdorff distance (HD) and mean surface distance (MSD) from 182 patients' field apertures used in the clinic. The average HD and MSD for the generated field apertures were 16 ± 7 and 7 ± 3 mm for the first approach, respectively, and 17 ± 7 and 7 ± 3 mm, respectively, for the second approach. The differences regarding HD and MSD between the first and the second approaches were 1 ± 2 and 1 ± 3 mm, respectively. A clinical review of the field aperture design, conducted using 30 patients, achieved a 100% acceptance rate for both the first and second approaches, and the plan review achieved a 100% acceptance rate for the first approach and a 93% acceptance rate for the second approach. The average acceptance rate for meeting lens dosimetric recommendations was 80% (left lens) and 77% (right lens) for the first approach, and 70% (both left and right lenses) for the second approach, compared with 50% (left lens) and 53% (right lens) for the clinical plans. CONCLUSION: This study provided an automated pipeline with two field aperture generation approaches to automatically generate WBRT treatment plans. Both quantitative and qualitative evaluations demonstrated that our novel pipeline was comparable with the original clinical plans.

Assessing the practicality of using a single knowledge-based planning model for multiple linac vendors.

PURPOSE: Knowledge-based planning (KBP) has been shown to be an effective tool in quality control for intensity-modulated radiation therapy treatment planning and generating high-quality plans. Previous studies have evaluated its ability to create consistent plans across institutions and between planners within the same institution as well as its use as teaching tool for inexperienced planners. This study evaluates whether planning quality is consistent when using a KBP model to plan across different treatment machines. MATERIALS AND METHODS: This study used a RapidPlan model (Varian Medical Systems) provided by the vendor, to which we added additional planning objectives, maximum dose limits, and planning structures, such that a clinically acceptable plan is achieved in a single optimization. This model was used to generate and optimize volumetric-modulated arc therapy plans for a cohort of 50 patients treated for head-neck cancer. Plans were generated using the following treatment machines: Varian 2100, Elekta Versa HD, and Varian Halcyon. A noninferiority testing methodology was used to evaluate the hypothesis that normal and target metrics in our autoplans were no worse than a set of clinically-acceptable baseline plans by a margin of 1.8 Gy or 3% dose-volume. The quality of these plans were also compared through the use of common clinical dose-volume histogram criteria. RESULTS: The Versa HD met our noninferiority criteria for 23 of 34 normal and target metrics; while the Halcyon and Varian 2100 machines met our criteria for 24 of 34 and 26 of 34 metrics, respectively. The experimental plans tended to have less volume coverage for prescription dose planning target volume and larger hotspot volumes. However, comparable plans were generated across different treatment machines. CONCLUSIONS: These results support the use of a head-neck RapidPlan models in centralized planning workflows that support clinics with different linac models/vendors, although some fine-tuning for targets may be necessary.

Statistical process control to monitor use of a web-based autoplanning tool.

PURPOSE: To investigate the use of statistical process control (SPC) for quality assurance of an integrated web-based autoplanning tool, Radiation Planning Assistant (RPA). METHODS: Automatically generated plans were downloaded and imported into two treatment planning systems (TPSs), RayStation and Eclipse, in which they were recalculated using fixed monitor units. The recalculated plans were then uploaded back to the RPA, and the mean dose differences for each contour between the original RPA and the TPSs plans were calculated. SPC was used to characterize the RPA plans in terms of two comparisons: RayStation TPS versus RPA and Eclipse TPS versus RPA for three anatomical sites, and variations in the machine parameters dosimetric leaf gap (DLG) and multileaf collimator transmission factor (MLC-TF) for two algorithms (Analytical Anisotropic Algorithm [AAA]) and Acuros in the Eclipse TPS. Overall, SPC was used to monitor the process of the RPA, while clinics would still perform their routine patient-specific QA. RESULTS: For RayStation, the average mean percent dose differences across all contours were 0.65% ± 1.05%, -2.09% ± 0.56%, and 0.28% ± 0.98% and average control limit ranges were 1.89% ± 1.32%, 2.16% ± 1.31%, and 2.65% ± 1.89% for the head and neck, cervix, and chest wall, respectively. In contrast, Eclipse's average mean percent dose differences across all contours were -0.62% ± 0.34%, 0.32% ± 0.23%, and -0.91% ± 0.98%, while average control limit ranges were 1.09% ± 0.77%, 3.69% ± 2.67%, 2.73% ± 1.86%, respectively. Averaging all contours and removing outliers, a 0% dose difference corresponded with a DLG value of 0.202 ± 0.019 cm and MLC-TF value of 0.020 ± 0.001 for Acuros and a DLG value of 0.135 ± 0.031 cm and MLC-TF value of 0.015 ± 0.001 for AAA. CONCLUSIONS: Differences in mean dose and control limits between RPA and two separately commissioned TPSs were determined. With varying control limits and means, SPC provides a flexible and useful process quality assurance tool for monitoring a complex automated system such as the RPA.

Development and validation of a checklist for use with automatically generated radiotherapy plans.

PURPOSE: To develop a checklist that improves the rate of error detection during the plan review of automatically generated radiotherapy plans. METHODS: A custom checklist was developed using guidance from American Association of Physicists in Medicine task groups 275 and 315 and the results of a failure modes and effects analysis of the Radiation Planning Assistant (RPA), an automated contouring and treatment planning tool. The preliminary checklist contained 90 review items for each automatically generated plan. In the first study, eight physicists were recruited from our institution who were familiar with the RPA. Each physicist reviewed 10 artificial intelligence-generated resident treatment plans from the RPA for safety and plan quality, five of which contained errors. Physicists performed plan checks, recorded errors, and rated each plan's clinical acceptability. Following a 2-week break, physicists reviewed 10 additional plans with a similar distribution of errors using our customized checklist. Participants then provided feedback on the usability of the checklist and it was modified accordingly. In a second study, this process was repeated with 14 senior medical physics residents who were randomly assigned to checklist or no checklist for their reviews. Each reviewed 10 plans, five of which contained errors, and completed the corresponding survey. RESULTS: In the first study, the checklist significantly improved the rate of error detection from 3.4 ± 1.1 to 4.4 ± 0.74 errors per participant without and with the checklist, respectively (p = 0.02). Error detection increased by 20% when the custom checklist was utilized. In the second study, 2.9 ± 0.84 and 3.5 ± 0.84 errors per participant were detected without and with the revised checklist, respectively (p = 0.08). Despite the lack of statistical significance for this cohort, error detection increased by 18% when the checklist was utilized. CONCLUSION: Our results indicate that the use of a customized checklist when reviewing automated treatment plans will result in improved patient safety.

Radiation Planning Assistant - A Web-based Tool to Support High-quality Radiotherapy in Clinics with Limited Resources.

Access to radiotherapy worldwide is limited. The Radiation Planning Assistant (RPA) is a fully automated, web-based tool that is being developed to offer fully automated radiotherapy treatment planning tools to clinics with limited resources. The goal is to help clinical teams scale their efforts, thus reaching more patients with cancer. The user connects to the RPA via a webpage, completes a Service Request (prescription and information about the radiotherapy targets), and uploads the patient's CT image set. The RPA offers two approaches to automated planning. In one-step planning, the system uses the Service Request and CT scan to automatically generate the necessary contours and treatment plan. In two-step planning, the user reviews and edits the automatically generated contours before the RPA continues to generate a volume-modulated arc therapy plan. The final plan is downloaded from the RPA website and imported into the user's local treatment planning system, where the dose is recalculated for the locally commissioned linac; if necessary, the plan is edited prior to approval for clinical use.

Addressing the Global Expertise Gap in Radiation Oncology: The Radiation Planning Assistant.

PURPOSE: Automation, including the use of artificial intelligence, has been identified as a possible opportunity to help reduce the gap in access and quality for radiotherapy and other aspects of cancer care. The Radiation Planning Assistant (RPA) project was conceived in 2015 (and funded in 2016) to use automated contouring and treatment planning algorithms to support the efforts of oncologists in low- and middle-income countries, allowing them to scale their efforts and treat more patients safely and efficiently (to increase access). DESIGN: In this review, we discuss the development of the RPA, with a particular focus on clinical acceptability and safety/risk across jurisdictions as these are important indicators for the successful future deployment of the RPA to increase radiotherapy availability and ameliorate global disparities in access to radiation oncology. RESULTS: RPA tools will be offered through a webpage, where users can upload computed tomography data sets and download automatically generated contours and treatment plans. All interfaces have been designed to maximize ease of use and minimize risk. The current version of the RPA includes automated contouring and planning for head and neck cancer, cervical cancer, breast cancer, and metastases to the brain. CONCLUSION: The RPA has been designed to bring high-quality treatment planning to more patients across the world, and it may encourage greater investment in treatment devices and other aspects of cancer treatment.

Using Failure Mode and Effects Analysis to Evaluate Risk in the Clinical Adoption of Automated Contouring and Treatment Planning Tools.

PURPOSE: In this study, we applied the failure mode and effects analysis (FMEA) approach to an automated radiation therapy contouring and treatment planning tool to assess, and subsequently limit, the risk of deploying automated tools. METHODS AND MATERIALS: Using an FMEA, we quantified the risks associated with the Radiation Planning Assistant (RPA), an automated contouring and treatment planning tool currently under development. A multidisciplinary team identified and scored each failure mode, using a combination of RPA plan data and experience for guidance. A 1-to-10 scale for severity, occurrence, and detectability of potential errors was used, following American Association of Physicists in Medicine Task Group 100 recommendations. High-risk failure modes were further explored to determine how the workflow could be improved to reduce the associated risk. RESULTS: Of 290 possible failure modes, we identified 126 errors that were unique to the RPA workflow, with a mean risk priority number (RPN) of 56.3 and a maximum RPN of 486. The top 10 failure modes were caused by automation bias, operator error, and software error. Twenty-one failure modes were above the action threshold of RPN = 125, leading to corrective actions. The workflow was modified to simplify the user interface and better training resources were developed, which highlight the importance of thorough review of the output of automated systems. After the changes, we rescored the high-risk errors, resulting in a final mean and maximum RPN of 33.7 and 288, respectively. CONCLUSIONS: We identified 126 errors specific to the automated workflow, most of which were caused by automation bias or operator error, which emphasized the need to simplify the user interface and ensure adequate user training. As a result of changes made to the software and the enhancement of training resources, the RPNs subsequently decreased, showing that FMEA is an effective way to assess and reduce risk associated with the deployment of automated planning tools.