Determining The Role Of Radiation Oncologist Demographic Factors On Segmentation Quality: Insights From A Crowd-Sourced Challenge Using Bayesian Estimation.

BACKGROUND: Medical image auto-segmentation is poised to revolutionize radiotherapy workflows. The quality of auto-segmentation training data, primarily derived from clinician observers, is of utmost importance. However, the factors influencing the quality of these clinician-derived segmentations have yet to be fully understood or quantified. Therefore, the purpose of this study was to determine the role of common observer demographic variables on quantitative segmentation performance. METHODS: Organ at risk (OAR) and tumor volume segmentations provided by radiation oncologist observers from the Contouring Collaborative for Consensus in Radiation Oncology public dataset were utilized for this study. Segmentations were derived from five separate disease sites comprised of one patient case each: breast, sarcoma, head and neck (H&N), gynecologic (GYN), and gastrointestinal (GI). Segmentation quality was determined on a structure-by-structure basis by comparing the observer segmentations with an expert-derived consensus gold standard primarily using the Dice Similarity Coefficient (DSC); surface DSC was investigated as a secondary metric. Metrics were stratified into binary groups based on previously established structure-specific expert-derived interobserver variability (IOV) cutoffs. Generalized linear mixed-effects models using Markov chain Monte Carlo Bayesian estimation were used to investigate the association between demographic variables and the binarized segmentation quality for each disease site separately. Variables with a highest density interval excluding zero - loosely analogous to frequentist significance - were considered to substantially impact the outcome measure. RESULTS: After filtering by practicing radiation oncologists, 574, 110, 452, 112, and 48 structure observations remained for the breast, sarcoma, H&N, GYN, and GI cases, respectively. The median percentage of observations that crossed the expert DSC IOV cutoff when stratified by structure type was 55% and 31% for OARs and tumor volumes, respectively. Bayesian regression analysis revealed tumor category had a substantial negative impact on binarized DSC for the breast (coefficient mean ± standard deviation: -0.97 ± 0.20), sarcoma (-1.04 ± 0.54), H&N (-1.00 ± 0.24), and GI (-2.95 ± 0.98) cases. There were no clear recurring relationships between segmentation quality and demographic variables across the cases, with most variables demonstrating large standard deviations and wide highest density intervals. CONCLUSION: Our study highlights substantial uncertainty surrounding conventionally presumed factors influencing segmentation quality. Future studies should investigate additional demographic variables, more patients and imaging modalities, and alternative metrics of segmentation acceptability.

Large scale crowdsourced radiotherapy segmentations across a variety of cancer anatomic sites.

Clinician generated segmentation of tumor and healthy tissue regions of interest (ROIs) on medical images is crucial for radiotherapy. However, interobserver segmentation variability has long been considered a significant detriment to the implementation of high-quality and consistent radiotherapy dose delivery. This has prompted the increasing development of automated segmentation approaches. However, extant segmentation datasets typically only provide segmentations generated by a limited number of annotators with varying, and often unspecified, levels of expertise. In this data descriptor, numerous clinician annotators manually generated segmentations for ROIs on computed tomography images across a variety of cancer sites (breast, sarcoma, head and neck, gynecologic, gastrointestinal; one patient per cancer site) for the Contouring Collaborative for Consensus in Radiation Oncology challenge. In total, over 200 annotators (experts and non-experts) contributed using a standardized annotation platform (ProKnow). Subsequently, we converted Digital Imaging and Communications in Medicine data into Neuroimaging Informatics Technology Initiative format with standardized nomenclature for ease of use. In addition, we generated consensus segmentations for experts and non-experts using the Simultaneous Truth and Performance Level Estimation method. These standardized, structured, and easily accessible data are a valuable resource for systematically studying variability in segmentation applications.

E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation Oncology crowdsourced initiative for multiobserver segmentation.

Purpose: Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a crowdsourced challenge engaging radiation oncologists across various expertise levels in segmentation. An obstacle to artificial intelligence (AI) development is the paucity of multiexpert datasets; consequently, we sought to characterize whether aggregate segmentations generated from multiple nonexperts could meet or exceed recognized expert agreement. Approach: Participants who contoured ≥ 1 region of interest (ROI) for the breast, sarcoma, head and neck (H&N), gynecologic (GYN), or gastrointestinal (GI) cases were identified as a nonexpert or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. STAPLE nonexpert ROIs were evaluated against STAPLE expert contours using Dice similarity coefficient (DSC). The expert interobserver DSC ( IODSC expert ) was calculated as an acceptability threshold between STAPLE nonexpert and STAPLE expert . To determine the number of nonexperts required to match the IODSC expert for each ROI, a single consensus contour was generated using variable numbers of nonexperts and then compared to the IODSC expert . Results: For all cases, the DSC values for STAPLE nonexpert versus STAPLE expert were higher than comparator expert IODSC expert for most ROIs. The minimum number of nonexpert segmentations needed for a consensus ROI to achieve IODSC expert acceptability criteria ranged between 2 and 4 for breast, 3 and 5 for sarcoma, 3 and 5 for H&N, 3 and 5 for GYN, and 3 for GI. Conclusions: Multiple nonexpert-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. Five nonexperts could potentially generate consensus segmentations for most ROIs with performance approximating experts, suggesting nonexpert segmentations as feasible cost-effective AI inputs.

Anatomical contouring variability in thoracic organs at risk.

The purpose of this study was to determine whether contouring thoracic organs at risk was consistent among medical dosimetrists and to identify how trends in dosimetrist׳s education and experience affected contouring accuracy. Qualitative and quantitative methods were used to contextualize the raw data that were obtained. A total of 3 different computed tomography (CT) data sets were provided to medical dosimetrists (N = 13) across 5 different institutions. The medical dosimetrists were directed to contour the lungs, heart, spinal cord, and esophagus. The medical dosimetrists were instructed to contour in line with their institutional standards and were allowed to use any contouring tool or technique that they would traditionally use. The contours from each medical dosimetrist were evaluated against "gold standard" contours drawn and validated by 2 radiation oncology physicians. The dosimetrist-derived contours were evaluated against the gold standard using both a Dice coefficient method and a penalty-based metric scoring system. A short survey was also completed by each medical dosimetrist to evaluate their individual contouring experience. There was no significant variation in the contouring consistency of the lungs and spinal cord. Intradosimetrist contouring was consistent for those who contoured the esophagus and heart correctly; however, medical dosimetrists with a poor metric score showed erratic and inconsistent methods of contouring.

Technical Note: Motion-perturbation method applied to dosimetry of dynamic MLC target tracking--A proof-of-concept.

PURPOSE: Previous studies show that dose to a moving target can be estimated using 4D measurement-guided dose reconstruction based on a process called virtual motion simulation, or VMS. A potential extension of VMS is to estimate dose during dynamic multileaf collimator (MLC)-tracking treatments. The authors introduce a modified VMS method and quantify its performance as proof-of-concept for tracking applications. METHODS: Direct measurements with a moving biplanar diode array were used to verify accuracy of the VMS dose estimates. A tracking environment for variably sized circular MLC apertures was simulated by sending preprogrammed control points to the MLC while simultaneously moving the accelerator treatment table. Sensitivity of the method to simulated tracking latency (0-700 ms) was also studied. Potential applicability of VMS to fast changing beam apertures was evaluated by modeling, based on the demonstrated dependence of the cumulative dose on the temporal dose gradient. RESULTS: When physical and virtual latencies were matched, the agreement rates (2% global/2 mm gamma) between the VMS and the biplanar dosimeter were above 96%. When compared to their own reference dose (0 induced latency), the agreement rates for VMS and biplanar array track closely up to 200 ms of induced latency with 10% low-dose cutoff threshold and 300 ms with 50% cutoff. Time-resolved measurements suggest that even in the modulated beams, the error in the cumulative dose introduced by the 200 ms VMS time resolution is not likely to exceed 0.5%. CONCLUSIONS: Based on current results and prior benchmarks of VMS accuracy, the authors postulate that this approach should be applicable to any MLC-tracking treatments where leaf speeds do not exceed those of the current Varian accelerators.

Motion as perturbation. II. Development of the method for dosimetric analysis of motion effects with fixed-gantry IMRT.

PURPOSE: In this work, the feasibility of implementing a motion-perturbation approach to accurately estimate volumetric dose in the presence of organ motion--previously demonstrated for VMAT--is studied for static gantry IMRT. The method's accuracy is improved for the voxels that have very low planned dose but acquire appreciable dose due to motion. The study describes the modified algorithm and its experimental validation and provides an example of a clinical application. METHODS: A contoured region-of-interest is propagated according to the predefined motion kernel throughout time-resolved 4D phantom dose grids. This timed series of 3D dose grids is produced by the measurement-guided dose reconstruction algorithm, based on an irradiation of a static ARCCHECK (AC) helical dosimeter array (Sun Nuclear Corp., Melbourne, FL). Each moving voxel collects dose over the dynamic simulation. The difference in dose-to-moving voxel vs dose-to-static voxel in-phantom forms the basis of a motion perturbation correction that is applied to the corresponding voxel in the patient dataset. A new method to synchronize the accelerator and dosimeter clocks, applicable to fixed-gantry IMRT, was developed. Refinements to the algorithm account for the excursion of low dose voxels into high dose regions, causing appreciable dose increase due to motion (LDVE correction). For experimental validation, four plans using TG-119 structure sets and objectives were produced using segmented IMRT direct machine parameters optimization in Pinnacle treatment planning system (v. 9.6, Philips Radiation Oncology Systems, Fitchburg, WI). All beams were delivered with the gantry angle of 0°. Each beam was delivered three times: (1) to the static AC centered on the room lasers; (2) to a static phantom containing a MAPCHECK2 (MC2) planar diode array dosimeter (Sun Nuclear); and (3) to the moving MC2 phantom. The motion trajectory was an ellipse in the IEC XY plane, with 3 and 1.5 cm axes. The period was 5 s, with the resulting average motion speed of 1.45 cm/s. The motion-perturbed high resolution (2 mm voxel) volumetric dose grids on the MC2 phantom were generated for each beam. From each grid, a coronal dose plane at the detector level was extracted and compared to the corresponding moving MC2 measurement, using gamma analysis with both global (G) and local (L) dose-error normalization. RESULTS: Using the TG-119 criteria of (3%G/3 mm), per beam average gamma analysis passing rates exceeded 95% in all cases. No individual beam had a passing rate below 91%. LDVE correction eliminated systematic disagreement patterns at the beams' aperture edges. In a representative example, application of LDVE correction improved (2%L/2 mm) gamma analysis passing rate for an IMRT beam from 74% to 98%. CONCLUSIONS: The effect of motion on the moving region-of-interest IMRT dose can be estimated with a standard, static phantom QA measurement, provided the motion characteristics are independently known from 4D CT or otherwise. The motion-perturbed absolute dose estimates were validated by the direct planar diode array measurements, and were found to reliably agree with them in a homogeneous phantom.

Cross-validation of two commercial methods for volumetric high-resolution dose reconstruction on a phantom for non-coplanar VMAT beams.

BACKGROUND AND PURPOSE: Delta(4) (ScandiDos AB, Uppsala, Sweden) and ArcCHECK with 3DVH software (Sun Nuclear Corp., Melbourne, FL, USA) are commercial quasi-three-dimensional diode dosimetry arrays capable of volumetric measurement-guided dose reconstruction. A method to reconstruct dose for non-coplanar VMAT beams with 3DVH is described. The Delta(4) 3D dose reconstruction on its own phantom for VMAT delivery has not been thoroughly evaluated previously, and we do so by comparison with 3DVH. MATERIALS AND METHODS: Reconstructed volumetric doses for VMAT plans delivered with different table angles were compared between the Delta(4) and 3DVH using gamma analysis. RESULTS: The average γ (2% local dose-error normalization/2mm) passing rate comparing the directly measured Delta(4) diode dose with 3DVH was 98.2 ± 1.6% (1SD). The average passing rate for the full volumetric comparison of the reconstructed doses on a homogeneous cylindrical phantom was 95.6 ± 1.5%. No dependence on the table angle was observed. CONCLUSIONS: Modified 3DVH algorithm is capable of 3D VMAT dose reconstruction on an arbitrary volume for the full range of table angles. Our comparison results between different dosimeters make a compelling case for the use of electronic arrays with high-resolution 3D dose reconstruction as primary means of evaluating spatial dose distributions during IMRT/VMAT verification.

Evaluating IMRT and VMAT dose accuracy: practical examples of failure to detect systematic errors when applying a commonly used metric and action levels.

PURPOSE: This study (1) examines a variety of real-world cases where systematic errors were not detected by widely accepted methods for IMRT/VMAT dosimetric accuracy evaluation, and (2) drills-down to identify failure modes and their corresponding means for detection, diagnosis, and mitigation. The primary goal of detailing these case studies is to explore different, more sensitive methods and metrics that could be used more effectively for evaluating accuracy of dose algorithms, delivery systems, and QA devices. METHODS: The authors present seven real-world case studies representing a variety of combinations of the treatment planning system (TPS), linac, delivery modality, and systematic error type. These case studies are typical to what might be used as part of an IMRT or VMAT commissioning test suite, varying in complexity. Each case study is analyzed according to TG-119 instructions for gamma passing rates and action levels for per-beam and/or composite plan dosimetric QA. Then, each case study is analyzed in-depth with advanced diagnostic methods (dose profile examination, EPID-based measurements, dose difference pattern analysis, 3D measurement-guided dose reconstruction, and dose grid inspection) and more sensitive metrics (2% local normalization/2 mm DTA and estimated DVH comparisons). RESULTS: For these case studies, the conventional 3%/3 mm gamma passing rates exceeded 99% for IMRT per-beam analyses and ranged from 93.9% to 100% for composite plan dose analysis, well above the TG-119 action levels of 90% and 88%, respectively. However, all cases had systematic errors that were detected only by using advanced diagnostic techniques and more sensitive metrics. The systematic errors caused variable but noteworthy impact, including estimated target dose coverage loss of up to 5.5% and local dose deviations up to 31.5%. Types of errors included TPS model settings, algorithm limitations, and modeling and alignment of QA phantoms in the TPS. Most of the errors were correctable after detection and diagnosis, and the uncorrectable errors provided useful information about system limitations, which is another key element of system commissioning. CONCLUSIONS: Many forms of relevant systematic errors can go undetected when the currently prevalent metrics for IMRT∕VMAT commissioning are used. If alternative methods and metrics are used instead of (or in addition to) the conventional metrics, these errors are more likely to be detected, and only once they are detected can they be properly diagnosed and rooted out of the system. Removing systematic errors should be a goal not only of commissioning by the end users but also product validation by the manufacturers. For any systematic errors that cannot be removed, detecting and quantifying them is important as it will help the physicist understand the limits of the system and work with the manufacturer on improvements. In summary, IMRT and VMAT commissioning, along with product validation, would benefit from the retirement of the 3%/3 mm passing rates as a primary metric of performance, and the adoption instead of tighter tolerances, more diligent diagnostics, and more thorough analysis.

Experimentally studied dynamic dose interplay does not meaningfully affect target dose in VMAT SBRT lung treatments.

PURPOSE: The effects of respiratory motion on the tumor dose can be divided into the gradient and interplay effects. While the interplay effect is likely to average out over a large number of fractions, it may play a role in hypofractionated [stereotactic body radiation therapy (SBRT)] treatments. This subject has been extensively studied for intensity modulated radiation therapy but less so for volumetric modulated arc therapy (VMAT), particularly in application to hypofractionated regimens. Also, no experimental study has provided full four-dimensional (4D) dose reconstruction in this scenario. The authors demonstrate how a recently described motion perturbation method, with full 4D dose reconstruction, is applied to describe the gradient and interplay effects during VMAT lung SBRT treatments. METHODS: VMAT dose delivered to a moving target in a patient can be reconstructed by applying perturbations to the treatment planning system-calculated static 3D dose. Ten SBRT patients treated with 6 MV VMAT beams in five fractions were selected. The target motion (motion kernel) was approximated by 3D rigid body translation, with the tumor centroids defined on the ten phases of the 4DCT. The motion was assumed to be periodic, with the period T being an average from the empirical 4DCT respiratory trace. The real observed tumor motion (total displacement ≤ 8 mm) was evaluated first. Then, the motion range was artificially increased to 2 or 3 cm. Finally, T was increased to 60 s. While not realistic, making T comparable to the delivery time elucidates if the interplay effect can be observed. For a single fraction, the authors quantified the interplay effect as the maximum difference in the target dosimetric indices, most importantly the near-minimum dose (D99%), between all possible starting phases. For the three- and five-fractions, statistical simulations were performed when substantial interplay was found. RESULTS: For the motion amplitudes and periods obtained from the 4DCT, the interplay effect is negligible (<0.2%). It is also small (0.9% average, 2.2% maximum) when the target excursion increased to 2-3 cm. Only with large motion and increased period (60 s) was a significant interplay effect observed, with D99% ranging from 16% low to 17% high. The interplay effect was statistically significantly lower for the three- and five-fraction statistical simulations. Overall, the gradient effect dominates the clinical situation. CONCLUSIONS: A novel method was used to reconstruct the volumetric dose to a moving tumor during lung SBRT VMAT deliveries. With the studied planning and treatment technique for realistic motion periods, regardless of the amplitude, the interplay has nearly no impact on the near-minimum dose. The interplay effect was observed, for study purposes only, with the period comparable to the VMAT delivery time.

Validation of measurement-guided 3D VMAT dose reconstruction on a heterogeneous anthropomorphic phantom.

3DVH software (Sun Nuclear Corp., Melbourne, FL) is capable of generating a volumetric patient VMAT dose by applying a volumetric perturbation algorithm based on comparing measurement-guided dose reconstruction and TPS-calculated dose to a cylindrical phantom. The primary purpose of this paper is to validate this dose reconstruction on an anthropomorphic heterogeneous thoracic phantom by direct comparison to independent measurements. The dosimetric insert to the phantom is novel, and thus the secondary goal is to demonstrate how it can be used for the hidden target end-to-end testing of VMAT treatments in lung. A dosimetric insert contains a 4 cm diameter unit-density spherical target located inside the right lung (0.21 g/cm(3) density). It has 26 slots arranged in two orthogonal directions, milled to hold optically stimulated luminescent dosimeters (OSLDs). Dose profiles in three cardinal orthogonal directions were obtained for five VMAT plans with varying degrees of modulation. After appropriate OSLD corrections were applied, 3DVH measurement-guided VMAT dose reconstruction agreed 100% with the measurements in the unit density target sphere at 3%/3 mm level (composite analysis) for all profile points for the four less-modulated VMAT plans, and for 96% of the points in the highly modulated C-shape plan (from TG-119). For this latter plan, while 3DVH shows acceptable agreement with independent measurements in the unit density target, in the lung disagreement with experiment is relatively high for both the TPS calculation and 3DVH reconstruction. For the four plans excluding the C-shape, 3%/3 mm overall composite analysis passing rates for 3DVH against independent measurement ranged from 93% to 100%. The C-shape plan was deliberately chosen as a stress test of the algorithm. The dosimetric spatial alignment hidden target test demonstrated the average distance to agreement between the measured and TPS profiles in the steep dose gradient area at the edge of the 2 cm target to be 1.0 ± 0.7, 0.3 ± 0.3, and 0.3 ± 0.3 mm for the IEC X, Y, and Z directions, respectively.

On the use of biomathematical models in patient-specific IMRT dose QA.

PURPOSE: To investigate the use of biomathematical models such as tumor control probability (TCP) and normal tissue complication probability (NTCP) as new quality assurance (QA) metrics. METHODS: Five different types of error (MLC transmission, MLC penumbra, MLC tongue and groove, machine output, and MLC position) were intentionally induced to 40 clinical intensity modulated radiation therapy (IMRT) patient plans (20 H&N cases and 20 prostate cases) to simulate both treatment planning system errors and machine delivery errors in the IMRT QA process. The changes in TCP and NTCP for eight different anatomic structures (H&N: CTV, GTV, both parotids, spinal cord, larynx; prostate: CTV, rectal wall) were calculated as the new QA metrics to quantify the clinical impact on patients. The correlation between the change in TCP∕NTCP and the change in selected DVH values was also evaluated. The relation between TCP∕NTCP change and the characteristics of the TCP∕NTCP curves is discussed. RESULTS: ΔTCP and ΔNTCP were summarized for each type of induced error and each structure. The changes/degradations in TCP and NTCP caused by the errors vary widely depending on dose patterns unique to each plan, and are good indicators of each plan's "robustness" to that type of error. CONCLUSIONS: In this in silico QA study the authors have demonstrated the possibility of using biomathematical models not only as patient-specific QA metrics but also as objective indicators that quantify, pretreatment, a plan's robustness with respect to possible error types.

Motion as a perturbation: measurement-guided dose estimates to moving patient voxels during modulated arc deliveries.

PURPOSE: To present a framework for measurement-guided VMAT dose reconstruction to moving patient voxels from a known motion kernel and the static phantom data, and to validate this perturbation-based approach with the proof-of-principle experiments. METHODS: As described previously, the VMAT 3D dose to a static patient can be estimated by applying a phantom measurement-guided perturbation to the treatment planning system (TPS)-calculated dose grid. The fraction dose to any voxel in the presence of motion, assuming the motion kernel is known, can be derived in a similar fashion by applying a measurement-guided motion perturbation. The dose to the diodes in a helical phantom is recorded at 50 ms intervals and is transformed into a series of time-resolved high-density volumetric dose grids. A moving voxel is propagated through this 4D dose space and the fraction dose to that voxel in the phantom is accumulated. The ratio of this motion-perturbed, reconstructed dose to the TPS dose in the phantom serves as a perturbation factor, applied to the TPS fraction dose to the similarly situated voxel in the patient. This approach was validated by the ion chamber and film measurements on four phantoms of different shape and structure: homogeneous and inhomogeneous cylinders, a homogeneous cube, and an anthropomorphic thoracic phantom. A 2D motion stage was used to simulate the motion. The stage position was synchronized with the beam start time with the respiratory gating simulator. The motion patterns were designed such that the motion speed was in the upper range of the expected tumor motion (1-1.4 cm∕s) and the range exceeded the normally observed limits (up to 5.7 cm). The conformal arc plans for X or Y motion (in the IEC 61217 coordinate system) consisted of manually created narrow (3 cm) rectangular strips moving in-phase (tracking) or phase-shifted by 90° (crossing) with respect to the phantom motion. The XY motion was tested with the computer-derived VMAT MLC sequences. For all phantoms and plans, time-resolved (10 Hz) ion chamber dose was collected. In addition, coronal (XY) films were exposed in the cube phantom to a VMAT beam with two different starting phases, and compared to the reconstructed motion-perturbed dose planes. RESULTS: For the X or Y motions with the moving strip and geometrical phantoms, the maximum difference between perturbation-reconstructed and ion chamber doses did not exceed 1.9%, and the average for any motion pattern∕starting phase did not exceed 1.3%. For the VMAT plans on the cubic and thoracic phantoms, one point exhibited a 3.5% error, while the remaining five were all within 1.1%. Across all the measurements (N = 22), the average disagreement was 0.5 ± 1.3% (1 SD). The films exhibited γ(3%∕3 mm) passing rates ≥90%. CONCLUSIONS: The dose to an arbitrary moving voxel in a patient can be estimated with acceptable accuracy for a VMAT delivery, by performing a single QA measurement with a cylindrical phantom and applying two consecutive perturbations to the TPS-calculated patient dose. The first one accounts for the differences between the planned and delivered static doses, while the second one corrects for the motion.

VMAT QA: measurement-guided 4D dose reconstruction on a patient.

PURPOSE: To develop and validate a volume-modulated arc therapy (VMAT) quality assurance (QA) tool that takes as input a time-resolved, low-density (∼10 mm) cylindrical surface dose map from a commercial helical diode array, and outputs a high density, volumetric, time-resolved dose matrix on an arbitrary patient dataset. This first validation study is limited to a homogeneous "patient." METHODS: A VMAT treatment is delivered to a diode array phantom (ARCCHECK, Sun Nuclear Corp., Melbourne, FL). 3DVH software (Sun Nuclear) derives the high-density volumetric dose using measurement-guided dose reconstruction (MGDR). MGDR cylindrical phantom results are then used to perturb the three-dimensional (3D) treatment planning dose on the patient dataset, producing a semiempirical volumetric dose grid. Four-dimensional (4D) dose reconstruction on the patient is also possible by morphing individual sub-beam doses instead of the composite. For conventional (3D) dose comparison two methods were developed, using the four plans (Multi-Target, C-shape, Mock Prostate, and Head and Neck), including their structures and objectives, from the AAPM TG-119 report. First, 3DVH and treatment planning system (TPS) cumulative point doses were compared to ion chamber in a cube water-equivalent phantom ("patient"). The shape of the phantom is different from the ARCCHECK and furthermore the targets were placed asymmetrically. Second, coronal and sagittal absolute film dose distributions in the cube were compared with 3DVH and TPS. For time-resolved (4D) comparisons, three tests were performed. First, volumetric dose differences were calculated between the 3D MGDR and cumulative time-resolved patient (4D MGDR) dose at the end of delivery, where they ideally should be identical. Second, time-resolved (10 Hz sampling rate) ion chamber doses were compared to cumulative point dose vs time curves from 4D MGDR. Finally, accelerator output was varied to assess the linearity of the 4D MGDR with global fluence change. RESULTS: Across four TG-119 plans, the average PTV point dose difference in the cube between 3DVH and ion chamber is 0.1 ± 1.0%. Average film vs TPS γ-analysis passing rates are 83.0%, 91.1%, and 98.4% for 1%∕2 mm, 2%∕2 mm, and 3%∕3 mm threshold combinations, respectively, while average film vs 3DVH γ-analysis passing rates are 88.6%, 96.1%, and 99.5% for the same respective criteria. 4D MGDR was also sufficiently accurate. First, for 99.5% voxels in each case, the doses from 3D and 4D MGDR at the end of delivery agree within 0.5% local dose-error∕1 mm distance. Moreover, all failing voxels are confined to the edge of the cylindrical reconstruction volume. Second, dose vs time curves track between the ion chamber and 4D MGDR within 1%. Finally, 4D MGDR dose changes linearly with the accelerator output: the difference between cumulative ion chamber and MGDR dose changed by no more than 1% (randomly) with the output variation range of 10%. CONCLUSIONS: Even for a well-commissioned TPS, comparison metrics show better agreement on average to MGDR than to TPS on the arbitrary-shaped measurable "patient." The method requires no more accelerator time than standard QA, while producing more clinically relevant information. Validation in a heterogeneous thoracic phantom is under way, as is the ultimate application of 4D MGDR to virtual motion studies.

Variations in the contouring of organs at risk: test case from a patient with oropharyngeal cancer.

PURPOSE: Anatomy contouring is critical in radiation therapy. Inaccuracy and variation in defining critical volumes will affect everything downstream: treatment planning, dose-volume histogram analysis, and contour-based visual guidance used in image-guided radiation therapy. This study quantified: (1) variation in the contouring of organs at risk (OAR) in a clinical test case and (2) corresponding effects on dosimetric metrics of highly conformal plans. METHODS AND MATERIALS: A common CT data set with predefined targets from a patient with oropharyngeal cancer was provided to a population of clinics, which were asked to (1) contour OARs and (2) design an intensity-modulated radiation therapy plan. Thirty-two acceptable plans were submitted as DICOM RT data sets, each generated by a different clinical team. Using those data sets, we quantified: (1) the OAR contouring variation and (2) the impact this variation has on dosimetric metrics. New technologies were employed, including a software tool to quantify three-dimensional structure comparisons. RESULTS: There was significant interclinician variation in OAR contouring. The degree of variation is organ-dependent. We found substantial dose differences resulting strictly from contouring variation (differences ranging from -289% to 56% for mean OAR dose; -22% to 35% for maximum dose). However, there appears to be a threshold in the OAR comparison metric beyond which the dose differences stabilize. CONCLUSIONS: The effects of interclinician variation in contouring organs-at-risk in the head and neck can be large and are organ-specific. Physicians need to be aware of the effect that variation in OAR contouring can play on the final treatment plan and not restrict their focus only to the target volumes.

Variation in external beam treatment plan quality: An inter-institutional study of planners and planning systems.

PURPOSE: This study quantifies variation in radiation treatment plan quality for plans generated by a population of treatment planners given very specific plan objectives. METHODS AND MATERIALS: A "Plan Quality Metric" (PQM) with 14 submetrics, each with a unique value function, was defined for a prostate treatment plan, serving as specific goals of a hypothetical "virtual physician." The exact PQM logic was distributed to a population of treatment planners (to remove ambiguity of plan goals or plan assessment methodology) as was a predefined computed tomographic image set and anatomic structure set (to remove anatomy delineation as a variable). Treatment planners used their clinical treatment planning system (TPS) to generate their best plan based on the specified goals and submitted their results for analysis. RESULTS: One hundred forty datasets were received and 125 plans accepted and analyzed. There was wide variability in treatment plan quality (defined as the ability of the planners and plans to meet the specified goals) quantified by the PQM. Despite the variability, the resulting PQM distributions showed no statistically significant difference between TPS employed, modality (intensity modulated radiation therapy versus arc), or education and certification status of the planner. The PQM results showed negligible correlation to number of beam angles, total monitor units, years of experience of the planner, or planner confidence. CONCLUSIONS: The ability of the treatment planners to meet the specified plan objectives (as quantified by the PQM) exhibited no statistical dependence on technologic parameters (TPS, modality, plan complexity), nor was the plan quality statistically different based on planner demographics (years of experience, confidence, certification, and education). Therefore, the wide variation in plan quality could be attributed to a general "planner skill" category that would lend itself to processes of continual improvement where best practices could be derived and disseminated to improve the mean quality and minimize the variation in any population of treatment planners.

Statistical variability and confidence intervals for planar dose QA pass rates.

PURPOSE: The most common metric for comparing measured to calculated dose, such as for pretreatment quality assurance of intensity-modulated photon fields, is a pass rate (%) generated using percent difference (%Diff), distance-to-agreement (DTA), or some combination of the two (e.g., gamma evaluation). For many dosimeters, the grid of analyzed points corresponds to an array with a low areal density of point detectors. In these cases, the pass rates for any given comparison criteria are not absolute but exhibit statistical variability that is a function, in part, on the detector sampling geometry. In this work, the authors analyze the statistics of various methods commonly used to calculate pass rates and propose methods for establishing confidence intervals for pass rates obtained with low-density arrays. METHODS: Dose planes were acquired for 25 prostate and 79 head and neck intensity-modulated fields via diode array and electronic portal imaging device (EPID), and matching calculated dose planes were created via a commercial treatment planning system. Pass rates for each dose plane pair (both centered to the beam central axis) were calculated with several common comparison methods: %Diff/DTA composite analysis and gamma evaluation, using absolute dose comparison with both local and global normalization. Specialized software was designed to selectively sample the measured EPID response (very high data density) down to discrete points to simulate low-density measurements. The software was used to realign the simulated detector grid at many simulated positions with respect to the beam central axis, thereby altering the low-density sampled grid. Simulations were repeated with 100 positional iterations using a 1 detector/cm(2) uniform grid, a 2 detector/cm(2) uniform grid, and similar random detector grids. For each simulation, %/DTA composite pass rates were calculated with various %Diff/DTA criteria and for both local and global %Diff normalization techniques. RESULTS: For the prostate and head/neck cases studied, the pass rates obtained with gamma analysis of high density dose planes were 2%-5% higher than respective %/DTA composite analysis on average (ranging as high as 11%), depending on tolerances and normalization. Meanwhile, the pass rates obtained via local normalization were 2%-12% lower than with global maximum normalization on average (ranging as high as 27%), depending on tolerances and calculation method. Repositioning of simulated low-density sampled grids leads to a distribution of possible pass rates for each measured/calculated dose plane pair. These distributions can be predicted using a binomial distribution in order to establish confidence intervals that depend largely on the sampling density and the observed pass rate (i.e., the degree of difference between measured and calculated dose). These results can be extended to apply to 3D arrays of detectors, as well. CONCLUSIONS: Dose plane QA analysis can be greatly affected by choice of calculation metric and user-defined parameters, and so all pass rates should be reported with a complete description of calculation method. Pass rates for low-density arrays are subject to statistical uncertainty (vs. the high-density pass rate), but these sampling errors can be modeled using statistical confidence intervals derived from the sampled pass rate and detector density. Thus, pass rates for low-density array measurements should be accompanied by a confidence interval indicating the uncertainty of each pass rate.

Moving from gamma passing rates to patient DVH-based QA metrics in pretreatment dose QA.

PURPOSE: The purpose of this work is to explore the usefulness of the gamma passing rate metric for per-patient, pretreatment dose QA and to validate a novel patient-dose∕DVH-based method and its accuracy and correlation. Specifically, correlations between: (1) gamma passing rates for three 3D dosimeter detector geometries vs clinically relevant patient DVH-based metrics; (2) Gamma passing rates of whole patient dose grids vs DVH-based metrics, (3) gamma passing rates filtered by region of interest (ROI) vs DVH-based metrics, and (4) the capability of a novel software algorithm that estimates corrected patient Dose-DVH based on conventional phantom QA data are analyzed. METHODS: Ninety six unique "imperfect" step-and-shoot IMRT plans were generated by applying four different types of errors on 24 clinical Head∕Neck patients. The 3D patient doses as well as the dose to a cylindrical QA phantom were then recalculated using an error-free beam model to serve as a simulated measurement for comparison. Resulting deviations to the planned vs simulated measured DVH-based metrics were generated, as were gamma passing rates for a variety of difference∕distance criteria covering: dose-in-phantom comparisons and dose-in-patient comparisons, with the in-patient results calculated both over the whole grid and per-ROI volume. Finally, patient dose and DVH were predicted using the conventional per-beam planar data as input into a commercial "planned dose perturbation" (PDP) algorithm, and the results of these predicted DVH-based metrics were compared to the known values. RESULTS: A range of weak to moderate correlations were found between clinically relevant patient DVH metrics (CTV-D95, parotid D(mean), spinal cord D1cc, and larynx D(mean)) and both 3D detector and 3D patient gamma passing rate (3%∕3 mm, 2%∕2 mm) for dose-in-phantom along with dose-in-patient for both whole patient volume and filtered per-ROI. There was considerable scatter in the gamma passing rate vs DVH-based metric curves. However, for the same input data, the PDP estimates were in agreement with actual patient DVH results. CONCLUSIONS: Gamma passing rate, even if calculated based on patient dose grids, has generally weak correlation to critical patient DVH errors. However, the PDP algorithm was shown to accurately predict the DVH impact using conventional planar QA results. Using patient-DVH-based metrics IMRT QA allows per-patient dose QA to be based on metrics that are both sensitive and specific. Further studies are now required to analyze new processes and action levels associated with DVH-based metrics to ensure effectiveness and practicality in the clinical setting.

Evaluation of a new VMAT QA device, or the "X" and "O" array geometries.

We introduce a logical process of three distinct phases to begin the evaluation of a new 3D dosimetry array. The array under investigation is a hollow cylinder phantom with diode detectors fixed in a helical shell forming an "O" axial detector cross section (ArcCHECK), with comparisons drawn to a previously studied 3D array with diodes fixed in two crossing planes forming an "X" axial cross section (Delta4). Phase I testing of the ArcCHECK establishes: robust relative calibration (response equalization) of the individual detectors, minor field size dependency of response not present in a 2D predecessor, and uncorrected angular response dependence in the axial plane. Phase II testing reveals vast differences between the two devices when studying fixed-width full circle arcs. These differences are primarily due to arc discretization by the TPS that produces low passing rates for the peripheral detectors of the ArcCHECK, but high passing rates for the Delta4. Similar, although less pronounced, effects are seen for the test VMAT plans modeled after the AAPM TG119 report. The very different 3D detector locations of the two devices, along with the knock-on effect of different percent normalization strategies, prove that the analysis results from the devices are distinct and noninterchangeable; they are truly measuring different things. The value of what each device measures, namely their correlation with--or ability to predict--clinically relevant errors in calculation and/or delivery of dose is the subject of future Phase III work.

Per-beam, planar IMRT QA passing rates do not predict clinically relevant patient dose errors.

PURPOSE: The purpose of this work is to determine the statistical correlation between per-beam, planar IMRT QA passing rates and several clinically relevant, anatomy-based dose errors for per-patient IMRT QA. The intent is to assess the predictive power of a common conventional IMRT QA performance metric, the Gamma passing rate per beam. METHODS: Ninety-six unique data sets were created by inducing four types of dose errors in 24 clinical head and neck IMRT plans, each planned with 6 MV Varian 120-leaf MLC linear accelerators using a commercial treatment planning system and step-and-shoot delivery. The error-free beams/plans were used as "simulated measurements" (for generating the IMRT QA dose planes and the anatomy dose metrics) to compare to the corresponding data calculated by the error-induced plans. The degree of the induced errors was tuned to mimic IMRT QA passing rates that are commonly achieved using conventional methods. RESULTS: Analysis of clinical metrics (parotid mean doses, spinal cord max and D1cc, CTV D95, and larynx mean) vs. IMRT QA Gamma analysis (3%/3 mm, 2/2, 1/1) showed that in all cases, there were only weak to moderate correlations (range of Pearson's r-values: -0.295 to 0.653). Moreover, the moderate correlations actually had positive Pearson's r-values (i.e., clinically relevant metric differences increased with increasing IMRT QA passing rate), indicating that some of the largest anatomy-based dose differences occurred in the cases of high IMRT QA passing rates, which may be called "false negatives." The results also show numerous instances of false positives or cases where low IMRT QA passing rates do not imply large errors in anatomy dose metrics. In none of the cases was there correlation consistent with high predictive power of planar IMRT passing rates, i.e., in none of the cases did high IMRT QA Gamma passing rates predict low errors in anatomy dose metrics or vice versa. CONCLUSIONS: There is a lack of correlation between conventional IMRT QA performance metrics (Gamma passing rates) and dose errors in anatomic regions-of-interest. The most common acceptance criteria and published actions levels therefore have insufficient, or at least unproven, predictive power for per-patient IMRT QA.

Evaluation of a fast method of EPID-based dosimetry for intensity-modulated radiation therapy.

Electronic portal imaging devices (EPIDs) could potentially be useful for intensity-modulated radiation therapy (IMRT) QA. The data density, high resolution, large active area, and efficiency of the MV EPID make it an attractive option. However, EPIDs were designed as imaging devices, not dosimeters, and as a result they do not inherently measure dose in tissue equivalent media. EPIDose (Sun Nuclear, Melbourne, FL) is a tool designed for the use of EPIDs in IMRT QA that uses raw MV EPID images (no additional build-up and independent of gantry angle, but with dark and flood field corrections applied) to estimate absolute dose planes normal to the beam axis in a homogeneous media (i.e. similar to conventional IMRT QA methods). However, because of the inherent challenges of the EPID-based dosimetry, validating and commissioning such a system must be done very carefully, by exploring the range of use cases and using well-proven "standards" for comparison. In this work, a multi-institutional study was performed to verify accurate EPID image to dose plane conversion over a variety of conditions. Converted EPID images were compared to 2D diode array absolute dose measurements for 188 fields from 28 clinical IMRT treatment plans. These plans were generated using a number of commercially available treatment planning systems (TPS) covering various treatment sites including prostate, head and neck, brain, and lung. The data included three beam energies (6, 10, and 15 MV) and both step-and-shoot and dynamic MLC fields. Out of 26,207 points of comparison over 188 fields analyzed, the average overall field pass rate was 99.7% when 3 mm/3% DTA criteria were used (range 94.0-100 per field). The pass rates for more stringent criteria were 97.8% for 2mm/2% DTA (range 82.0-100 per field), and 84.6% for 1 mm/1% DTA (range 54.7-100 per field). Individual patient-specific sites as well, as different beam energies, followed similar trends to the overall pass rates.