Assessing Interobserver Variability in the Delineation of Structures in Radiation Oncology: A Systematic Review.

PURPOSE: The delineation of target volumes and organs at risk is the main source of uncertainty in radiation therapy. Numerous interobserver variability (IOV) studies have been conducted, often with unclear methodology and nonstandardized reporting. We aimed to identify the parameters chosen in conducting delineation IOV studies and assess their performances and limits. METHODS AND MATERIALS: We conducted a systematic literature review to highlight major points of heterogeneity and missing data in IOV studies published between 2018 and 2021. For the main used metrics, we did in silico analyses to assess their limits in specific clinical situations. RESULTS: All disease sites were represented in the 66 studies examined. Organs at risk were studied independently of tumor site in 29% of reviewed IOV studies. In 65% of studies, statistical analyses were performed. No gold standard (GS; ie, reference) was defined in 36% of studies. A single expert was considered as the GS in 21% of studies, without testing intraobserver variability. All studies reported both absolute and relative indices, including the Dice similarity coefficient (DSC) in 68% and the Hausdorff distance (HD) in 42%. Limitations were shown in silico for small structures when using the DSC and dependence on irregular shapes when using the HD. Variations in DSC values were large between studies, and their thresholds were inconsistent. Most studies (51%) included 1 to 10 cases. The median number of observers or experts was 7 (range, 2-35). The intraclass correlation coefficient was reported in only 9% of cases. Investigating the feasibility of studying IOV in delineation, a minimum of 8 observers with 3 cases, or 11 observers with 2 cases, was required to demonstrate moderate reproducibility. CONCLUSIONS: Implementation of future IOV studies would benefit from a more standardized methodology: clear definitions of the gold standard and metrics and a justification of the tradeoffs made in the choice of the number of observers and number of delineated cases should be provided.

International assessment of interobserver reproducibility of flap delineation in head and neck carcinoma.

Background: Several reports have suggested that radiotherapy after reconstructive surgery for head and neck cancer (HNC), could have deleterious effects on the flaps with respect to functional outcomes. To predict and prevent toxicities, flap delineation should be accurate and reproducible. The objective of the present study was to evaluate the interobserver variability of frequent types of flaps used in HNC, based on the recent GORTEC atlas.Materials and methods: Each member of an international working group (WG) consisting of 14 experts delineated the flaps on a CT set from six patients. Each patient had one of the five most commonly used flaps in HNC: a regional pedicled pectoralis major myocutaneous flap, a local pedicled rotational soft tissue facial artery musculo-mucosal (FAMM) (2 patients), a fasciocutaneous radial forearm free flap, a soft tissue anterolateral thigh (ALT) free flap, or a fibular free flap. The WG's contours were compared to a reference contour, validated by a surgeon and a radiologist specializing in HNC. Contours were considered as reproducible if the median Dice Similarity Coefficient (DSC) was > 0.7.Results: The median volumes of the six flaps delineated by the WG were close to the reference contour value, with approximately 50 cc for the pectoral, fibula, and ALT flaps, 20 cc for the radial forearm, and up to 10 cc for the FAMM. The volumetric ratio was thus close to the optimal value of 100% for all flaps. The median DSC obtained by the WG compared to the reference for the pectoralis flap, the FAMM, the radial forearm flap, ALT flap, and the fibular flap were 0.82, 0.40, 0.76, 0.81, and 0.76, respectively.Conclusions: This study showed that the delineation of four main flaps used for HNC was reproducible. The delineation of the FAMM, however, requires close cooperation between radiologist, surgeon and radiation oncologist because of the poor visibility of this flap on CT and its small size.