Toward a standard for the evaluation of PET-Auto-Segmentation methods following the recommendations of AAPM task group No. 211: Requirements and implementation.

PURPOSE: The aim of this paper is to define the requirements and describe the design and implementation of a standard benchmark tool for evaluation and validation of PET-auto-segmentation (PET-AS) algorithms. This work follows the recommendations of Task Group 211 (TG211) appointed by the American Association of Physicists in Medicine (AAPM). METHODS: The recommendations published in the AAPM TG211 report were used to derive a set of required features and to guide the design and structure of a benchmarking software tool. These items included the selection of appropriate representative data and reference contours obtained from established approaches and the description of available metrics. The benchmark was designed in a way that it could be extendable by inclusion of bespoke segmentation methods, while maintaining its main purpose of being a standard testing platform for newly developed PET-AS methods. An example of implementation of the proposed framework, named PETASset, was built. In this work, a selection of PET-AS methods representing common approaches to PET image segmentation was evaluated within PETASset for the purpose of testing and demonstrating the capabilities of the software as a benchmark platform. RESULTS: A selection of clinical, physical, and simulated phantom data, including "best estimates" reference contours from macroscopic specimens, simulation template, and CT scans was built into the PETASset application database. Specific metrics such as Dice Similarity Coefficient (DSC), Positive Predictive Value (PPV), and Sensitivity (S), were included to allow the user to compare the results of any given PET-AS algorithm to the reference contours. In addition, a tool to generate structured reports on the evaluation of the performance of PET-AS algorithms against the reference contours was built. The variation of the metric agreement values with the reference contours across the PET-AS methods evaluated for demonstration were between 0.51 and 0.83, 0.44 and 0.86, and 0.61 and 1.00 for DSC, PPV, and the S metric, respectively. Examples of agreement limits were provided to show how the software could be used to evaluate a new algorithm against the existing state-of-the art. CONCLUSIONS: PETASset provides a platform that allows standardizing the evaluation and comparison of different PET-AS methods on a wide range of PET datasets. The developed platform will be available to users willing to evaluate their PET-AS methods and contribute with more evaluation datasets.

Classification and evaluation strategies of auto-segmentation approaches for PET: Report of AAPM task group No. 211.

PURPOSE: The purpose of this educational report is to provide an overview of the present state-of-the-art PET auto-segmentation (PET-AS) algorithms and their respective validation, with an emphasis on providing the user with help in understanding the challenges and pitfalls associated with selecting and implementing a PET-AS algorithm for a particular application. APPROACH: A brief description of the different types of PET-AS algorithms is provided using a classification based on method complexity and type. The advantages and the limitations of the current PET-AS algorithms are highlighted based on current publications and existing comparison studies. A review of the available image datasets and contour evaluation metrics in terms of their applicability for establishing a standardized evaluation of PET-AS algorithms is provided. The performance requirements for the algorithms and their dependence on the application, the radiotracer used and the evaluation criteria are described and discussed. Finally, a procedure for algorithm acceptance and implementation, as well as the complementary role of manual and auto-segmentation are addressed. FINDINGS: A large number of PET-AS algorithms have been developed within the last 20 years. Many of the proposed algorithms are based on either fixed or adaptively selected thresholds. More recently, numerous papers have proposed the use of more advanced image analysis paradigms to perform semi-automated delineation of the PET images. However, the level of algorithm validation is variable and for most published algorithms is either insufficient or inconsistent which prevents recommending a single algorithm. This is compounded by the fact that realistic image configurations with low signal-to-noise ratios (SNR) and heterogeneous tracer distributions have rarely been used. Large variations in the evaluation methods used in the literature point to the need for a standardized evaluation protocol. CONCLUSIONS: Available comparison studies suggest that PET-AS algorithms relying on advanced image analysis paradigms provide generally more accurate segmentation than approaches based on PET activity thresholds, particularly for realistic configurations. However, this may not be the case for simple shape lesions in situations with a narrower range of parameters, where simpler methods may also perform well. Recent algorithms which employ some type of consensus or automatic selection between several PET-AS methods have potential to overcome the limitations of the individual methods when appropriately trained. In either case, accuracy evaluation is required for each different PET scanner and scanning and image reconstruction protocol. For the simpler, less robust approaches, adaptation to scanning conditions, tumor type, and tumor location by optimization of parameters is necessary. The results from the method evaluation stage can be used to estimate the contouring uncertainty. All PET-AS contours should be critically verified by a physician. A standard test, i.e., a benchmark dedicated to evaluating both existing and future PET-AS algorithms needs to be designed, to aid clinicians in evaluating and selecting PET-AS algorithms and to establish performance limits for their acceptance for clinical use. The initial steps toward designing and building such a standard are undertaken by the task group members.

Radiotherapy volume delineation using dynamic [18F]-FDG PET/CT imaging in patients with oropharyngeal cancer: a pilot study.

PURPOSE: Delineation of gross tumour volume in 3D is a critical step in the radiotherapy (RT) treatment planning for oropharyngeal cancer (OPC). Static [18F]-FDG PET/CT imaging has been suggested as a method to improve the reproducibility of tumour delineation, but it suffers from low specificity. We undertook this pilot study in which dynamic features in time-activity curves (TACs) of [18F]-FDG PET/CT images were applied to help the discrimination of tumour from inflammation and adjacent normal tissue. METHODS: Five patients with OPC underwent dynamic [18F]-FDG PET/CT imaging in treatment position. Voxel-by-voxel analysis was performed to evaluate seven dynamic features developed with the knowledge of differences in glucose metabolism in different tissue types and visual inspection of TACs. The Gaussian mixture model and K-means algorithms were used to evaluate the performance of the dynamic features in discriminating tumour voxels compared to the performance of standardized uptake values obtained from static imaging. RESULTS: Some dynamic features showed a trend towards discrimination of different metabolic areas but lack of consistency means that clinical application is not recommended based on these results alone. CONCLUSIONS: Impact of inflammatory tissue remains a problem for volume delineation in RT of OPC, but a simple dynamic imaging protocol proved practicable and enabled simple data analysis techniques that show promise for complementing the information in static uptake values.

Reply to the comments on "comparative study with new accuracy metrics for target volume contouring in PET image guided radiation therapy".

This communication is submitted in response to the letter of van den Hoff and Hofheinz (2013). Based on findings in their earlier study (Hofheinz , 2010) the letter criticizes the use of a physical positron emission tomography (PET) phantom with "cold wall" volumes of interest, in part of the evaluation of PET segmentation tools in our experiment reported in this issue (Shepherd , 2012). In addition, the letter raises concerns about the low number of independent expert (manual) delineations used in Shepherd , (2012) to assess accuracy of tumor segmentation in patient images, and disambiguates the details of one of the segmentation methods involved in Shepherd , (2012).

Comparative assessment of segmentation algorithms for tumor delineation on a test-retest [(11)C]choline dataset.

PURPOSE: Many methods have been proposed for tumor segmentation from positron emission tomography images. Because of the increasingly important role that [(11)C]choline is playing in oncology and because no study has compared segmentation methods on this tracer, the authors assessed several segmentation algorithms on a [(11)C]choline test-retest dataset. METHODS: Fixed and adaptive threshold-based methods, fuzzy C-means (FCM), Canny's edge detection method, the watershed transform, and the fuzzy locally adaptive Bayesian algorithm (FLAB) were used. Test-retest [(11)C]choline scans of nine patients with breast cancer were considered and the percent test-retest variability %VAR(TEST-RETEST) of tumor volume (TV) was employed to assess the results. The same methods were then applied to two denoised datasets generated by applying either a Gaussian filter or the wavelet transform. RESULTS: The (semi)automated methods FCM, FLAB, and Canny emerged as the best ones in terms of TV reproducibility. For these methods, the %root mean square error %RMSE of %VAR(TEST-RETEST), defined as %RMSE= variance+mean(2), was in the range 10%-21.2%, depending on the dataset and algorithm. Threshold-based methods gave TV estimates which were extremely variable, particularly on the unsmoothed data; their performance improved on the denoised datasets, whereas smoothing did not have a remarkable impact on the (semi)automated methods. TV variability was comparable to that of SUV(MAX) and SUV(MEAN) (range 14.7%-21.9% for %RMSE of %VAR(TEST-RETEST), after the exclusion of one outlier, 40%-43% when the outlier was included). CONCLUSIONS: The TV variability obtained with the best methods was similar to the one reported for TV in previous [(18)F]FDG and [(18)F]FLT studies and to the one of SUV(MAX)∕SUV(MEAN) on the authors' [(11)C]choline dataset. The good reproducibility of [(11)C]choline TV warrants further studies to test whether TV could predict early response to treatment and survival, as for [(18)F]FDG, to complement∕substitute the use of SUV(MAX) and SUV(MEAN).

Comparative study with new accuracy metrics for target volume contouring in PET image guided radiation therapy.

The impact of PET on radiation therapy is held back by poor methods of defining functional volumes of interest. Many new software tools are being proposed for contouring target volumes but the different approaches are not adequately compared and their accuracy is poorly evaluated due to the illdefinition of ground truth. This paper compares the largest cohort to date of established, emerging and proposed PET contouring methods, in terms of accuracy and variability. We emphasise spatial accuracy and present a new metric that addresses the lack of unique ground truth. 30 methods are used at 13 different institutions to contour functional VOIs in clinical PET/CT and a custom-built PET phantom representing typical problems in image guided radiotherapy. Contouring methods are grouped according to algorithmic type, level of interactivity and how they exploit structural information in hybrid images. Experiments reveal benefits of high levels of user interaction, as well as simultaneous visualisation of CT images and PET gradients to guide interactive procedures. Method-wise evaluation identifies the danger of over-automation and the value of prior knowledge built into an algorithm.

Interactive lesion segmentation with shape priors from offline and online learning.

In medical image segmentation, tumors and other lesions demand the highest levels of accuracy but still call for the highest levels of manual delineation. One factor holding back automatic segmentation is the exemption of pathological regions from shape modelling techniques that rely on high-level shape information not offered by lesions. This paper introduces two new statistical shape models (SSMs) that combine radial shape parameterization with machine learning techniques from the field of nonlinear time series analysis. We then develop two dynamic contour models (DCMs) using the new SSMs as shape priors for tumor and lesion segmentation. From training data, the SSMs learn the lower level shape information of boundary fluctuations, which we prove to be nevertheless highly discriminant. One of the new DCMs also uses online learning to refine the shape prior for the lesion of interest based on user interactions. Classification experiments reveal superior sensitivity and specificity of the new shape priors over those previously used to constrain DCMs. User trials with the new interactive algorithms show that the shape priors are directly responsible for improvements in accuracy and reductions in user demand.

Gaussian process models of dynamic PET for functional volume definition in radiation oncology.

In routine oncologic positron emission tomography (PET), dynamic information is discarded by time-averaging the signal to produce static images of the "standardised uptake value" (SUV). Defining functional volumes of interest (VOIs) in terms of SUV is flawed, as values are affected by confounding factors and the chosen time window, and SUV images are not sensitive to functional heterogeneity of pathological tissues. Also, SUV iso-contours are highly affected by the choice of threshold and no threshold, or other SUV-based segmentation method, is universally accepted for a given VOI type. Gaussian Process (GP) time series models describe macro-scale dynamic behavior arising from countless interacting micro-scale processes, as is the case for PET signals from heterogeneous tissue. We use GPs to model time-activity curves (TACs) from dynamic PET and to define functional volumes for PET oncology. Probabilistic methods of tissue discrimination are presented along with novel contouring methods for functional VOI segmentation. We demonstrate the value of GP models for voxel classification and VOI contouring of diseased and metastatic tissues with functional heterogeneity in prostate PET. Classification experiments reveal superior sensitivity and specificity over SUV calculation and a TAC-based method proposed in recent literature. Contouring experiments reveal differences in shape between gold-standard and GP VOIs and correlation with kinetic models shows that the novel VOIs contain extra clinically relevant information compared to SUVs alone. We conclude that the proposed models offer a principled data analysis technique that improves on SUVs for oncologic VOI definition. Continuing research will generalize GP models for different oncology tracers and imaging protocols with the ultimate goal of clinical use including treatment planning.

Unsupervised tumour segmentation in PET based on local and global intensity fitting active surface and alpha matting.

This paper proposes an unsupervised tumour segmentation scheme for PET data. The method computes the volume of interests (VOIs) with subpixel precision by considering the limited resolution and partial volume effect. Firstly, it uses local and global intensity active surface modelling to segment VOIs, then an alpha matting method is used to eliminate false negative classification and refine the segmentation results. We have validated our method on real PET images of head-and-neck cancer patients as well as images of a custom designed PET phantom. Experiments show that our method can generate more accurate segmentation results compared with alternative approaches.

Stochastic model for heart-rate fluctuations.

A normal human heart rate shows complex fluctuations in time, which is natural, because the heart rate is controlled by a large number of different feedback control loops. These unpredictable fluctuations have been shown to display fractal dynamics, long-term correlations, and 1/f noise. These characterizations are statistical and they have been widely studied and used, but much less is known about the detailed time evolution (dynamics) of the heart-rate control mechanism. Here we show that a simple one-dimensional Langevin-type stochastic difference equation can accurately model the heart-rate fluctuations in a time scale from minutes to hours. The model consists of a deterministic nonlinear part and a stochastic part typical to Gaussian noise, and both parts can be directly determined from the measured heart-rate data. Studies of 27 healthy subjects reveal that in most cases, the deterministic part has a form typically seen in bistable systems: there are two stable fixed points and one unstable one.