An automated methodology for whole-body, multimodality tracking of individual cancer lesions.

Objective. Manual analysis of individual cancer lesions to assess disease response is clinically impractical and requires automated lesion tracking methodologies. However, no methodology has been developed for whole-body individual lesion tracking, across an arbitrary number of scans, and acquired with various imaging modalities.Approach. This study introduces a lesion tracking methodology and benchmarked it using 2368Ga-DOTATATE PET/CT and PET/MR images of eight neuroendocrine tumor patients. The methodology consists of six steps: (1) alignment of multiple scans via image registration, (2) body-part labeling, (3) automatic lesion-wise dilation, (4) clustering of lesions based on local lesion shape metrics, (5) assignment of lesion tracks, and (6) output of a lesion graph. Registration performance was evaluated via landmark distance, lesion matching accuracy was evaluated between each image pair, and lesion tracking accuracy was evaluated via identical track ratio. Sensitivity studies were performed to evaluate the impact of lesion dilation (fixed versus automatic dilation), anatomic location, image modalities (inter- versus intra-modality), registration mode (direct versus indirect registration), and track size (number of time-points and lesions) on lesion matching and tracking performance.Main results. Manual contouring yielded 956 lesions, 1570 lesion-matching decisions, and 493 lesion tracks. The median residual registration error was 2.5 mm. The automatic lesion dilation led to 0.90 overall lesion matching accuracy, and an 88% identical track ratio. The methodology is robust regarding anatomic locations, image modalities, and registration modes. The number of scans had a moderate negative impact on the identical track ratio (94% for 2 scans, 91% for 3 scans, and 81% for 4 scans). The number of lesions substantially impacted the identical track ratio (93% for 2 nodes versus 54% for ≥5 nodes).Significance. The developed methodology resulted in high lesion-matching accuracy and enables automated lesion tracking in PET/CT and PET/MR.

Multi-organ segmentation of CT via convolutional neural network: impact of training setting and scanner manufacturer.

Objective. Automated organ segmentation on CT images can enable the clinical use of advanced quantitative software devices, but model performance sensitivities must be understood before widespread adoption can occur. The goal of this study was to investigate performance differences between Convolutional Neural Networks (CNNs) trained to segment one (single-class) versus multiple (multi-class) organs, and between CNNs trained on scans from a single manufacturer versus multiple manufacturers.Methods. The multi-class CNN was trained on CT images obtained from 455 whole-body PET/CT scans (413 for training, 42 for testing) taken with Siemens, GE, and Phillips PET/CT scanners where 16 organs were segmented. The multi-class CNN was compared to 16 smaller single-class CNNs trained using the same data, but with segmentations of only one organ per model. In addition, CNNs trained on Siemens-only (N = 186) and GE-only (N = 219) scans (manufacturer-specific) were compared with CNNs trained on data from both Siemens and GE scanners (manufacturer-mixed). Segmentation performance was quantified using five performance metrics, including the Dice Similarity Coefficient (DSC).Results. The multi-class CNN performed well compared to previous studies, even in organs usually considered difficult auto-segmentation targets (e.g., pancreas, bowel). Segmentations from the multi-class CNN were significantly superior to those from smaller single-class CNNs in most organs, and the 16 single-class models took, on average, six times longer to segment all 16 organs compared to the single multi-class model. The manufacturer-mixed approach achieved minimally higher performance over the manufacturer-specific approach.Significance. A CNN trained on contours of multiple organs and CT data from multiple manufacturers yielded high-quality segmentations. Such a model is an essential enabler of image processing in a software device that quantifies and analyzes such data to determine a patient's treatment response. To date, this activity of whole organ segmentation has not been adopted due to the intense manual workload and time required.

Performance of an automated registration-based method for longitudinal lesion matching and comparison to inter-reader variability.

Objective.Patients with metastatic disease are followed throughout treatment with medical imaging, and accurately assessing changes of individual lesions is critical to properly inform clinical decisions. The goal of this work was to assess the performance of an automated lesion-matching algorithm in comparison to inter-reader variability (IRV) of matching lesions between scans of metastatic cancer patients.Approach.Forty pairs of longitudinal PET/CT and CT scans were collected and organized into four cohorts: lung cancers, head and neck cancers, lymphomas, and advanced cancers. Cases were also divided by cancer burden: low-burden (<10 lesions), intermediate-burden (10-29), and high-burden (30+). Two nuclear medicine physicians conducted independent reviews of each scan-pair and manually matched lesions. Matching differences between readers were assessed to quantify the IRV of lesion matching. The two readers met to form a consensus, which was considered a gold standard and compared against the output of an automated lesion-matching algorithm. IRV and performance of the automated method were quantified using precision, recall, F1-score, and the number of differences.Main results.The performance of the automated method did not differ significantly from IRV for any metric in any cohort (p> 0.05, Wilcoxon paired test). In high-burden cases, the F1-score (median [range]) was 0.89 [0.63, 1.00] between the automated method and reader consensus and 0.93 [0.72, 1.00] between readers. In low-burden cases, F1-scores were 1.00 [0.40, 1.00] and 1.00 [0.40, 1.00], for the automated method and IRV, respectively. Automated matching was significantly more efficient than either reader (p< 0.001). In high-burden cases, median matching time for the readers was 60 and 30 min, respectively, while automated matching took a median of 3.9 minSignificance.The automated lesion-matching algorithm was successful in performing lesion matching, meeting the benchmark of IRV. Automated lesion matching can significantly expedite and improve the consistency of longitudinal lesion-matching.

Quantitative imaging biomarkers of immune-related adverse events in immune-checkpoint blockade-treated metastatic melanoma patients: a pilot study.

PURPOSE: To develop quantitative molecular imaging biomarkers of immune-related adverse event (irAE) development in malignant melanoma (MM) patients receiving immune-checkpoint inhibitors (ICI) imaged with 18F-FDG PET/CT. METHODS: 18F-FDG PET/CT images of 58 MM patients treated with anti-PD-1 or anti-CTLA-4 ICI were retrospectively analyzed for indication of irAE. Three target organs, most commonly affected by irAE, were considered: bowel, lung, and thyroid. Patient charts were reviewed to identify which patients experienced irAE, irAE grade, and time to irAE diagnosis. Target organs were segmented using a convolutional neural network (CNN), and novel quantitative imaging biomarkers - SUV percentiles (SUVX%) of 18F-FDG uptake within the target organs - were correlated with the clinical irAE status. Area under the receiver-operating characteristic curve (AUROC) was used to quantify irAE detection performance. Patients who did not experience irAE were used to establish normal ranges for target organ 18F-FDG uptake. RESULTS: A total of 31% (18/58) patients experienced irAE in the three target organs: bowel (n=6), lung (n=5), and thyroid (n=9). Optimal percentiles for identifying irAE were bowel (SUV95%, AUROC=0.79), lung (SUV95%, AUROC=0.98), and thyroid (SUV75%, AUROC=0.88). Optimal cut-offs for irAE detection were bowel (SUV95%>2.7 g/mL), lung (SUV95%>1.7 g/mL), and thyroid (SUV75%>2.1 g/mL). Normal ranges (95% confidence interval) for the SUV percentiles in patients without irAE were bowel [1.74, 2.86 g/mL], lung [0.73, 1.46 g/mL], and thyroid [0.86, 1.99 g/mL]. CONCLUSIONS: Increased 18F-FDG uptake within irAE-affected organs provides predictive information about the development of irAE in MM patients receiving ICI and represents a potential quantitative imaging biomarker for irAE. Some irAE can be detected on 18F-FDG PET/CT well before clinical symptoms appear.

Image intensity histograms as imaging biomarkers: application to immune-related colitis.

Purpose.To investigate image intensity histograms as a potential source of useful imaging biomarkers in both a clinical example of detecting immune-related colitis (irColitis) in18F-FDG PET/CT images of immunotherapy patients and an idealized case of classifying digital reference objects (DRO).Methods.Retrospective analysis of bowel18F-FDG uptake in N = 40 patients receiving immune checkpoint inhibitors was conducted. A CNN trained to segment the bowel was used to generate the histogram of bowel18F-FDG uptake, and percentiles of the histogram were considered as potential metrics for detecting inflammation associated with irColitis. A model of the colon was also considered using cylindrical DRO. Classification of DRO with different intensity distributions was undertaken under varying geometry and noise settings.Results.The most predictive biomarker of irColitis was the 95th percentile of the bowel SUV histogram (SUV95%). Patients later diagnosed with irColitis had a significantly higher increase in SUV95%from baseline to first on-treatment PET than patients who did not experience irColitis (p = 0.02). An increase in SUV95%> + 40% separated pre-irColitis change from normal variability with a sensitivity of 75% and specificity of 88%. Furthermore, histogram percentiles were ideal metrics for classifying 'hot center' and 'cold center' DRO, and were robust to varying DRO geometry and noise, and to the presence of spoiler volumes unrelated to the detection task.Conclusions.The 95th percentile of the bowel SUV histogram was the optimal metric for detecting irColitis on18F-FDG PET/CT. Image intensity histograms are a promising source of imaging biomarkers for clinical tasks.

Interpretation and visualization techniques for deep learning models in medical imaging.

Deep learning (DL) approaches to medical image analysis tasks have recently become popular; however, they suffer from a lack of human interpretability critical for both increasing understanding of the methods' operation and enabling clinical translation. This review summarizes currently available methods for performing image model interpretation and critically evaluates published uses of these methods for medical imaging applications. We divide model interpretation in two categories: (1) understanding model structure and function and (2) understanding model output. Understanding model structure and function summarizes ways to inspect the learned features of the model and how those features act on an image. We discuss techniques for reducing the dimensionality of high-dimensional data and cover autoencoders, both of which can also be leveraged for model interpretation. Understanding model output covers attribution-based methods, such as saliency maps and class activation maps, which produce heatmaps describing the importance of different parts of an image to the model prediction. We describe the mathematics behind these methods, give examples of their use in medical imaging, and compare them against one another. We summarize several published toolkits for model interpretation specific to medical imaging applications, cover limitations of current model interpretation methods, provide recommendations for DL practitioners looking to incorporate model interpretation into their task, and offer general discussion on the importance of model interpretation in medical imaging contexts.

FDG PET/CT for Assessment of Immune Therapy: Opportunities and Understanding Pitfalls.

Immune checkpoint blockade has demonstrated the ability to modulate the immune system to produce durable responses in a wide range of cancers and has significantly impacted the standard of care. However, many cancer patients still do not respond to immune checkpoint blockade or have a limited duration of antitumor responses. Moreover, immune-related adverse events caused by immune checkpoint blockade can be severe and debilitating for some patients, limiting continuation of therapy and resulting in severe autoimmune conditions. Standard-of-care conventional anatomic imaging modalities and tumor response criteria have limitations to adequately assess tumor responses, especially early in the course of therapy, for risk-adapted clinical management to inform care of patients treated with immunotherapy. Molecular imaging with position emission tomography (PET) provides a noninvasive functional biomarker of tumor response, and of immune activation, for patients on immune-based therapies to help address these needs. 18F-FDG (FDG) PET/CT is readily available clinically and a number of studies have evaluated the use of this agent for assessment of prognosis, treatment response and immune activation for patients treated with immune checkpoint blockade. In this review paper, we discuss the current oncologic applications and imaging needs of cancer immunotherapy, recent studies applying FDG PET/CT for tumor response assessment, and evaluation of immune-related adverse events for improving clinical management. We largely focus on metastatic melanoma; however, we generalize where applicable to immunotherapy in other tumor types. We also briefly discuss PET imaging and quantitation as well as emerging non-FDG PET imaging radiotracers for cancer immunotherapy imaging.