Revolutionizing Digital Pathology With the Power of Generative Artificial Intelligence and Foundation Models.

Digital pathology has transformed the traditional pathology practice of analyzing tissue under a microscope into a computer vision workflow. Whole-slide imaging allows pathologists to view and analyze microscopic images on a computer monitor, enabling computational pathology. By leveraging artificial intelligence (AI) and machine learning (ML), computational pathology has emerged as a promising field in recent years. Recently, task-specific AI/ML (eg, convolutional neural networks) has risen to the forefront, achieving above-human performance in many image-processing and computer vision tasks. The performance of task-specific AI/ML models depends on the availability of many annotated training datasets, which presents a rate-limiting factor for AI/ML development in pathology. Task-specific AI/ML models cannot benefit from multimodal data and lack generalization, eg, the AI models often struggle to generalize to new datasets or unseen variations in image acquisition, staining techniques, or tissue types. The 2020s are witnessing the rise of foundation models and generative AI. A foundation model is a large AI model trained using sizable data, which is later adapted (or fine-tuned) to perform different tasks using a modest amount of task-specific annotated data. These AI models provide in-context learning, can self-correct mistakes, and promptly adjust to user feedback. In this review, we provide a brief overview of recent advances in computational pathology enabled by task-specific AI, their challenges and limitations, and then introduce various foundation models. We propose to create a pathology-specific generative AI based on multimodal foundation models and present its potentially transformative role in digital pathology. We describe different use cases, delineating how it could serve as an expert companion of pathologists and help them efficiently and objectively perform routine laboratory tasks, including quantifying image analysis, generating pathology reports, diagnosis, and prognosis. We also outline the potential role that foundation models and generative AI can play in standardizing the pathology laboratory workflow, education, and training.

The predictive value of pretherapy [68Ga]Ga-DOTA-TATE PET and biomarkers in [177Lu]Lu-PRRT tumor dosimetry.

PURPOSE: Metastatic neuroendocrine tumors (NETs) overexpressing type 2 somatostatin receptors are the target for peptide receptor radionuclide therapy (PRRT) through the theragnostic pair of 68Ga/177Lu-DOTATATE. The main purpose of this study was to develop machine learning models to predict therapeutic tumor dose using pre therapy 68Ga -PET and clinicopathological biomarkers. METHODS: We retrospectively analyzed 90 segmented metastatic NETs from 25 patients (M14/F11, age 63.7 ± 9.5, range 38-76) treated by 177Lu-DOTATATE at our institute. Patients underwent both pretherapy [68Ga]Ga-DOTA-TATE PET/CT and four timepoints SPECT/CT at ~ 4, 24, 96, and 168 h post-177Lu-DOTATATE infusion. Tumors were segmented by a radiologist on baseline CT or MRI and transferred to co-registered PET/CT and SPECT/CT, and normal organs were segmented by deep learning-based method on CT of the PET and SPECT. The SUV metrics and tumor-to-normal tissue SUV ratios (SUV_TNRs) were calculated from 68Ga -PET at the contour-level. Posttherapy dosimetry was performed based on the co-registration of SPECT/CTs to generate time-integrated-activity, followed by an in-house Monte Carlo-based absorbed dose estimation. The correlation between delivered 177Lu Tumor absorbed dose and PET-derived metrics along with baseline clinicopathological biomarkers (such as Creatinine, Chromogranin A and prior therapies) were evaluated. Multiple interpretable machine-learning algorithms were developed to predict tumor dose using these pretherapy information. Model performance on a nested tenfold cross-validation was evaluated in terms of coefficient of determination (R2), mean-absolute-error (MAE), and mean-relative-absolute-error (MRAE). RESULTS: SUVmean showed a significant correlation (q-value < 0.05) with absorbed dose (Spearman ρ = 0.64), followed by TLSUVmean (SUVmean of total-lesion-burden) and SUVpeak (ρ = 0.45 and 0.41, respectively). The predictive value of PET-SUVmean in estimation of posttherapy absorbed dose was stronger compared to PET-SUVpeak, and SUV_TNRs in terms of univariate analysis (R2 = 0.28 vs. R2 ≤ 0.12). An optimal trivariate random forest model composed of SUVmean, TLSUVmean, and total liver SUVmean (normal and tumoral liver) provided the best performance in tumor dose prediction with R2 = 0.64, MAE = 0.73 Gy/GBq, and MRAE = 0.2. CONCLUSION: Our preliminary results demonstrate the feasibility of using baseline PET images for prediction of absorbed dose prior to 177Lu-PRRT. Machine learning models combining multiple PET-based metrics performed better than using a single SUV value and using other investigated clinicopathological biomarkers. Developing such quantitative models forms the groundwork for the role of 68Ga -PET not only for the implementation of personalized treatment planning but also for patient stratification in the era of precision medicine.

Enhanced immune activation within the tumor microenvironment and circulation of female high-risk melanoma patients and improved survival with adjuvant CTLA4 blockade compared to males.

BACKGROUND: We hypothesized that a gender difference in clinical response may exist to adjuvant CTLA4 blockade with ipilimumab versus high-dose IFNα (HDI). We investigated differences in candidate immune biomarkers in the circulation and tumor microenvironment (TME). PATIENTS AND METHODS: This gender-based analysis was nested within the E1609 trial that tested adjuvant therapy with ipilimumab 3 mg/kg (ipi3) and 10 mg/kg (ipi10) versus HDI in high risk resected melanoma. We investigated gender differences in treatment efficacy with ipi3 and ipi10 versus HDI while adjusting for age, stage, ECOG performance (PS), ulceration, primary tumor status and lymph node number. Forest plots were created to compare overall survival (OS) and relapse free survival (RFS) between ipi and HDI. Gene expression profiling (GEP) was performed on tumors of 718 (454 male, 264 female) patients. Similarly, serum and peripheral blood mononuclear cells (PBMC) samples were tested for soluble and cellular biomarkers (N = 321 patients; 109 female and 212 male). RESULTS: The subgroups of female, stage IIIC, PS = 1, ulcerated primary, in-transit metastasis demonstrated significant improvement in RFS and/or OS with ipi3 versus HDI. Female gender was significant for both OS and RFS and was further explored. In the RFS comparison, a multivariate Cox regression model including significant variables indicated a significant interaction between gender and treatment (P = 0.024). In peripheral blood, percentages of CD3+ T cells (P = 0.024) and CD3+ CD4+ helper T cells (P = 0.0001) were higher in females compared to males. Trends toward higher circulating levels of IL1ß (P = 0.07) and IL6 (P = 0.06) were also found in females. Males had higher percentages of monocytes (P = 0.03) with trends toward higher percentages of regulatory T cells (T-reg). Tumor GEP analysis supported enhanced infiltration with immune cells including gammadelta T cells (P = 0.005), NK cells (P = 0.01), dendritic cells (P = 0.01), CD4+ T cells (P = 0.03), CD8+ T cells (P = 0.03) and T-reg (P = 0.008) in the tumors of females compared to males and a higher T-effector and IFNγ gene signature score (P = 0.0244). CONCLUSION: Female gender was associated with adjuvant CTLA4 blockade clinical benefits and female patients were more likely to have evidence of type1 immune activation within the TME and the circulation. Trial registration ClinicalTrials.gov NCT01274338. Registered 11 January 2011, https://www. CLINICALTRIALS: gov/ct2/show/NCT01274338.

Quantitative Molecular Positron Emission Tomography Imaging Using Advanced Deep Learning Techniques.

The widespread availability of high-performance computing and the popularity of artificial intelligence (AI) with machine learning and deep learning (ML/DL) algorithms at the helm have stimulated the development of many applications involving the use of AI-based techniques in molecular imaging research. Applications reported in the literature encompass various areas, including innovative design concepts in positron emission tomography (PET) instrumentation, quantitative image reconstruction and analysis techniques, computer-aided detection and diagnosis, as well as modeling and prediction of outcomes. This review reflects the tremendous interest in quantitative molecular imaging using ML/DL techniques during the past decade, ranging from the basic principles of ML/DL techniques to the various steps required for obtaining quantitatively accurate PET data, including algorithms used to denoise or correct for physical degrading factors as well as to quantify tracer uptake and metabolic tumor volume for treatment monitoring or radiation therapy treatment planning and response prediction.This review also addresses future opportunities and current challenges facing the adoption of ML/DL approaches and their role in multimodality imaging.

Cluster model incorporating heterogeneous dose distribution of partial parotid irradiation for radiotherapy induced xerostomia prediction with machine learning methods.

PURPOSE: A cluster model incorporating heterogeneous dose distribution within the parotid gland was developed and validated retrospectively for radiotherapy (RT) induced xerostomia prediction with machine learning (ML) techniques. METHODS: Sixty clusters were obtained at 1 Gy step size with threshold doses ranging from 1 to 60 Gy, for each of the enrolled 155 patients with HNC from three institutions. Feature clusters were selected with the neighborhood component analysis (NCA) and subsequently fed into four supervised ML models for xerostomia prediction comparison: support vector machines (SVM), k-nearest neighbor (kNN), naïve Bayes (NB), and random forest (RF). The predictive performance of each model was evaluated using cross validation resampling with the area-under-the-curves (AUC) of the receiver-operating-characteristic (ROC). The xerostomia predicting capacity using testing data was assessed with accuracy, sensitivity, and specificity for these models and three cluster connectivity choices. Mean dose based logistic regression served as the benchmark for evaluation. RESULTS: Feature clusters identified by NCA fell in three threshold dose ranges: 5-15Gy, 25-35Gy, and 45-50Gy. Mean dose predictive power was 15% lower than that of the cluster model using the logistic regression classifier. Model validation demonstrated that kNN model outperformed slightly other three models but no substantial difference was observed. Applying the fine-tuned models to testing data yielded that the mean accuracy from SVM, kNN and NB models were between 0.68 and 0.7 while that of RF was ∼0.6. SVM model yielded the best sensitivity (0.76) and kNN model delivered consistent sensitivity and specificity. This is consistent with cross validation. Clusters calculated with three connectivity choices exhibited minimally different predictions. CONCLUSION: Compared to mean dose, the proposed cluster model has shown its improvement as the xerostomia predictor. When combining with ML techniques, it could provide a clinically useful tool for xerostomia prediction and facilitate decision making during radiotherapy planning for patients with HNC.

The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping.

Background Radiomic features may quantify characteristics present in medical imaging. However, the lack of standardized definitions and validated reference values have hampered clinical use. Purpose To standardize a set of 174 radiomic features. Materials and Methods Radiomic features were assessed in three phases. In phase I, 487 features were derived from the basic set of 174 features. Twenty-five research teams with unique radiomics software implementations computed feature values directly from a digital phantom, without any additional image processing. In phase II, 15 teams computed values for 1347 derived features using a CT image of a patient with lung cancer and predefined image processing configurations. In both phases, consensus among the teams on the validity of tentative reference values was measured through the frequency of the modal value and classified as follows: less than three matches, weak; three to five matches, moderate; six to nine matches, strong; 10 or more matches, very strong. In the final phase (phase III), a public data set of multimodality images (CT, fluorine 18 fluorodeoxyglucose PET, and T1-weighted MRI) from 51 patients with soft-tissue sarcoma was used to prospectively assess reproducibility of standardized features. Results Consensus on reference values was initially weak for 232 of 302 features (76.8%) at phase I and 703 of 1075 features (65.4%) at phase II. At the final iteration, weak consensus remained for only two of 487 features (0.4%) at phase I and 19 of 1347 features (1.4%) at phase II. Strong or better consensus was achieved for 463 of 487 features (95.1%) at phase I and 1220 of 1347 features (90.6%) at phase II. Overall, 169 of 174 features were standardized in the first two phases. In the final validation phase (phase III), most of the 169 standardized features could be excellently reproduced (166 with CT; 164 with PET; and 164 with MRI). Conclusion A set of 169 radiomics features was standardized, which enabled verification and calibration of different radiomics software. © RSNA, 2020 Online supplemental material is available for this article. See also the editorial by Kuhl and Truhn in this issue.

Machine Learning and Imaging Informatics in Oncology.

In the era of personalized and precision medicine, informatics technologies utilizing machine learning (ML) and quantitative imaging are witnessing a rapidly increasing role in medicine in general and in oncology in particular. This expanding role ranges from computer-aided diagnosis to decision support of treatments with the potential to transform the current landscape of cancer management. In this review, we aim to provide an overview of ML methodologies and imaging informatics techniques and their recent application in modern oncology. We will review example applications of ML in oncology from the literature, identify current challenges and highlight future potentials.

Oncology Informatics: Status Quo and Outlook.

Oncology has undergone rapid progress, with emerging developments in areas including cancer stem cells, molecularly targeted therapies, genomic analyses, and individually tailored immunotherapy. These advances have expanded the tools available in the fight against cancer. Some of these have seen broad media coverage resulting in justified public attention. However, these achievements have only been possible due to rapid developments in the expanding field of biomedical informatics and information technology (IT). Artificial intelligence, radiomics, electronic health records, and electronic patient-reported outcome measures (ePROMS) are only a few of the developments enabling further progress in oncology. The promising impact of IT in oncology will only become reality through a multidisciplinary approach to the complex challenges ahead.

Machine learning for radiomics-based multimodality and multiparametric modeling.

Due to the recent developments of both hardware and software technologies, multimodality medical imaging techniques have been increasingly applied in clinical practice and research studies. Previously, the application of multimodality imaging in oncology has been mainly related to combining anatomical and functional imaging to improve diagnostic specificity and/or target definition, such as positron emission tomography/computed tomography (PET/CT) and single-photon emission CT (SPECT)/CT. More recently, the fusion of various images, such as multiparametric magnetic resonance imaging (MRI) sequences, different PET tracer images, PET/MRI, has become more prevalent, which has enabled more comprehensive characterization of the tumor phenotype. In order to take advantage of these valuable multimodal data for clinical decision making using radiomics, we present two ways to implement the multimodal image analysis, namely radiomic (handcrafted feature) based and deep learning (machine learned feature) based methods. Applying advanced machine (deep) learning algorithms across multimodality images have shown better results compared with single modality modeling for prognostic and/or prediction of clinical outcomes. This holds great potentials for providing more personalized treatment for patients and achieve better outcomes.

Integrating radiomics into clinical trial design.

In radiomics, quantitative features that describe phenotypic tumor characteristics are derived from radiographic images. Because radiomics generates information from routine medical images, it is a powerful way to non-invasively examine the spatial and temporal heterogeneity of disease, and thus has potential to significantly impact clinical trial design, execution, and ultimately patient care. The aim of this review article is to discuss how radiomics may address some of the current challenges in clinical randomized control trials, and the difficulties of integrating robust and repeatable radiomics analysis into trial design. Each step of the radiomics process, including image acquisition and reconstruction, image segmentation, feature extraction, and computational analysis, requires extensive standardization in order to be successfully incorporated into clinical trials and inform clinical decision making. By addressing these challenges, the potential of radiomics may be realized.

A prediction model for early death in non-small cell lung cancer patients following curative-intent chemoradiotherapy.

BACKGROUND: Early death after a treatment can be seen as a therapeutic failure. Accurate prediction of patients at risk for early mortality is crucial to avoid unnecessary harm and reducing costs. The goal of our work is two-fold: first, to evaluate the performance of a previously published model for early death in our cohorts. Second, to develop a prognostic model for early death prediction following radiotherapy. MATERIAL AND METHODS: Patients with NSCLC treated with chemoradiotherapy or radiotherapy alone were included in this study. Four different cohorts from different countries were available for this work (N = 1540). The previous model used age, gender, performance status, tumor stage, income deprivation, no previous treatment given (yes/no) and body mass index to make predictions. A random forest model was developed by learning on the Maastro cohort (N = 698). The new model used performance status, age, gender, T and N stage, total tumor volume (cc), total tumor dose (Gy) and chemotherapy timing (none, sequential, concurrent) to make predictions. Death within 4 months of receiving the first radiotherapy fraction was used as the outcome. RESULTS: Early death rates ranged from 6 to 11% within the four cohorts. The previous model performed with AUC values ranging from 0.54 to 0.64 on the validation cohorts. Our newly developed model had improved AUC values ranging from 0.62 to 0.71 on the validation cohorts. CONCLUSIONS: Using advanced machine learning methods and informative variables, prognostic models for early mortality can be developed. Development of accurate prognostic tools for early mortality is important to inform patients about treatment options and optimize care.

Perspectives on making big data analytics work for oncology.

Oncology, with its unique combination of clinical, physical, technological, and biological data provides an ideal case study for applying big data analytics to improve cancer treatment safety and outcomes. An oncology treatment course such as chemoradiotherapy can generate a large pool of information carrying the 5Vs hallmarks of big data. This data is comprised of a heterogeneous mixture of patient demographics, radiation/chemo dosimetry, multimodality imaging features, and biological markers generated over a treatment period that can span few days to several weeks. Efforts using commercial and in-house tools are underway to facilitate data aggregation, ontology creation, sharing, visualization and varying analytics in a secure environment. However, open questions related to proper data structure representation and effective analytics tools to support oncology decision-making need to be addressed. It is recognized that oncology data constitutes a mix of structured (tabulated) and unstructured (electronic documents) that need to be processed to facilitate searching and subsequent knowledge discovery from relational or NoSQL databases. In this context, methods based on advanced analytics and image feature extraction for oncology applications will be discussed. On the other hand, the classical p (variables)≫n (samples) inference problem of statistical learning is challenged in the Big data realm and this is particularly true for oncology applications where p-omics is witnessing exponential growth while the number of cancer incidences has generally plateaued over the past 5-years leading to a quasi-linear growth in samples per patient. Within the Big data paradigm, this kind of phenomenon may yield undesirable effects such as echo chamber anomalies, Yule-Simpson reversal paradox, or misleading ghost analytics. In this work, we will present these effects as they pertain to oncology and engage small thinking methodologies to counter these effects ranging from incorporating prior knowledge, using information-theoretic techniques to modern ensemble machine learning approaches or combination of these. We will particularly discuss the pros and cons of different approaches to improve mining of big data in oncology.

Modeling non-genetic information dynamics in cells using reservoir computing.

Virtually all cells use energy-driven, ion-specific membrane pumps to maintain large transmembrane gradients of Na+, K+, Cl-, Mg++, and Ca++, but the corresponding evolutionary benefit remains unclear. We propose that these gradients enable a dynamic and versatile biological system that acquires, analyzes, and responds to environmental information. We hypothesize that environmental signals are transmitted into the cell by ion fluxes along pre-existing gradients through gated ion-specific membrane channels. The consequent changes in cytoplasmic ion concentration can generate a local response or orchestrate global/regional cellular dynamics through wire-like ion fluxes along pre-existing and self-assembling cytoskeleton to engage the endoplasmic reticulum, mitochondria, and nucleus.

Cell identity revealed by precise cell cycle state mapping links data modalities.

Several methods for cell cycle inference from sequencing data exist and are widely adopted. In contrast, methods for classification of cell cycle state from imaging data are scarce. We have for the first time integrated sequencing and imaging derived cell cycle pseudo-times for assigning 449 imaged cells to 693 sequenced cells at an average resolution of 3.4 and 2.4 cells for sequencing and imaging data respectively. Data integration revealed thousands of pathways and organelle features that are correlated with each other, including several previously known interactions and novel associations. The ability to assign the transcriptome state of a profiled cell to its closest living relative, which is still actively growing and expanding opens the door for genotype-phenotype mapping at single cell resolution forward in time.

Magnetic Resonance-Guided Cancer Therapy Radiomics and Machine Learning Models for Response Prediction.

Magnetic resonance imaging (MRI) is known for its accurate soft tissue delineation of tumors and normal tissues. This development has significantly impacted the imaging and treatment of cancers. Radiomics is the process of extracting high-dimensional features from medical images. Several studies have shown that these extracted features may be used to build machine-learning models for the prediction of treatment outcomes of cancer patients. Various feature selection techniques and machine models interrogate the relevant radiomics features for predicting cancer treatment outcomes. This study aims to provide an overview of MRI radiomics features used in predicting clinical treatment outcomes with machine learning techniques. The review includes examples from different disease sites. It will also discuss the impact of magnetic field strength, sample size, and other characteristics on outcome prediction performance.

Presurgical Upgrade Prediction of DCIS to Invasive Ductal Carcinoma Using Time-dependent Deep Learning Models with DCE MRI.

Purpose To determine whether time-dependent deep learning models can outperform single time point models in predicting preoperative upgrade of ductal carcinoma in situ (DCIS) to invasive malignancy at dynamic contrast-enhanced (DCE) breast MRI without a lesion segmentation prerequisite. Materials and Methods In this exploratory study, 154 cases of biopsy-proven DCIS (25 upgraded at surgery and 129 not upgraded) were selected consecutively from a retrospective cohort of preoperative DCE MRI in women with a mean age of 59 years at time of diagnosis from 2012 to 2022. Binary classification was implemented with convolutional neural network (CNN)-long short-term memory (LSTM) architectures benchmarked against traditional CNNs without manual segmentation of the lesions. Combinatorial performance analysis of ResNet50 versus VGG16-based models was performed with each contrast phase. Binary classification area under the receiver operating characteristic curve (AUC) was reported. Results VGG16-based models consistently provided better holdout test AUCs than did ResNet50 in CNN and CNN-LSTM studies (multiphase test AUC, 0.67 vs 0.59, respectively, for CNN models [P = .04] and 0.73 vs 0.62 for CNN-LSTM models [P = .008]). The time-dependent model (CNN-LSTM) provided a better multiphase test AUC over single time point (CNN) models (0.73 vs 0.67; P = .04). Conclusion Compared with single time point architectures, sequential deep learning algorithms using preoperative DCE MRI improved prediction of DCIS lesions upgraded to invasive malignancy without the need for lesion segmentation. Keywords: MRI, Dynamic Contrast-enhanced, Breast, Convolutional Neural Network (CNN) Supplemental material is available for this article. © RSNA, 2024.

A joint ESTRO and AAPM guideline for development, clinical validation and reporting of artificial intelligence models in radiation therapy.

BACKGROUND AND PURPOSE: Artificial Intelligence (AI) models in radiation therapy are being developed with increasing pace. Despite this, the radiation therapy community has not widely adopted these models in clinical practice. A cohesive guideline on how to develop, report and clinically validate AI algorithms might help bridge this gap. METHODS AND MATERIALS: A Delphi process with all co-authors was followed to determine which topics should be addressed in this comprehensive guideline. Separate sections of the guideline, including Statements, were written by subgroups of the authors and discussed with the whole group at several meetings. Statements were formulated and scored as highly recommended or recommended. RESULTS: The following topics were found most relevant: Decision making, image analysis, volume segmentation, treatment planning, patient specific quality assurance of treatment delivery, adaptive treatment, outcome prediction, training, validation and testing of AI model parameters, model availability for others to verify, model quality assurance/updates and upgrades, ethics. Key references were given together with an outlook on current hurdles and possibilities to overcome these. 19 Statements were formulated. CONCLUSION: A cohesive guideline has been written which addresses main topics regarding AI in radiation therapy. It will help to guide development, as well as transparent and consistent reporting and validation of new AI tools and facilitate adoption.