MIM-CyCIF: masked imaging modeling for enhancing cyclic immunofluorescence (CyCIF) with panel reduction and imputation.

Cyclic Immunofluorescence (CyCIF) can quantify multiple biomarkers, but panel capacity is limited by technical challenges. We propose a computational panel reduction approach that can impute the information content from 25 markers using only 9 markers, learning co-expression and morphological patterns while concurrently increasing speed and panel content and decreasing cost. We demonstrate strong correlations in predictions and generalizability across breast and colorectal cancer, illustrating applicability of our approach to diverse tissue types.

Ultra high content analyses of circulating and tumor associated hybrid cells reveal phenotypic heterogeneity.

Persistently high, worldwide mortality from cancer highlights the unresolved challenges of disease surveillance and detection that impact survival. Development of a non-invasive, blood-based biomarker would transform survival from cancer. We demonstrate the functionality of ultra-high content analyses of a newly identified population of tumor cells that are hybrids between neoplastic and immune cells in patient matched tumor and peripheral blood specimens. Using oligonucleotide conjugated antibodies (Ab-oligo) permitting cyclic immunofluorescence (cyCIF), we present analyses of phenotypes among tumor and peripheral blood hybrid cells. Interestingly, the majority of circulating hybrid cell (CHC) subpopulations were not identified in tumor-associated hybrids. These results highlight the efficacy of ultra-high content phenotypic analyses using Ab-oligo based cyCIF applied to both tumor and peripheral blood specimens. The combination of a multiplex phenotypic profiling platform that is gentle enough to analyze blood to detect and evaluate disseminated tumor cells represents a novel approach to exploring novel tumor biology and potential utility for developing the population as a blood-based biomarker in cancer.

Quantitative image analysis pipeline for detecting circulating hybrid cells in immunofluorescence images with human-level accuracy.

Circulating hybrid cells (CHCs) are a newly discovered, tumor-derived cell population found in the peripheral blood of cancer patients and are thought to contribute to tumor metastasis. However, identifying CHCs by immunofluorescence (IF) imaging of patient peripheral blood mononuclear cells (PBMCs) is a time-consuming and subjective process that currently relies on manual annotation by laboratory technicians. Additionally, while IF is relatively easy to apply to tissue sections, its application to PBMC smears presents challenges due to the presence of biological and technical artifacts. To address these challenges, we present a robust image analysis pipeline to automate the detection and analysis of CHCs in IF images. The pipeline incorporates quality control to optimize specimen preparation protocols and remove unwanted artifacts, leverages a ß-variational autoencoder (VAE) to learn meaningful latent representations of single-cell images, and employs a support vector machine (SVM) classifier to achieve human-level CHC detection. We created a rigorously labeled IF CHC data set including nine patients and two disease sites with the assistance of 10 annotators to evaluate the pipeline. We examined annotator variation and bias in CHC detection and provided guidelines to optimize the accuracy of CHC annotation. We found that all annotators agreed on CHC identification for only 65% of the cells in the data set and had a tendency to underestimate CHC counts for regions of interest (ROIs) containing relatively large amounts of cells (>50,000) when using the conventional enumeration method. On the other hand, our proposed approach is unbiased to ROI size. The SVM classifier trained on the ß-VAE embeddings achieved an F1 score of 0.80, matching the average performance of human annotators. Our pipeline enables researchers to explore the role of CHCs in cancer progression and assess their potential as a clinical biomarker for metastasis. Further, we demonstrate that the pipeline can identify discrete cellular phenotypes among PBMCs, highlighting its utility beyond CHCs.

Single-cell morphodynamical trajectories enable prediction of gene expression accompanying cell state change.

Extracellular signals induce changes to molecular programs that modulate multiple cellular phenotypes, including proliferation, motility, and differentiation status. The connection between dynamically adapting phenotypic states and the molecular programs that define them is not well understood. Here we develop data-driven models of single-cell phenotypic responses to extracellular stimuli by linking gene transcription levels to "morphodynamics" - changes in cell morphology and motility observable in time-lapse image data. We adopt a dynamics-first view of cell state by grouping single-cell trajectories into states with shared morphodynamic responses. The single-cell trajectories enable development of a first-of-its-kind computational approach to map live-cell dynamics to snapshot gene transcript levels, which we term MMIST, Molecular and Morphodynamics-Integrated Single-cell Trajectories. The key conceptual advance of MMIST is that cell behavior can be quantified based on dynamically defined states and that extracellular signals alter the overall distribution of cell states by altering rates of switching between states. We find a cell state landscape that is bound by epithelial and mesenchymal endpoints, with distinct sequences of epithelial to mesenchymal transition (EMT) and mesenchymal to epithelial transition (MET) intermediates. The analysis yields predictions for gene expression changes consistent with curated EMT gene sets and provides a prediction of thousands of RNA transcripts through extracellular signal-induced EMT and MET with near-continuous time resolution. The MMIST framework leverages true single-cell dynamical behavior to generate molecular-level omics inferences and is broadly applicable to other biological domains, time-lapse imaging approaches and molecular snapshot data.

3D multiplexed tissue imaging reconstruction and optimized region of interest (ROI) selection through deep learning model of channels embedding.

Introduction: Tissue-based sampling and diagnosis are defined as the extraction of information from certain limited spaces and its diagnostic significance of a certain object. Pathologists deal with issues related to tumor heterogeneity since analyzing a single sample does not necessarily capture a representative depiction of cancer, and a tissue biopsy usually only presents a small fraction of the tumor. Many multiplex tissue imaging platforms (MTIs) make the assumption that tissue microarrays (TMAs) containing small core samples of 2-dimensional (2D) tissue sections are a good approximation of bulk tumors although tumors are not 2D. However, emerging whole slide imaging (WSI) or 3D tumor atlases that use MTIs like cyclic immunofluorescence (CyCIF) strongly challenge this assumption. In spite of the additional insight gathered by measuring the tumor microenvironment in WSI or 3D, it can be prohibitively expensive and time-consuming to process tens or hundreds of tissue sections with CyCIF. Even when resources are not limited, the criteria for region of interest (ROI) selection in tissues for downstream analysis remain largely qualitative and subjective as stratified sampling requires the knowledge of objects and evaluates their features. Despite the fact TMAs fail to adequately approximate whole tissue features, a theoretical subsampling of tissue exists that can best represent the tumor in the whole slide image. Methods: To address these challenges, we propose deep learning approaches to learn multi-modal image translation tasks from two aspects: 1) generative modeling approach to reconstruct 3D CyCIF representation and 2) co-embedding CyCIF image and Hematoxylin and Eosin (H&E) section to learn multi-modal mappings by a cross-domain translation for minimum representative ROI selection. Results and discussion: We demonstrate that generative modeling enables a 3D virtual CyCIF reconstruction of a colorectal cancer specimen given a small subset of the imaging data at training time. By co-embedding histology and MTI features, we propose a simple convex optimization for objective ROI selection. We demonstrate the potential application of ROI selection and the efficiency of its performance with respect to cellular heterogeneity.

MIM-CyCIF: Masked Imaging Modeling for Enhancing Cyclic Immunofluorescence (CyCIF) with Panel Reduction and Imputation.

CyCIF can quantify multiple biomarkers, but panel capacity is limited by technical challenges. We propose a computational panel reduction approach that can impute the information content from 25 markers using only 9 markers, learning co-expression and morphological patterns while concurrently increasing speed and panel content and decreasing cost. We demonstrate strong correlations in predictions and generalizability across breast and colorectal cancer, illustrating applicability of our approach to diverse tissue types.

MYC Deregulation and PTEN Loss Model Tumor and Stromal Heterogeneity of Aggressive Triple-Negative Breast Cancer.

Triple-negative breast cancer (TNBC) patients have a poor prognosis and few treatment options. Mouse models of TNBC are important for development of new therapies, however, few mouse models represent the complexity of TNBC. Here, we develop a female TNBC murine model by mimicking two common TNBC mutations with high co-occurrence: amplification of the oncogene MYC and deletion of the tumor suppressor PTEN. This Myc;Ptenfl model develops heterogeneous triple-negative mammary tumors that display histological and molecular features commonly found in human TNBC. Our research involves deep molecular and spatial analyses on Myc;Ptenfl tumors including bulk and single-cell RNA-sequencing, and multiplex tissue-imaging. Through comparison with human TNBC, we demonstrate that this genetic mouse model develops mammary tumors with differential survival and therapeutic responses that closely resemble the inter- and intra-tumoral and microenvironmental heterogeneity of human TNBC, providing a pre-clinical tool for assessing the spectrum of patient TNBC biology and drug response.

Dual-modality imaging of immunofluorescence and imaging mass cytometry for whole-slide imaging and accurate segmentation.

Imaging mass cytometry (IMC) is a powerful technique capable of detecting over 30 markers on a single slide. It has been increasingly used for single-cell-based spatial phenotyping in a wide range of samples. However, it only acquires a rectangle field of view (FOV) with a relatively small size and low image resolution, which hinders downstream analysis. Here, we reported a highly practical dual-modality imaging method that combines high-resolution immunofluorescence (IF) and high-dimensional IMC on the same tissue slide. Our computational pipeline uses the whole-slide image (WSI) of IF as a spatial reference and integrates small-FOV IMC into a WSI of IMC. The high-resolution IF images enable accurate single-cell segmentation to extract robust high-dimensional IMC features for downstream analysis. We applied this method in esophageal adenocarcinoma of different stages, identified the single-cell pathology landscape via reconstruction of WSI IMC images, and demonstrated the advantage of the dual-modality imaging strategy.

Quantitative image analysis pipeline for detecting circulating hybrid cells in immunofluorescence images with human-level accuracy.

Circulating hybrid cells (CHCs) are a newly discovered, tumor-derived cell population identified in the peripheral blood of cancer patients and are thought to contribute to tumor metastasis. However, identifying CHCs by immunofluorescence (IF) imaging of patient peripheral blood mononuclear cells (PBMCs) is a time-consuming and subjective process that currently relies on manual annotation by laboratory technicians. Additionally, while IF is relatively easy to apply to tissue sections, its application on PBMC smears presents challenges due to the presence of biological and technical artifacts. To address these challenges, we present a robust image analysis pipeline to automate the detection and analyses of CHCs in IF images. The pipeline incorporates quality control to optimize specimen preparation protocols and remove unwanted artifacts, leverages a ß-variational autoencoder (VAE) to learn meaningful latent representations of single-cell images and employs a support vector machine (SVM) classifier to achieve human-level CHC detection. We created a rigorously labeled IF CHC dataset including 9 patients and 2 disease sites with the assistance of 10 annotators to evaluate the pipeline. We examined annotator variation and bias in CHC detection and then provided guidelines to optimize the accuracy of CHC annotation. We found that all annotators agreed on CHC identification for only 65% of the cells in the dataset and had a tendency to underestimate CHC counts for regions of interest (ROI) containing relatively large amounts of cells (>50,000) when using conventional enumeration methods. On the other hand, our proposed approach is unbiased to ROI size. The SVM classifier trained on the ß-VAE encodings achieved an F1 score of 0.80, matching the average performance of annotators. Our pipeline enables researchers to explore the role of CHCs in cancer progression and assess their potential as a clinical biomarker for metastasis. Further, we demonstrate that the pipeline can identify discrete cellular phenotypes among PBMCs, highlighting its utility beyond CHCs.

MIM-CyCIF: Masked Imaging Modeling for Enhancing Cyclic Immunofluorescence (CyCIF) with Panel Reduction and Imputation.

CyCIF quantifies multiple biomarkers, but panel capacity is compromised by technical challenges including tissue loss. We propose a computational panel reduction, inferring surrogate CyCIF data from a subset of biomarkers. Our model reconstructs the information content from 25 markers using only 9 markers, learning co-expression and morphological patterns. We demonstrate strong correlations in predictions and generalizability across breast and colorectal cancer tissue microarrays, illustrating broader applicability to diverse tissue types.

Addressing persistent challenges in digital image analysis of cancerous tissues.

The National Cancer Institute (NCI) supports many research programs and consortia, many of which use imaging as a major modality for characterizing cancerous tissue. A trans-consortia Image Analysis Working Group (IAWG) was established in 2019 with a mission to disseminate imaging-related work and foster collaborations. In 2022, the IAWG held a virtual hackathon focused on addressing challenges of analyzing high dimensional datasets from fixed cancerous tissues. Standard image processing techniques have automated feature extraction, but the next generation of imaging data requires more advanced methods to fully utilize the available information. In this perspective, we discuss current limitations of the automated analysis of multiplexed tissue images, the first steps toward deeper understanding of these limitations, what possible solutions have been developed, any new or refined approaches that were developed during the Image Analysis Hackathon 2022, and where further effort is required. The outstanding problems addressed in the hackathon fell into three main themes: 1) challenges to cell type classification and assessment, 2) translation and visual representation of spatial aspects of high dimensional data, and 3) scaling digital image analyses to large (multi-TB) datasets. We describe the rationale for each specific challenge and the progress made toward addressing it during the hackathon. We also suggest areas that would benefit from more focus and offer insight into broader challenges that the community will need to address as new technologies are developed and integrated into the broad range of image-based modalities and analytical resources already in use within the cancer research community.

Automated large volume sample preparation for vEM.

New developments in electron microscopy technology, improved efficiency of detectors, and artificial intelligence applications for data analysis over the past decade have increased the use of volume electron microscopy (vEM) in the life sciences field. Moreover, sample preparation methods are continuously being modified by investigators to improve final sample quality, increase electron density, combine imaging technologies, and minimize the introduction of artifacts into specimens under study. There are a variety of conventional bench protocols that a researcher can utilize, though most of these protocols require several days. In this work, we describe the utilization of an automated specimen processor, the mPrep™ ASP-2000™, to prepare samples for vEM that are compatible with focused ion beam scanning electron microscopy (FIB-SEM), serial block face scanning electron microscopy (SBF-SEM), and array tomography (AT). The protocols described here aimed for methods that are completed in a much shorter period of time while minimizing the exposure of the operator to hazardous and toxic chemicals and improving the reproducibility of the specimens' heavy metal staining, all without compromising the quality of the data acquired using backscattered electrons during SEM imaging. As a control, we have included a widely used sample bench protocol and have utilized it as a comparator for image quality analysis, both qualitatively and using image quality analysis metrics.

Morphodynamical cell state description via live-cell imaging trajectory embedding.

Time-lapse imaging is a powerful approach to gain insight into the dynamic responses of cells, but the quantitative analysis of morphological changes over time remains challenging. Here, we exploit the concept of "trajectory embedding" to analyze cellular behavior using morphological feature trajectory histories-that is, multiple time points simultaneously, rather than the more common practice of examining morphological feature time courses in single timepoint (snapshot) morphological features. We apply this approach to analyze live-cell images of MCF10A mammary epithelial cells after treatment with a panel of microenvironmental perturbagens that strongly modulate cell motility, morphology, and cell cycle behavior. Our morphodynamical trajectory embedding analysis constructs a shared cell state landscape revealing ligand-specific regulation of cell state transitions and enables quantitative and descriptive models of single-cell trajectories. Additionally, we show that incorporation of trajectories into single-cell morphological analysis enables (i) systematic characterization of cell state trajectories, (ii) better separation of phenotypes, and (iii) more descriptive models of ligand-induced differences as compared to snapshot-based analysis. This morphodynamical trajectory embedding is broadly applicable to the quantitative analysis of cell responses via live-cell imaging across many biological and biomedical applications.

SEG: Segmentation Evaluation in absence of Ground truth labels.

Identifying individual cells or nuclei is often the first step in the analysis of multiplex tissue imaging (MTI) data. Recent efforts to produce plug-and-play, end-to-end MTI analysis tools such as MCMICRO1- though groundbreaking in their usability and extensibility - are often unable to provide users guidance regarding the most appropriate models for their segmentation task among an endless proliferation of novel segmentation methods. Unfortunately, evaluating segmentation results on a user's dataset without ground truth labels is either purely subjective or eventually amounts to the task of performing the original, time-intensive annotation. As a consequence, researchers rely on models pre-trained on other large datasets for their unique tasks. Here, we propose a methodological approach for evaluating MTI nuclei segmentation methods in absence of ground truth labels by scoring relatively to a larger ensemble of segmentations. To avoid potential sensitivity to collective bias from the ensemble approach, we refine the ensemble via weighted average across segmentation methods, which we derive from a systematic model ablation study. First, we demonstrate a proof-of-concept and the feasibility of the proposed approach to evaluate segmentation performance in a small dataset with ground truth annotation. To validate the ensemble and demonstrate the importance of our method-specific weighting, we compare the ensemble's detection and pixel-level predictions - derived without supervision - with the data's ground truth labels. Second, we apply the methodology to an unlabeled larger tissue microarray (TMA) dataset, which includes a diverse set of breast cancer phenotypes, and provides decision guidelines for the general user to more easily choose the most suitable segmentation methods for their own dataset by systematically evaluating the performance of individual segmentation approaches in the entire dataset.

Dual-modality imaging of immunofluorescence and imaging mass cytometry for whole slide imaging with accurate single-cell segmentation.

Imaging mass cytometry (IMC) is a powerful multiplexed tissue imaging technology that allows simultaneous detection of more than 30 makers on a single slide. It has been increasingly used for singlecell-based spatial phenotyping in a wide range of samples. However, it only acquires a small, rectangle field of view (FOV) with a low image resolution that hinders downstream analysis. Here, we reported a highly practical dual-modality imaging method that combines high-resolution immunofluorescence (IF) and high-dimensional IMC on the same tissue slide. Our computational pipeline uses the whole slide image (WSI) of IF as a spatial reference and integrates small FOVs IMC into a WSI of IMC. The high-resolution IF images enable accurate single-cell segmentation to extract robust high-dimensional IMC features for downstream analysis. We applied this method in esophageal adenocarcinoma of different stages, identified the single-cell pathology landscape via reconstruction of WSI IMC images, and demonstrated the advantage of the dual-modality imaging strategy. Motivation: Highly multiplexed tissue imaging allows visualization of the spatially resolved expression of multiple proteins at the single-cell level. Although imaging mass cytometry (IMC) using metal isotope-conjugated antibodies has a significant advantage of low background signal and absence of autofluorescence or batch effect, it has a low resolution that hampers accurate cell segmentation and results in inaccurate feature extraction. In addition, IMC only acquires mm 2 -sized rectangle regions, which limits its application and efficiency when studying larger clinical samples with non-rectangle shapes. To maximize the research output of IMC, we developed the dual-modality imaging method based on a highly practical and technical improvement requiring no extra specialized equipment or agents and proposed a comprehensive computational pipeline that combines IF and IMC. The proposed method greatly improves the accuracy of cell segmentation and downstream analysis and is able to obtain whole slide image IMC to capture the comprehensive cellular landscape of large tissue sections.

Circulating Neoplastic-Immune Hybrid Cells Predict Metastatic Progression in Uveal Melanoma.

Background: Uveal melanoma is an aggressive cancer with high metastatic risk. Recently, we identified a circulating cancer cell population that co-expresses neoplastic and leukocyte antigens, termed circulating hybrid cells (CHCs). In other cancers, CHCs are more numerous and better predict oncologic outcomes compared to circulating tumor cells (CTCs). We sought to investigate the potential of CHCs as a prognostic biomarker in uveal melanoma. Methods: We isolated peripheral blood monocular cells from uveal melanoma patients at the time of primary treatment and used antibodies against leukocyte and melanoma markers to identify and enumerate CHCs and CTCs by immunocytochemistry. Results: Using a multi-marker approach to capture the heterogeneous disseminated tumor cell population, detection of CHCs was highly sensitive in uveal melanoma patients regardless of disease stage. CHCs were detected in 100% of stage I-III uveal melanoma patients (entire cohort, n = 68), whereas CTCs were detected in 58.8% of patients. CHCs were detected at levels statically higher than CTCs across all stages (p = 0.05). Moreover, CHC levels, but not CTCs, predicted 3 year progression-free survival (p < 0.03) and overall survival (p < 0.04). Conclusion: CHCs are a novel and promising prognostic biomarker in uveal melanoma.

Exploring approaches for predictive cancer patient digital twins: Opportunities for collaboration and innovation.

We are rapidly approaching a future in which cancer patient digital twins will reach their potential to predict cancer prevention, diagnosis, and treatment in individual patients. This will be realized based on advances in high performance computing, computational modeling, and an expanding repertoire of observational data across multiple scales and modalities. In 2020, the US National Cancer Institute, and the US Department of Energy, through a trans-disciplinary research community at the intersection of advanced computing and cancer research, initiated team science collaborative projects to explore the development and implementation of predictive Cancer Patient Digital Twins. Several diverse pilot projects were launched to provide key insights into important features of this emerging landscape and to determine the requirements for the development and adoption of cancer patient digital twins. Projects included exploring approaches to using a large cohort of digital twins to perform deep phenotyping and plan treatments at the individual level, prototyping self-learning digital twin platforms, using adaptive digital twin approaches to monitor treatment response and resistance, developing methods to integrate and fuse data and observations across multiple scales, and personalizing treatment based on cancer type. Collectively these efforts have yielded increased insights into the opportunities and challenges facing cancer patient digital twin approaches and helped define a path forward. Given the rapidly growing interest in patient digital twins, this manuscript provides a valuable early progress report of several CPDT pilot projects commenced in common, their overall aims, early progress, lessons learned and future directions that will increasingly involve the broader research community.

Computational multiplex panel reduction to maximize information retention in breast cancer tissue microarrays.

Recent state-of-the-art multiplex imaging techniques have expanded the depth of information that can be captured within a single tissue sample by allowing for panels with dozens of markers. Despite this increase in capacity, space on the panel is still limited due to technical artifacts, tissue loss, and long imaging acquisition time. As such, selecting which markers to include on a panel is important, since removing important markers will result in a loss of biologically relevant information, but identifying redundant markers will provide a room for other markers. To address this, we propose computational approaches to determine the amount of shared information between markers and select an optimally reduced panel that captures maximum amount of information with the fewest markers. Here we examine several panel selection approaches and evaluate them based on their ability to reconstruct the full panel images and information within breast cancer tissue microarray datasets using cyclic immunofluorescence as a proof of concept. We show that all methods perform adequately and can re-capture cell types using only 18 of 25 markers (72% of the original panel size). The correlation-based selection methods achieved the best single-cell marker mean intensity predictions with a Spearman correlation of 0.90 with the reduced panel. Using the proposed methods shown here, it is possible for researchers to design more efficient multiplex imaging panels that maximize the amount of information retained with the limited number of markers with respect to certain evaluation metrics and architecture biases.

Deciphering the Immune Complexity in Esophageal Adenocarcinoma and Pre-Cancerous Lesions With Sequential Multiplex Immunohistochemistry and Sparse Subspace Clustering Approach.

Esophageal adenocarcinoma (EAC) develops from a chronic inflammatory environment across four stages: intestinal metaplasia, known as Barrett's esophagus, low- and high-grade dysplasia, and adenocarcinoma. Although the genomic characteristics of this progression have been well defined via large-scale DNA sequencing, the dynamics of various immune cell subsets and their spatial interactions in their tumor microenvironment remain unclear. Here, we applied a sequential multiplex immunohistochemistry (mIHC) platform with computational image analysis pipelines that allow for the detection of 10 biomarkers in one formalin-fixed paraffin-embedded (FFPE) tissue section. Using this platform and quantitative image analytics, we studied changes in the immune landscape during disease progression based on 40 normal and diseased areas from endoscopic mucosal resection specimens of chemotherapy treatment- naïve patients, including normal esophagus, metaplasia, low- and high-grade dysplasia, and adenocarcinoma. The results revealed a steady increase of FOXP3+ T regulatory cells and a CD163+ myelomonocytic cell subset. In parallel to the manual gating strategy applied for cell phenotyping, we also adopted a sparse subspace clustering (SSC) algorithm allowing the automated cell phenotyping of mIHC-based single-cell data. The algorithm successfully identified comparable cell types, along with significantly enriched FOXP3 T regulatory cells and CD163+ myelomonocytic cells as found in manual gating. In addition, SCC identified a new CSF1R+CD1C+ myeloid lineage, which not only was previously unknown in this disease but also increases with advancing disease stages. This study revealed immune dynamics in EAC progression and highlighted the potential application of a new multiplex imaging platform, combined with computational image analysis on routine clinical FFPE sections, to investigate complex immune populations in tumor ecosystems.