Towards Generalizability and Robustness in Biological Object Detection in Electron Microscopy Images.

Machine learning approaches have the potential for meaningful impact in the biomedical field. However, there are often challenges unique to biomedical data that prohibits the adoption of these innovations. For example, limited data, data volatility, and data shifts all compromise model robustness and generalizability. Without proper tuning and data management, deploying machine learning models in the presence of unaccounted for corruptions leads to reduced or misleading performance. This study explores techniques to enhance model generalizability through iterative adjustments. Specifically, we investigate a detection tasks using electron microscopy images and compare models trained with different normalization and augmentation techniques. We found that models trained with Group Normalization or texture data augmentation outperform other normalization techniques and classical data augmentation, enabling them to learn more generalized features. These improvements persist even when models are trained and tested on disjoint datasets acquired through diverse data acquisition protocols. Results hold true for transformerand convolution-based detection architectures. The experiments show an impressive 29% boost in average precision, indicating significant enhancements in the model's generalizibality. This underscores the models' capacity to effectively adapt to diverse datasets and demonstrates their increased resilience in real-world applications.

Efficient semi-supervised semantic segmentation of electron microscopy cancer images with sparse annotations.

Electron microscopy (EM) enables imaging at nanometer resolution and can shed light on how cancer evolves to develop resistance to therapy. Acquiring these images has become a routine task; however, analyzing them is now the bottleneck, as manual structure identification is very time-consuming and can take up to several months for a single sample. Deep learning approaches offer a suitable solution to speed up the analysis. In this work, we present a study of several state-of-the-art deep learning models for the task of segmenting nuclei and nucleoli in volumes from tumor biopsies. We compared previous results obtained with the ResUNet architecture to the more recent UNet++, FracTALResNet, SenFormer, and CEECNet models. In addition, we explored the utilization of unlabeled images through semi-supervised learning with Cross Pseudo Supervision. We have trained and evaluated all of the models on sparse manual labels from three fully annotated in-house datasets that we have made available on demand, demonstrating improvements in terms of 3D Dice score. From the analysis of these results, we drew conclusions on the relative gains of using more complex models, semi-supervised learning as well as next steps for the mitigation of the manual segmentation bottleneck.

Automated large volume sample preparation for vEM.

New developments in electron microscopy technology, improved efficiency of detectors, and artificial intelligence applications for data analysis over the past decade have increased the use of volume electron microscopy (vEM) in the life sciences field. Moreover, sample preparation methods are continuously being modified by investigators to improve final sample quality, increase electron density, combine imaging technologies, and minimize the introduction of artifacts into specimens under study. There are a variety of conventional bench protocols that a researcher can utilize, though most of these protocols require several days. In this work, we describe the utilization of an automated specimen processor, the mPrep™ ASP-2000™, to prepare samples for vEM that are compatible with focused ion beam scanning electron microscopy (FIB-SEM), serial block face scanning electron microscopy (SBF-SEM), and array tomography (AT). The protocols described here aimed for methods that are completed in a much shorter period of time while minimizing the exposure of the operator to hazardous and toxic chemicals and improving the reproducibility of the specimens' heavy metal staining, all without compromising the quality of the data acquired using backscattered electrons during SEM imaging. As a control, we have included a widely used sample bench protocol and have utilized it as a comparator for image quality analysis, both qualitatively and using image quality analysis metrics.

Efficient semi-supervised semantic segmentation of electron microscopy cancer images with sparse annotations.

Electron microscopy (EM) enables imaging at a resolution of nanometers and can shed light on how cancer evolves to develop resistance to therapy. Acquiring these images has become a routine task.However, analyzing them is now a bottleneck, as manual structure identification is very time-consuming and can take up to several months for a single sample. Deep learning approaches offer a suitable solution to speed up the analysis. In this work, we present a study of several state-of-the-art deep learning models for the task of segmenting nuclei and nucleoli in volumes from tumor biopsies. We compared previous results obtained with the ResUNet architecture to the more recent UNet++, FracTALResNet, SenFormer, and CEECNet models. In addition, we explored the utilization of unlabeled images through semi-supervised learning with Cross Pseudo Supervision. We have trained and evaluated all of the models on sparse manual labels from three fully annotated in-house datasets that we have made available on demand, demonstrating improvements in terms of 3D Dice score. From the analysis of these results, we drew conclusions on the relative gains of using more complex models, and semi-supervised learning as well as the next steps for the mitigation of the manual segmentation bottleneck.

Segmentation of cellular ultrastructures on sparsely labeled 3D electron microscopy images using deep learning.

Focused ion beam-scanning electron microscopy (FIB-SEM) images can provide a detailed view of the cellular ultrastructure of tumor cells. A deeper understanding of their organization and interactions can shed light on cancer mechanisms and progression. However, the bottleneck in the analysis is the delineation of the cellular structures to enable quantitative measurements and analysis. We mitigated this limitation using deep learning to segment cells and subcellular ultrastructure in 3D FIB-SEM images of tumor biopsies obtained from patients with metastatic breast and pancreatic cancers. The ultrastructures, such as nuclei, nucleoli, mitochondria, endosomes, and lysosomes, are relatively better defined than their surroundings and can be segmented with high accuracy using a neural network trained with sparse manual labels. Cell segmentation, on the other hand, is much more challenging due to the lack of clear boundaries separating cells in the tissue. We adopted a multi-pronged approach combining detection, boundary propagation, and tracking for cell segmentation. Specifically, a neural network was employed to detect the intracellular space; optical flow was used to propagate cell boundaries across the z-stack from the nearest ground truth image in order to facilitate the separation of individual cells; finally, the filopodium-like protrusions were tracked to the main cells by calculating the intersection over union measure for all regions detected in consecutive images along z-stack and connecting regions with maximum overlap. The proposed cell segmentation methodology resulted in an average Dice score of 0.93. For nuclei, nucleoli, and mitochondria, the segmentation achieved Dice scores of 0.99, 0.98, and 0.86, respectively. The segmentation of FIB-SEM images will enable interpretative rendering and provide quantitative image features to be associated with relevant clinical variables.

An omic and multidimensional spatial atlas from serial biopsies of an evolving metastatic breast cancer.

Mechanisms of therapeutic resistance and vulnerability evolve in metastatic cancers as tumor cells and extrinsic microenvironmental influences change during treatment. To support the development of methods for identifying these mechanisms in individual people, here we present an omic and multidimensional spatial (OMS) atlas generated from four serial biopsies of an individual with metastatic breast cancer during 3.5 years of therapy. This resource links detailed, longitudinal clinical metadata that includes treatment times and doses, anatomic imaging, and blood-based response measurements to clinical and exploratory analyses, which includes comprehensive DNA, RNA, and protein profiles; images of multiplexed immunostaining; and 2- and 3-dimensional scanning electron micrographs. These data report aspects of heterogeneity and evolution of the cancer genome, signaling pathways, immune microenvironment, cellular composition and organization, and ultrastructure. We present illustrative examples of how integrative analyses of these data reveal potential mechanisms of response and resistance and suggest novel therapeutic vulnerabilities.

Multiscale cardiac imaging spanning the whole heart and its internal cellular architecture in a small animal model.

Cardiac pumping depends on the morphological structure of the heart, but also on its subcellular (ultrastructural) architecture, which enables cardiac contraction. In cases of congenital heart defects, localized ultrastructural disruptions that increase the risk of heart failure are only starting to be discovered. This is in part due to a lack of technologies that can image the three-dimensional (3D) heart structure, to assess malformations; and its ultrastructure, to assess organelle disruptions. We present here a multiscale, correlative imaging procedure that achieves high-resolution images of the whole heart, using 3D micro-computed tomography (micro-CT); and its ultrastructure, using 3D scanning electron microscopy (SEM). In a small animal model (chicken embryo), we achieved uniform fixation and staining of the whole heart, without losing ultrastructural preservation on the same sample, enabling correlative multiscale imaging. Our approach enables multiscale studies in models of congenital heart disease and beyond.

The heart is our hardest-working organ and beats around 100,000 times a day, pumping blood through a vast system of vessels to all areas of the body. Specialized heart cells make the heart contract rhythmically, enabling it to work efficiently. Contractile molecules inside these cells, called myofibrils, align within the heart cells, and heart cells align to each other, so that the heart tissue contracts effectively. However, when the heart has defects or is diseased this organization can be lost, and the heart may no longer pump blood efficiently, leading to sometimes life-threatening complications. For example, around one in a hundred newborn babies suffer from congenital heart defects, and despite medical advances, these conditions remain the main cause of non-infectious mortality in children. Many cases of congenital heart disease are diagnosed before a baby is born during an ultrasound scan. However, these scans, as well as subsequent diagnostic tools, lack the precision to detect problems within the heart cells. Now, Rykiel et al. used two complementary imaging techniques known as micro-computed tomography and scanning electron microscopy to acquire pictures of the whole heart as well as of the organization inside the heart cells. This made it possible to capture the structure of the heart tissue at both micrometer (the whole heart) and nanometer resolution (the inside of the cells), and to study what happens within the heart and its cells when the heart has a defect. Rykiel et al. tested the imaging technology on the hearts of chicken embryos, at stages equivalent to a five to six-month-old human fetus, and compared a healthy heart with a heart with a defect called tetralogy of Fallot. They found that the tissues in the heart with a defect had a sponge-like appearance, with increased space in between cells. Moreover, the myofibrils of the heart with a defect were aligned differently compared to those in the normal heart. More research is needed to fully understand what happens when the heart has a defect. However, the imaging technology used in this study offers the possibility of examining the heart at an unprecedented level of detail. This will deepen our understanding of how structural heart defects arise and how they affect the pumping of the heart, and will give us clues to design better treatments for patients with heart defects and other heart anomalies.

A workflow for visualizing human cancer biopsies using large-format electron microscopy.

Recent developments in large format electron microscopy have enabled generation of images that provide detailed ultrastructural information on normal and diseased cells and tissues. Analyses of these images increase our understanding of cellular organization and interactions and disease-related changes therein. In this manuscript, we describe a workflow for two-dimensional (2D) and three-dimensional (3D) imaging, including both optical and scanning electron microscopy (SEM) methods, that allow pathologists and cancer biology researchers to identify areas of interest from human cancer biopsies. The protocols and mounting strategies described in this workflow are compatible with 2D large format EM mapping, 3D focused ion beam-SEM and serial block face-SEM. The flexibility to use diverse imaging technologies available at most academic institutions makes this workflow useful and applicable for most life science samples. Volumetric analysis of the biopsies studied here revealed morphological, organizational and ultrastructural aspects of the tumor cells and surrounding environment that cannot be revealed by conventional 2D EM imaging. Our results indicate that although 2D EM is still an important tool in many areas of diagnostic pathology, 3D images of ultrastructural relationships between both normal and cancerous cells, in combination with their extracellular matrix, enables cancer researchers and pathologists to better understand the progression of the disease and identify potential therapeutic targets.

National Cancer Institute Think-Tank Meeting Report on Proteomic Cartography and Biomarkers at the Single-Cell Level: Interrogation of Premalignant Lesions.

A Think-Tank Meeting was convened by the National Cancer Institute (NCI) to solicit experts' opinion on the development and application of multiomic single-cell analyses, and especially single-cell proteomics, to improve the development of a new generation of biomarkers for cancer risk, early detection, diagnosis, and prognosis as well as to discuss the discovery of new targets for prevention and therapy. It is anticipated that such markers and targets will be based on cellular, subcellular, molecular, and functional aberrations within the lesion and within individual cells. Single-cell proteomic data will be essential for the establishment of new tools with searchable and scalable features that include spatial and temporal cartographies of premalignant and malignant lesions. Challenges and potential solutions that were discussed included (i) The best way/s to analyze single-cells from fresh and preserved tissue; (ii) Detection and analysis of secreted molecules and from single cells, especially from a tissue slice; (iii) Detection of new, previously undocumented cell type/s in the premalignant and early stage cancer tissue microenvironment; (iv) Multiomic integration of data to support and inform proteomic measurements; (v) Subcellular organelles-identifying abnormal structure, function, distribution, and location within individual premalignant and malignant cells; (vi) How to improve the dynamic range of single-cell proteomic measurements for discovery of differentially expressed proteins and their post-translational modifications (PTM); (vii) The depth of coverage measured concurrently using single-cell techniques; (viii) Quantitation - absolute or semiquantitative? (ix) Single methodology or multiplexed combinations? (x) Application of analytical methods for identification of biologically significant subsets; (xi) Data visualization of N-dimensional data sets; (xii) How to construct intercellular signaling networks in individual cells within premalignant tumor microenvironments (TME); (xiii) Associations between intrinsic cellular processes and extrinsic stimuli; (xiv) How to predict cellular responses to stress-inducing stimuli; (xv) Identification of new markers for prediction of progression from precursor, benign, and localized lesions to invasive cancer, based on spatial and temporal changes within individual cells; (xvi) Identification of new targets for immunoprevention or immunotherapy-identification of neoantigens and surfactome of individual cells within a lesion.

A fully integrated, three-dimensional fluorescence to electron microscopy correlative workflow.

While fluorescence microscopy provides tools for highly specific labeling and sensitive detection, its resolution limit and lack of general contrast has hindered studies of cellular structure and protein localization. Recent advances in correlative light and electron microscopy (CLEM), including the fully integrated CLEM workflow instrument, the FEI CorrSight with MAPS, have allowed for a more reliable, reproducible, and quicker approach to correlate three-dimensional time-lapse confocal fluorescence data, with three-dimensional focused ion beam-scanning electron microscopy data. Here we demonstrate the entire integrated CLEM workflow using fluorescently tagged MCF7 breast cancer cells.