A deep learning-based toolkit for 3D nuclei segmentation and quantitative analysis in cellular and tissue context.

We present a new set of computational tools that enable accurate and widely applicable 3D segmentation of nuclei in various 3D digital organs. We have developed an approach for ground truth generation and iterative training of 3D nuclear segmentation models, which we applied to popular CellPose, PlantSeg and StarDist algorithms. We provide two high-quality models trained on plant nuclei that enable 3D segmentation of nuclei in datasets obtained from fixed or live samples, acquired from different plant and animal tissues, and stained with various nuclear stains or fluorescent protein-based nuclear reporters. We also share a diverse high-quality training dataset of about 10,000 nuclei. Furthermore, we advanced the MorphoGraphX analysis and visualization software by, among other things, providing a method for linking 3D segmented nuclei to their surrounding cells in 3D digital organs. We found that the nuclear-to-cell volume ratio varies between different ovule tissues and during the development of a tissue. Finally, we extended the PlantSeg 3D segmentation pipeline with a proofreading tool that uses 3D segmented nuclei as seeds to correct cell segmentation errors in difficult-to-segment tissues.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Sensing their plasma membrane curvature allows migrating cells to circumvent obstacles.

To navigate through diverse tissues, migrating cells must balance persistent self-propelled motion with adaptive behaviors to circumvent obstacles. We identify a curvature-sensing mechanism underlying obstacle evasion in immune-like cells. Specifically, we propose that actin polymerization at the advancing edge of migrating cells is inhibited by the curvature-sensitive BAR domain protein Snx33 in regions with inward plasma membrane curvature. The genetic perturbation of this machinery reduces the cells' capacity to evade obstructions combined with faster and more persistent cell migration in obstacle-free environments. Our results show how cells can read out their surface topography and utilize actin and plasma membrane biophysics to interpret their environment, allowing them to adaptively decide if they should move ahead or turn away. On the basis of our findings, we propose that the natural diversity of BAR domain proteins may allow cells to tune their curvature sensing machinery to match the shape characteristics in their environment.

Embryo-uterine interaction coordinates mouse embryogenesis during implantation.

Embryo implantation into the uterus marks a key transition in mammalian development. In mice, implantation is mediated by the trophoblast and is accompanied by a morphological transition from the blastocyst to the egg cylinder. However, the roles of trophoblast-uterine interactions in embryo morphogenesis during implantation are poorly understood due to inaccessibility in utero and the remaining challenges to recapitulate it ex vivo from the blastocyst. Here, we engineer a uterus-like microenvironment to recapitulate peri-implantation development of the whole mouse embryo ex vivo and reveal essential roles of the physical embryo-uterine interaction. We demonstrate that adhesion between the trophoblast and the uterine matrix is required for in utero-like transition of the blastocyst to the egg cylinder. Modeling the implanting embryo as a wetting droplet links embryo shape dynamics to the underlying changes in trophoblast adhesion and suggests that the adhesion-mediated tension release facilitates egg cylinder formation. Light-sheet live imaging and the experimental control of the engineered uterine geometry and trophoblast velocity uncovers the coordination between trophoblast motility and embryo growth, where the trophoblast delineates space for embryo morphogenesis.

MorphoFeatures for unsupervised exploration of cell types, tissues, and organs in volume electron microscopy.

Electron microscopy (EM) provides a uniquely detailed view of cellular morphology, including organelles and fine subcellular ultrastructure. While the acquisition and (semi-)automatic segmentation of multicellular EM volumes are now becoming routine, large-scale analysis remains severely limited by the lack of generally applicable pipelines for automatic extraction of comprehensive morphological descriptors. Here, we present a novel unsupervised method for learning cellular morphology features directly from 3D EM data: a neural network delivers a representation of cells by shape and ultrastructure. Applied to the full volume of an entire three-segmented worm of the annelid Platynereis dumerilii, it yields a visually consistent grouping of cells supported by specific gene expression profiles. Integration of features across spatial neighbours can retrieve tissues and organs, revealing, for example, a detailed organisation of the animal foregut. We envision that the unbiased nature of the proposed morphological descriptors will enable rapid exploration of very different biological questions in large EM volumes, greatly increasing the impact of these invaluable, but costly resources.

Convolutional networks for supervised mining of molecular patterns within cellular context.

Cryo-electron tomograms capture a wealth of structural information on the molecular constituents of cells and tissues. We present DeePiCt (deep picker in context), an open-source deep-learning framework for supervised segmentation and macromolecular complex localization in cryo-electron tomography. To train and benchmark DeePiCt on experimental data, we comprehensively annotated 20 tomograms of Schizosaccharomyces pombe for ribosomes, fatty acid synthases, membranes, nuclear pore complexes, organelles, and cytosol. By comparing DeePiCt to state-of-the-art approaches on this dataset, we show its unique ability to identify low-abundance and low-density complexes. We use DeePiCt to study compositionally distinct subpopulations of cellular ribosomes, with emphasis on their contextual association with mitochondria and the endoplasmic reticulum. Finally, applying pre-trained networks to a HeLa cell tomogram demonstrates that DeePiCt achieves high-quality predictions in unseen datasets from different biological species in a matter of minutes. The comprehensively annotated experimental data and pre-trained networks are provided for immediate use by the community.

An ex vivo system to study cellular dynamics underlying mouse peri-implantation development.

Upon implantation, mammalian embryos undergo major morphogenesis and key developmental processes such as body axis specification and gastrulation. However, limited accessibility obscures the study of these crucial processes. Here, we develop an ex vivo Matrigel-collagen-based culture to recapitulate mouse development from E4.5 to E6.0. Our system not only recapitulates embryonic growth, axis initiation, and overall 3D architecture in 49% of the cases, but its compatibility with light-sheet microscopy also enables the study of cellular dynamics through automatic cell segmentation. We find that, upon implantation, release of the increasing tension in the polar trophectoderm is necessary for its constriction and invagination. The resulting extra-embryonic ectoderm plays a key role in growth, morphogenesis, and patterning of the neighboring epiblast, which subsequently gives rise to all embryonic tissues. This 3D ex vivo system thus offers unprecedented access to peri-implantation development for in toto monitoring, measurement, and spatiotemporally controlled perturbation, revealing a mechano-chemical interplay between extra-embryonic and embryonic tissues.

Volume electron microscopy.

Life exists in three dimensions, but until the turn of the century most electron microscopy methods provided only 2D image data. Recently, electron microscopy techniques capable of delving deep into the structure of cells and tissues have emerged, collectively called volume electron microscopy (vEM). Developments in vEM have been dubbed a quiet revolution as the field evolved from established transmission and scanning electron microscopy techniques, so early publications largely focused on the bioscience applications rather than the underlying technological breakthroughs. However, with an explosion in the uptake of vEM across the biosciences and fast-paced advances in volume, resolution, throughput and ease of use, it is timely to introduce the field to new audiences. In this Primer, we introduce the different vEM imaging modalities, the specialized sample processing and image analysis pipelines that accompany each modality and the types of information revealed in the data. We showcase key applications in the biosciences where vEM has helped make breakthrough discoveries and consider limitations and future directions. We aim to show new users how vEM can support discovery science in their own research fields and inspire broader uptake of the technology, finally allowing its full adoption into mainstream biological imaging.

Profiling cellular diversity in sponges informs animal cell type and nervous system evolution.

The evolutionary origin of metazoan cell types such as neurons and muscles is not known. Using whole-body single-cell RNA sequencing in a sponge, an animal without nervous system and musculature, we identified 18 distinct cell types. These include nitric oxide­sensitive contractile pinacocytes, amoeboid phagocytes, and secretory neuroid cells that reside in close contact with digestive choanocytes that express scaffolding and receptor proteins. Visualizing neuroid cells by correlative x-ray and electron microscopy revealed secretory vesicles and cellular projections enwrapping choanocyte microvilli and cilia. Our data show a communication system that is organized around sponge digestive chambers, using conserved modules that became incorporated into the pre- and postsynapse in the nervous systems of other animals.

Deep learning enables fast and dense single-molecule localization with high accuracy.

Single-molecule localization microscopy (SMLM) has had remarkable success in imaging cellular structures with nanometer resolution, but standard analysis algorithms require sparse emitters, which limits imaging speed and labeling density. Here, we overcome this major limitation using deep learning. We developed DECODE (deep context dependent), a computational tool that can localize single emitters at high density in three dimensions with highest accuracy for a large range of imaging modalities and conditions. In a public software benchmark competition, it outperformed all other fitters on 12 out of 12 datasets when comparing both detection accuracy and localization error, often by a substantial margin. DECODE allowed us to acquire fast dynamic live-cell SMLM data with reduced light exposure and to image microtubules at ultra-high labeling density. Packaged for simple installation and use, DECODE will enable many laboratories to reduce imaging times and increase localization density in SMLM.

Mechanical competition alters the cellular interpretation of an endogenous genetic program.

The intrinsic genetic program of a cell is not sufficient to explain all of the cell's activities. External mechanical stimuli are increasingly recognized as determinants of cell behavior. In the epithelial folding event that constitutes the beginning of gastrulation in Drosophila, the genetic program of the future mesoderm leads to the establishment of a contractile actomyosin network that triggers apical constriction of cells and thereby tissue folding. However, some cells do not constrict but instead stretch, even though they share the same genetic program as their constricting neighbors. We show here that tissue-wide interactions force these cells to expand even when an otherwise sufficient amount of apical, active actomyosin is present. Models based on contractile forces and linear stress-strain responses do not reproduce experimental observations, but simulations in which cells behave as ductile materials with nonlinear mechanical properties do. Our models show that this behavior is a general emergent property of actomyosin networks in a supracellular context, in accordance with our experimental observations of actin reorganization within stretching cells.

Whole-body integration of gene expression and single-cell morphology.

Animal bodies are composed of cell types with unique expression programs that implement their distinct locations, shapes, structures, and functions. Based on these properties, cell types assemble into specific tissues and organs. To systematically explore the link between cell-type-specific gene expression and morphology, we registered an expression atlas to a whole-body electron microscopy volume of the nereid Platynereis dumerilii. Automated segmentation of cells and nuclei identifies major cell classes and establishes a link between gene activation, chromatin topography, and nuclear size. Clustering of segmented cells according to gene expression reveals spatially coherent tissues. In the brain, genetically defined groups of neurons match ganglionic nuclei with coherent projections. Besides interneurons, we uncover sensory-neurosecretory cells in the nereid mushroom bodies, which thus qualify as sensory organs. They furthermore resemble the vertebrate telencephalon by molecular anatomy. We provide an integrated browser as a Fiji plugin for remote exploration of all available multimodal datasets.

Developing open-source software for bioimage analysis: opportunities and challenges.

Fast-paced innovations in imaging have resulted in single systems producing exponential amounts of data to be analyzed. Computational methods developed in computer science labs have proven to be crucial for analyzing these data in an unbiased and efficient manner, reaching a prominent role in most microscopy studies. Still, their use usually requires expertise in bioimage analysis, and their accessibility for life scientists has therefore become a bottleneck. Open-source software for bioimage analysis has developed to disseminate these computational methods to a wider audience, and to life scientists in particular. In recent years, the influence of many open-source tools has grown tremendously, helping tens of thousands of life scientists in the process. As creators of successful open-source bioimage analysis software, we here discuss the motivations that can initiate development of a new tool, the common challenges faced, and the characteristics required for achieving success.

Deep learning-enhanced light-field imaging with continuous validation.

Visualizing dynamic processes over large, three-dimensional fields of view at high speed is essential for many applications in the life sciences. Light-field microscopy (LFM) has emerged as a tool for fast volumetric image acquisition, but its effective throughput and widespread use in biology has been hampered by a computationally demanding and artifact-prone image reconstruction process. Here, we present a framework for artificial intelligence-enhanced microscopy, integrating a hybrid light-field light-sheet microscope and deep learning-based volume reconstruction. In our approach, concomitantly acquired, high-resolution two-dimensional light-sheet images continuously serve as training data and validation for the convolutional neural network reconstructing the raw LFM data during extended volumetric time-lapse imaging experiments. Our network delivers high-quality three-dimensional reconstructions at video-rate throughput, which can be further refined based on the high-resolution light-sheet images. We demonstrate the capabilities of our approach by imaging medaka heart dynamics and zebrafish neural activity with volumetric imaging rates up to 100 Hz.

Integration of Cell Growth and Asymmetric Division during Lateral Root Initiation in Arabidopsis thaliana.

Lateral root formation determines to a large extent the ability of plants to forage their environment and thus their growth. In Arabidopsis thaliana and other angiosperms, lateral root initiation requires radial cell expansion and several rounds of anticlinal cell divisions that give rise to a central core of small cells, which express different markers than the larger surrounding cells. These small central cells then switch their plane of divisions to periclinal and give rise to seemingly morphologically similar daughter cells that have different identities and establish the different cell types of the new root. Although the execution of these anticlinal and periclinal divisions is tightly regulated and essential for the correct development of the lateral root, we know little about their geometrical features. Here, we generate a four-dimensional reconstruction of the first stages of lateral root formation and analyze the geometric features of the anticlinal and periclinal divisions. We identify that the periclinal divisions of the small central cells are morphologically dissimilar and asymmetric. We show that mother cell volume is different when looking at anticlinal vs. periclinal divisions and the repeated anticlinal divisions do not lead to reduction in cell volume, although cells are shorter. Finally, we show that cells undergoing a periclinal division are characterized by a strong cell expansion. Our results indicate that cells integrate growth and division to precisely partition their volume upon division during the first two stages of lateral root formation.

A digital 3D reference atlas reveals cellular growth patterns shaping the Arabidopsis ovule.

A fundamental question in biology is how morphogenesis integrates the multitude of processes that act at different scales, ranging from the molecular control of gene expression to cellular coordination in a tissue. Using machine-learning-based digital image analysis, we generated a three-dimensional atlas of ovule development in Arabidopsis thaliana, enabling the quantitative spatio-temporal analysis of cellular and gene expression patterns with cell and tissue resolution. We discovered novel morphological manifestations of ovule polarity, a new mode of cell layer formation, and previously unrecognized subepidermal cell populations that initiate ovule curvature. The data suggest an irregular cellular build-up of WUSCHEL expression in the primordium and new functions for INNER NO OUTER in restricting nucellar cell proliferation and the organization of the interior chalaza. Our work demonstrates the analytical power of a three-dimensional digital representation when studying the morphogenesis of an organ of complex architecture that eventually consists of 1900 cells.