Whole-body integration of gene expression and single-cell morphology.

Animal bodies are composed of cell types with unique expression programs that implement their distinct locations, shapes, structures, and functions. Based on these properties, cell types assemble into specific tissues and organs. To systematically explore the link between cell-type-specific gene expression and morphology, we registered an expression atlas to a whole-body electron microscopy volume of the nereid Platynereis dumerilii. Automated segmentation of cells and nuclei identifies major cell classes and establishes a link between gene activation, chromatin topography, and nuclear size. Clustering of segmented cells according to gene expression reveals spatially coherent tissues. In the brain, genetically defined groups of neurons match ganglionic nuclei with coherent projections. Besides interneurons, we uncover sensory-neurosecretory cells in the nereid mushroom bodies, which thus qualify as sensory organs. They furthermore resemble the vertebrate telencephalon by molecular anatomy. We provide an integrated browser as a Fiji plugin for remote exploration of all available multimodal datasets.

Embryo-uterine interaction coordinates mouse embryogenesis during implantation.

Embryo implantation into the uterus marks a key transition in mammalian development. In mice, implantation is mediated by the trophoblast and is accompanied by a morphological transition from the blastocyst to the egg cylinder. However, the roles of trophoblast-uterine interactions in embryo morphogenesis during implantation are poorly understood due to inaccessibility in utero and the remaining challenges to recapitulate it ex vivo from the blastocyst. Here, we engineer a uterus-like microenvironment to recapitulate peri-implantation development of the whole mouse embryo ex vivo and reveal essential roles of the physical embryo-uterine interaction. We demonstrate that adhesion between the trophoblast and the uterine matrix is required for in utero-like transition of the blastocyst to the egg cylinder. Modeling the implanting embryo as a wetting droplet links embryo shape dynamics to the underlying changes in trophoblast adhesion and suggests that the adhesion-mediated tension release facilitates egg cylinder formation. Light-sheet live imaging and the experimental control of the engineered uterine geometry and trophoblast velocity uncovers the coordination between trophoblast motility and embryo growth, where the trophoblast delineates space for embryo morphogenesis.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Convolutional networks for supervised mining of molecular patterns within cellular context.

Cryo-electron tomograms capture a wealth of structural information on the molecular constituents of cells and tissues. We present DeePiCt (deep picker in context), an open-source deep-learning framework for supervised segmentation and macromolecular complex localization in cryo-electron tomography. To train and benchmark DeePiCt on experimental data, we comprehensively annotated 20 tomograms of Schizosaccharomyces pombe for ribosomes, fatty acid synthases, membranes, nuclear pore complexes, organelles, and cytosol. By comparing DeePiCt to state-of-the-art approaches on this dataset, we show its unique ability to identify low-abundance and low-density complexes. We use DeePiCt to study compositionally distinct subpopulations of cellular ribosomes, with emphasis on their contextual association with mitochondria and the endoplasmic reticulum. Finally, applying pre-trained networks to a HeLa cell tomogram demonstrates that DeePiCt achieves high-quality predictions in unseen datasets from different biological species in a matter of minutes. The comprehensively annotated experimental data and pre-trained networks are provided for immediate use by the community.

Deep learning enables fast and dense single-molecule localization with high accuracy.

Single-molecule localization microscopy (SMLM) has had remarkable success in imaging cellular structures with nanometer resolution, but standard analysis algorithms require sparse emitters, which limits imaging speed and labeling density. Here, we overcome this major limitation using deep learning. We developed DECODE (deep context dependent), a computational tool that can localize single emitters at high density in three dimensions with highest accuracy for a large range of imaging modalities and conditions. In a public software benchmark competition, it outperformed all other fitters on 12 out of 12 datasets when comparing both detection accuracy and localization error, often by a substantial margin. DECODE allowed us to acquire fast dynamic live-cell SMLM data with reduced light exposure and to image microtubules at ultra-high labeling density. Packaged for simple installation and use, DECODE will enable many laboratories to reduce imaging times and increase localization density in SMLM.

Deep learning-enhanced light-field imaging with continuous validation.

Visualizing dynamic processes over large, three-dimensional fields of view at high speed is essential for many applications in the life sciences. Light-field microscopy (LFM) has emerged as a tool for fast volumetric image acquisition, but its effective throughput and widespread use in biology has been hampered by a computationally demanding and artifact-prone image reconstruction process. Here, we present a framework for artificial intelligence-enhanced microscopy, integrating a hybrid light-field light-sheet microscope and deep learning-based volume reconstruction. In our approach, concomitantly acquired, high-resolution two-dimensional light-sheet images continuously serve as training data and validation for the convolutional neural network reconstructing the raw LFM data during extended volumetric time-lapse imaging experiments. Our network delivers high-quality three-dimensional reconstructions at video-rate throughput, which can be further refined based on the high-resolution light-sheet images. We demonstrate the capabilities of our approach by imaging medaka heart dynamics and zebrafish neural activity with volumetric imaging rates up to 100 Hz.

Microscopy-based assay for semi-quantitative detection of SARS-CoV-2 specific antibodies in human sera: A semi-quantitative, high throughput, microscopy-based assay expands existing approaches to measure SARS-CoV-2 specific antibody levels in human sera.

Emergence of the novel pathogenic coronavirus SARS-CoV-2 and its rapid pandemic spread presents challenges that demand immediate attention. Here, we describe the development of a semi-quantitative high-content microscopy-based assay for detection of three major classes (IgG, IgA, and IgM) of SARS-CoV-2 specific antibodies in human samples. The possibility to detect antibodies against the entire viral proteome together with a robust semi-automated image analysis workflow resulted in specific, sensitive and unbiased assay that complements the portfolio of SARS-CoV-2 serological assays. Sensitive, specific and quantitative serological assays are urgently needed for a better understanding of humoral immune response against the virus as a basis for developing public health strategies to control viral spread. The procedure described here has been used for clinical studies and provides a general framework for the application of quantitative high-throughput microscopy to rapidly develop serological assays for emerging virus infections.

ilastik: interactive machine learning for (bio)image analysis.

We present ilastik, an easy-to-use interactive tool that brings machine-learning-based (bio)image analysis to end users without substantial computational expertise. It contains pre-defined workflows for image segmentation, object classification, counting and tracking. Users adapt the workflows to the problem at hand by interactively providing sparse training annotations for a nonlinear classifier. ilastik can process data in up to five dimensions (3D, time and number of channels). Its computational back end runs operations on-demand wherever possible, allowing for interactive prediction on data larger than RAM. Once the classifiers are trained, ilastik workflows can be applied to new data from the command line without further user interaction. We describe all ilastik workflows in detail, including three case studies and a discussion on the expected performance.

Integration of Cell Growth and Asymmetric Division during Lateral Root Initiation in Arabidopsis thaliana.

Lateral root formation determines to a large extent the ability of plants to forage their environment and thus their growth. In Arabidopsis thaliana and other angiosperms, lateral root initiation requires radial cell expansion and several rounds of anticlinal cell divisions that give rise to a central core of small cells, which express different markers than the larger surrounding cells. These small central cells then switch their plane of divisions to periclinal and give rise to seemingly morphologically similar daughter cells that have different identities and establish the different cell types of the new root. Although the execution of these anticlinal and periclinal divisions is tightly regulated and essential for the correct development of the lateral root, we know little about their geometrical features. Here, we generate a four-dimensional reconstruction of the first stages of lateral root formation and analyze the geometric features of the anticlinal and periclinal divisions. We identify that the periclinal divisions of the small central cells are morphologically dissimilar and asymmetric. We show that mother cell volume is different when looking at anticlinal vs. periclinal divisions and the repeated anticlinal divisions do not lead to reduction in cell volume, although cells are shorter. Finally, we show that cells undergoing a periclinal division are characterized by a strong cell expansion. Our results indicate that cells integrate growth and division to precisely partition their volume upon division during the first two stages of lateral root formation.

Segmenting and Tracking Multiple Dividing Targets Using ilastik.

Tracking crowded cells or other targets in biology is often a challenging task due to poor signal-to-noise ratio, mutual occlusion, large displacements, little discernibility, and the ability of cells to divide. We here present an open source implementation of conservation tracking (Schiegg et al., IEEE international conference on computer vision (ICCV). IEEE, New York, pp 2928-2935, 2013) in the ilastik software framework. This robust tracking-by-assignment algorithm explicitly makes allowance for false positive detections, undersegmentation, and cell division. We give an overview over the underlying algorithm and parameters, and explain the use for a light sheet microscopy sequence of a Drosophila embryo. Equipped with this knowledge, users will be able to track targets of interest in their own data.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

MorphoFeatures for unsupervised exploration of cell types, tissues, and organs in volume electron microscopy.

Electron microscopy (EM) provides a uniquely detailed view of cellular morphology, including organelles and fine subcellular ultrastructure. While the acquisition and (semi-)automatic segmentation of multicellular EM volumes are now becoming routine, large-scale analysis remains severely limited by the lack of generally applicable pipelines for automatic extraction of comprehensive morphological descriptors. Here, we present a novel unsupervised method for learning cellular morphology features directly from 3D EM data: a neural network delivers a representation of cells by shape and ultrastructure. Applied to the full volume of an entire three-segmented worm of the annelid Platynereis dumerilii, it yields a visually consistent grouping of cells supported by specific gene expression profiles. Integration of features across spatial neighbours can retrieve tissues and organs, revealing, for example, a detailed organisation of the animal foregut. We envision that the unbiased nature of the proposed morphological descriptors will enable rapid exploration of very different biological questions in large EM volumes, greatly increasing the impact of these invaluable, but costly resources.

Sensing their plasma membrane curvature allows migrating cells to circumvent obstacles.

To navigate through diverse tissues, migrating cells must balance persistent self-propelled motion with adaptive behaviors to circumvent obstacles. We identify a curvature-sensing mechanism underlying obstacle evasion in immune-like cells. Specifically, we propose that actin polymerization at the advancing edge of migrating cells is inhibited by the curvature-sensitive BAR domain protein Snx33 in regions with inward plasma membrane curvature. The genetic perturbation of this machinery reduces the cells' capacity to evade obstructions combined with faster and more persistent cell migration in obstacle-free environments. Our results show how cells can read out their surface topography and utilize actin and plasma membrane biophysics to interpret their environment, allowing them to adaptively decide if they should move ahead or turn away. On the basis of our findings, we propose that the natural diversity of BAR domain proteins may allow cells to tune their curvature sensing machinery to match the shape characteristics in their environment.

An ex vivo system to study cellular dynamics underlying mouse peri-implantation development.

Upon implantation, mammalian embryos undergo major morphogenesis and key developmental processes such as body axis specification and gastrulation. However, limited accessibility obscures the study of these crucial processes. Here, we develop an ex vivo Matrigel-collagen-based culture to recapitulate mouse development from E4.5 to E6.0. Our system not only recapitulates embryonic growth, axis initiation, and overall 3D architecture in 49% of the cases, but its compatibility with light-sheet microscopy also enables the study of cellular dynamics through automatic cell segmentation. We find that, upon implantation, release of the increasing tension in the polar trophectoderm is necessary for its constriction and invagination. The resulting extra-embryonic ectoderm plays a key role in growth, morphogenesis, and patterning of the neighboring epiblast, which subsequently gives rise to all embryonic tissues. This 3D ex vivo system thus offers unprecedented access to peri-implantation development for in toto monitoring, measurement, and spatiotemporally controlled perturbation, revealing a mechano-chemical interplay between extra-embryonic and embryonic tissues.

Volume electron microscopy.

Life exists in three dimensions, but until the turn of the century most electron microscopy methods provided only 2D image data. Recently, electron microscopy techniques capable of delving deep into the structure of cells and tissues have emerged, collectively called volume electron microscopy (vEM). Developments in vEM have been dubbed a quiet revolution as the field evolved from established transmission and scanning electron microscopy techniques, so early publications largely focused on the bioscience applications rather than the underlying technological breakthroughs. However, with an explosion in the uptake of vEM across the biosciences and fast-paced advances in volume, resolution, throughput and ease of use, it is timely to introduce the field to new audiences. In this Primer, we introduce the different vEM imaging modalities, the specialized sample processing and image analysis pipelines that accompany each modality and the types of information revealed in the data. We showcase key applications in the biosciences where vEM has helped make breakthrough discoveries and consider limitations and future directions. We aim to show new users how vEM can support discovery science in their own research fields and inspire broader uptake of the technology, finally allowing its full adoption into mainstream biological imaging.

The Mutex Watershed and its Objective: Efficient, Parameter-Free Graph Partitioning.

Image partitioning, or segmentation without semantics, is the task of decomposing an image into distinct segments, or equivalently to detect closed contours. Most prior work either requires seeds, one per segment; or a threshold; or formulates the task as multicut / correlation clustering, an NP-hard problem. Here, we propose an efficient algorithm for graph partitioning, the "Mutex Watershed". Unlike seeded watershed, the algorithm can accommodate not only attractive but also repulsive cues, allowing it to find a previously unspecified number of segments without the need for explicit seeds or a tunable threshold. We also prove that this simple algorithm solves to global optimality an objective function that is intimately related to the multicut / correlation clustering integer linear programming formulation. The algorithm is deterministic, very simple to implement, and has empirically linearithmic complexity. When presented with short-range attractive and long-range repulsive cues from a deep neural network, the Mutex Watershed gives the best results currently known for the competitive ISBI 2012 EM segmentation benchmark.