Petascale pipeline for precise alignment of images from serial section electron microscopy.

The reconstruction of neural circuits from serial section electron microscopy (ssEM) images is being accelerated by automatic image segmentation methods. Segmentation accuracy is often limited by the preceding step of aligning 2D section images to create a 3D image stack. Precise and robust alignment in the presence of image artifacts is challenging, especially as datasets are attaining the petascale. We present a computational pipeline for aligning ssEM images with several key elements. Self-supervised convolutional nets are trained via metric learning to encode and align image pairs, and they are used to initialize iterative fine-tuning of alignment. A procedure called vector voting increases robustness to image artifacts or missing image data. For speedup the series is divided into blocks that are distributed to computational workers for alignment. The blocks are aligned to each other by composing transformations with decay, which achieves a global alignment without resorting to a time-consuming global optimization. We apply our pipeline to a whole fly brain dataset, and show improved accuracy relative to prior state of the art. We also demonstrate that our pipeline scales to a cubic millimeter of mouse visual cortex. Our pipeline is publicly available through two open source Python packages.

Comparative connectomics and escape behavior in larvae of closely related Drosophila species.

Evolution has generated an enormous variety of morphological, physiological, and behavioral traits in animals. How do behaviors evolve in different directions in species equipped with similar neurons and molecular components? Here we adopted a comparative approach to investigate the similarities and differences of escape behaviors in response to noxious stimuli and their underlying neural circuits between closely related drosophilid species. Drosophilids show a wide range of escape behaviors in response to noxious cues, including escape crawling, stopping, head casting, and rolling. Here we find that D. santomea, compared with its close relative D. melanogaster, shows a higher probability of rolling in response to noxious stimulation. To assess whether this behavioral difference could be attributed to differences in neural circuitry, we generated focused ion beam-scanning electron microscope volumes of the ventral nerve cord of D. santomea to reconstruct the downstream partners of mdIV, a nociceptive sensory neuron in D. melanogaster. Along with partner interneurons of mdVI (including Basin-2, a multisensory integration neuron necessary for rolling) previously identified in D. melanogaster, we identified two additional partners of mdVI in D. santomea. Finally, we showed that joint activation of one of the partners (Basin-1) and a common partner (Basin-2) in D. melanogaster increased rolling probability, suggesting that the high rolling probability in D. santomea is mediated by the additional activation of Basin-1 by mdIV. These results provide a plausible mechanistic explanation for how closely related species exhibit quantitative differences in the likelihood of expressing the same behavior.

Architecture and dynamics of a desmosome-endoplasmic reticulum complex.

The endoplasmic reticulum (ER) forms a dynamic network that contacts other cellular membranes to regulate stress responses, calcium signalling and lipid transfer. Here, using high-resolution volume electron microscopy, we find that the ER forms a previously unknown association with keratin intermediate filaments and desmosomal cell-cell junctions. Peripheral ER assembles into mirror image-like arrangements at desmosomes and exhibits nanometre proximity to keratin filaments and the desmosome cytoplasmic plaque. ER tubules exhibit stable associations with desmosomes, and perturbation of desmosomes or keratin filaments alters ER organization, mobility and expression of ER stress transcripts. These findings indicate that desmosomes and the keratin cytoskeleton regulate the distribution, function and dynamics of the ER network. Overall, this study reveals a previously unknown subcellular architecture defined by the structural integration of ER tubules with an epithelial intercellular junction.

Local shape descriptors for neuron segmentation.

We present an auxiliary learning task for the problem of neuron segmentation in electron microscopy volumes. The auxiliary task consists of the prediction of local shape descriptors (LSDs), which we combine with conventional voxel-wise direct neighbor affinities for neuron boundary detection. The shape descriptors capture local statistics about the neuron to be segmented, such as diameter, elongation, and direction. On a study comparing several existing methods across various specimen, imaging techniques, and resolutions, auxiliary learning of LSDs consistently increases segmentation accuracy of affinity-based methods over a range of metrics. Furthermore, the addition of LSDs promotes affinity-based segmentation methods to be on par with the current state of the art for neuron segmentation (flood-filling networks), while being two orders of magnitudes more efficient-a critical requirement for the processing of future petabyte-sized datasets.

Rapid reconstruction of neural circuits using tissue expansion and light sheet microscopy.

Brain function is mediated by the physiological coordination of a vast, intricately connected network of molecular and cellular components. The physiological properties of neural network components can be quantified with high throughput. The ability to assess many animals per study has been critical in relating physiological properties to behavior. By contrast, the synaptic structure of neural circuits is presently quantifiable only with low throughput. This low throughput hampers efforts to understand how variations in network structure relate to variations in behavior. For neuroanatomical reconstruction, there is a methodological gulf between electron microscopic (EM) methods, which yield dense connectomes at considerable expense and low throughput, and light microscopic (LM) methods, which provide molecular and cell-type specificity at high throughput but without synaptic resolution. To bridge this gulf, we developed a high-throughput analysis pipeline and imaging protocol using tissue expansion and light sheet microscopy (ExLLSM) to rapidly reconstruct selected circuits across many animals with single-synapse resolution and molecular contrast. Using Drosophila to validate this approach, we demonstrate that it yields synaptic counts similar to those obtained by EM, enables synaptic connectivity to be compared across sex and experience, and can be used to correlate structural connectivity, functional connectivity, and behavior. This approach fills a critical methodological gap in studying variability in the structure and function of neural circuits across individuals within and between species.

A scalable and modular automated pipeline for stitching of large electron microscopy datasets.

Serial-section electron microscopy (ssEM) is the method of choice for studying macroscopic biological samples at extremely high resolution in three dimensions. In the nervous system, nanometer-scale images are necessary to reconstruct dense neural wiring diagrams in the brain, so -called connectomes. The data that can comprise of up to 108 individual EM images must be assembled into a volume, requiring seamless 2D registration from physical section followed by 3D alignment of the stitched sections. The high throughput of ssEM necessitates 2D stitching to be done at the pace of imaging, which currently produces tens of terabytes per day. To achieve this, we present a modular volume assembly software pipeline ASAP (Assembly Stitching and Alignment Pipeline) that is scalable to datasets containing petabytes of data and parallelized to work in a distributed computational environment. The pipeline is built on top of the Render Trautman and Saalfeld (2019) services used in the volume assembly of the brain of adult Drosophila melanogaster (Zheng et al. 2018). It achieves high throughput by operating only on image meta-data and transformations. ASAP is modular, allowing for easy incorporation of new algorithms without significant changes in the workflow. The entire software pipeline includes a complete set of tools for stitching, automated quality control, 3D section alignment, and final rendering of the assembled volume to disk. ASAP has been deployed for continuous stitching of several large-scale datasets of the mouse visual cortex and human brain samples including one cubic millimeter of mouse visual cortex (Yin et al. 2020); Microns Consortium et al. (2021) at speeds that exceed imaging. The pipeline also has multi-channel processing capabilities and can be applied to fluorescence and multi-modal datasets like array tomography.

Transverse endoplasmic reticulum expansion in hereditary spastic paraplegia corticospinal axons.

Hereditary spastic paraplegias (HSPs) comprise a large group of inherited neurologic disorders affecting the longest corticospinal axons (SPG1-86 plus others), with shared manifestations of lower extremity spasticity and gait impairment. Common autosomal dominant HSPs are caused by mutations in genes encoding the microtubule-severing ATPase spastin (SPAST; SPG4), the membrane-bound GTPase atlastin-1 (ATL1; SPG3A) and the reticulon-like, microtubule-binding protein REEP1 (REEP1; SPG31). These proteins bind one another and function in shaping the tubular endoplasmic reticulum (ER) network. Typically, mouse models of HSPs have mild, later onset phenotypes, possibly reflecting far shorter lengths of their corticospinal axons relative to humans. Here, we have generated a robust, double mutant mouse model of HSP in which atlastin-1 is genetically modified with a K80A knock-in (KI) missense change that abolishes its GTPase activity, whereas its binding partner Reep1 is knocked out. Atl1KI/KI/Reep1-/- mice exhibit early onset and rapidly progressive declines in several motor function tests. Also, ER in mutant corticospinal axons dramatically expands transversely and periodically in a mutation dosage-dependent manner to create a ladder-like appearance, on the basis of reconstructions of focused ion beam-scanning electron microscopy datasets using machine learning-based auto-segmentation. In lockstep with changes in ER morphology, axonal mitochondria are fragmented and proportions of hypophosphorylated neurofilament H and M subunits are dramatically increased in Atl1KI/KI/Reep1-/- spinal cord. Co-occurrence of these findings links ER morphology changes to alterations in mitochondrial morphology and cytoskeletal organization. Atl1KI/KI/Reep1-/- mice represent an early onset rodent HSP model with robust behavioral and cellular readouts for testing novel therapies.

EASI-FISH for thick tissue defines lateral hypothalamus spatio-molecular organization.

Determining the spatial organization and morphological characteristics of molecularly defined cell types is a major bottleneck for characterizing the architecture underpinning brain function. We developed Expansion-Assisted Iterative Fluorescence In Situ Hybridization (EASI-FISH) to survey gene expression in brain tissue, as well as a turnkey computational pipeline to rapidly process large EASI-FISH image datasets. EASI-FISH was optimized for thick brain sections (300 µm) to facilitate reconstruction of spatio-molecular domains that generalize across brains. Using the EASI-FISH pipeline, we investigated the spatial distribution of dozens of molecularly defined cell types in the lateral hypothalamic area (LHA), a brain region with poorly defined anatomical organization. Mapping cell types in the LHA revealed nine spatially and molecularly defined subregions. EASI-FISH also facilitates iterative reanalysis of scRNA-seq datasets to determine marker-genes that further dissociated spatial and morphological heterogeneity. The EASI-FISH pipeline democratizes mapping molecularly defined cell types, enabling discoveries about brain organization.

Whole-cell organelle segmentation in volume electron microscopy.

Cells contain hundreds of organelles and macromolecular assemblies. Obtaining a complete understanding of their intricate organization requires the nanometre-level, three-dimensional reconstruction of whole cells, which is only feasible with robust and scalable automatic methods. Here, to support the development of such methods, we annotated up to 35 different cellular organelle classes-ranging from endoplasmic reticulum to microtubules to ribosomes-in diverse sample volumes from multiple cell types imaged at a near-isotropic resolution of 4 nm per voxel with focused ion beam scanning electron microscopy (FIB-SEM)1. We trained deep learning architectures to segment these structures in 4 nm and 8 nm per voxel FIB-SEM volumes, validated their performance and showed that automatic reconstructions can be used to directly quantify previously inaccessible metrics including spatial interactions between cellular components. We also show that such reconstructions can be used to automatically register light and electron microscopy images for correlative studies. We have created an open data and open-source web repository, 'OpenOrganelle', to share the data, computer code and trained models, which will enable scientists everywhere to query and further improve automatic reconstruction of these datasets.

Automatic detection of synaptic partners in a whole-brain Drosophila electron microscopy data set.

We develop an automatic method for synaptic partner identification in insect brains and use it to predict synaptic partners in a whole-brain electron microscopy dataset of the fruit fly. The predictions can be used to infer a connectivity graph with high accuracy, thus allowing fast identification of neural pathways. To facilitate circuit reconstruction using our results, we develop CIRCUITMAP, a user interface add-on for the circuit annotation tool CATMAID.

An unbiased template of the Drosophila brain and ventral nerve cord.

The fruit fly Drosophila melanogaster is an important model organism for neuroscience with a wide array of genetic tools that enable the mapping of individual neurons and neural subtypes. Brain templates are essential for comparative biological studies because they enable analyzing many individuals in a common reference space. Several central brain templates exist for Drosophila, but every one is either biased, uses sub-optimal tissue preparation, is imaged at low resolution, or does not account for artifacts. No publicly available Drosophila ventral nerve cord template currently exists. In this work, we created high-resolution templates of the Drosophila brain and ventral nerve cord using the best-available technologies for imaging, artifact correction, stitching, and template construction using groupwise registration. We evaluated our central brain template against the four most competitive, publicly available brain templates and demonstrate that ours enables more accurate registration with fewer local deformations in shorter time.

Computational methods for stitching, alignment, and artifact correction of serial section data.

Imaging large samples at the resolution offered by electron microscopy is typically achieved by sequentially recording overlapping tiles that are later combined to seamless mosaics. Mosaics of serial sections are aligned to reconstruct three-dimensional volumes. To achieve this, image distortions and artifacts as introduced during sample preparation or imaging need to be removed. In this chapter, we will discuss typical sources of artifacts and distortion, and we will learn how to use the open source software TrakEM2 to correct them.

Cortical column and whole-brain imaging with molecular contrast and nanoscale resolution.

Optical and electron microscopy have made tremendous inroads toward understanding the complexity of the brain. However, optical microscopy offers insufficient resolution to reveal subcellular details, and electron microscopy lacks the throughput and molecular contrast to visualize specific molecular constituents over millimeter-scale or larger dimensions. We combined expansion microscopy and lattice light-sheet microscopy to image the nanoscale spatial relationships between proteins across the thickness of the mouse cortex or the entire Drosophila brain. These included synaptic proteins at dendritic spines, myelination along axons, and presynaptic densities at dopaminergic neurons in every fly brain region. The technology should enable statistically rich, large-scale studies of neural development, sexual dimorphism, degree of stereotypy, and structural correlations to behavior or neural activity, all with molecular contrast.

Large Scale Image Segmentation with Structured Loss Based Deep Learning for Connectome Reconstruction.

We present a method combining affinity prediction with region agglomeration, which improves significantly upon the state of the art of neuron segmentation from electron microscopy (EM) in accuracy and scalability. Our method consists of a 3D U-Net, trained to predict affinities between voxels, followed by iterative region agglomeration. We train using a structured loss based on Malis, encouraging topologically correct segmentations obtained from affinity thresholding. Our extension consists of two parts: First, we present a quasi-linear method to compute the loss gradient, improving over the original quadratic algorithm. Second, we compute the gradient in two separate passes to avoid spurious gradient contributions in early training stages. Our predictions are accurate enough that simple learning-free percentile-based agglomeration outperforms more involved methods used earlier on inferior predictions. We present results on three diverse EM datasets, achieving relative improvements over previous results of 27, 15, and 250 percent. Our findings suggest that a single method can be applied to both nearly isotropic block-face EM data and anisotropic serial sectioned EM data. The runtime of our method scales linearly with the size of the volume and achieves a throughput of $\sim$∼ 2.6 seconds per megavoxel, qualifying our method for the processing of very large datasets.

A Complete Electron Microscopy Volume of the Brain of Adult Drosophila melanogaster.

Drosophila melanogaster has a rich repertoire of innate and learned behaviors. Its 100,000-neuron brain is a large but tractable target for comprehensive neural circuit mapping. Only electron microscopy (EM) enables complete, unbiased mapping of synaptic connectivity; however, the fly brain is too large for conventional EM. We developed a custom high-throughput EM platform and imaged the entire brain of an adult female fly at synaptic resolution. To validate the dataset, we traced brain-spanning circuitry involving the mushroom body (MB), which has been extensively studied for its role in learning. All inputs to Kenyon cells (KCs), the intrinsic neurons of the MB, were mapped, revealing a previously unknown cell type, postsynaptic partners of KC dendrites, and unexpected clustering of olfactory projection neurons. These reconstructions show that this freely available EM volume supports mapping of brain-spanning circuits, which will significantly accelerate Drosophila neuroscience. VIDEO ABSTRACT.

Building bridges between cellular and molecular structural biology.

The integration of cellular and molecular structural data is key to understanding the function of macromolecular assemblies and complexes in their in vivo context. Here we report on the outcomes of a workshop that discussed how to integrate structural data from a range of public archives. The workshop identified two main priorities: the development of tools and file formats to support segmentation (that is, the decomposition of a three-dimensional volume into regions that can be associated with defined objects), and the development of tools to support the annotation of biological structures.

Whole-brain serial-section electron microscopy in larval zebrafish.

High-resolution serial-section electron microscopy (ssEM) makes it possible to investigate the dense meshwork of axons, dendrites, and synapses that form neuronal circuits. However, the imaging scale required to comprehensively reconstruct these structures is more than ten orders of magnitude smaller than the spatial extents occupied by networks of interconnected neurons, some of which span nearly the entire brain. Difficulties in generating and handling data for large volumes at nanoscale resolution have thus restricted vertebrate studies to fragments of circuits. These efforts were recently transformed by advances in computing, sample handling, and imaging techniques, but high-resolution examination of entire brains remains a challenge. Here, we present ssEM data for the complete brain of a larval zebrafish (Danio rerio) at 5.5 days post-fertilization. Our approach utilizes multiple rounds of targeted imaging at different scales to reduce acquisition time and data management requirements. The resulting dataset can be analysed to reconstruct neuronal processes, permitting us to survey all myelinated axons (the projectome). These reconstructions enable precise investigations of neuronal morphology, which reveal remarkable bilateral symmetry in myelinated reticulospinal and lateral line afferent axons. We further set the stage for whole-brain structure-function comparisons by co-registering functional reference atlases and in vivo two-photon fluorescence microscopy data from the same specimen. All obtained images and reconstructions are provided as an open-access resource.

PreMosa: extracting 2D surfaces from 3D microscopy mosaics.

MOTIVATION: A significant focus of biological research is to understand the development, organization and function of tissues. A particularly productive area of study is on single layer epithelial tissues in which the adherence junctions of cells form a 2D manifold that is fluorescently labeled. Given the size of the tissue, a microscope must collect a mosaic of overlapping 3D stacks encompassing the stained surface. Downstream interpretation is greatly simplified by preprocessing such a dataset as follows: (i) extracting and mapping the stained manifold in each stack into a single 2D projection plane, (ii) correcting uneven illumination artifacts, (iii) stitching the mosaic planes into a single, large 2D image and (iv) adjusting the contrast. RESULTS: We have developed PreMosa, an efficient, fully automatic pipeline to perform the four preprocessing tasks above resulting in a single 2D image of the stained manifold across which contrast is optimized and illumination is even. Notable features are as follows. First, the 2D projection step employs a specially developed algorithm that actually finds the manifold in the stack based on maximizing contrast, intensity and smoothness. Second, the projection step comes first, implying all subsequent tasks are more rapidly solved in 2D. And last, the mosaic melding employs an algorithm that globally adjusts contrasts amongst the 2D tiles so as to produce a seamless, high-contrast image. We conclude with an evaluation using ground-truth datasets and present results on datasets from Drosophila melanogaster wings and Schmidtae mediterranea ciliary components. AVAILABILITY AND IMPLEMENTATION: PreMosa is available under https://cblasse.github.io/premosa. CONTACT: blasse@mpi-cbg.de or myers@mpi-cbg.de. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.