Content-aware frame interpolation (CAFI): deep learning-based temporal super-resolution for fast bioimaging.

The development of high-resolution microscopes has made it possible to investigate cellular processes in 3D and over time. However, observing fast cellular dynamics remains challenging because of photobleaching and phototoxicity. Here we report the implementation of two content-aware frame interpolation (CAFI) deep learning networks, Zooming SlowMo and Depth-Aware Video Frame Interpolation, that are highly suited for accurately predicting images in between image pairs, therefore improving the temporal resolution of image series post-acquisition. We show that CAFI is capable of understanding the motion context of biological structures and can perform better than standard interpolation methods. We benchmark CAFI's performance on 12 different datasets, obtained from four different microscopy modalities, and demonstrate its capabilities for single-particle tracking and nuclear segmentation. CAFI potentially allows for reduced light exposure and phototoxicity on the sample for improved long-term live-cell imaging. The models and the training and testing data are available via the ZeroCostDL4Mic platform.

Induction of lysosomal and mitochondrial biogenesis by AMPK phosphorylation of FNIP1.

Cells respond to mitochondrial poisons with rapid activation of the adenosine monophosphate-activated protein kinase (AMPK), causing acute metabolic changes through phosphorylation and prolonged adaptation of metabolism through transcriptional effects. Transcription factor EB (TFEB) is a major effector of AMPK that increases expression of lysosome genes in response to energetic stress, but how AMPK activates TFEB remains unresolved. We demonstrate that AMPK directly phosphorylates five conserved serine residues in folliculin-interacting protein 1 (FNIP1), suppressing the function of the folliculin (FLCN)-FNIP1 complex. FNIP1 phosphorylation is required for AMPK to induce nuclear translocation of TFEB and TFEB-dependent increases of peroxisome proliferator-activated receptor gamma coactivator 1-alpha (PGC1α) and estrogen-related receptor alpha (ERRα) messenger RNAs. Thus, mitochondrial damage triggers AMPK-FNIP1-dependent nuclear translocation of TFEB, inducing sequential waves of lysosomal and mitochondrial biogenesis.

Roadmap on Deep Learning for Microscopy.

Through digital imaging, microscopy has evolved from primarily being a means for visual observation of life at the micro- and nano-scale, to a quantitative tool with ever-increasing resolution and throughput. Artificial intelligence, deep neural networks, and machine learning are all niche terms describing computational methods that have gained a pivotal role in microscopy-based research over the past decade. This Roadmap is written collectively by prominent researchers and encompasses selected aspects of how machine learning is applied to microscopy image data, with the aim of gaining scientific knowledge by improved image quality, automated detection, segmentation, classification and tracking of objects, and efficient merging of information from multiple imaging modalities. We aim to give the reader an overview of the key developments and an understanding of possibilities and limitations of machine learning for microscopy. It will be of interest to a wide cross-disciplinary audience in the physical sciences and life sciences.

Local shape descriptors for neuron segmentation.

We present an auxiliary learning task for the problem of neuron segmentation in electron microscopy volumes. The auxiliary task consists of the prediction of local shape descriptors (LSDs), which we combine with conventional voxel-wise direct neighbor affinities for neuron boundary detection. The shape descriptors capture local statistics about the neuron to be segmented, such as diameter, elongation, and direction. On a study comparing several existing methods across various specimen, imaging techniques, and resolutions, auxiliary learning of LSDs consistently increases segmentation accuracy of affinity-based methods over a range of metrics. Furthermore, the addition of LSDs promotes affinity-based segmentation methods to be on par with the current state of the art for neuron segmentation (flood-filling networks), while being two orders of magnitudes more efficient-a critical requirement for the processing of future petabyte-sized datasets.

Structured cerebellar connectivity supports resilient pattern separation.

The cerebellum is thought to help detect and correct errors between intended and executed commands1,2 and is critical for social behaviours, cognition and emotion3-6. Computations for motor control must be performed quickly to correct errors in real time and should be sensitive to small differences between patterns for fine error correction while being resilient to noise7. Influential theories of cerebellar information processing have largely assumed random network connectivity, which increases the encoding capacity of the network's first layer8-13. However, maximizing encoding capacity reduces the resilience to noise7. To understand how neuronal circuits address this fundamental trade-off, we mapped the feedforward connectivity in the mouse cerebellar cortex using automated large-scale transmission electron microscopy and convolutional neural network-based image segmentation. We found that both the input and output layers of the circuit exhibit redundant and selective connectivity motifs, which contrast with prevailing models. Numerical simulations suggest that these redundant, non-random connectivity motifs increase the resilience to noise at a negligible cost to the overall encoding capacity. This work reveals how neuronal network structure can support a trade-off between encoding capacity and redundancy, unveiling principles of biological network architecture with implications for the design of artificial neural networks.

Automatic detection of synaptic partners in a whole-brain Drosophila electron microscopy data set.

We develop an automatic method for synaptic partner identification in insect brains and use it to predict synaptic partners in a whole-brain electron microscopy dataset of the fruit fly. The predictions can be used to infer a connectivity graph with high accuracy, thus allowing fast identification of neural pathways. To facilitate circuit reconstruction using our results, we develop CIRCUITMAP, a user interface add-on for the circuit annotation tool CATMAID.

The Neuroanatomical Ultrastructure and Function of a Biological Ring Attractor.

Neural representations of head direction (HD) have been discovered in many species. Theoretical work has proposed that the dynamics associated with these representations are generated, maintained, and updated by recurrent network structures called ring attractors. We evaluated this theorized structure-function relationship by performing electron-microscopy-based circuit reconstruction and RNA profiling of identified cell types in the HD system of Drosophila melanogaster. We identified motifs that have been hypothesized to maintain the HD representation in darkness, update it when the animal turns, and tether it to visual cues. Functional studies provided support for the proposed roles of individual excitatory or inhibitory circuit elements in shaping activity. We also discovered recurrent connections between neuronal arbors with mixed pre- and postsynaptic specializations. Our results confirm that the Drosophila HD network contains the core components of a ring attractor while also revealing unpredicted structural features that might enhance the network's computational power.

Large Scale Image Segmentation with Structured Loss Based Deep Learning for Connectome Reconstruction.

We present a method combining affinity prediction with region agglomeration, which improves significantly upon the state of the art of neuron segmentation from electron microscopy (EM) in accuracy and scalability. Our method consists of a 3D U-Net, trained to predict affinities between voxels, followed by iterative region agglomeration. We train using a structured loss based on Malis, encouraging topologically correct segmentations obtained from affinity thresholding. Our extension consists of two parts: First, we present a quasi-linear method to compute the loss gradient, improving over the original quadratic algorithm. Second, we compute the gradient in two separate passes to avoid spurious gradient contributions in early training stages. Our predictions are accurate enough that simple learning-free percentile-based agglomeration outperforms more involved methods used earlier on inferior predictions. We present results on three diverse EM datasets, achieving relative improvements over previous results of 27, 15, and 250 percent. Our findings suggest that a single method can be applied to both nearly isotropic block-face EM data and anisotropic serial sectioned EM data. The runtime of our method scales linearly with the size of the volume and achieves a throughput of $\sim$∼ 2.6 seconds per megavoxel, qualifying our method for the processing of very large datasets.