Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning.

Light-field microscopy has emerged as a technique of choice for high-speed volumetric imaging of fast biological processes. However, artifacts, nonuniform resolution and a slow reconstruction speed have limited its full capabilities for in toto extraction of dynamic spatiotemporal patterns in samples. Here, we combined a view-channel-depth (VCD) neural network with light-field microscopy to mitigate these limitations, yielding artifact-free three-dimensional image sequences with uniform spatial resolution and high-video-rate reconstruction throughput. We imaged neuronal activities across moving Caenorhabditis elegans and blood flow in a beating zebrafish heart at single-cell resolution with volumetric imaging rates up to 200 Hz.

Adaptive super-resolution enabled on-chip contact microscopy.

We demonstrate an adaptive super-resolution based contact imaging on a CMOS chip to achieve subcellular spatial resolution over a large field of view of ∼24 mm2. By using regular LED illumination, we acquire the single lower-resolution image of the objects placed approximate to the sensor with unit magnification. For the raw contact-mode lens-free image, the pixel size of the sensor chip limits the spatial resolution. We develop a hybrid supervised-unsupervised strategy to train a super-resolution network, circumventing the missing of in-situ ground truth, effectively recovering a much higher resolution image of the objects, permitting sub-micron spatial resolution to be achieved across the entire sensor chip active area. We demonstrate the success of this approach by imaging the proliferation dynamics of cells directly cultured on the chip.

Semantic redundancy-aware implicit neural compression for multidimensional biomedical image data.

The surge in advanced imaging techniques has generated vast biomedical image data with diverse dimensions in space, time and spectrum, posing big challenges to conventional compression techniques in image storage, transmission, and sharing. Here, we propose an intelligent image compression approach with the first-proved semantic redundancy of biomedical data in the implicit neural function domain. This Semantic redundancy based Implicit Neural Compression guided with Saliency map (SINCS) can notably improve the compression efficiency for arbitrary-dimensional image data in terms of compression ratio and fidelity. Moreover, with weight transfer and residual entropy coding strategies, it shows improved compression speed while maintaining high quality. SINCS yields high quality compression with over 2000-fold compression ratio on 2D, 2D-T, 3D, 4D biomedical images of diverse targets ranging from single virus to entire human organs, and ensures reliable downstream tasks, such as object segmentation and quantitative analyses, to be conducted at high efficiency.

A practical guide to deep-learning light-field microscopy for 3D imaging of biological dynamics.

Here, we present a step-by-step protocol for the implementation of deep-learning-enhanced light-field microscopy enabling 3D imaging of instantaneous biological processes. We first provide the instructions to build a light-field microscope (LFM) capable of capturing optically encoded dynamic signals. Then, we detail the data processing and model training of a view-channel-depth (VCD) neural network, which enables instant 3D image reconstruction from a single 2D light-field snapshot. Finally, we describe VCD-LFM imaging of several model organisms and demonstrate image-based quantitative studies on neural activities and cardio-hemodynamics. For complete details on the use and execution of this protocol, please refer to Wang et al. (2021).1.

Video-rate 3D imaging of living cells using Fourier view-channel-depth light field microscopy.

Interrogation of subcellular biological dynamics occurring in a living cell often requires noninvasive imaging of the fragile cell with high spatiotemporal resolution across all three dimensions. It thereby poses big challenges to modern fluorescence microscopy implementations because the limited photon budget in a live-cell imaging task makes the achievable performance of conventional microscopy approaches compromise between their spatial resolution, volumetric imaging speed, and phototoxicity. Here, we incorporate a two-stage view-channel-depth (VCD) deep-learning reconstruction strategy with a Fourier light-field microscope based on diffractive optical element to realize fast 3D super-resolution reconstructions of intracellular dynamics from single diffraction-limited 2D light-filed measurements. This VCD-enabled Fourier light-filed imaging approach (F-VCD), achieves video-rate (50 volumes per second) 3D imaging of intracellular dynamics at a high spatiotemporal resolution of ~180 nm × 180 nm × 400 nm and strong noise-resistant capability, with which light field images with a signal-to-noise ratio (SNR) down to -1.62 dB could be well reconstructed. With this approach, we successfully demonstrate the 4D imaging of intracellular organelle dynamics, e.g., mitochondria fission and fusion, with ~5000 times of observation.

Deep-learning on-chip light-sheet microscopy enabling video-rate volumetric imaging of dynamic biological specimens.

Volumetric imaging of dynamic signals in a large, moving, and light-scattering specimen is extremely challenging, owing to the requirement on high spatiotemporal resolution and difficulty in obtaining high-contrast signals. Here we report that through combining a microfluidic chip-enabled digital scanning light-sheet illumination strategy with deep-learning based image restoration, we can realize isotropic 3D imaging of a whole crawling Drosophila larva on an ordinary inverted microscope at a single-cell resolution and a high volumetric imaging rate up to 20 Hz. Enabled with high performances even unmet by current standard light-sheet fluorescence microscopes, we in toto record the neural activities during the forward and backward crawling of a 1st instar larva, and successfully correlate the calcium spiking of motor neurons with the locomotion patterns.