Image stitching algorithm for super-resolution localization microscopy combined with fluorescence noise prior.

Super-resolution panoramic pathological imaging provides a powerful tool for biologists to observe the ultrastructure of samples. Localization data can maintain the essential ultrastructural information of biological samples with a small storage space, and also provides a new opportunity for stitching super-resolution images. However, the existing image stitching methods based on localization data cannot accurately calculate the registration offset of sample regions with no or few structural points and thus lead to registration errors. Here, we proposed a stitching framework called PNanoStitcher. The framework fully utilizes the distribution characteristics of the background fluorescence noise in the stitching region and solves the stitching failure in sample regions with no or few structural points. We verified our method using both simulated and experimental datasets, and compared it with existing stitching methods. PNanoStitcher achieved superior stitching results on biological samples with no structural and few structural regions. The study provides an important driving force for the development of super-resolution digital pathology.

Real-time data processing in colorimetry camera-based single-molecule localization microscopy via CPU-GPU-FPGA heterogeneous computation.

Because conventional low-light cameras used in single-molecule localization microscopy (SMLM) do not have the ability to distinguish colors, it is often necessary to employ a dedicated optical system and/or a complicated image analysis procedure to realize multi-color SMLM. Recently, researchers explored the potential of a new kind of low-light camera called colorimetry camera as an alternative detector in multi-color SMLM, and achieved two-color SMLM under a simple optical system, with a comparable cross-talk to the best reported values. However, extracting images from all color channels is a necessary but lengthy process in colorimetry camera-based SMLM (called CC-STORM), because this process requires the sequential traversal of a massive number of pixels. By taking advantage of the parallelism and pipeline characteristics of FPGA, in this paper, we report an updated multi-color SMLM method called HCC-STORM, which integrated the data processing tasks in CC-STORM into a home-built CPU-GPU-FPGA heterogeneous computing platform. We show that, without scarifying the original performance of CC-STORM, the execution speed of HCC-STORM was increased by approximately three times. Actually, in HCC-STORM, the total data processing time for each raw image with 1024 × 1024 pixels was 26.9 ms. This improvement enabled real-time data processing for a field of view of 1024 × 1024 pixels and an exposure time of 30 ms (a typical exposure time in CC-STORM). Furthermore, to reduce the difficulty of deploying algorithms into the heterogeneous computing platform, we also report the necessary interfaces for four commonly used high-level programming languages, including C/C++, Python, Java, and Matlab. This study not only pushes forward the mature of CC-STORM, but also presents a powerful computing platform for tasks with heavy computation load.

Robust autofocus method based on patterned active illumination and image cross-correlation analysis.

For the effectiveness of a computer-aided diagnosis system, the quality of whole-slide image (WSI) is the foundation, and a useful autofocus method is an important part of ensuring the quality of WSI. The existing autofocus methods need to balance focusing speed and focusing accuracy, and need to be optimized separately for different samples or scenes. In this paper, a robust autofocus method based on fiber bundle illumination and image normalization analysis is proposed. For various application scenes, it meets the requirements of autofocusing through active illumination, such as bright field imaging and fluorescence imaging. For different structures on samples, it ensures the autofocusing accuracy through image analysis. The experimental results imply that the autofocusing method in this paper can effectively track the change of the distance from the sample to the focal plane and significantly improve the WSI quality.

Deep learning using a residual deconvolutional network enables real-time high-density single-molecule localization microscopy.

High-density localization based on deep learning is a very effective method to accelerate single molecule localization microscopy (SMLM). Compared with traditional high-density localization methods, deep learning-based methods enable a faster data processing speed and a higher localization accuracy. However, the reported high-density localization methods based on deep learning are still not fast enough to enable real time data processing for large batches of raw images, which is probably due to the heavy computational burden and computation complexity in the U-shape architecture used in these models. Here we propose a high-density localization method called FID-STORM, which is based on an improved residual deconvolutional network for the real-time processing of raw images. In FID-STORM, we use a residual network to extract the features directly from low-resolution raw images rather than the U-shape network from interpolated images. We also use a model fusion from TensorRT to further accelerate the inference of the model. In addition, we process the sum of the localization images directly on GPU to obtain an additional speed gain. Using simulated and experimental data, we verified that the FID-STORM method achieves a processing speed of 7.31 ms/frame at 256 × 256 pixels @ Nvidia RTX 2080 Ti graphic card, which is shorter than the typical exposure time of 10∼30 ms, thus enabling real-time data processing in high-density SMLM. Moreover, compared with a popular interpolated image-based method called Deep-STORM, FID-STORM enables a speed gain of ∼26 times, without loss of reconstruction accuracy. We also provided an ImageJ plugin for our new method.

Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging.

Single-molecule localization microscopy in a typical wide-field setup has been widely used for investigating subcellular structures with super resolution; however, field-dependent aberrations restrict the field of view (FOV) to only tens of micrometers. Here, we present a deep-learning method for precise localization of spatially variant point emitters (FD-DeepLoc) over a large FOV covering the full chip of a modern sCMOS camera. Using a graphic processing unit-based vectorial point spread function (PSF) fitter, we can fast and accurately model the spatially variant PSF of a high numerical aperture objective in the entire FOV. Combined with deformable mirror-based optimal PSF engineering, we demonstrate high-accuracy three-dimensional single-molecule localization microscopy over a volume of ~180 × 180 × 5 µm3, allowing us to image mitochondria and nuclear pore complexes in entire cells in a single imaging cycle without hardware scanning; a 100-fold increase in throughput compared to the state of the art.

Real-time image resolution measurement for single molecule localization microscopy.

Recent advancements in single molecule localization microscopy (SMLM) have demonstrated outstanding potential applications in high-throughput and high-content screening imaging. One major limitation to such applications is to find a way to optimize imaging throughput without scarifying image quality, especially the homogeneity in image resolution, during the imaging of hundreds of field-of-views (FOVs) in heterogeneous samples. Here we introduce a real-time image resolution measurement method for SMLM to solve this problem. This method is under the heuristic framework of overall image resolution that counts on localization precision and localization density. Rather than estimating the mean localization density after completing the entire SMLM process, this method uses the spatial Poisson process to model the random activation of molecules and thus determines the localization density in real-time. We demonstrate that the method is valid in real-time resolution measurement and is effective in guaranteeing homogeneous image resolution across multiple representative FOVs with optimized imaging throughput.

ResNet-based image inpainting method for enhancing the imaging speed of single molecule localization microscopy.

Single molecule localization microscopy (SMLM) is a mainstream method in the field of super-resolution fluorescence microscopy that can achieve a spatial resolution of 20∼30 nm through a simple optical system. SMLM usually requires thousands of raw images to reconstruct a super-resolution image, and thus suffers from a slow imaging speed. Recently, several methods based on image inpainting have been developed to enhance the imaging speed of SMLM. However, these image inpainting methods may also produce erroneous local features (or called image artifacts), for example, incorrectly joined or split filaments. In this study, we use the ResNet generator, a network with strong local feature extraction capability, to replace the popularly-used U-Net generator to minimize the image artifact problem in current image inpainting methods, and develop an image inpainting method called DI-STORM. We validate our method using both simulated and experimental data, and demonstrate that DI-STORM has the best acceleration capability and produces the least artifacts in the repaired images, as compared with VDSR (the simplest CNN-based image inpainting method in SMLM) and ANNA-PALM (the best GAN-based image inpainting method in SMLM). We believe that DI-STORM could facilitate the application of deep learning-based image inpainting methods for SMLM.

Pushing the colorimetry camera-based fluorescence microscopy to low light imaging by denoising and dye combination.

Colorimetry camera-based fluorescence microscopy (CCFM) is a single-frame imaging method for observing multiple biological events simultaneously. Compared with the traditional multi-color fluorescence microscopy methods based on sequential excitation or spectral splitting, the CCFM method simplifies multi-color fluorescence imaging experiments, while keeping a high spatial resolution. However, when the level of the detected fluorescence signal decreases, the image quality, the demosaicking algorithm precision, and the discrimination of fluorescence channels on the colorimetry camera will also decrease. Thus, CCFM has a poor color resolution under a low signal level. For example, the crosstalk will be higher than 10% when the signal is less than 100 photons/pixel. To solve this problem, we developed a new algorithm that combines sCMOS noise correction with demosaicking, and a dye selection method based on the spectral response characteristics of the colorimetry camera. By combining the above two strategies, low crosstalk can be obtained with 4 ∼ 6 fold fewer fluorescence photons, and low light single-frame four-color fluorescence imaging was successfully performed on fixed cos-7 cells. This study expands the power of the CCFM method, and provides a simple and efficient way for various bioimaging applications in low-light conditions.

Influence of drift correction precision on super-resolution localization microscopy.

Super-resolution localization microscopy (SRLM) breaks the diffraction limit successfully and improves the resolution of optical imaging systems by nearly an order of magnitude. However, SRLM typically takes several minutes or longer to collect a sufficient number of image frames that are required for reconstructing a final super-resolution image. During this long image acquisition period, system drift should be tightly controlled to ensure the imaging quality; thus, several drift correction methods have been developed. However, it is still unclear whether the performance of these methods is able to ensure sufficient image quality in SRLM. Without a clear answer to this question, it is hard to choose a suitable drift correction method for a specific SRLM experiment. In this paper, we use both theoretical analysis and simulation to investigate the relationship among drift correction precision, localization precision, and position estimation precision. We propose a concept of relative localization precision for evaluating the effect of drift correction on imaging resolution, which would help to select an appropriate drift correction method for a specific experiment.

LuckyProfiler: an ImageJ plug-in capable of quantifying FWHM resolution easily and effectively for super-resolution images.

Quantifying the resolution of a super-resolution image is vital for biologists trying to apply super-resolution microscopy in various research fields. Among the reported image resolution estimation methods, the one that calculates the full width at half maximum (FWHM) of line profile, called FWHM resolution, continues the traditional resolution criteria and has been popularly used by many researchers. However, quantifying the FWHM resolution of a super-resolution image is a time-consuming, labor-intensive, and error-prone process because this method typically involves a manual and careful selection of one or several of the smallest structures. In this paper, we investigate the influencing factors in FWHM resolution quantification systematically and present an ImageJ plug-in called LuckyProfiler for biologists so that they can have an easy and effective way of quantifying the FWHM resolution of super-resolution images.

PCIe-based FPGA-GPU heterogeneous computation for real-time multi-emitter fitting in super-resolution localization microscopy.

Real-time multi-emitter fitting is a key technology for advancing super-resolution localization microscopy (SRLM), especially when it is necessary to achieve dynamic imaging quality control and/or optimization of experimental conditions. However, with the increase of activation densities, the requirements in the computing resources would increase rapidly due to the complexity of the fitting algorithms, making it difficult to realize real-time multi-emitter fitting for emitter density more than 0.6 mol/µm2 in large field of view (FOV), even after acceleration with the popular Graphics Processing Unit (GPU) computation. Here we adopt the task parallelism strategy in computer science to construct a Peripheral Component Interconnect Express (PCIe) based all-in-one heterogeneous computing platform (AIO-HCP), where the data between two major parallel computing hardware, Field Programmable Gate Array (FPGA) and GPU, are interacted directly and executed simultaneously. Using simulated and experimental data, we verify that AIO-HCP could achieve a data throughput of up to ∼ 1.561 GB/s between FPGA and GPU. With this new platform, we develop a multi-emitter fitting method, called AIO-STORM, under big data stream parallel scheduling. We show that AIO-STORM is capable of providing real-time image processing on raw images with 100 µm × 100 µm FOV, 10 ms exposure time and 5.5 mol/µm2 structure density, without scarifying image quality. This study overcomes the data throughput limitation of heterogeneous devices, demonstrates the power of the PCIe-based heterogeneous computation platform, and offers opportunities for multi-scale stitching of super-resolution images.

Single-shot multi-color fluorescence microscopy via a colorimetry camera.

Multi-color fluorescence microscopy presents highly detailed biological samples interactively. However, current multi-color methods suffer from an intricate optical setup, complicated image analysis, or a long acquisition time. To address these issues, here we develop a simple multi-color method based on a customized colorimetry camera to enable the detection of multiple structures from single-shot acquisition. The unfiltered channel (W pixels) and color channels (R, G, B, and NIR pixels) in this customized camera simultaneously provide a broad detection wavelength range and high detection sensitivity. We built a simple optical setup by replacing the monochrome camera in a basic fluorescence microscopy system with a colorimetry camera, and developed effective image analysis procedures to reconstruct a multi-color image from a single frame of a raw image. We demonstrated single-shot four-color wide-field fluorescence imaging on fixed cos-7 cells with < 5% cross talk, which is comparable to the best reported values. Our method greatly simplifies both the optical system and image analysis in the widely used method of multi-color fluorescence microscopy, thus offering an effective and easy way to study multiple objects at the same time.

A labeling strategy with effective preservation of fluorophores for expansion single-molecule localization microscopy (Ex-SMLM).

Expansion microscopy (ExM) significantly improves the resolution of conventional diffraction-limited optical microscopy by using physically expanding biological samples. Combining ExM with single-molecule localization microscopy (SMLM) could further enhance the resolving power of SMLM, which is typically in the order of 20-30 nm. However, to make this combination successful, we need to solve three key issues related to sample preparation, including mainly hydrogel shrinking in an ionic photoswitching buffer, fluorescence photobleaching due to a free-radical reaction and reduced labelling efficiency from protease digestion. Re-embedding polyacrylamide gel or using an improved photoswitching buffer with a low ionic strength is able to minimize or even solve the hydrogel shrinking problem, while the development of post-expansion labelling approaches avoids fluorescence bleaching. However, the preservation of protein epitopes (which determines the labelling efficiency) remains to be challenging. In this paper, we propose to tackle this challenge by introducing the highly selective and stable biotin-streptavidin interaction into the post-expansion labelling strategy. After upgrading the popular immunolabelling linkage scheme from Epitope-Primary antibody-Secondary antibody-Fluorophores to Epitope-Primary antibody-Secondary antibody-Biotin-Streptavidin-Fluorophores, we were able to label protein epitopes with biotin, which was stable during the expansion process, and thus avoid the troublesome problem in preserving protein epitopes or antibodies. We demonstrate that combining Ex-SMLM with the new post-expansion linkage scheme enables new possibilities in resolving the detailed arrangement of Nup133 proteins in the nuclear pore complex, which helps researchers to observe a clearer structure. This study provides new opportunities for studying the ultrastructural details of subcellular organelles or even biomacromolecules, using the conventional SMLM system.

Accelerating multi-emitter localization in super-resolution localization microscopy with FPGA-GPU cooperative computation.

The real-time multi-emitter localization method is essential for advancing high-throughput super-resolution localization microscopy (HT-SRLM). In the past decade, the graphics processing unit (GPU) computation has been dominantly used to accelerate the execution speed of the multi-emitter localization method. However, if HT-SRLM is combined with a scientific complementary metal-oxide-semiconductor (sCMOS) camera working at full frame rate, real-time image processing is still difficult to achieve using this acceleration approach, thus resulting in a massive data storage challenge and even system crash. Here we take advantage of the cooperative acceleration power of field programming gate array (FPGA) computation and GPU computation, and propose a method called HCP-STORM to enable real-time multi-emitter localization. Using simulated images, we verified that HCP-STORM is capable of providing real-time image processing for raw images from a representative Hamamatsu Flash 4 V3 sCMOS camera working at full frame rate (that is, 2048×2048 pixels @ 10 ms exposure time). Using experimental images, we prove that HCP-STORM is 25 times faster than QC-STORM and 295 times faster than ThunderSTORM, with a small but acceptable degradation in image quality. This study shows the potential of FPGA-GPU cooperative computation in accelerating multi-emitter localization, and pushes a significant step toward the maturity of HT-SRLM technology.

High-precision 3D drift correction with differential phase contrast images.

Single molecule localization microscopy (SMLM) usually requires long image acquisition time at the order of minutes and thus suffers from sample drift, which deteriorates image quality. A drift estimation method with high precision is typically used in SMLM, which can be further combined with a drift compensation device to enable active microscope stabilization. Among all the reported methods, the drift estimation method based on bright-field image correlation requires no extra sample preparation or complicated modification to the imaging setup. However, the performance of this method is limited by the contrast of bright-field images, especially for the structures without sufficient features. In this paper, we proposed to use differential phase contrast (DPC) microscopy to enhance the image contrast and presented a 3D drift correction method with higher precision and robustness. This DPC-based drift correction method is suitable even for biological samples without clear morphological features. We demonstrated that this method can achieve a correction precision of < 6 nm in both the lateral direction and axial direction. Using SMLM imaging of microtubules, we verified that this method provides a comparable drift estimation performance as redundant cross-correlation.

Two-color super-resolution localization microscopy via joint encoding of emitter location and color.

Multi-color super-resolution localization microscopy (SRLM) provides great opportunities for studying the structural and functional details of biological samples. However, current multi-color SRLM methods either suffer from medium to high crosstalk, or require a dedicated optical system and a complicated image analysis procedure. To address these problems, here we propose a completely different method to realize multi-color SRLM. This method is built upon a customized RGBW camera with a repeated pattern of filtered (Red, Green, Blue and Near-infrared) and unfiltered (White) pixels. With a new insight that RGBW camera is advantageous for color recognition instead of color reproduction, we developed a joint encoding scheme of emitter location and color. By combing this RGBW camera with the joint encoding scheme and a simple optical set-up, we demonstrated two-color SRLM with ∼20 nm resolution and < 2% crosstalk (which is comparable to the best-reported values). This study significantly reduces the complexity of two-color SRLM (and potentially multi-color SRLM), and thus offers good opportunities for general biomedical research laboratories to use multi-color SRLM, which is currently mastered only by well-trained researchers.

Computational framework for generating large panoramic super-resolution images from localization microscopy.

Combining super-resolution localization microscopy with pathology creates new opportunities for biomedical researches. This combination requires a suitable image mosaic method for generating a panoramic image from many overlapping super-resolution images. However, current image mosaic methods are not suitable for this purpose. Here we proposed a computational framework and developed an image mosaic method called NanoStitcher. We generated ground truth datasets and defined criteria to evaluate this computational framework. We used both simulated and experimental datasets to prove that NanoStitcher exhibits better performance than two representative image mosaic methods. This study is helpful for the mature of super-resolution digital pathology.

Characterizing and correcting camera noise in back-illuminated sCMOS cameras.

With promising properties of fast imaging speed, large field-of-view, relative low cost and many others, back-illuminated sCMOS cameras have been receiving intensive attention for low light level imaging in the past several years. However, due to the pixel-to-pixel difference of camera noise (called noise non-uniformity) in sCMOS cameras, researchers may hesitate to use them in some application fields, and sometimes wonder whether they should optimize the noise non-uniformity of their sCMOS cameras before using them in a specific application scenario. In this paper, we systematically characterize the impact of different types of sCMOS noise on image quality and perform corrections to these types of sCMOS noise using three representative algorithms (PURE, NCS and MLEsCMOS). We verify that it is possible to use appropriate correction methods to push the non-uniformity of major types of camera noise, including readout noise, offset, and photon response, to a satisfactory level for conventional microscopy and single molecule localization microscopy. We further find out that, after these corrections, global read noise becomes a major concern that limits the imaging performance of back-illuminated sCMOS cameras. We believe this study provides new insights into the understanding of camera noise in back-illuminated sCMOS cameras, and also provides useful information for future development of this promising camera technology.

Localization of the WD repeat-containing protein 5 to the Virion Assembly Compartment Facilitates Human Cytomegalovirus Assembly.

We previously reported that human cytomegalovirus (HCMV) utilizes the cellular protein WD repeat-containing protein 5 (WDR5) to facilitate capsid nuclear egress. Here, we further show that HCMV infection results in WDR5 localization in a juxtanuclear region, and that its localization to this cellular site is associated with viral replication and late viral gene expression. Furthermore, WDR5 accumulated in the virion assembly compartment (vAC) and co-localized with vAC markers of gamma-tubulin (γ-tubulin), early endosomes, and viral vAC marker proteins pp65, pp28, and glycoprotein B (gB). WDR5 co-immunoprecipitated with multiple virion proteins, including MCP, pp150, pp65, pIRS1, and pTRS1, which may explain WDR5 accumulation in the vAC during infection. WDR5 fractionated with virions either in the presence or absence of Triton X-100 and was present in purified viral particles, suggesting that WDR5 was incorporated into HCMV virions. Thus, WDR5 localized to the vAC and was incorporated into virions, raising the possibility that in addition to capsid nuclear egress, WDR5 could also participate in cytoplasmic HCMV virion morphogenesis.Importance Human cytomegalovirus (HCMV) has a large (∼235-kb) genome that contains over 170 ORFs and exploits numerous cellular factors to facilitate its replication. In the late phase of HCMV infection cytoplasmic membranes are reorganized to establish the virion assembly compartment (vAC), which has been shown to necessary for efficient assembly of progeny virions. We previously reported that WDR5 facilitates HCMV nuclear egress. Here, we show that WDR5 is localized to the vAC and incorporated into virions, perhaps contributing to efficient virion maturation. Thus, findings in this study identified a potential role for WDR5 in HCMV assembly in the cytoplasmic phase of virion morphogenesis.

Rethinking resolution estimation in fluorescence microscopy: from theoretical resolution criteria to super-resolution microscopy.

Resolution is undoubtedly the most important parameter in optical microscopy by providing an estimation on the maximum resolving power of a certain optical microscope. For centuries, the resolution of an optical microscope is generally considered to be limited only by the numerical aperture of the optical system and the wavelength of light. However, since the invention and popularity of various advanced fluorescence microscopy techniques, especially super-resolution fluorescence microscopy, many new methods have been proposed for estimating the resolution, leading to confusions for researchers who need to quantify the resolution of their fluorescence microscopes. In this paper, we firstly summarize the early concepts and criteria for predicting the resolution limit of an ideal optical system. Then, we discuss some important influence factors that deteriorate the resolution of a certain fluorescence microscope. Finally, we provide methods and examples on how to measure the resolution of a fluorescence microscope from captured fluorescence images. This paper aims to answer as best as possible the theoretical and practical issues regarding the resolution estimation in fluorescence microscopy.