Real-time image resolution measurement for single molecule localization microscopy.

Recent advancements in single molecule localization microscopy (SMLM) have demonstrated outstanding potential applications in high-throughput and high-content screening imaging. One major limitation to such applications is to find a way to optimize imaging throughput without scarifying image quality, especially the homogeneity in image resolution, during the imaging of hundreds of field-of-views (FOVs) in heterogeneous samples. Here we introduce a real-time image resolution measurement method for SMLM to solve this problem. This method is under the heuristic framework of overall image resolution that counts on localization precision and localization density. Rather than estimating the mean localization density after completing the entire SMLM process, this method uses the spatial Poisson process to model the random activation of molecules and thus determines the localization density in real-time. We demonstrate that the method is valid in real-time resolution measurement and is effective in guaranteeing homogeneous image resolution across multiple representative FOVs with optimized imaging throughput.

Influence of drift correction precision on super-resolution localization microscopy.

Super-resolution localization microscopy (SRLM) breaks the diffraction limit successfully and improves the resolution of optical imaging systems by nearly an order of magnitude. However, SRLM typically takes several minutes or longer to collect a sufficient number of image frames that are required for reconstructing a final super-resolution image. During this long image acquisition period, system drift should be tightly controlled to ensure the imaging quality; thus, several drift correction methods have been developed. However, it is still unclear whether the performance of these methods is able to ensure sufficient image quality in SRLM. Without a clear answer to this question, it is hard to choose a suitable drift correction method for a specific SRLM experiment. In this paper, we use both theoretical analysis and simulation to investigate the relationship among drift correction precision, localization precision, and position estimation precision. We propose a concept of relative localization precision for evaluating the effect of drift correction on imaging resolution, which would help to select an appropriate drift correction method for a specific experiment.

PCIe-based FPGA-GPU heterogeneous computation for real-time multi-emitter fitting in super-resolution localization microscopy.

Real-time multi-emitter fitting is a key technology for advancing super-resolution localization microscopy (SRLM), especially when it is necessary to achieve dynamic imaging quality control and/or optimization of experimental conditions. However, with the increase of activation densities, the requirements in the computing resources would increase rapidly due to the complexity of the fitting algorithms, making it difficult to realize real-time multi-emitter fitting for emitter density more than 0.6 mol/µm2 in large field of view (FOV), even after acceleration with the popular Graphics Processing Unit (GPU) computation. Here we adopt the task parallelism strategy in computer science to construct a Peripheral Component Interconnect Express (PCIe) based all-in-one heterogeneous computing platform (AIO-HCP), where the data between two major parallel computing hardware, Field Programmable Gate Array (FPGA) and GPU, are interacted directly and executed simultaneously. Using simulated and experimental data, we verify that AIO-HCP could achieve a data throughput of up to ∼ 1.561 GB/s between FPGA and GPU. With this new platform, we develop a multi-emitter fitting method, called AIO-STORM, under big data stream parallel scheduling. We show that AIO-STORM is capable of providing real-time image processing on raw images with 100 µm × 100 µm FOV, 10 ms exposure time and 5.5 mol/µm2 structure density, without scarifying image quality. This study overcomes the data throughput limitation of heterogeneous devices, demonstrates the power of the PCIe-based heterogeneous computation platform, and offers opportunities for multi-scale stitching of super-resolution images.

Accelerating multi-emitter localization in super-resolution localization microscopy with FPGA-GPU cooperative computation.

The real-time multi-emitter localization method is essential for advancing high-throughput super-resolution localization microscopy (HT-SRLM). In the past decade, the graphics processing unit (GPU) computation has been dominantly used to accelerate the execution speed of the multi-emitter localization method. However, if HT-SRLM is combined with a scientific complementary metal-oxide-semiconductor (sCMOS) camera working at full frame rate, real-time image processing is still difficult to achieve using this acceleration approach, thus resulting in a massive data storage challenge and even system crash. Here we take advantage of the cooperative acceleration power of field programming gate array (FPGA) computation and GPU computation, and propose a method called HCP-STORM to enable real-time multi-emitter localization. Using simulated images, we verified that HCP-STORM is capable of providing real-time image processing for raw images from a representative Hamamatsu Flash 4 V3 sCMOS camera working at full frame rate (that is, 2048×2048 pixels @ 10 ms exposure time). Using experimental images, we prove that HCP-STORM is 25 times faster than QC-STORM and 295 times faster than ThunderSTORM, with a small but acceptable degradation in image quality. This study shows the potential of FPGA-GPU cooperative computation in accelerating multi-emitter localization, and pushes a significant step toward the maturity of HT-SRLM technology.

High-precision 3D drift correction with differential phase contrast images.

Single molecule localization microscopy (SMLM) usually requires long image acquisition time at the order of minutes and thus suffers from sample drift, which deteriorates image quality. A drift estimation method with high precision is typically used in SMLM, which can be further combined with a drift compensation device to enable active microscope stabilization. Among all the reported methods, the drift estimation method based on bright-field image correlation requires no extra sample preparation or complicated modification to the imaging setup. However, the performance of this method is limited by the contrast of bright-field images, especially for the structures without sufficient features. In this paper, we proposed to use differential phase contrast (DPC) microscopy to enhance the image contrast and presented a 3D drift correction method with higher precision and robustness. This DPC-based drift correction method is suitable even for biological samples without clear morphological features. We demonstrated that this method can achieve a correction precision of < 6 nm in both the lateral direction and axial direction. Using SMLM imaging of microtubules, we verified that this method provides a comparable drift estimation performance as redundant cross-correlation.

Two-color super-resolution localization microscopy via joint encoding of emitter location and color.

Multi-color super-resolution localization microscopy (SRLM) provides great opportunities for studying the structural and functional details of biological samples. However, current multi-color SRLM methods either suffer from medium to high crosstalk, or require a dedicated optical system and a complicated image analysis procedure. To address these problems, here we propose a completely different method to realize multi-color SRLM. This method is built upon a customized RGBW camera with a repeated pattern of filtered (Red, Green, Blue and Near-infrared) and unfiltered (White) pixels. With a new insight that RGBW camera is advantageous for color recognition instead of color reproduction, we developed a joint encoding scheme of emitter location and color. By combing this RGBW camera with the joint encoding scheme and a simple optical set-up, we demonstrated two-color SRLM with ∼20 nm resolution and < 2% crosstalk (which is comparable to the best-reported values). This study significantly reduces the complexity of two-color SRLM (and potentially multi-color SRLM), and thus offers good opportunities for general biomedical research laboratories to use multi-color SRLM, which is currently mastered only by well-trained researchers.

Computational framework for generating large panoramic super-resolution images from localization microscopy.

Combining super-resolution localization microscopy with pathology creates new opportunities for biomedical researches. This combination requires a suitable image mosaic method for generating a panoramic image from many overlapping super-resolution images. However, current image mosaic methods are not suitable for this purpose. Here we proposed a computational framework and developed an image mosaic method called NanoStitcher. We generated ground truth datasets and defined criteria to evaluate this computational framework. We used both simulated and experimental datasets to prove that NanoStitcher exhibits better performance than two representative image mosaic methods. This study is helpful for the mature of super-resolution digital pathology.