Deep learning permits imaging of multiple structures with the same fluorophores.

Fluorescence microscopy, which employs fluorescent tags to label and observe cellular structures and their dynamics, is a powerful tool for life sciences. However, due to the spectral overlap between different dyes, a limited number of structures can be separately labeled and imaged for live-cell applications. In addition, the conventional sequential channel imaging procedure is quite time consuming, as it needs to switch either different lasers or filters. Here, we propose a novel double-structure network (DBSN) that consists of multiple connected models, which can extract six distinct subcellular structures from three raw images with only two separate fluorescent labels. DBSN combines the intensity-balance model to compensate for uneven fluorescent labels for different structures and the structure-separation model to extract multiple different structures with the same fluorescent labels. Therefore, DBSN breaks the bottleneck of the existing technologies and holds immense potential applications in the field of cell biology.

A deep learning based method for automatic analysis of high-throughput droplet digital PCR images.

Droplet digital PCR (ddPCR) is a technique for absolute quantification of nucleic acid molecules and is widely used in biomedical research and clinical diagnosis. ddPCR partitions the reaction solution containing target molecules into a large number of independent microdroplets for amplification and performs quantitative analysis of target molecules by calculating the proportion of positive droplets by the principle of Poisson distribution. Accurate recognition of positive droplets in ddPCR images is of great importance to guarantee the accuracy of target nucleic acid quantitative analysis. However, hand-designed operators are sensitive to interference and have disadvantages such as low contrast, uneven illumination, low sample copy number, and noise, and their accuracy and robustness still need to be improved. Herein, we developed a deep learning-based high-throughput ddPCR droplet detection framework for robust and accurate ddPCR image analysis, and the experimental results show that our method achieves excellent performance in the recognition of positive droplets (99.71%) within a limited time. By combining the Hough transform and a convolutional neural network (CNN), our novel method can automatically filter out invalid droplets that are difficult to be identified by local or global encoding methods and realize high-precision localization and classification of droplets in ddPCR images under variable exposure, contrast, and uneven illumination conditions without the need for image pre-processing and normalization processes.

Three-dimensional super-resolved live cell imaging through polarized multi-angle TIRF.

Measuring three-dimensional nanoscale cellular structures is challenging, especially when the structure is dynamic. Owing to the informative total internal reflection fluorescence (TIRF) imaging under varied illumination angles, multi-angle (MA) TIRF has been examined to offer a nanoscale axial and a subsecond temporal resolution. However, conventional MA-TIRF still performs badly in lateral resolution and fails to characterize the depth image in densely distributed regions. Here, we emphasize the lateral super-resolution in the MA-TIRF, exampled by simply introducing polarization modulation into the illumination procedure. Equipped with a sparsity and accelerated proximal algorithm, we examine a more precise 3D sample structure compared with previous methods, enabling live cell imaging with a temporal resolution of 2 s and recovering high-resolution mitochondria fission and fusion processes. We also shared the recovery program, which is the first open-source recovery code for MA-TIRF, to the best of our knowledge.

Prior Visual-guided Self-supervised Learning Enables Color Vignetting Correction for High-throughput Microscopic Imaging.

Vignetting constitutes a prevalent optical degradation that significantly compromises the quality of biomedical microscopic imaging. However, a robust and efficient vignetting correction methodology in multi-channel microscopic images remains absent at present. In this paper, we take advantage of a prior knowledge about the homogeneity of microscopic images and radial attenuation property of vignetting to develop a self-supervised deep learning algorithm that achieves complex vignetting removal in color microscopic images. Our proposed method, vignetting correction lookup table (VCLUT), is trainable on both single and multiple images, which employs adversarial learning to effectively transfer good imaging conditions from the user visually defined central region of its own light field to the entire image. To illustrate its effectiveness, we performed individual correction experiments on data from five distinct biological specimens. The results demonstrate that VCLUT exhibits enhanced performance compared to classical methods. We further examined its performance as a multi-image-based approach on a pathological dataset, revealing its advantage over other stateof-the-art approaches in both qualitative and quantitative measurements. Moreover, it uniquely possesses the capacity for generalization across various levels of vignetting intensity and an ultra-fast model computation capability, rendering it well-suited for integration into high-throughput imaging pipelines of digital microscopy.

Optogenetic Strategies for Optimizing the Performance of Phospholipids Biosensors.

High-performance biosensors play a crucial role in elucidating the intricate spatiotemporal regulatory roles and dynamics of membrane phospholipids. However, enhancing the sensitivity and imaging performance remains a significant challenge. Here, optogenetic-based strategies are presented to optimize phospholipid biosensors. These strategies involves presequestering unbound biosensors in the cell nucleus and regulating their cytosolic levels with blue light to minimize background signal interference in phospholipid detection, particularly under conditions of high expression levels of biosensor. Furthermore, optically controlled phase separation and the SunTag system are employed to generate punctate probes for substrate detection, thereby amplifying biosensor signals and enhancing visualization of the detection process. These improved phospholipid biosensors hold great potential for enhancing the understanding of the spatiotemporal dynamics and regulatory roles of membrane lipids in live cells and the methodological insights in this study might be valuable for developing other high-performance biosensors.

Boosting microscopic object detection via feature activation map guided poisson blending.

Microscopic examination of visible components based on micrographs is the gold standard for testing in biomedical research and clinical diagnosis. The application of object detection technology in bioimages not only improves the efficiency of the analyst but also provides decision support to ensure the objectivity and consistency of diagnosis. However, the lack of large annotated datasets is a significant impediment in rapidly deploying object detection models for microscopic formed elements detection. Standard augmentation methods used in object detection are not appropriate because they are prone to destroy the original micro-morphological information to produce counterintuitive micrographs, which is not conducive to build the trust of analysts in the intelligent system. Here, we propose a feature activation map-guided boosting mechanism dedicated to microscopic object detection to improve data efficiency. Our results show that the boosting mechanism provides solid gains in the object detection model deployed for microscopic formed elements detection. After image augmentation, the mean Average Precision (mAP) of baseline and strong baseline of the Chinese herbal medicine micrograph dataset are increased by 16.3% and 5.8% respectively. Similarly, on the urine sediment dataset, the boosting mechanism resulted in an improvement of 8.0% and 2.6% in mAP of the baseline and strong baseline maps respectively. Moreover, the method shows strong generalizability and can be easily integrated into any main-stream object detection model. The performance enhancement is interpretable, making it more suitable for microscopic biomedical applications.

Deep learning-enabled fast DNA-PAINT imaging in cells.

DNA-based point accumulation in nanoscale topography (DNA-PAINT) is a well-established technique for single-molecule localization microscopy (SMLM), enabling resolution of up to a few nanometers. Traditionally, DNA-PAINT involves the utilization of tens of thousands of single-molecule fluorescent images to generate a single super-resolution image. This process can be time-consuming, which makes it unfeasible for many researchers. Here, we propose a simplified DNA-PAINT labeling method and a deep learning-enabled fast DNA-PAINT imaging strategy for subcellular structures, such as microtubules. By employing our method, super-resolution reconstruction can be achieved with only one-tenth of the raw data previously needed, along with the option of acquiring the widefield image. As a result, DNA-PAINT imaging is significantly accelerated, making it more accessible to a wider range of biological researchers.

Augmented Super-Resolution Radial Fluctuations (aSRRF) Pushing the Limits of Structured Illumination Microscopy.

Structured illumination microscopy (SIM) is a versatile super-resolution technique known for its compatibility with a wide range of probes and fast implementation. While 3D SIM is capable of achieving a spatial resolution of ∼120 nm laterally and ∼300 nm axially, attempting to further enhance the resolution through methods such as nonlinear SIM or 4-beam SIM introduces complexities in optical configurations, increased phototoxicity, and reduced temporal resolution. Here, we have developed a novel method that combines SIM with augmented super-resolution radial fluctuations (aSRRF) utilizing a single image through image augmentation. By applying aSRRF reconstruction to SIM images, we can enhance the SIM resolution to ∼50 nm isotopically, without requiring any modifications to the optical system or sample acquisition process. Additionaly, we have incorporated the aSRRF approach into an ImageJ plugin and demonstrated its versatility across various fluorescence microscopy images, showcasing a remarkable two-fold resolution increase.

Imaging Single-Vesicle Exocytosis with Total Internal Reflection Fluorescence Microscopy (TIRFM).

Total internal reflection fluorescence microscopy (TIRFM) provides extremely thin optical sectioning with excellent signal-to-noise ratios, which allows for visualization of membrane dynamics at the cell surface with superb spatiotemporal resolution. In this chapter, TIRFM is used to record and analyze exocytosis of single glucose transporter-4 (GLUT4) containing vesicles in 3T3-L1 adipocytes.

DNA-PAINT Imaging Accelerated by Machine Learning.

DNA point accumulation in nanoscale topography (DNA-PAINT) is an easy-to-implement approach for localization-based super-resolution imaging. Conventional DNA-PAINT imaging typically requires tens of thousands of frames of raw data to reconstruct one super-resolution image, which prevents its potential application for live imaging. Here, we introduce a new DNA-PAINT labeling method that allows for imaging of microtubules with both DNA-PAINT and widefield illumination. We develop a U-Net-based neural network, namely, U-PAINT to accelerate DNA-PAINT imaging from a widefield fluorescent image and a sparse single-molecule localization image. Compared with the conventional method, U-PAINT only requires one-tenth of the original raw data, which permits fast imaging and reconstruction of super-resolution microtubules and can be adopted to analyze other SMLM datasets. We anticipate that this machine learning method enables faster and even live-cell DNA-PAINT imaging in the future.

A survey on applications of deep learning in microscopy image analysis.

Advanced microscopy enables us to acquire quantities of time-lapse images to visualize the dynamic characteristics of tissues, cells or molecules. Microscopy images typically vary in signal-to-noise ratios and include a wealth of information which require multiple parameters and time-consuming iterative algorithms for processing. Precise analysis and statistical quantification are often needed for the understanding of the biological mechanisms underlying these dynamic image sequences, which has become a big challenge in the field. As deep learning technologies develop quickly, they have been applied in bioimage processing more and more frequently. Novel deep learning models based on convolution neural networks have been developed and illustrated to achieve inspiring outcomes. This review article introduces the applications of deep learning algorithms in microscopy image analysis, which include image classification, region segmentation, object tracking and super-resolution reconstruction. We also discuss the drawbacks of existing deep learning-based methods, especially on the challenges of training datasets acquisition and evaluation, and propose the potential solutions. Furthermore, the latest development of augmented intelligent microscopy that based on deep learning technology may lead to revolution in biomedical research.

Deep learning enables structured illumination microscopy with low light levels and enhanced speed.

Structured illumination microscopy (SIM) surpasses the optical diffraction limit and offers a two-fold enhancement in resolution over diffraction limited microscopy. However, it requires both intense illumination and multiple acquisitions to produce a single high-resolution image. Using deep learning to augment SIM, we obtain a five-fold reduction in the number of raw images required for super-resolution SIM, and generate images under extreme low light conditions (at least 100× fewer photons). We validate the performance of deep neural networks on different cellular structures and achieve multi-color, live-cell super-resolution imaging with greatly reduced photobleaching.

Development of fan-shaped tracker for single particle tracking.

With the development of super-resolution fluorescence microscopy, complex dynamic processes in living cells can be observed and recorded with unprecedented temporal and spatial resolution. Single particle tracking (SPT) is the most important step to explore the relationship between the spatio-temporal dynamics of subcellular molecules and their functions. Although previous studies have developed SPT algorithms to quantitatively analyze particle dynamics in cell, traditional tracking methods have poor performance when dealing with intersecting trajectories. This can be attributed to two main reasons: (a) they do not have point compensation process for overlapping objects; (b) they use inefficient motion prediction models. In this paper, we present a novel fan-shaped tracker (FsT) algorithm to reconstruct the trajectories of subcellular vesicles in living cells. We proposed a customized point compensation method for overlapping objects based on the fan-shaped motion trend of the particles. Furthermore, we validated the performance of the FsT in both simulated time-lapse movies with variable imaging quality and in real vesicle moving images. Meanwhile, we compared the performance of FsT with other five state-of-the-art tracking algorithms by using commonly defined measures. The results showed that our FsT achieves better performance in high signal-to-noise ratio conditions and in tracking of overlapping objects. We anticipate that our FsT method will have vast applications in tracking of moving objects in cell.

Optogenetic control of epithelial-mesenchymal transition in cancer cells.

Epithelial-mesenchymal transition (EMT) is one of the most important mechanisms in the initiation and promotion of cancer cell metastasis. The phosphoinositide 3-kinase (PI3K) signaling pathway has been demonstrated to be involved in TGF-ß induced EMT, but the complicated TGF-ß signaling network makes it challenging to dissect the important role of PI3K on regulation of EMT process. Here, we applied optogenetic controlled PI3K module (named 'Opto-PI3K'), which based on CRY2 and the N-terminal of CIB1 (CIBN), to rapidly and reversibly control the endogenous PI3K activity in cancer cells with light. By precisely modulating the kinetics of PI3K activation, we found that E-cadherin is an important downstream target of PI3K signaling. Compared with TGF-ß treatment, Opto-PI3K had more potent effect in down-regulation of E-cadherin expression, which was demonstrated to be regulated in a light dose-dependent manner. Surprisingly, sustained PI3K activation induced partial EMT state in A549 cells that is highly reversible. Furthermore, we demonstrated that Opto-PI3K only partially mimicked TGF-ß effects on promotion of cell migration in vitro. These results reveal the importance of PI3K signaling in TGF-ß induced EMT, suggesting other TGF-ß regulated signaling pathways are necessary for the full and irreversible promotion of EMT in cancer cells. In addition, our study implicates the great promise of optogenetics in cancer research for mapping input-output relationships in oncogenic pathways.

Multi-color live-cell super-resolution volume imaging with multi-angle interference microscopy.

Imaging and tracking of near-surface three-dimensional volumetric nanoscale dynamic processes of live cells remains a challenging problem. In this paper, we propose a multi-color live-cell near-surface-volume super-resolution microscopy method that combines total internal reflection fluorescence structured illumination microscopy with multi-angle evanescent light illumination. We demonstrate that our approach of multi-angle interference microscopy is perfectly adapted to studying subcellular dynamics of mitochondria and microtubule architectures during cell migration.