Fourier Ptychographic Neural Network Combined with Zernike Aberration Recovery and Wirtinger Flow Optimization.

Fourier ptychographic microscopy, as a computational imaging method, can reconstruct high-resolution images but suffers optical aberration, which affects its imaging quality. For this reason, this paper proposes a network model for simulating the forward imaging process in the Tensorflow framework using samples and coherent transfer functions as the input. The proposed model improves the introduced Wirtinger flow algorithm, retains the central idea, simplifies the calculation process, and optimizes the update through back propagation. In addition, Zernike polynomials are used to accurately estimate aberration. The simulation and experimental results show that this method can effectively improve the accuracy of aberration correction, maintain good correction performance under complex scenes, and reduce the influence of optical aberration on imaging quality.

A Physics-Inspired Deep Learning Framework for an Efficient Fourier Ptychographic Microscopy Reconstruction under Low Overlap Conditions.

Two-dimensional observation of biological samples at hundreds of nanometers resolution or even below is of high interest for many sensitive medical applications. Recent advances have been obtained over the last ten years with computational imaging. Among them, Fourier Ptychographic Microscopy is of particular interest because of its important super-resolution factor. In complement to traditional intensity images, phase images are also produced. A large set of N raw images (with typically N = 225) is, however, required because of the reconstruction process that is involved. In this paper, we address the problem of FPM image reconstruction using a few raw images only (here, N = 37) as is highly desirable to increase microscope throughput. In contrast to previous approaches, we develop an algorithmic approach based on a physics-informed optimization deep neural network and statistical reconstruction learning. We demonstrate its efficiency with the help of simulations. The forward microscope image formation model is explicitly introduced in the deep neural network model to optimize its weights starting from an initialization that is based on statistical learning. The simulation results that are presented demonstrate the conceptual benefits of the approach. We show that high-quality images are effectively reconstructed without any appreciable resolution degradation. The learning step is also shown to be mandatory.

Peripheral Blood Leukocyte Detection Based on an Improved Detection Transformer Algorithm.

The combination of a blood cell analyzer and artificial microscopy to detect white blood cells is used in hospitals. Blood cell analyzers not only have large throughput, but they also cannot detect cell morphology; although artificial microscopy has high accuracy, it is inefficient and prone to missed detections. In view of the above problems, a method based on Fourier ptychographic microscopy (FPM) and deep learning to detect peripheral blood leukocytes is proposed in this paper. Firstly, high-resolution and wide-field microscopic images of human peripheral blood cells are obtained using the FPM system, and the cell image data are enhanced with DCGANs (deep convolution generative adversarial networks) to construct datasets for performance evaluation. Then, an improved DETR (detection transformer) algorithm is proposed to improve the detection accuracy of small white blood cell targets; that is, the residual module Conv Block in the feature extraction part of the DETR network is improved to reduce the problem of information loss caused by downsampling. Finally, CIOU (complete intersection over union) is introduced as the bounding box loss function, which avoids the problem that GIOU (generalized intersection over union) is difficult to optimize when the two boxes are far away and the convergence speed is faster. The experimental results show that the mAP of the improved DETR algorithm in the detection of human peripheral white blood cells is 0.936. In addition, this algorithm is compared with other convolutional neural networks in terms of average accuracy, parameters, and number of inference frames per second, which verifies the feasibility of this method in microscopic medical image detection.

Fourier Ptychographic Microscopic Reconstruction Method Based on Residual Hybrid Attention Network.

Fourier ptychographic microscopy (FPM) is a novel technique for computing microimaging that allows imaging of samples such as pathology sections. However, due to the influence of systematic errors and noise, the quality of reconstructed images using FPM is often poor, and the reconstruction efficiency is low. In this paper, a hybrid attention network that combines spatial attention mechanisms with channel attention mechanisms into FPM reconstruction is introduced. Spatial attention can extract fine spatial features and reduce redundant features while, combined with residual channel attention, it adaptively readjusts the hierarchical features to achieve the conversion of low-resolution complex amplitude images to high-resolution ones. The high-resolution images generated by this method can be applied to medical cell recognition, segmentation, classification, and other related studies, providing a better foundation for relevant research.

Investigating the Joint Amplitude and Phase Imaging of Stained Samples in Automatic Diagnosis.

The diagnosis of many diseases relies, at least on first intention, on an analysis of blood smears acquired with a microscope. However, image quality is often insufficient for the automation of such processing. A promising improvement concerns the acquisition of enriched information on samples. In particular, Quantitative Phase Imaging (QPI) techniques, which allow the digitization of the phase in complement to the intensity, are attracting growing interest. Such imaging allows the exploration of transparent objects not visible in the intensity image using the phase image only. Another direction proposes using stained images to reveal some characteristics of the cells in the intensity image; in this case, the phase information is not exploited. In this paper, we question the interest of using the bi-modal information brought by intensity and phase in a QPI acquisition when the samples are stained. We consider the problem of detecting parasitized red blood cells for diagnosing malaria from stained blood smears using a Deep Neural Network (DNN). Fourier Ptychographic Microscopy (FPM) is used as the computational microscopy framework to produce QPI images. We show that the bi-modal information enhances the detection performance by 4% compared to the intensity image only when the convolution in the DNN is implemented through a complex-based formalism. This proves that the DNN can benefit from the bi-modal enhanced information. We conjecture that these results should extend to other applications processed through QPI acquisition.

Deep Multi-Feature Transfer Network for Fourier Ptychographic Microscopy Imaging Reconstruction.

Fourier ptychographic microscopy (FPM) is a potential imaging technique, which is used to achieve wide field-of-view (FOV), high-resolution and quantitative phase information. The LED array is used to irradiate the samples from different angles to obtain the corresponding low-resolution intensity images. However, the performance of reconstruction still suffers from noise and image data redundancy, which needs to be considered. In this paper, we present a novel Fourier ptychographic microscopy imaging reconstruction method based on a deep multi-feature transfer network, which can achieve good anti-noise performance and realize high-resolution reconstruction with reduced image data. First, in this paper, the image features are deeply extracted through transfer learning ResNet50, Xception and DenseNet121 networks, and utilize the complementarity of deep multiple features and adopt cascaded feature fusion strategy for channel merging to improve the quality of image reconstruction; then the pre-upsampling is used to reconstruct the network to improve the texture details of the high-resolution reconstructed image. We validate the performance of the reported method via both simulation and experiment. The model has good robustness to noise and blurred images. Better reconstruction results are obtained under the conditions of short time and low resolution. We hope that the end-to-end mapping method of neural network can provide a neural-network perspective to solve the FPM reconstruction.

Fourier Ptychographic Microscopy 10 Years on: A Review.

Fourier ptychographic microscopy (FPM) emerged as a prominent imaging technique in 2013, attracting significant interest due to its remarkable features such as precise phase retrieval, expansive field of view (FOV), and superior resolution. Over the past decade, FPM has become an essential tool in microscopy, with applications in metrology, scientific research, biomedicine, and inspection. This achievement arises from its ability to effectively address the persistent challenge of achieving a trade-off between FOV and resolution in imaging systems. It has a wide range of applications, including label-free imaging, drug screening, and digital pathology. In this comprehensive review, we present a concise overview of the fundamental principles of FPM and compare it with similar imaging techniques. In addition, we present a study on achieving colorization of restored photographs and enhancing the speed of FPM. Subsequently, we showcase several FPM applications utilizing the previously described technologies, with a specific focus on digital pathology, drug screening, and three-dimensional imaging. We thoroughly examine the benefits and challenges associated with integrating deep learning and FPM. To summarize, we express our own viewpoints on the technological progress of FPM and explore prospective avenues for its future developments.

Classifying breast cancer and fibroadenoma tissue biopsies from paraffined stain-free slides by fractal biomarkers in Fourier Ptychographic Microscopy.

Breast cancer is one of the most spread and monitored pathologies in high-income countries. After breast biopsy, histological tissue is stored in paraffin, sectioned and mounted. Conventional inspection of tissue slides under benchtop light microscopes involves paraffin removal and staining, typically with H&E. Then, expert pathologists are called to judge the stained slides. However, paraffin removal and staining are operator-dependent, time and resources consuming processes that can generate ambiguities due to non-uniform staining. Here we propose a novel method that can work directly on paraffined stain-free slides. We use Fourier Ptychography as a quantitative phase-contrast microscopy method, which allows accessing a very wide field of view (i.e., mm2) in one single image while guaranteeing high lateral resolution (i.e., 0.5 µm). This imaging method is multi-scale, since it enables looking at the big picture, i.e. the complex tissue structure and connections, with the possibility to zoom-in up to the single-cell level. To handle this informative image content, we introduce elements of fractal geometry as multi-scale analysis method. We show the effectiveness of fractal features in describing and classifying fibroadenoma and breast cancer tissue slides from ten patients with very high accuracy. We reach 94.0 ± 4.2% test accuracy in classifying single images. Above all, we show that combining the decisions of the single images, each patient's slide can be classified with no error. Besides, fractal geometry returns a guide map to help pathologist to judge the different tissue portions based on the likelihood these can be associated to a breast cancer or fibroadenoma biomarker. The proposed automatic method could significantly simplify the steps of tissue analysis and make it independent from the sample preparation, the skills of the lab operator and the pathologist.

Adaptive correction method of hybrid aberrations in Fourier ptychographic microscopy.

Significance: Fourier ptychographic microscopy (FPM) enables quantitative phase imaging with a large field-of-view and high resolution by acquiring a series of low-resolution intensity images corresponding to different spatial frequencies stitched together in the Fourier domain. However, the presence of various aberrations in an imaging system can significantly degrade the quality of reconstruction results. The imaging performance and efficiency of the existing embedded optical pupil function recovery (EPRY-FPM) aberration correction algorithm are low due to the optimization strategy. Aim: An aberration correction method (AA-P algorithm) based on an improved phase recovery strategy is proposed to improve the reconstruction image quality. Approach: This algorithm uses adaptive modulation factors, which are added while updating iterations to optimize the spectral function and optical pupil function updates of the samples, respectively. The effectiveness of the proposed algorithm is verified through simulations and experiments using an open-source biological sample dataset. Results: Experimental results show that the proposed AA-P algorithm in an optical system with hybrid aberrations, recovered complex amplitude images with clearer contours and higher phase contrast. The image reconstruction quality was improved by 82.6% when compared with the EPRY-FPM algorithm. Conclusions: The proposed AA-P algorithm can reconstruct better results with faster convergence, and the recovered optical pupil function can better characterize the aberration of the imaging system. Thus, our method is expected to reduce the strict requirements of wavefront aberration for the current FPM.

Fast and robust Fourier ptychographic microscopy with position misalignment correction.

Significance: Fourier ptychographic microscopy (FPM) is a new, developing computational imaging technology. It can realize the quantitative phase imaging of a wide field of view and high-resolution (HR) simultaneously by means of multi-angle illumination via a light emitting diode (LED) array, combined with a phase recovery algorithm and the synthetic aperture principle. However, in the FPM reconstruction process, LED position misalignment affects the quality of the reconstructed image, and the reconstruction efficiency of the existing LED position correction algorithms needs to be improved. Aim: This study aims to improve the FPM correction method based on simulated annealing (SA) and proposes a position misalignment correction method (AA-C algorithm) using an improved phase recovery strategy. Approach: The spectrum function update strategy was optimized by adding an adaptive control factor, and the reconstruction efficiency of the algorithm was improved. Results: The experimental results show that the proposed method is effective and robust for position misalignment correction of LED arrays in FPM, and the convergence speed can be improved by 21.2% and 54.9% compared with SC-FPM and PC-FPM, respectively. Conclusions: These results can reduce the requirement of the FPM system for LED array accuracy and improve robustness.

Using symmetric illumination and color camera to achieve high throughput Fourier ptychographic microscopy.

This study aims to develop a high throughput Fourier ptychographic microscopy (FPM) technique based on symmetric illumination and a color detector, which is able to accelerate image acquisition by up to 12 times. As an emerging technology, the efficiency of FPM is limited by its data acquisition process, especially for color microscope image reconstruction. To overcome this, we built an FPM prototype equipped with a color camera and a 4×/0.13 NA objective lens. During the image acquisition, two symmetric LEDs illuminate the sample simultaneously using white light, which doubles the light intensity and reduces the total captured raw patterns by half. A standard USAF 1951 resolution target was used to measure the system's modulation transfer function (MTF) curve, and the H&E-stained ovarian cancer samples were then imaged to assess the feature qualities depicted on the reconstructed images. The results showed that the measured MTF curves of red, green, and blue channels are generally comparable to the corresponding curves generated by conventional FPM, while symmetric illumination FPM preserves more tissue details, which is superior to the results captured by conventional 20×/0.4 NA objective lens. This investigation initially verified the feasibility of symmetric illumination based color FPM.

Beyond conventional microscopy: Observing kidney tissues by means of fourier ptychography.

Kidney microscopy is a mainstay in studying the morphological structure, physiology and pathology of kidney tissues, as histology provides important results for a reliable diagnosis. A microscopy modality providing at same time high-resolution images and a wide field of view could be very useful for analyzing the whole architecture and the functioning of the renal tissue. Recently, Fourier Ptychography (FP) has been proofed to yield images of biology samples such as tissues and in vitro cells while providing high resolution and large field of view, thus making it a unique and attractive opportunity for histopathology. Moreover, FP offers tissue imaging with high contrast assuring visualization of small desirable features, although with a stain-free mode that avoids any chemical process in histopathology. Here we report an experimental measuring campaign for creating the first comprehensive and extensive collection of images of kidney tissues captured by this FP microscope. We show that FP microscopy unlocks a new opportunity for the physicians to observe and judge renal tissue slides through the novel FP quantitative phase-contrast microscopy. Phase-contrast images of kidney tissue are analyzed by comparing them with the corresponding renal images taken under a conventional bright-field microscope both for stained and unstained tissue samples of different thicknesses. In depth discussion on the advantages and limitations of this new stain-free microscopy modality is reported, showing its usefulness over the classical light microscopy and opening a potential route for using FP in clinical practice for histopathology of kidney.

Physics-based learning with channel attention for Fourier ptychographic microscopy.

Fourier ptychographic microscopy (FPM) is a computational imaging technology for large field-of-view, high resolution and quantitative phase imaging. In FPM, low-resolution intensity images captured with angle-varying illumination are synthesized in Fourier space with phase retrieval approaches. However, system errors such as pupil aberration and light-emitting diode (LED) intensity error seriously affect the reconstruction performance. In this article, we propose a physics-based neural network with channel attention for FPM reconstruction. With the channel attention module, which is introduced into physics-based neural networks for the first time, the spatial distribution of LED intensity can be adaptively corrected. Besides, the channel attention module is used to synthesize different Zernike modes and recover the pupil function. Detailed simulations and experiments are carried out to validate the effectiveness and robustness of the proposed method. The results demonstrate that our method achieves better performance in high-resolution complex field reconstruction, LED intensity correction and pupil function recovery compared with the state-of-art methods. The combination with deep neural network structures like channel attention modules significantly enhance the performance of physics-based neural networks and will promote the application of FPM in practical use.

Fourier Ptychographic Microscopy via Alternating Direction Method of Multipliers.

Fourier ptychographic microscopy (FPM) has risen as a promising computational imaging technique that breaks the trade-off between high resolution and large field of view (FOV). Its reconstruction is normally formulated as a blind phase retrieval problem, where both the object and probe have to be recovered from phaseless measured data. However, the stability and reconstruction quality may dramatically deteriorate in the presence of noise interference. Herein, we utilized the concept of alternating direction method of multipliers (ADMM) to solve this problem (termed ADMM-FPM) by breaking it into multiple subproblems, each of which may be easier to deal with. We compared its performance against existing algorithms in both simulated and practical FPM platform. It is found that ADMM-FPM method belongs to a global optimization algorithm with a high degree of parallelism and thus results in a more stable and robust phase recovery under noisy conditions. We anticipate that ADMM will rekindle interest in FPM as more modifications and innovations are implemented in the future.

Accurate and stable two-step LED position calibration method for Fourier ptychographic microscopy.

SIGNIFICANCE: Fourier ptychography microscopy (FPM) is a computational optical imaging technology that employs angularly varying illuminations and a phase retrieval algorithm to achieve a wide field of view and high-resolution imaging simultaneously. In the FPM, LED position error will reduce the quality of the reconstructed high-resolution image. To correct the LED positions, current methods consider each of the LED positions as independent and use an optimization algorithm to get each of the positions. When the positional misalignment is large or the search position falls into a local optimal value, the current methods may lack stability and accuracy. AIM: We improve the model of the LED position and propose an accurate and stable two-step correction scheme (tcFPM) to calibrate the LED position error. APPROACH: The improved LED positions model combines the overall offset, which represents the relative deviation of the LED array and the optical axis, with the slight deviation of each LED's independent position. In the tcFPM, the overall offset of the LED array is corrected at first, which obtains an approximate value of the overall offset of the LED array. Then the position of each LED is precisely adjusted, which obtains the slight offset of each LED. RESULTS: This LED position error model is more in line with the actual situation. The simulation and experimental results show that the method has high accuracy in correcting the LED position. Furthermore, the reconstruction process of tcFPM is more stable and significantly improves the quality of the reconstruction results, which is compared with some LED position error correction methods. CONCLUSIONS: An LED position error correction technology is proposed, which has a stable iterative process and improves the reconstruction accuracy of complex amplitude.

Neural network model assisted Fourier ptychography with Zernike aberration recovery and total variation constraint.

SIGNIFICANCE: Fourier ptychography (FP) is a computational imaging approach that achieves high-resolution reconstruction. Inspired by neural networks, many deep-learning-based methods are proposed to solve FP problems. However, the performance of FP still suffers from optical aberration, which needs to be considered. AIM: We present a neural network model for FP reconstructions that can make proper estimation toward aberration and achieve artifact-free reconstruction. APPROACH: Inspired by the iterative reconstruction of FP, we design a neural network model that mimics the forward imaging process of FP via TensorFlow. The sample and aberration are considered as learnable weights and optimized through back-propagation. Especially, we employ the Zernike terms instead of aberration to decrease the optimization freedom of pupil recovery and perform a high-accuracy estimation. Owing to the auto-differentiation capabilities of the neural network, we additionally utilize total variation regularization to improve the visual quality. RESULTS: We validate the performance of the reported method via both simulation and experiment. Our method exhibits higher robustness against sophisticated optical aberrations and achieves better image quality by reducing artifacts. CONCLUSIONS: The forward neural network model can jointly recover the high-resolution sample and optical aberration in iterative FP reconstruction. We hope our method that can provide a neural-network perspective to solve iterative-based coherent or incoherent imaging problems.

An efficient Fourier ptychographic microscopy method based on optimized pattern of LED angle illumination.

Considering angle diversity and synthetic aperture, Fourier ptychographic microscopy (FPM) could address contradiction of high resolution and wide field of view. However, in the conventional FPM method, large capture quantity leads to poor efficiency. So, an efficient FPM method based on optimized pattern of LED angle illumination is proposed. Firstly, from position relationship between the LED, aperture and sample, we obtain the theoretical expandable spectrum range and the spectrum distribution in Fourier space of all LEDs. Secondly, we u se image quality assessment methods to extract differential expressions between an arbitrary LED illumination and full LEDs illumination. Thirdly, the optimized angle illumination strategy is achieved according to the analysis of different expressions. Based on this method, we design a rhombus-based illumination method for the LED array to accelerate the FPM efficiency. Finally, we validate the effectiveness and efficiency of our method with both simulated and real experiments. The results indicate that our method can effectively improve the efficiency for FPM without sacrificing image reconstruction quality.

Fourier Ptychographic Microscopy for Rapid, High-Resolution Imaging of Circulating Tumor Cells Enriched by Microfiltration.

Examining the hematogenous compartment for evidence of metastasis has increased significantly within the oncology research community in recent years, due to the development of technologies aimed at the enrichment of circulating tumor cells (CTCs), the subpopulation of primary tumor cells that gain access to the circulatory system and are responsible for colonization at distant sites. In contrast to other technologies, filtration-based CTC enrichment, which exploits differences in size between larger tumor cells and surrounding smaller, non-tumor blood cells, has the potential to improve CTC characterization through isolation of tumor cell populations with greater molecular heterogeneity. However, microscopic analysis of uneven filtration surfaces containing CTCs is laborious, time-consuming, and inconsistent, preventing widespread use of filtration-based enrichment technologies. Here, integrated with a microfiltration-based CTC and rare cell enrichment device we have previously described, we present a protocol for Fourier Ptychographic Microscopy (FPM), a method that, unlike many automated imaging platforms, produces high-speed, high-resolution images that can be digitally refocused, allowing users to observe objects of interest present on multiple focal planes within the same image frame. The development of a cost-effective and high-throughput CTC analysis system for filtration-based enrichment technologies could have profound clinical implications for improved CTC detection and analysis.