Roadmap on computational methods in optical imaging and holography [invited].

Computational methods have been established as cornerstones in optical imaging and holography in recent years. Every year, the dependence of optical imaging and holography on computational methods is increasing significantly to the extent that optical methods and components are being completely and efficiently replaced with computational methods at low cost. This roadmap reviews the current scenario in four major areas namely incoherent digital holography, quantitative phase imaging, imaging through scattering layers, and super-resolution imaging. In addition to registering the perspectives of the modern-day architects of the above research areas, the roadmap also reports some of the latest studies on the topic. Computational codes and pseudocodes are presented for computational methods in a plug-and-play fashion for readers to not only read and understand but also practice the latest algorithms with their data. We believe that this roadmap will be a valuable tool for analyzing the current trends in computational methods to predict and prepare the future of computational methods in optical imaging and holography. Supplementary Information: The online version contains supplementary material available at 10.1007/s00340-024-08280-3.

Rapid flowing cells localization enabled by spatiotemporal manipulation of their holographic patterns.

Lab-on-a-Chip microfluidic devices present an innovative and cost-effective platform in the current trend of miniaturization and simplification of imaging flow cytometry; they are excellent candidates for high-throughput single-cell analysis. In such microfluidic platforms, cell tracking becomes a fundamental tool for investigating biophysical processes, from intracellular dynamics to the characterization of cell motility and migration. However, high-throughput and long-term cell tracking puts a high demand on the consumption of computing resources. Here, we propose a novel strategy to achieve rapid 3D cell localizations along the microfluidic channel. This method is based on the spatiotemporal manipulation of recorded holographic interference fringes, and it allows fast and precise localization of cells without performing complete holographic reconstruction. Conventional holographic tracking is typically based on the phase contrast obtained by decoupling the calculation of optical axial and transverse coordinates. Computing time and resource consumption may increase because all the frames need to be calculated in the Fourier domain. In our proposed method, the 2D transverse positions are directly located by morphological calculation based on the hologram. The complex-amplitude wavefronts are directly reconstructed by spatiotemporal phase shifting to calculate the axial position by the refocusing criterion. Only spatial calculation is considered in the proposed method. We demonstrate that the computational time of transverse tracking is only one-tenth of the conventional method, while the total computational time of the proposed method decreases up to 54% with respect to the conventional approach. The proposed approach can open the route for analyzing flow cytometry in quantitative phase microscopy assays.

Thin and lightweight camera based on Pancake lens and deep learning.

Computational imaging using a Pancake lens can help reduce the size of optical systems by folded optics. However, Pancake cameras frequently exhibit inferior image quality due to stray light, low light transmission, and spatially varying aberrations. In this Letter, we propose a thin and lightweight camera comprising a polarization-based catadioptric Pancake lens and a Fourier Position encoding Network (FPNet). The camera achieves high-quality imaging at an f-number of 0.4 and an expansive 88° field of view. The FPNet encodes the positional order of the point spread functions, mitigating global optical image degradation and improving image quality by 10.13 dB in PSNR. The Pancake camera and FPNet have potential applications in mobile photography and virtual/augmented reality.

Non-convex optimization for inverse problem solving in computer-generated holography.

Computer-generated holography is a promising technique that modulates user-defined wavefronts with digital holograms. Computing appropriate holograms with faithful reconstructions is not only a problem closely related to the fundamental basis of holography but also a long-standing challenge for researchers in general fields of optics. Finding the exact solution of a desired hologram to reconstruct an accurate target object constitutes an ill-posed inverse problem. The general practice of single-diffraction computation for synthesizing holograms can only provide an approximate answer, which is subject to limitations in numerical implementation. Various non-convex optimization algorithms are thus designed to seek an optimal solution by introducing different constraints, frameworks, and initializations. Herein, we overview the optimization algorithms applied to computer-generated holography, incorporating principles of hologram synthesis based on alternative projections and gradient descent methods. This is aimed to provide an underlying basis for optimized hologram generation, as well as insights into the cutting-edge developments of this rapidly evolving field for potential applications in virtual reality, augmented reality, head-up display, data encryption, laser fabrication, and metasurface design.

CFZA camera: a high-resolution lensless imaging technique based on compound Fresnel zone aperture.

In lensless imaging using a Fresnel zone aperture (FZA), it is generally believed that the resolution is limited by the outermost ring breadth of the FZA. The limitation has the potential to be broken according to the multi-order property of binary FZAs. In this Letter, we propose to use a high-order component of the FZA as the point spread function (PSF) to develop a high-order transfer function backpropagation (HBP) algorithm to enhance the resolution. The proportion of high-order diffraction energy is low, leading to severe defocus noise in the reconstructed image. To address this issue, we propose a Compound FZA (CFZA), which merges two partial FZAs operating at different orders as the mask to strike a balance between the noise and resolution. Experimental results verify that the CFZA-based camera has a resolution that is double that of a traditional FZA-based camera with an identical outer ring breadth and can be reconstructed with high quality by a single HBP without calibration. Our method offers a cost-effective solution for achieving high-resolution imaging, expanding the potential applications of FZA-based lensless imaging in a variety of areas.

Quantitative phase imaging based on holography: trends and new perspectives.

In 1948, Dennis Gabor proposed the concept of holography, providing a pioneering solution to a quantitative description of the optical wavefront. After 75 years of development, holographic imaging has become a powerful tool for optical wavefront measurement and quantitative phase imaging. The emergence of this technology has given fresh energy to physics, biology, and materials science. Digital holography (DH) possesses the quantitative advantages of wide-field, non-contact, precise, and dynamic measurement capability for complex-waves. DH has unique capabilities for the propagation of optical fields by measuring light scattering with phase information. It offers quantitative visualization of the refractive index and thickness distribution of weak absorption samples, which plays a vital role in the pathophysiology of various diseases and the characterization of various materials. It provides a possibility to bridge the gap between the imaging and scattering disciplines. The propagation of wavefront is described by the complex amplitude. The complex-value in the complex-domain is reconstructed from the intensity-value measurement by camera in the real-domain. Here, we regard the process of holographic recording and reconstruction as a transformation between complex-domain and real-domain, and discuss the mathematics and physical principles of reconstruction. We review the DH in underlying principles, technical approaches, and the breadth of applications. We conclude with emerging challenges and opportunities based on combining holographic imaging with other methodologies that expand the scope and utility of holographic imaging even further. The multidisciplinary nature brings technology and application experts together in label-free cell biology, analytical chemistry, clinical sciences, wavefront sensing, and semiconductor production.

Deep learning sheds new light on non-orthogonal optical multiplexing.

A deep neural network for non-orthogonal input channel encoding is proposed to recover speckle images through a multimode fiber. This novel approach could shed new light on the non-orthogonal optical multiplexing over a scattering medium.

Decoding of compressive data pages for optical data storage utilizing FFDNet.

Coded aperture-based compression has proven to be an effective approach for high-density cold data storage. Nevertheless, its limited decoding speed represents a significant challenge for its broader application. We introduce a novel, to the best of our knowledge, decoding method leveraging the fast and flexible denoising network (FFDNet), capable of decoding a coded aperture-based compressive data page within 30.64 s. The practicality of the method has been confirmed in the decoding of monochromatic photo arrays, full-color photos, and dynamic videos. In experimental trials, the variance between decoded results obtained via the FFDNet-based method and the FFDNet-absent method in terms of average PSNR is less than 1 dB, while realizing a decoding speed enhancement of over 100-fold when employing the FFDNet-based method.

Computational Imaging: The Next Revolution for Biophotonics and Biomedicine.

This Editorial is the preface for the topical collection of "Computational Imaging for Biophotonics and Biomedicine", which collates the 12 contributions listed in Table 1 [...].

Adaptive layer-based computer-generated holograms.

We propose a novel, to the best of our knowledge, and fast adaptive layer-based (ALB) method for generating a computer-generated hologram (CGH) with accurate depth information. A complex three-dimensional (3D) object is adaptively divided into layers along the depth direction according to its own non-uniformly distributed depth coordinates, which reduces the depth error caused by the conventional layer-based method. Each adaptive layer generates a single-layer hologram using the angular spectrum method for diffraction, and the final hologram of a complex three-dimensional object is obtained by superimposing all the adaptive layer holograms. A hologram derived with the proposed method is referred to as an adaptive layer-based hologram (ALBH). Our demonstration shows that the desired reconstruction can be achieved with 52 adaptive layers in 8.7 s, whereas the conventional method requires 397 layers in 74.9 s.

Super-resolution lensless on-chip microscopy based on array illumination and sub-pixel shift search.

The resolution of a lensless on-chip microscopy system is constrained by the pixel size of image sensors. This Letter introduces a super-resolution on-chip microscopy system based on a compact array light source illumination and sub-pixel shift search. The system utilizes a closely spaced array light source composed by four RGB LED modules, sequentially illuminating the sample. A sub-pixel shift search algorithm is proposed, which determines the sub-pixel shift by comparing the frequency of captured low-resolution holograms. Leveraging this sub-pixel shift, a super-resolution reconstruction algorithm is introduced, building upon a multi-wavelength phase retrieval method, enabling the rapid super-resolution reconstruction of holograms with the region-of-interest. The system and algorithms presented herein obviate the need for a displacement control platform and calibration of the illumination angles of the light source, facilitating a super-resolution phase reconstruction under partially coherent illumination.

Lensless imaging in one shot using the complex degree of coherence obtained by multiaperture interferences.

The van Cittert-Zernike theorem states that the Fourier transform of the intensity distribution function of a distant, incoherent source is equal to the complex degree of coherence. In this Letter, we present a method for measuring the complex degree of coherence in one shot by recording the interference patterns produced by multiple aperture pairs. The intensity of the sample is obtained by Fourier transforming the complex degree of coherence. The experimental verification by using a simple object is presented together with a discussion on how the method could be improved for imaging more complex samples.

Single-shot deterministic complex amplitude imaging with a single-layer metalens.

Conventional imaging systems can only capture light intensity. Meanwhile, the lost phase information may be critical for a variety of applications such as label-free microscopy and optical metrology. Existing phase retrieval techniques typically require a bulky setup, multiframe measurements, or prior information of the target scene. Here, we proposed an extremely compact system for complex amplitude imaging, leveraging the extreme versatility of a single-layer metalens to generate spatially multiplexed and polarization phase-shifted point spread functions. Combining the metalens with a polarization camera, the system can simultaneously record four polarization shearing interference patterns along both in-plane directions, thus allowing the deterministic reconstruction of the complex amplitude light field in a single shot. Using an incoherent light-emitting diode as the illumination, we experimentally demonstrated speckle-noise-free complex amplitude imaging for both static and moving objects with tailored magnification ratio and field of view. The miniaturized and robust system may open the door for complex amplitude imaging in portable devices for point-of-care applications.

Deep nonlocal low-rank regularization for complex-domain pixel super-resolution.

Pixel super-resolution (PSR) has emerged as a promising technique to break the sampling limit for phase imaging systems. However, due to the inherent nonconvexity of phase retrieval problem and super-resolution process, PSR algorithms are sensitive to noise, leading to reconstruction quality inevitably deteriorating. Following the plug-and-play framework, we introduce the nonlocal low-rank (NLR) regularization for accurate and robust PSR, achieving a state-of-the-art performance. Inspired by the NLR prior, we further develop the complex-domain nonlocal low-rank network (CNLNet) regularization to perform nonlocal similarity matching and low-rank approximation in the deep feature domain rather than the spatial domain of conventional NLR. Through visual and quantitative comparisons, CNLNet-based reconstruction shows an average 1.4 dB PSNR improvement over conventional NLR, outperforming existing algorithms under various scenarios.

Coded aperture-based compressive data page for optical data storage.

In an era of data explosion, optical data storage provides an alternative solution for cold data storage due to its energy-saving and cost-effective features. However, its data density is still insufficient for zettabyte-scale cold data storage. Here, a coded aperture-based compressive data page with a compression ratio of ≤0.125 is proposed. Based on two frameworks-weighted nuclear norm minimization (WNNM) and alternating direction method of multipliers (ADMM)-the decoded quality of the compressive data page is ensured by utilizing sparsity priors. In experiments, compressive data pages of a monochromatic photo-array, full-color photo, and dynamic video are accurately decoded.

DGE-CNN: 2D-to-3D holographic display based on a depth gradient extracting module and ZCNN network.

Holography is a crucial technique for the ultimate three-dimensional (3D) display, because it renders all optical cues from the human visual system. However, the shortage of 3D contents strictly restricts the extensive application of holographic 3D displays. In this paper, a 2D-to-3D-display system by deep learning-based monocular depth estimation is proposed. By feeding a single RGB image of a 3D scene into our designed DGE-CNN network, a corresponding display-oriented 3D depth map can be accurately generated for layer-based computer-generated holography. With simple parameter adjustment, our system can adapt the distance range of holographic display according to specific requirements. The high-quality and flexible holographic 3D display can be achieved based on a single RGB image without 3D rendering devices, permitting potential human-display interactive applications such as remote education, navigation, and medical treatment.

Amp-vortex edge-camera: a lensless multi-modality imaging system with edge enhancement.

We demonstrate a lensless imaging system with edge-enhanced imaging constructed with a Fresnel zone aperture (FZA) mask placed 3 mm away from a CMOS sensor. We propose vortex back-propagation (vortex-BP) and amplitude vortex-BP algorithms for the FZA-based lensless imaging system to remove the noise and achieve the fast reconstruction of high contrast edge enhancement. Directionally controlled anisotropic edge enhancement can be achieved with our proposed superimposed vortex-BP algorithm. With different reconstruction algorithms, the proposed amp-vortex edge-camera in this paper can achieve 2D bright filed imaging, isotropic, and directional controllable anisotropic edge-enhanced imaging with incoherent light illumination, by a single-shot captured hologram. The effect of edge detection is the same as optical edge detection, which is the re-distribution of light energy. Noise-free in-focus edge detection can be achieved by using back-propagation, without a de-noise algorithm, which is an advantage over other lensless imaging technologies. This is expected to be widely used in autonomous driving, artificial intelligence recognition in consumer electronics, etc.

Adaptive constraints by morphological operations for single-shot digital holography.

Digital holography provides access to quantitative measurement of the entire complex field, which is indispensable for the investigation of wave-matter interactions. The emerging iterative phase retrieval approach enables to solve the inverse imaging problem only from the given intensity measurements and physical constraints. However, enforcing imprecise constraints limits the reconstruction accuracy and convergence speed. Here, we propose an advanced iterative phase retrieval framework for single-shot in-line digital holography that incorporates adaptive constraints, which achieves optimized convergence behavior, high-fidelity and twin-image-free reconstruction. In conjunction with morphological operations which can extract the object structure while eliminating the irrelevant part such as artifacts and noise, adaptive constraints allow the support region to be accurately estimated and automatically updated at each iteration. Numerical reconstruction of complex-valued objects and the capability of noise immunity are investigated. The improved reconstruction performance of this approach is experimentally validated. Such flexible and versatile framework has promising applications in biomedicine, X-ray coherent diffractive imaging and wavefront sensing.

Analysis of reconstruction quality for computer-generated holograms using a model free of circular-convolution error.

Continuous complex-amplitude computer-generated holograms (CGHs) are converted to discrete amplitude-only or phase-only ones in practical applications to cater for the characteristics of spatial light modulators (SLMs). To describe the influence of the discretization correctly, a refined model that eliminates the circular-convolution error is proposed to emulate the propagation of the wavefront during the formation and reconstruction of a CGH. The effects of several significant factors, including quantized amplitude and phase, zero-padding rate, random phase, resolution, reconstruction distance, wavelength, pixel pitch, phase modulation deviation and pixel-to-pixel interaction, are discussed. Based on evaluations, the optimal quantization for both available and future SLM devices is suggested.

Polarimetric calibrated robust dual-SLM complex-amplitude computer-generated holography.

Liquid crystal on silicon (LCoS) is a widely used spatial light modulator (SLM) in computer-generated holography (CGH). However, the phase-modulating profile of LCoS is often not ideally uniform in application, bringing about undesired intensity fringes. In this study, we overcome this problem by proposing a highly robust dual-SLM complex-amplitude CGH technique, which incorporates a polarimetric mode and a diffractive mode. The polarimetric mode linearizes the general phase modulations of the two SLMs separately, while the diffractive mode uses camera-in-the-loop optimization to achieve improved holographic display. Experimental results show the effectiveness of our proposal in improving reconstructing accuracy by 21.12% in peak signal-to-noise ratio (PSNR) and 50.74% in structure similarity index measure (SSIM), using LCoS SLMs with originally non-uniform phase-modulating profiles.