Roadmap on computational methods in optical imaging and holography [invited].

Computational methods have been established as cornerstones in optical imaging and holography in recent years. Every year, the dependence of optical imaging and holography on computational methods is increasing significantly to the extent that optical methods and components are being completely and efficiently replaced with computational methods at low cost. This roadmap reviews the current scenario in four major areas namely incoherent digital holography, quantitative phase imaging, imaging through scattering layers, and super-resolution imaging. In addition to registering the perspectives of the modern-day architects of the above research areas, the roadmap also reports some of the latest studies on the topic. Computational codes and pseudocodes are presented for computational methods in a plug-and-play fashion for readers to not only read and understand but also practice the latest algorithms with their data. We believe that this roadmap will be a valuable tool for analyzing the current trends in computational methods to predict and prepare the future of computational methods in optical imaging and holography. Supplementary Information: The online version contains supplementary material available at 10.1007/s00340-024-08280-3.

Hybrid diffractive optics design via hardware-in-the-loop methodology for achromatic extended-depth-of-field imaging.

End-to-end optimization of diffractive optical elements (DOEs) profile through a digital differentiable model combined with computational imaging have gained an increasing attention in emerging applications due to the compactness of resultant physical setups. Despite recent works have shown the potential of this methodology to design optics, its performance in physical setups is still limited and affected by manufacturing artefacts of DOE, mismatch between simulated and resultant experimental point spread functions, and calibration errors. Additionally, the computational burden of the digital differentiable model to effectively design the DOE is increasing, thus limiting the size of the DOE that can be designed. To overcome the above mentioned limitations, a co-design of hybrid optics and image reconstruction algorithm is produced following the end-to-end hardware-in-the-loop strategy, using for optimization a convolutional neural network equipped with quantitative and qualitative loss functions. The optics of the imaging system consists on the phase-only spatial light modulator (SLM) as DOE and refractive lens. SLM phase-pattern is optimized by applying the Hardware-in-the-loop technique, which helps to eliminate the mismatch between numerical modelling and physical reality of image formation as light propagation is not numerically modelled but is physically done. Comparison with compound multi-lens optics of a last generation smartphone and a mirrorless commercial cameras show that the proposed system is advanced in all-in-focus sharp imaging for a depth range 0.4-1.9 m.

Power-balanced hybrid optics boosted design for achromatic extended depth-of-field imaging via optimized mixed OTF.

A power-balanced hybrid optical imaging system has a diffractive computational camera, introduced in this paper, with image formation by a refractive lens and multilevel phase mask (MPM). This system provides a long focal depth with low chromatic aberrations thanks to MPM and a high energy light concentration due to the refractive lens. We introduce the concept of optical power balance between the lens and MPM, which controls the contribution of each element to modulate the incoming light. Additional features of our MPM design are the inclusion of the quantization of the MPM's shape on the number of levels and the Fresnel order (thickness) using a smoothing function. To optimize the optical power balance as well as the MPM, we built a fully differentiable image formation model for joint optimization of optical and imaging parameters for the proposed camera using neural network techniques. We also optimized a single Wiener-like optical transfer function (OTF) invariant to depth to reconstruct a sharp image. We numerically and experimentally compare the designed system with its counterparts, lensless and just-lens optical systems, for the visible wavelength interval (400-700) nm and the depth-of-field range (0.5-∞ m for numerical and 0.5-2 m for experimental). We believe the attained results demonstrate that the proposed system equipped with the optimal OTF overcomes its counterparts--even when they are used with optimized OTF--in terms of the reconstruction quality for off-focus distances. The simulation results also reveal that optimizing the optical power balance, Fresnel order, and the number of levels parameters are essential for system performance attaining an improvement of up to 5 dB of PSNR using the optimized OTF compared to its counterpart lensless setup.

Lensless hyperspectral phase imaging in a self-reference setup based on Fourier transform spectroscopy and noise suppression.

A novel phase retrieval algorithm for broadband hyperspectral phase imaging from noisy intensity observations is proposed. It utilizes advantages of the Fourier transform spectroscopy in the self-referencing optical setup and provides additional, beyond spectral intensity distribution, reconstruction of the investigated object's phase. The noise amplification Fellgett's disadvantage is relaxed by the application of a sparse wavefront noise filtering embedded in the proposed algorithm. The algorithm reliability is proved by simulation tests and by results of physical experiments for transparent objects. These tests demonstrate precise phase imaging and object depth (profile) reconstruction.

Single exposure lensless subpixel phase imaging: optical system design, modelling, and experimental study.

Design and optimization of lensless phase-retrieval optical system with phase modulation of free-space propagation wavefront is proposed for subpixel imaging to achieve super-resolution reconstruction. Contrary to the traditional super-resolution phase-retrieval, the method in this paper requires a single observation only and uses the advanced Super-Resolution Sparse Phase Amplitude Retrieval (SR-SPAR) iterative technique which contains optimized sparsity based filters and multi-scale filters. The successful object imaging relies on modulation of the object wavefront with a random phase-mask, which generates coded diffracted intensity pattern, allowing us to extract subpixel information. The system's noise-robustness was investigated and verified. The super-resolution phase-imaging is demonstrated by simulations and physical experiments. The simulations included high quality reconstructions with super-resolution factor of 5, and acceptable at factor up to 9. By physical experiments 3 µm details were resolved, which are 2.3 times smaller than the resolution following from the Nyquist-Shannon sampling theorem.

Hyperspectral data denoising for terahertz pulse time-domain holography.

We investigated data denoising in hyperspectral terahertz pulse time-domain holography. Using the block-matching algorithms adapted for spatio-temporal and spatio-spectral volumetric data we studied and optimized parameters of these algorithms to improve phase image reconstruction quality. We propose a sequential application of the two algorithms oriented on work in temporal and spectral domains. Experimental data demonstrate the improvement in the quality of the resultant time-domain images as well as phase images and object's relief. The simulation results are proved by comparison with the experimental ones.

Terahertz pulse time-domain holography with balance detection: complex-domain sparse imaging.

We investigated the peculiarities of the terahertz pulse time-domain holography principle in the case of raster scanning with the balance detection system. The noise in this system represents a Skellam distribution model, which differentiates it from systems based on a photoconductive antenna. We analyzed this Skellam model and provided both numerical and experimental investigations. We found that the variance of the noise in the balance detection system does not depend on the true signal. Complex-domain images obtained in this model are filtered by block-matching algorithms adapted for spatio-temporal and spatiospectral volumetric data. We presented a new cube complex-domain filter algorithm that uses block matching in all 3D data sets simultaneously in spatial and frequency coordinates. A combination of temporal and complex-domain filters allows us to expand the dynamic range of terahertz frequencies for which we can obtain amplitude/phase information. Experimental data demonstrate an improvement in the quality of the resultant images both in the time domain and complex-spectral domain. The simulation and experimental results are in good agreement.

Spectral Object Recognition in Hyperspectral Holography with Complex-Domain Denoising.

In this paper, we have applied a recently developed complex-domain hyperspectral denoiser for the object recognition task, which is performed by the correlation analysis of investigated objects' spectra with the fingerprint spectra from the same object. Extensive experiments carried out on noisy data from digital hyperspectral holography demonstrate a significant enhancement of the recognition accuracy of signals masked by noise, when the advanced noise suppression is applied.

Dictionary Learning Phase Retrieval from Noisy Diffraction Patterns.

This paper proposes a novel algorithm for image phase retrieval, i.e., for recovering complex-valued images from the amplitudes of noisy linear combinations (often the Fourier transform) of the sought complex images. The algorithm is developed using the alternating projection framework and is aimed to obtain high performance for heavily noisy (Poissonian or Gaussian) observations. The estimation of the target images is reformulated as a sparse regression, often termed sparse coding, in the complex domain. This is accomplished by learning a complex domain dictionary from the data it represents via matrix factorization with sparsity constraints on the code (i.e., the regression coefficients). Our algorithm, termed dictionary learning phase retrieval (DLPR), jointly learns the referred to dictionary and reconstructs the unknown target image. The effectiveness of DLPR is illustrated through experiments conducted on complex images, simulated and real, where it shows noticeable advantages over the state-of-the-art competitors.

High-accuracy off-axis wavefront reconstruction from noisy data: local least square with multiple adaptive windows.

A variational algorithm to object wavefront reconstruction from noisy intensity observations is developed for the off-axis holography scenario with imaging in the acquisition plane. The algorithm is based on the local least square technique proposed in paper [J. Opt. Soc. Am. A21, 367 (2004)]. First, multiple reconstructions of the wavefront are produced for various size and various directional windows applied for localization of estimation. At the second stage, a special statistical rule is applied in order to select the best window size estimate for each pixel of the image and for each of the directional windows. At the third final stage the estimates of the different directions obtained for each pixel are aggregated in the final one. Simulation experiments and real data processing prove that the developed algorithm demonstrate the performance of the extraordinary quality and accuracy for both the phase and amplitude of the object wavefront.

Super-resolution in a defocused plenoptic camera: a wave-optics-based approach.

Plenoptic cameras enable the capture of a light field with a single device. However, with traditional light field rendering procedures, they can provide only low-resolution two-dimensional images. Super-resolution is considered to overcome this drawback. In this study, we present a super-resolution method for the defocused plenoptic camera (Plenoptic 1.0), where the imaging system is modeled using wave optics principles and utilizing low-resolution depth information of the scene. We are particularly interested in super-resolution of in-focus and near in-focus scene regions, which constitute the most challenging cases. The simulation results show that the employed wave-optics model makes super-resolution possible for such regions as long as sufficiently accurate depth information is available.

Sparse ptychographical coherent diffractive imaging from noisy measurements.

Ptychography is a lensless coherent diffractive imaging that uses intensity measurements of multiple diffraction patterns collected with a localized illumination probe from overlapping regions of an object. An iterative algorithm is proposed that is targeted on optimal processing noisy measurements. The noise suppression is enabled by two instruments: first, the maximum-likelihood technique formulated for Poissonian (photon-counting) measurements, and, second, sparse approximation of the phase and magnitude of the object and probe. It is shown that the maximum-likelihood estimate of the wavefield at the sensor plane for noisy measurements is essentially different from the famous Gerchberg-Saxton-Fienup solution, where the magnitude of the estimate is replaced by the square root of the intensity measurement. The simulation experiments demonstrate the state-of-the-art performance of the proposed algorithm both numerically and visually.

Phase retrieval in 4f optical system: background compensation and sparse regularization of object with binary amplitude.

Generally, wave field reconstructions obtained by phase-retrieval algorithms are noisy, blurred, and corrupted by various artifacts such as irregular waves, spots, etc. These distortions, arising due to many factors, such as nonidealities of the optical system (misalignment, focusing errors), dust on optical elements, reflections, and vibration, are hard to localize and specify. It is assumed that there is a cumulative disturbance called "background," which describes mentioned distortions in the coherent imaging system manifested at the sensor plane. Here we propose a novel iterative phase-retrieval algorithm compensating for these distortions in the optical system. An estimate of this background is obtained via special calibration experiments, and then it is used for the object reconstruction. The algorithm is based on the maximum likelihood approach targeting on the optimal object reconstruction from noisy data and imaging enhancement using a priori information on the object amplitude. In this work we demonstrate the compensation of the distortions of the optical trace for a complex-valued object with a binary amplitude. The developed algorithm results in state-of-the-art filtering, and sharp reconstruction imaging of the object amplitude can be achieved.

Miniature color camera via flat hybrid meta-optics.

The race for miniature color cameras using flat meta-optics has rapidly developed the end-to-end design framework using neural networks. Although a large body of work has shown the potential of this methodology, the reported performance is still limited due to fundamental limitations coming from meta-optics, mismatch between simulated and resultant experimental point spread functions, and calibration errors. Here, we use a HIL optics design methodology to solve these limitations and demonstrate a miniature color camera via flat hybrid meta-optics (refractive + meta-mask). The resulting camera achieves high-quality full-color imaging for a 5-mm aperture optics with a focal length of 5 mm. We observed a superior quality of the images captured by the hybrid meta-optical camera compared to a compound multi-lens optics of a mirrorless commercial camera.

Compressive sensing computational ghost imaging.

The computational ghost imaging with a phase spatial light modulator (SLM) for wave field coding is considered. A transmission-mask amplitude object is reconstructed from multiple intensity observations. Compressive techniques are used in order to gain a successful image reconstruction with a number of observations (measurement experiments), which is smaller than the image size. Maximum likelihood style algorithms are developed, respectively, for Poissonian and approximate Gaussian modeling of random observations. A sparse and overcomplete modeling of the object enables the advanced high accuracy and sharp imaging. Numerical experiments demonstrate that an approximative Gaussian distribution with an invariant variance results in the algorithm that is efficient for Poissonian observations.

High-accuracy wave field reconstruction: decoupled inverse imaging with sparse modeling of phase and amplitude.

We apply a nonlocal adaptive spectral transform for sparse modeling of phase and amplitude of a coherent wave field. The reconstruction of this wave field from complex-valued Gaussian noisy observations is considered. The problem is formulated as a multiobjective constrained optimization. The developed iterative algorithm decouples the inversion of the forward propagation operator and the filtering of phase and amplitude of the wave field. It is demonstrated by simulations that the performance of the algorithm, both visually and numerically, is the current state of the art.

Phase retrieval via spatial light modulator phase modulation in 4f optical setup: numerical inverse imaging with sparse regularization for phase and amplitude.

The 4f optical setup is considered with a wave field modulation by a spatial light modulator located in the focal plane of the first lens. Phase as well as amplitude of the wave field are reconstructed from noisy multiple-intensity observations. The reconstruction is optimal due to a constrained maximum likelihood formulation of the problem. The proposed algorithm is iterative with decoupling of the inverse of the forward propagation of the wave field and the filtering of phase and amplitude. The sparse modeling of phase and amplitude enables the advanced high-accuracy filtering and sharp imaging of the complex-valued wave field. Artifacts typical for the conventional algorithms (wiggles, ringing, waves, etc.) and attributed to optical diffraction can be suppressed by the proposed algorithm.

Wave field reconstruction from multiple plane intensity-only data: augmented lagrangian algorithm.

A complex-valued wave field is reconstructed from intensity-only measurements given at multiple observation planes parallel to the object plane. The phase-retrieval algorithm is obtained from the constrained maximum likelihood approach provided that the additive noise is gaussian. The forward propagation from the object plane to the measurement plane is treated as a constraint in the proposed variational setting of reconstruction. The developed iterative algorithm is based on an augmented lagrangian technique. An advanced performance of the algorithm is demonstrated by numerical simulations.

Backward discrete wave field propagation modeling as an inverse problem: toward perfect reconstruction of wave field distributions.

We consider reconstruction of a wave field distribution in an input/object plane from data in an output/diffraction (sensor) plane. We provide digital modeling both for the forward and backward wave field propagation. A novel algebraic matrix form of the discrete diffraction transform (DDT) originated in Katkovnik et al. [Appl. Opt. 47, 3481 (2008)] is proposed for the forward modeling that is aliasing free and precise for pixelwise invariant object and sensor plane distributions. This "matrix DDT" is a base for formalization of the object wave field reconstruction (backward propagation) as an inverse problem. The transfer matrices of the matrix DDT are used for calculations as well as for the analysis of conditions when the perfect reconstruction of the object wave field distribution is possible. We show by simulation that the developed inverse propagation algorithm demonstrates an improved accuracy as compared with the standard convolutional and discrete Fresnel transform algorithms.

Absolute phase estimation: adaptive local denoising and global unwrapping.

The paper attacks absolute phase estimation with a two-step approach: the first step applies an adaptive local denoising scheme to the modulo-2 pi noisy phase; the second step applies a robust phase unwrapping algorithm to the denoised modulo-2 pi phase obtained in the first step. The adaptive local modulo-2 pi phase denoising is a new algorithm based on local polynomial approximations. The zero-order and the first-order approximations of the phase are calculated in sliding windows of varying size. The zero-order approximation is used for pointwise adaptive window size selection, whereas the first-order approximation is used to filter the phase in the obtained windows. For phase unwrapping, we apply the recently introduced robust (in the sense of discontinuity preserving) PUMA unwrapping algorithm [IEEE Trans. Image Process.16, 698 (2007)] to the denoised wrapped phase. Simulations give evidence that the proposed algorithm yields state-of-the-art performance, enabling strong noise attenuation while preserving image details.