Focus Issue Introduction: 3D Image Acquisition and Display: Technology, Perception and Applications.

This Feature Issue of Optics Express is organized in conjunction with the 2022 Optica conference on 3D Image Acquisition and Display: Technology, Perception and Applications which was held in hybrid format from 11 to 15, July 2022 as part of the Imaging and Applied Optics Congress and Optical Sensors and Sensing Congress 2022 in Vancouver, Canada. This Feature Issue presents 31 articles which cover the topics and scope of the 2022 3D Image Acquisition and Display conference. This Introduction provides a summary of these published articles that appear in this Feature Issue.

Resolution limit in opto-digital systems revisited.

The resolution limit achievable with an optical system is a fundamental piece of information when characterizing its performance, mainly in case of microscopy imaging. Usually this information is given in the form of a distance, often expressed in microns, or in the form of a cutoff spatial frequency, often expressed in line pairs per mm. In modern imaging systems, where the final image is collected by pixelated digital cameras, the resolution limit is determined by the performance of both, the optical systems and the digital sensor. Usually, one of these factors is considered to be prevalent over the other for estimating the spatial resolution, leading to the global performance of the imaging system ruled by either the classical Abbe resolution limit, based on physical diffraction, or by the Nyquist resolution limit, based on the digital sensor features. This estimation fails significantly to predict the global performance of opto-digital imaging systems, like 3D microscopes, where none of the factors is negligible. In that case, which indeed is the most common, neither the Abbe formula nor the Nyquist formula provide by themselves a reliable prediction for the resolution limit. This is a serious drawback since systems designers often use those formulae as design input parameters. Aiming to overcome this lack, a simple mathematical expression obtained by finely articulating the Abbe and Nyquist formulas, to easily predict the spatial resolution limit of opto-digital imaging systems, is proposed here. The derived expression is tested experimentally, and shows to be valid in a broad range of opto-digital combinations.

Machine Learning-Based View Synthesis in Fourier Lightfield Microscopy.

Current interest in Fourier lightfield microscopy is increasing, due to its ability to acquire 3D images of thick dynamic samples. This technique is based on simultaneously capturing, in a single shot, and with a monocular setup, a number of orthographic perspective views of 3D microscopic samples. An essential feature of Fourier lightfield microscopy is that the number of acquired views is low, due to the trade-off relationship existing between the number of views and their corresponding lateral resolution. Therefore, it is important to have a tool for the generation of a high number of synthesized view images, without compromising their lateral resolution. In this context we investigate here the use of a neural radiance field view synthesis method, originally developed for its use with macroscopic scenes acquired with a moving (or an array of static) digital camera(s), for its application to the images acquired with a Fourier lightfield microscope. The results obtained and presented in this paper are analyzed in terms of lateral resolution and of continuous and realistic parallax. We show that, in terms of these requirements, the proposed technique works efficiently in the case of the epi-illumination microscopy mode.

Fourier lightfield microscopy: a practical design guide.

In this work, a practical guide for the design of a Fourier lightfield microscope is reported. The fundamentals of the Fourier lightfield are presented and condensed on a set of contour plots from which the user can select the design values of the spatial resolution, the field of view, and the depth of field, as function of the specifications of the hardware of the host microscope. This work guides the reader to select the parameters of the infinity-corrected microscope objective, the optical relay lenses, the aperture stop, the microlens array, and the digital camera. A user-friendly graphic calculator is included to ease the design, even to those who are not familiar with the lightfield technology. The guide is aimed to simplify the design process of a Fourier lightfield microscope, which sometimes could be a daunting task, and in this way, to invite the widespread use of this technology. An example of a design and experimental results on imaging different types of samples is also presented.

Handheld and Cost-Effective Fourier Lightfield Microscope.

In this work, the design, building, and testing of the most portable, easy-to-build, robust, handheld, and cost-effective Fourier Lightfield Microscope (FLMic) to date is reported. The FLMic is built by means of a surveillance camera lens and additional off-the-shelf optical elements, resulting in a cost-effective FLMic exhibiting all the regular sought features in lightfield microscopy, such as refocusing and gathering 3D information of samples by means of a single-shot approach. The proposed FLMic features reduced dimensions and light weight, which, combined with its low cost, turn the presented FLMic into a strong candidate for in-field application where 3D imaging capabilities are pursued. The use of cost-effective optical elements has a relatively low impact on the optical performance, regarding the figures dictated by the theory, while its price can be at least 100 times lower than that of a regular FLMic. The system operability is tested in both bright-field and fluorescent modes by imaging a resolution target, a honeybee wing, and a knot of dyed cotton fibers.

The Lightfield Microscope Eyepiece.

Lightfield microscopy has raised growing interest in the last few years. Its ability to get three-dimensional information about the sample in a single shot makes it suitable for many applications in which time resolution is fundamental. In this paper we present a novel device, which is capable of converting any conventional microscope into a lightfield microscope. Based on the Fourier integral microscope concept, we designed the lightfield microscope eyepiece. This is coupled to the eyepiece port, to let the user exploit all the host microscope's components (objective turret, illumination systems, translation stage, etc.) and get a 3D reconstruction of the sample. After the optical design, a proof-of-concept device was built with off-the-shelf optomechanical components. Here, its optical performances are demonstrated, which show good matching with the theoretical ones. Then, the pictures of different samples taken with the lightfield eyepiece are shown, along with the corresponding reconstructions. We demonstrated the functioning of the lightfield eyepiece and lay the foundation for the development of a commercial device that works with any microscope.

Three-dimensional real-time darkfield imaging through Fourier lightfield microscopy.

We report a protocol that takes advantage of the Fourier lightfield microscopy concept for providing 3D darkfield images of volumetric samples in a single-shot. This microscope takes advantage of the Fourier lightfield configuration, in which a lens array is placed at the Fourier plane of the microscope objective, providing a direct multiplexing of the spatio-angular information of the sample. Using the proper illumination beam, the system collects the light scattered by the sample while the background light is blocked out. This produces a set of orthographic perspective images with shifted spatial-frequency components that can be recombined to produce a 3D darkfield image. Applying the adequate reconstruction algorithm high-contrast darkfield optical sections are calculated in real time. The presented method is applied for fast volumetric reconstructions of unstained 3D samples.

Roadmap on 3D integral imaging: sensing, processing, and display.

This Roadmap article on three-dimensional integral imaging provides an overview of some of the research activities in the field of integral imaging. The article discusses various aspects of the field including sensing of 3D scenes, processing of captured information, and 3D display and visualization of information. The paper consists of a series of 15 sections from the experts presenting various aspects of the field on sensing, processing, displays, augmented reality, microscopy, object recognition, and other applications. Each section represents the vision of its author to describe the progress, potential, vision, and challenging issues in this field.

What about computational super-resolution in fluorescence Fourier light field microscopy?

Recently, Fourier light field microscopy was proposed to overcome the limitations in conventional light field microscopy by placing a micro-lens array at the aperture stop of the microscope objective instead of the image plane. In this way, a collection of orthographic views from different perspectives are directly captured. When inspecting fluorescent samples, the sensitivity and noise of the sensors are a major concern and large sensor pixels are required to cope with low-light conditions, which implies under-sampling issues. In this context, we analyze the sampling patterns in Fourier light field microscopy to understand to what extent computational super-resolution can be triggered during deconvolution in order to improve the resolution of the 3D reconstruction of the imaged data.

Fast and robust phase-shift estimation in two-dimensional structured illumination microscopy.

A method of determining unknown phase-shifts between elementary images in two-dimensional Structured Illumination Microscopy (2D-SIM) is presented. The proposed method is based on the comparison of the peak intensity of spectral components. These components correspond to the inherent structured illumination spectral content and the residual component that appears from wrongly estimated phase-shifts. The estimation of the phase-shifts is carried out by finding the absolute maximum of a function defined as the normalized peak intensity difference in the Fourier domain. This task is performed by an optimization method providing a fast estimation of the phase-shift. The algorithm stability and robustness are tested for various levels of noise and contrasts of the structured illumination pattern. Furthermore, the proposed approach reduces the number of computations compared to other existing techniques. The method is supported by the theoretical calculations and validated by means of simulated and experimental results.

Robust Depth Estimation for Light Field Microscopy.

Light field technologies have seen a rise in recent years and microscopy is a field where such technology has had a deep impact. The possibility to provide spatial and angular information at the same time and in a single shot brings several advantages and allows for new applications. A common goal in these applications is the calculation of a depth map to reconstruct the three-dimensional geometry of the scene. Many approaches are applicable, but most of them cannot achieve high accuracy because of the nature of such images: biological samples are usually poor in features and do not exhibit sharp colors like natural scene. Due to such conditions, standard approaches result in noisy depth maps. In this work, a robust approach is proposed where accurate depth maps can be produced exploiting the information recorded in the light field, in particular, images produced with Fourier integral Microscope. The proposed approach can be divided into three main parts. Initially, it creates two cost volumes using different focal cues, namely correspondences and defocus. Secondly, it applies filtering methods that exploit multi-scale and super-pixels cost aggregation to reduce noise and enhance the accuracy. Finally, it merges the two cost volumes and extracts a depth map through multi-label optimization.

Large Depth-of-Field Integral Microscopy by Use of a Liquid Lens.

Integral microscopy is a 3D imaging technique that permits the recording of spatial and angular information of microscopic samples. From this information it is possible to calculate a collection of orthographic views with full parallax and to refocus computationally, at will, through the 3D specimen. An important drawback of integral microscopy, especially when dealing with thick samples, is the limited depth of field (DOF) of the perspective views. This imposes a significant limitation on the depth range of computationally refocused images. To overcome this problem, we propose here a new method that is based on the insertion, at the pupil plane of the microscope objective, of an electrically controlled liquid lens (LL) whose optical power can be changed by simply tuning the voltage. This new apparatus has the advantage of controlling the axial position of the objective focal plane while keeping constant the essential parameters of the integral microscope, that is, the magnification, the numerical aperture and the amount of parallax. Thus, given a 3D sample, the new microscope can provide a stack of integral images with complementary depth ranges. The fusion of the set of refocused images permits to enlarge the reconstruction range, obtaining images in focus over the whole region.

New Method of Microimages Generation for 3D Display.

In this paper, we propose a new method for the generation of microimages, which processes real 3D scenes captured with any method that permits the extraction of its depth information. The depth map of the scene, together with its color information, is used to create a point cloud. A set of elemental images of this point cloud is captured synthetically and from it the microimages are computed. The main feature of this method is that the reference plane of displayed images can be set at will, while the empty pixels are avoided. Another advantage of the method is that the center point of displayed images and also their scale and field of view can be set. To show the final results, a 3D InI display prototype is implemented through a tablet and a microlens array. We demonstrate that this new technique overcomes the drawbacks of previous similar ones and provides more flexibility setting the characteristics of the final image.

Phase-shifting by means of an electronically tunable lens: quantitative phase imaging of biological specimens with digital holographic microscopy.

The use of an electronically tunable lens (ETL) to produce controlled phase shifts in interferometric arrangements is shown. The performance of the ETL as a phase-shifting device is experimentally validated in phase-shifting digital holographic microscopy. Quantitative phase maps of a section of the thorax of a Drosophila melanogaster fly and of human red blood cells have been obtained using our proposal. The experimental results validate the possibility of using the ETL as a reliable phase-shifter device.

Mapping electron-beam-injected trapped charge with scattering scanning near-field optical microscopy.

Scattering scanning near-field optical microscopy (s-SNOM) has been demonstrated as a valuable tool for mapping the optical and optoelectronic properties of materials with nanoscale resolution. Here we report experimental evidence that trapped electric charges injected by an electron beam at the surface of dielectric samples affect the sample-dipole interaction, which has direct impact on the s-SNOM image content. Nanoscale mapping of the surface trapped charge holds significant potential for the precise tailoring of the electrostatic properties of dielectric and semiconductive samples, such as hydroxyapatite, which has particular importance with respect to biomedical applications. The methodology developed here is highly relevant to semiconductor device fabrication as well.

Free-depths reconstruction with synthetic impulse response in integral imaging.

Integral Imaging provides spatial and angular information of three-dimensional (3D) objects, which can be used both for 3D display and for computational post-processing purposes. In order to recover the depth information from an integral image, several algorithms have been developed. In this paper, we propose a new free depth synthesis and reconstruction method based on the two-dimensional (2D) deconvolution between the integral image and a simplified version of the periodic impulse response function (IRF) of the system. The period of the IRF depends directly on the axial position within the object space. Then, we can retrieve the depth information by performing the deconvolution with computed impulse responses with different periods. In addition, alternative reconstructions can be obtained by deconvolving with non-conventional synthetic impulse responses. Our experiments show the feasibility of the proposed method as well as its potential applications.

Spatial light modulator phase mask implementation of wavefront encoded 3D computational-optical microscopy.

Spatial light modulator (SLM) implementation of wavefront encoding enables various types of engineered point-spread functions (PSFs), including the generalized-cubic and squared-cubic phase mask wavefront encoded (WFE) PSFs, shown to reduce the impact of sample-induced spherical aberration in fluorescence microscopy. This investigation validates dynamic experimental parameter variation of these WFE-PSFs. We find that particular design parameter bounds exist, within which the divergence of computed and experimental WFE-PSFs is of the same order of magnitude as that of computed and experimental conventional PSFs, such that model-based approaches for solving the inverse imaging problem can be applied to a wide range of SLM-WFE systems. Interferometric measurements were obtained to evaluate the SLM implementation of the numeric mask. Agreement between experiment and theory in terms of a wrapped phase, 0-2π, validates the phase mask implementation and allows characterization of the SLM response. These measurements substantiate experimental practice of computational-optical microscope imaging with an SLM-engineered PSF.

Resolution enhancement in integral microscopy by physical interpolation.

Integral-imaging technology has demonstrated its capability for computing depth images from the microimages recorded after a single shot. This capability has been shown in macroscopic imaging and also in microscopy. Despite the possibility of refocusing different planes from one snap-shot is crucial for the study of some biological processes, the main drawback in integral imaging is the substantial reduction of the spatial resolution. In this contribution we report a technique, which permits to increase the two-dimensional spatial resolution of the computed depth images in integral microscopy by a factor of √2. This is made by a double-shot approach, carried out by means of a rotating glass plate, which shifts the microimages in the sensor plane. We experimentally validate the resolution enhancement as well as we show the benefit of applying the technique to biological specimens.

Physical compensation of phase curvature in digital holographic microscopy by use of programmable liquid lens.

Quantitative phase measurements obtained with digital holographic microscopes are strongly dependent on the optical arrangement of the imaging system. The nontelecentric operation provides phase measurements affected by a parabolic phase factor and requires numerical postprocessing, which does not always remove all the perturbation. Accurate phase measurements are achieved by using the imaging system in telecentric mode. Unfortunately, this condition is not accomplished when a commercial microscope is used as the imaging system. In this paper, we present an approach for obtaining accurate phase measurements in nontelecentric imaging systems without the need for numerical postprocessing. The method uses an electrically tunable liquid lens to illuminate the sample so that the perturbing parabolic wavefront is cancelled out. Experimental holograms of a Fresnel lens and a section of the thorax of a Drosophila melanogaster fly are captured to verify the proposed method.

Three-dimensional display by smart pseudoscopic-to-orthoscopic conversion with tunable focus.

The original aim of the integral-imaging concept, reported by Gabriel Lippmann more than a century ago, is the capture of images of 3D scenes for their projection onto an autostereoscopic display. In this paper we report a new algorithm for the efficient generation of microimages for their direct projection onto an integral-imaging monitor. Like our previous algorithm, the smart pseudoscopic-to-orthoscopic conversion (SPOC) algorithm, this algorithm produces microimages ready to produce 3D display with full parallax. However, this new algorithm is much simpler than the previous one, produces microimages free of black pixels, and permits fixing at will, between certain limits, the reference plane and the field of view of the displayed 3D scene. Proofs of concept are illustrated with 3D capture and 3D display experiments.