Neural étendue expander for ultra-wide-angle high-fidelity holographic display.

Holographic displays can generate light fields by dynamically modulating the wavefront of a coherent beam of light using a spatial light modulator, promising rich virtual and augmented reality applications. However, the limited spatial resolution of existing dynamic spatial light modulators imposes a tight bound on the diffraction angle. As a result, modern holographic displays possess low étendue, which is the product of the display area and the maximum solid angle of diffracted light. The low étendue forces a sacrifice of either the field-of-view (FOV) or the display size. In this work, we lift this limitation by presenting neural étendue expanders. This new breed of optical elements, which is learned from a natural image dataset, enables higher diffraction angles for ultra-wide FOV while maintaining both a compact form factor and the fidelity of displayed contents to human viewers. With neural étendue expanders, we experimentally achieve 64 × étendue expansion of natural images in full color, expanding the FOV by an order of magnitude horizontally and vertically, with high-fidelity reconstruction quality (measured in PSNR) over 29 dB on retinal-resolution images.

Additive fabrication of SiO2-based micro-optics with lag-free depth and reduced roughness.

Ultra-thin optical components with high design flexibility are required for various applications in today's optical and imaging systems, and this is why the use of diffractive optical elements (DOEs) is rapidly increasing. They can be used for multiple optical systems because of their compact size, increased design flexibility, and ease of mass production. Unfortunately, most existing DOEs are fabricated using conventional etching-based methods, resulting in high surface roughness and aspect ratio-dependent etching rate. Furthermore, when small feature size and large feature size patterns co-exist in the same DOE design, the etching depth differs significantly in the same design, called reactive-ion etching (RIE) lag. All these artifacts lead to a reduction in the diffraction efficiency of DOEs. To overcome the drawbacks of etching-based fabrication methods, we propose an alternative method for fabricating DOEs without RIE lag and with improved surface smoothness. The method consists of additively growing multilevel microstructures of SiO2 material deposited by the plasma-enhanced chemical vapor deposition (PECVD) method onto the substrate followed by liftoff. We demonstrate the effectiveness of the fabrication methods with representative DOEs for imaging and laser beam shaping applications.

Adaptive differentiable grids for cryo-electron tomography reconstruction and denoising.

Motivation: Tilt-series cryo-electron tomography is a powerful tool widely used in structural biology to study 3D structures of micro-organisms, macromolecular complexes, etc. Still, the reconstruction process remains an arduous task due to several challenges: The missing-wedge acquisition, sample misalignment and motion, the need to process large data, and, especially, a low signal-to-noise ratio. Results: Inspired by the recently introduced neural representations, we propose an adaptive learning-based representation of the density field of the captured sample. This representation consists of an octree structure, where each node represents a 3D density grid optimized from the captured projections during the training process. This optimization is performed using a loss that combines a differentiable image formation model with different regularization terms: total variation, boundary consistency, and a cross-nodes non-local constraint. The final reconstruction is obtained by interpolating the learned density grid at the desired voxel positions. The evaluation of our approach using captured data of viruses and cells shows that our proposed representation is well adapted to handle missing wedges, and improves the signal-to-noise ratio of the reconstructed tomogram. The reconstruction quality is highly improved in comparison to the state-of-the-art methods, while using the lowest computing time footprint. Availability and implementation: The code is available on Github at https://github.com/yuanhaowang1213/adaptivediffgrid_ex.

Aberration-Aware Depth-From-Focus.

Computer vision methods for depth estimation usually use simple camera models with idealized optics. For modern machine learning approaches, this creates an issue when attempting to train deep networks with simulated data, especially for focus-sensitive tasks like Depth-from-Focus. In this work, we investigate the domain gap caused by off-axis aberrations that will affect the decision of the best-focused frame in a focal stack. We then explore bridging this domain gap through aberration-aware training (AAT). Our approach involves a lightweight network that models lens aberrations at different positions and focus distances, which is then integrated into the conventional network training pipeline. We evaluate the generality of network models on both synthetic and real-world data. The experimental results demonstrate that the proposed AAT scheme can improve depth estimation accuracy without fine-tuning the model for different datasets. The code will be available in github.com/vccimaging/Aberration-Aware-Depth-from-Focus.

Hexagonal diffractive optical elements.

Diffractive optical elements (DOEs) have widespread applications in optics, ranging from point spread function engineering to holographic display. Conventionally, DOE design relies on Cartesian simulation grids, resulting in square features in the final design. Unfortunately, Cartesian grids provide an anisotropic sampling of the plane, and the resulting square features can be challenging to fabricate with high fidelity using methods such as photolithography. To address these limitations, we explore the use of hexagonal grids as a new grid structure for DOE design and fabrication. In this study, we demonstrate wave propagation simulation using an efficient hexagonal coordinate system and compare simulation accuracy with the standard Cartesian sampling scheme. Additionally, we have implemented algorithms for the inverse DOE design. The resulting hexagonal DOEs, encoded with wavefront information for holograms, are fabricated and experimentally compared to their Cartesian counterparts. Our findings indicate that employing hexagonal grids enhances holographic imaging quality. The exploration of new grid structures holds significant potential for advancing optical technology across various domains, including imaging, microscopy, photography, lighting, and virtual reality.

Diffractive lensless imaging with optimized Voronoi-Fresnel phase.

Lensless cameras are a class of imaging devices that shrink the physical dimensions to the very close vicinity of the image sensor by replacing conventional compound lenses with integrated flat optics and computational algorithms. Here we report a diffractive lensless camera with spatially-coded Voronoi-Fresnel phase to achieve superior image quality. We propose a design principle of maximizing the acquired information in optics to facilitate the computational reconstruction. By introducing an easy-to-optimize Fourier domain metric, Modulation Transfer Function volume (MTFv), which is related to the Strehl ratio, we devise an optimization framework to guide the optimization of the diffractive optical element. The resulting Voronoi-Fresnel phase features an irregular array of quasi-Centroidal Voronoi cells containing a base first-order Fresnel phase function. We demonstrate and verify the imaging performance for photography applications with a prototype Voronoi-Fresnel lensless camera on a 1.6-megapixel image sensor in various illumination conditions. Results show that the proposed design outperforms existing lensless cameras, and could benefit the development of compact imaging systems that work in extreme physical conditions.

HTRSD: Hybrid Taylor Rayleigh-Sommerfeld diffraction.

Computing wave propagation is of the utmost importance in computational optics, especially three-dimensional optical imaging and computer-generated hologram. The angular spectrum method, based on fast Fourier transforms, is one of the efficient approaches; however, it induces sampling issues. We report a Hybrid Taylor Rayleigh-Sommerfeld diffraction (HTRSD) that achieves more accurate and faster wave propagation than the widely used angular spectrum method.

ChromoEnhancer: An Artificial-Intelligence-Based Tool to Enhance Neoplastic Karyograms as an Aid for Effective Analysis.

Cytogenetics laboratory tests are among the most important procedures for the diagnosis of genetic diseases, especially in the area of hematological malignancies. Manual chromosomal karyotyping methods are time consuming and labor intensive and, hence, expensive. Therefore, to alleviate the process of analysis, several attempts have been made to enhance karyograms. The current chromosomal image enhancement is based on classical image processing. This approach has its limitations, one of which is that it has a mandatory application to all chromosomes, where customized application to each chromosome is ideal. Moreover, each chromosome needs a different level of enhancement, depending on whether a given area is from the chromosome itself or it is just an artifact from staining. The analysis of poor-quality karyograms, which is a difficulty faced often in preparations from cancer samples, is time consuming and might result in missing the abnormality or difficulty in reporting the exact breakpoint within the chromosome. We developed ChromoEnhancer, a novel artificial-intelligence-based method to enhance neoplastic karyogram images. The method is based on Generative Adversarial Networks (GANs) with a data-centric approach. GANs are known for the conversion of one image domain to another. We used GANs to convert poor-quality karyograms into good-quality images. Our method of karyogram enhancement led to robust routine cytogenetic analysis and, therefore, to accurate detection of cryptic chromosomal abnormalities. To evaluate ChromoEnahancer, we randomly assigned a subset of the enhanced images and their corresponding original (unenhanced) images to two independent cytogeneticists to measure the karyogram quality and the elapsed time to complete the analysis, using four rating criteria, each scaled from 1 to 5. Furthermore, we compared the enhanced images with our method to the original ones, using quantitative measures (PSNR and SSIM metrics).

GPU Accelerated 3D Tomographic Reconstruction and Visualization from Noisy Electron Microscopy Tilt-Series.

We present a novel framework for 3D tomographic reconstruction and visualization of tomograms from noisy electron microscopy tilt-series. Our technique takes as an input aligned tilt-series from cryogenic electron microscopy and creates denoised 3D tomograms using a proximal jointly-optimized approach that iteratively performs reconstruction and denoising, relieving the users of the need to select appropriate denoising algorithms in the pre-reconstruction or post-reconstruction steps. The whole process is accelerated by exploiting parallelism on modern GPUs, and the results can be visualized immediately after the reconstruction using volume rendering tools incorporated in the framework. We show that our technique can be used with multiple combinations of reconstruction algorithms and regularizers, thanks to the flexibility provided by proximal algorithms. Additionally, the reconstruction framework is open-source and can be easily extended with additional reconstruction and denoising methods. Furthermore, our approach enables visualization of reconstruction error throughout the iterative process within the reconstructed tomogram and on projection planes of the input tilt-series. We evaluate our approach in comparison with state-of-the-art approaches and additionally show how our error visualization can be used for reconstruction evaluation.

Etch-free additive lithographic fabrication methods for reflective and transmissive micro-optics.

With the widespread application of micro-optics in a large range of areas, versatile high quality fabrication methods for diffractive optical elements (DOEs) have always been desired by both the research community and by industry. Traditionally, multi-level DOEs are fabricated by a repetitive combination of photolithography and reactive-ion etching (RIE). The optical phase accuracy and micro-surface quality are severely affected by various etching artifacts, e.g., RIE lag, aspect ratio dependent etching rates, and etching artifacts in the RIE steps. Here we propose an alternative way to fabricate DOEs by additively growing multi-level microstructures onto the substrate. Depth accuracy, surface roughness, uniformity and smoothness are easily controlled to high accuracy by a combination of deposition and lift-off, rather than etching. Uniform depths can be realized for both micrometer and millimeter scale features that are simultaneously present in the designs. The grown media can either be used directly as a reflective DOE, or as a master stamp for nanoimprinting refractive designs. We demonstrate the effectiveness of the fabrication methods with representative reflective and transmissive DOEs for imaging and display applications.

Towards self-calibrated lens metrology by differentiable refractive deflectometry.

Deflectometry, as a non-contact, fully optical metrology method, is difficult to apply to refractive elements due to multi-surface entanglement and precise pose alignment. Here, we present a computational self-calibration approach to measure parametric lenses using dual-camera refractive deflectometry, achieved by an accurate, differentiable, and efficient ray tracing framework for modeling the metrology setup, based on which damped least squares is utilized to estimate unknown lens shape and pose parameters. We successfully demonstrate both synthetic and experimental results on singlet lens surface curvature and asphere-freeform metrology in a transmissive setting.

Modeling classical wavefront sensors.

We present an image formation model for deterministic phase retrieval in propagation-based wavefront sensing, unifying analysis for classical wavefront sensors such as Shack-Hartmann (slopes tracking) and curvature sensors (based on Transport-of-Intensity Equation). We show how this model generalizes commonly seen formulas, including Transport-of-Intensity Equation, from small distances and beyond. Using this model, we analyze theoretically achievable lateral wavefront resolution in propagation-based deterministic wavefront sensing. Finally, via a prototype masked wavefront sensor, we show simultaneous bright field and phase imaging numerically recovered in real-time from a single-shot measurement.

Quantitative Phase and Intensity Microscopy Using Snapshot White Light Wavefront Sensing.

Phase imaging techniques are an invaluable tool in microscopy for quickly examining thin transparent specimens. Existing methods are limited to either simple and inexpensive methods that produce only qualitative phase information (e.g. phase contrast microscopy, DIC), or significantly more elaborate and expensive quantitative methods. Here we demonstrate a low-cost, easy to implement microscopy setup for quantitative imaging of phase and bright field amplitude using collimated white light illumination.

Imaging with SPADs and DMDs: Seeing through Diffraction-Photons.

This paper addresses the problem of imaging in the presence of diffraction-photons. Diffraction-photons arise from the low contrast ratio of DMDs (∼1000:1), and very much degrade the quality of images captured by SPAD-based systems. Herein, a joint illumination-deconvolution scheme is designed to overcome diffraction-photons, enabling the acquisition of intensity and depth images. Additionally, a proof-of-concept experiment is conducted to demonstrate the viability of the designed scheme. It is shown that by co-designing the illumination and deconvolution phases of imaging, one can substantially overcome diffraction-photons.

Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification.

Convolutional neural networks (CNNs) excel in a wide variety of computer vision applications, but their high performance also comes at a high computational cost. Despite efforts to increase efficiency both algorithmically and with specialized hardware, it remains difficult to deploy CNNs in embedded systems due to tight power budgets. Here we explore a complementary strategy that incorporates a layer of optical computing prior to electronic computing, improving performance on image classification tasks while adding minimal electronic computational cost or processing time. We propose a design for an optical convolutional layer based on an optimized diffractive optical element and test our design in two simulations: a learned optical correlator and an optoelectronic two-layer CNN. We demonstrate in simulation and with an optical prototype that the classification accuracies of our optical systems rival those of the analogous electronic implementations, while providing substantial savings on computational cost.

Discriminative Transfer Learning for General Image Restoration.

Recently, several discriminative learning approaches have been proposed for effective image restoration, achieving convincing trade-off between image quality and computational efficiency. However, these methods require separate training for each restoration task (e.g., denoising, deblurring, demosaicing) and problem condition (e.g., noise level of input images). This makes it time-consuming and difficult to encompass all tasks and conditions during training. In this paper, we propose a discriminative transfer learning method that incorporates formal proximal optimization and discriminative learning for general image restoration. The method requires a single-pass discriminative training and allows for reuse across various problems and conditions while achieving an efficiency comparable to previous discriminative approaches. Furthermore, after being trained, our model can be easily transferred to new likelihood terms to solve untrained tasks, or be combined with existing priors to further improve image restoration quality.

Statistical model for OCT image denoising.

Optical coherence tomography (OCT) is a non-invasive technique with a large array of applications in clinical imaging and biological tissue visualization. However, the presence of speckle noise affects the analysis of OCT images and their diagnostic utility. In this article, we introduce a new OCT denoising algorithm. The proposed method is founded on a numerical optimization framework based on maximum-a-posteriori estimate of the noise-free OCT image. It combines a novel speckle noise model, derived from local statistics of empirical spectral domain OCT (SD-OCT) data, with a Huber variant of total variation regularization for edge preservation. The proposed approach exhibits satisfying results in terms of speckle noise reduction as well as edge preservation, at reduced computational cost.

Ultra-high resolution coded wavefront sensor.

Wavefront sensors and more general phase retrieval methods have recently attracted a lot of attention in a host of application domains, ranging from astronomy to scientific imaging and microscopy. In this paper, we introduce a new class of sensor, the Coded Wavefront Sensor, which provides high spatio-temporal resolution using a simple masked sensor under white light illumination. Specifically, we demonstrate megapixel spatial resolution and phase accuracy better than 0.1 wavelengths at reconstruction rates of 50 Hz or more, thus opening up many new applications from high-resolution adaptive optics to real-time phase retrieval in microscopy.

The Light Field Attachment: Turning a DSLR into a Light Field Camera Using a Low Budget Camera Ring.

We propose a concept for a lens attachment that turns a standard DSLR camera and lens into a light field camera. The attachment consists of eight low-resolution, low-quality side cameras arranged around the central high-quality SLR lens. Unlike most existing light field camera architectures, this design provides a high-quality 2D image mode, while simultaneously enabling a new high-quality light field mode with a large camera baseline but little added weight, cost, or bulk compared with the base DSLR camera. From an algorithmic point of view, the high-quality light field mode is made possible by a new light field super-resolution method that first improves the spatial resolution and image quality of the side cameras and then interpolates additional views as needed. At the heart of this process is a super-resolution method that we call iterative Patch- And Depth-based Synthesis (iPADS), which combines patch-based and depth-based synthesis in a novel fashion. Experimental results obtained for both real captured data and synthetic data confirm that our method achieves substantial improvements in super-resolution for side-view images as well as the high-quality and view-coherent rendering of dense and high-resolution light fields.

Encoded diffractive optics for full-spectrum computational imaging.

Diffractive optical elements can be realized as ultra-thin plates that offer significantly reduced footprint and weight compared to refractive elements. However, such elements introduce severe chromatic aberrations and are not variable, unless used in combination with other elements in a larger, reconfigurable optical system. We introduce numerically optimized encoded phase masks in which different optical parameters such as focus or zoom can be accessed through changes in the mechanical alignment of a ultra-thin stack of two or more masks. Our encoded diffractive designs are combined with a new computational approach for self-calibrating imaging (blind deconvolution) that can restore high-quality images several orders of magnitude faster than the state of the art without pre-calibration of the optical system. This co-design of optics and computation enables tunable, full-spectrum imaging using thin diffractive optics.