Freeform metasurface color router for deep submicron pixel image sensors.

Advances in imaging technologies have led to a high demand for ultracompact, high-resolution image sensors. However, color filter-based image sensors, now miniaturized to deep submicron pixel sizes, face challenges such as low signal-to-noise ratio due to fewer photons per pixel and inherent efficiency limitations from color filter arrays. Here, we demonstrate a freeform metasurface color router that achieves ultracompact pixel sizes while overcoming the efficiency limitations of conventional architectures by splitting and focusing visible light instead of filtering. This development is enabled by a fully differentiable topology optimization framework to maximize the use of the design space while ensuring fabrication feasibility and robustness to fabrication errors. The metasurface can distribute an average of 85% of incident visible light according to the Bayer pattern with a pixel size of 0.6 µm. The device and design methodology enable the compact, high-sensitivity, and high-resolution image sensors for various modern technologies and pave the way for the advanced photonic device design.

HoloSR: deep learning-based super-resolution for real-time high-resolution computer-generated holograms.

This study presents HoloSR, a novel deep learning-based super-resolution approach designed to produce high-resolution computer-generated holograms from low-resolution RGBD images, enabling the real-time production of realistic three-dimensional images. The HoloSR combines the enhanced deep super-resolution network with resize and convolution layers, facilitating the direct generation of high-resolution computer-generated holograms without requiring additional interpolation. Various upscaling scales, extending up to ×4, are evaluated to assess the performance of our method. Quantitative metrics such as structural similarity and peak signal-to-noise ratio are employed to measure the quality of the reconstructed images. Our simulation and experimental results demonstrate that HoloSR successfully achieves super-resolution by generating high-resolution holograms from low-resolution RGBD inputs with supervised and unsupervised learning.

Waveguide holography for 3D augmented reality glasses.

Near-eye displays are fundamental technology in the next generation computing platforms for augmented reality and virtual reality. However, there are remaining challenges to deliver immersive and comfortable visual experiences to users, such as compact form factor, solving vergence-accommodation conflict, and achieving a high resolution with a large eyebox. Here we show a compact holographic near-eye display concept that combines the advantages of waveguide displays and holographic displays to overcome the challenges towards true 3D holographic augmented reality glasses. By modeling the coherent light interactions and propagation via the waveguide combiner, we demonstrate controlling the output wavefront using a spatial light modulator located at the input coupler side. The proposed method enables 3D holographic displays via exit-pupil expanding waveguide combiners, providing a large software-steerable eyebox. It also offers additional advantages such as resolution enhancement capability by suppressing phase discontinuities caused by pupil replication process. We build prototypes to verify the concept with experimental results and conclude the paper with discussion.

Scalable manufacturing of high-index atomic layer-polymer hybrid metasurfaces for metaphotonics in the visible.

Metalenses are attractive alternatives to conventional bulky refractive lenses owing to their superior light-modulating performance and sub-micrometre-scale thicknesses; however, limitations in existing fabrication techniques, including high cost, low throughput and small patterning area, have hindered their mass production. Here we demonstrate low-cost and high-throughput mass production of large-aperture visible metalenses using deep-ultraviolet argon fluoride immersion lithography and wafer-scale nanoimprint lithography. Once a 12″ master stamp is imprinted, hundreds of centimetre-scale metalenses can be fabricated using a thinly coated high-index film to enhance light confinement, resulting in a substantial increase in conversion efficiency. As a proof of concept, an ultrathin virtual reality device created with the printed metalens demonstrates its potential towards the scalable manufacturing of metaphotonic devices.

Accelerating a spatially varying aberration correction of holographic displays with low-rank approximation.

Correction of spatially varying aberrations in holographic displays often requires intractable computational loads. In this Letter, we introduce a low-rank approximation method that decomposes sub-holograms into a small number of modes, thereby reformulating the computer-generated hologram calculation into a summation of a few convolutions. The low-rank approximation is carried out with two different algorithms: the Karhunen-Loeve transform as the optimum solution with respect to the mean-squared error criterion and a novel, to the best of our knowledge, optimization method to provide uniform image quality over the entire field of view. The proposed method is two orders of magnitude faster than the conventional point-wise integration method in our experimental setup, with comparable image quality.

Multi-illumination 3D holographic display using a binary mask.

We introduce a novel, to the best of our knowledge, method to increase the bandwidth in holographic displays. Here, multi-angle illumination using multiple laser diodes (LDs) is adopted to expand the limited diffraction angle of the spatial light modulator (SLM). To solve the problem of signal repetitions caused by sharing the same SLM pattern, we use a random binary mask (BM). We demonstrate via simulations and experiments that our method effectively increases the bandwidth with sufficient image quality. Furthermore, the speckle noise, a critical issue of the holographic display that decreases the contrast and is potentially harmful to eyes, is reduced by the advantage of incoherent summation in the reconstruction plane. We believe that this method is a practical approach that can expand the bandwidth of the holographic display by alleviating the bottleneck of hardware limitations.

Design of highly perceptible dual-resonance all-dielectric metasurface colorimetric sensor via deep neural networks.

Colorimetric sensing, which provides effective detection of bio-molecular signals with one's naked eye, is an exceptionally promising sensing technique in that it enables convenient detection and simplification of entire sensing system. Though colorimetric sensors based on all-dielectric nanostructures have potential to exhibit distinct color variations enabling manageable detection due to their trivial intrinsic loss, there is crucial limitation that the sensitivity to environmental changes lags behind their plasmonic counterparts because of relatively small region of near field-analyte interaction of the dielectric Mie-type resonator. To overcome this challenge, we proposed all-dielectric metasurface colorimetric sensor which exhibits dual-resonance in the visible region. Thereafter, we confirmed with simulation that, in the elaborately designed dual-Lorentzian-type spectra, highly perceptible variations of structural color were manifested even in minute change of peripheral refractive index. In addition to verifying physical effectiveness of the superior colorimetric sensing performance appearing in the dual-resonance type sensor, by combining advanced optimization technique utilizing deep neural networks, we attempted to maximize sensing performance while obtaining dramatic improvement of design efficiency. Through well-trained deep neural network that accurately simulates the input target spectrum, we numerically verified that designed colorimetric sensor shows a remarkable sensing resolution distinguishable up to change of refractive index of 0.0086.

Expanding energy envelope in holographic display via mutually coherent multi-directional illumination.

Holographic display is considered as the most promising three-dimensional (3D) display due to its unique feature of reconstructing arbitrary wavefronts. However, the limited étendue, which hinders the immersive experience of observers, remains a major unresolved issue in holographic display technique. In this paper, we propose a novel approach to tweak the constraints of étendue by expanding the energy envelope in holographic display via mutually coherent multi-illumination. The proposed concept contains both a light source design for generating a mutually coherent multi-directional wave and a computer-generated hologram optimization framework for providing high-resolution 3D holograms. To verify the proposed approach, a benchtop prototype of a holographic near-eye display providing an intrinsic large exit-pupil is implemented. The experimental results clearly show that the exit-pupil is effectively expanded by four times and an appropriate viewpoint image is reconstructed according to the view position.

Color curved hologram calculation method based on angle multiplexing.

In this paper, a method of color curved hologram calculation based on angle multiplexing is proposed. The relationship between the wavelength, center angle and sampling interval of the curved holograms is analyzed for the first time by analyzing the reconstruction process of the curved holograms with different wavelengths. Based on this relationship, the color curved holograms are calculated by compensating phase to the complex amplitude distribution of the planar holograms. To eliminate the chromatic aberration, the curved holograms of different objects with the same color channel are respectively used for angle multiplexing and phase compensation, and then the color composed curved hologram is generated. Different color objects without chromatic aberration can be reconstructed by bending the composed curved hologram into different central angles. The experimental results verify the feasibility of the proposed method. Besides, the application of the proposed method in augmented reality display is also shown.

Learning-based compensation of spatially varying aberrations for holographic display [Invited].

We propose a hologram generation technique to compensate for spatially varying aberrations of holographic displays through machine learning. The image quality of the holographic display is severely degraded when there exist optical aberrations due to misalignment of optical elements or off-axis projection. One of the main advantages of holographic display is that aberrations can be compensated for without additional optical elements. Conventionally, computer-generated holograms for compensation are synthesized through a point-wise integration method, which requires large computational loads. Here, we propose to replace the integration with a combination of fast-Fourier-transform-based convolutions and forward computation of a deep neural network. The point-wise integration method took approximately 95.14 s to generate a hologram of 1024×1024pixels, while the proposed method took about 0.13 s, which corresponds to ×732 computation speed improvement. Furthermore, the aberration compensation by the proposed method is verified through experiments.

High-contrast, speckle-free, true 3D holography via binary CGH optimization.

Holography is a promising approach to implement the three-dimensional (3D) projection beyond the present two-dimensional technology. True 3D holography requires abilities of arbitrary 3D volume projection with high-axial resolution and independent control of all 3D voxels. However, it has been challenging to implement the true 3D holography with high-reconstruction quality due to the speckle. Here, we propose the practical solution to realize speckle-free, high-contrast, true 3D holography by combining random-phase, temporal multiplexing, binary holography, and binary optimization. We adopt the random phase for the true 3D implementation to achieve the maximum axial resolution with fully independent control of the 3D voxels. We develop the high-performance binary hologram optimization framework to minimize the binary quantization noise, which provides accurate and high-contrast reconstructions for 2D as well as 3D cases. Utilizing the fast operation of binary modulation, the full-color high-framerate holographic video projection is realized while the speckle noise of random phase is overcome by temporal multiplexing. Our high-quality true 3D holography is experimentally verified by projecting multiple arbitrary dense images simultaneously. The proposed method can be adopted in various applications of holography, where we show additional demonstration that realistic true 3D hologram in VR and AR near-eye displays. The realization will open a new path towards the next generation of holography.

Off-axis camera-in-the-loop optimization with noise reduction strategy for high-quality hologram generation.

In this Letter, we introduce a noise reduction (NR) strategy in the off-axis camera-in-the-loop (CITL) optimization for high-quality hologram generation. Our proposal adopts the Gaussian blur in the NR strategy to suppress the high-frequency noise and improve the optimization convergence. A double-hologram generation technique is used to reduce the noise further. The off-axis system's aberrations are eliminated by integrating the aberration compensation method as well. Compared with the original CITL method, the image quality of the proposed method is improved by approximately 5.5 dB in the optical experiment.

Volumetric Head-Mounted Display With Locally Adaptive Focal Blocks.

A commercial head-mounted display (HMD) for virtual reality (VR) presents three-dimensional imagery with a fixed focal distance. The VR HMD with a fixed focus can cause visual discomfort to an observer. In this article, we propose a novel design of a compact VR HMD supporting near-correct focus cues over a wide depth of field (from 18 cm to optical infinity). The proposed HMD consists of a low-resolution binary backlight, a liquid crystal display panel, and focus-tunable lenses. In the proposed system, the backlight locally illuminates the display panel that is floated by the focus-tunable lens at a specific distance. The illumination moment and the focus-tunable lens' focal power are synchronized to generate focal blocks at the desired distances. The distance of each focal block is determined by depth information of three-dimensional imagery to provide near-correct focus cues. We evaluate the focus cue fidelity of the proposed system considering the fill factor and resolution of the backlight. Finally, we verify the display performance with experimental results.

Distortion corrected tomographic near-eye displays using light field optimization.

Several multifocal displays have been proposed to provide accurate accommodation cues. However, multifocal displays have an undesirable feature, which is especially emphasized in near-eye displays configuration, that the field of views (FOVs) of the virtual planes change over depth. We demonstrate that this change in FOV causes image distortions, which reduces overall image quality, and depth perception error due to the variation of image sizes according to depths. Here, we introduce a light field optimization technique to compensate for magnification variations among the focal planes. Our approach alleviates image distortions, especially noticeable in the contents with large depth discontinuity, and reconstructs the image size to precise depths, while maintaining a specific tolerance length for the target eye relief. To verify the feasibility of the algorithm, we employ this optimization method for the tomographic near-eye display system to acquire the optimal image and backlight sequences for a volumetric scene. In general, we confirm that the structural similarity index measure of reconstructed images against ground truth increases by 20% when the eye relief is 15 mm, and the accommodation cue is appropriately stimulated at the target depth with our proposed method.

Optimization of computer-generated holograms featuring phase randomness control.

In this Letter, we introduce a computer-generated hologram (CGH) optimization technique that can control the randomness of the reconstructed phase. The phase randomness significantly affects the eyebox size and depth of field in holographic near-eye displays. Our proposal is to synthesize the CGH through the sum of two terms computed from the target scene with a random phase. We set a weighting pattern for summation as the optimization variable, which enables the CGH to reflect the random phase during optimization. We evaluate the proposed algorithm on single-depth and multi-depth contents, and the performance is validated via simulations and experiments.

Vision-correcting holographic display: evaluation of aberration correcting hologram.

Vision-correcting displays are key to achieving physical and physiological comforts to the users with refractive errors. Among such displays are holographic displays, which can provide a high-resolution vision-adaptive solution with complex wavefront modulation. However, none of the existing hologram rendering techniques have considered the optical properties of the human eye nor evaluated the significance of vision correction. Here, we introduce vision-correcting holographic display and hologram acquisition that integrates user-dependent prescriptions and a physical model of the optics, enabling the correction of on-axis and off-axis aberrations. Experimental and empirical evaluations of the vision-correcting holographic displays show the competence of holographic corrections over the conventional vision correction solutions.

Occlusion-capable see-through display without the screen-door effect using a photochromic mask.

Conventional occlusion-capable see-through display systems have many practical limitations such as the form factor, narrow field of view, screen-door effect, and diffraction of a real scene. In this Letter, we propose an occlusion-capable see-through display using lens arrays and a photochromic plate. By imaging the occlusion mask on the photochromic plate with near-UV light, the visible light transmittance of the plate changes. Since no black matrix lies on the photochromic plate, our system provides a clear real scene view without the grid structure of the pixels and can prevent diffraction defects of the real scene. We also alleviate the drawback of a narrow field of view using the lens arrays for a reduced form factor.

Compact tomographic near-eye display using a MEMS scanning mirror.

We propose a compact tomographic near-eye display by combining a micro-electro-mechanical systems (MEMS) scanning mirror device, focus tunable lens, and a single light-emitting diode source. A holographic optical element was used to elaborately focus the light source into the MEMS scanning mirror while providing further miniaturization. We implemented a drastically downsized multifocal display system that expresses a depth range of 4.8 D with eight focal planes by synchronizing these modules. Furthermore, the planar images are optimized to achieve correct retinal scenes at each accommodation state. The simulated and experimental results verify that the suggested near-eye display system can provide three-dimensional virtual images while showing physical feasibility.

Ultrahigh-definition volumetric light field projection.

We introduce a projection-type light field display featuring effective light modulation. By combining a tomographic display with integral imaging (InIm) technology, a novel optical design is capable of an autostereoscopic light field projector. Here, the tomographic approach generates a high-resolution volumetric scene, and InIm makes it possible for the volumetric scene to be reconstructed on a large screen through a projection. Since all the processes are realized optically without digital processing, our system can overcome the performance limitations associated with the number of pixels in the conventional InIm displays. We built a prototype display and demonstrated that our optical design has the potential of massive resolution with a full-parallax in a single device.