High-Quality 3D Visualization System for Light-Field Microscopy with Fine-Scale Shape Measurement through Accurate 3D Surface Data.

We propose a light-field microscopy display system that provides improved image quality and realistic three-dimensional (3D) measurement information. Our approach acquires both high-resolution two-dimensional (2D) and light-field images of the specimen sequentially. We put forward a matting Laplacian-based depth estimation algorithm to obtain nearly realistic 3D surface data, allowing the calculation of depth data, which is relatively close to the actual surface, and measurement information from the light-field images of specimens. High-reliability area data of the focus measure map and spatial affinity information of the matting Laplacian are used to estimate nearly realistic depths. This process represents a reference value for the light-field microscopy depth range that was not previously available. A 3D model is regenerated by combining the depth data and the high-resolution 2D image. The element image array is rendered through a simplified direction-reversal calculation method, which depends on user interaction from the 3D model and is displayed on the 3D display device. We confirm that the proposed system increases the accuracy of depth estimation and measurement and improves the quality of visualization and 3D display images.

Comprehensive High-Quality Three-Dimensional Display System Based on a Simplified Light-Field Image Acquisition Method and a Full-Connected Deep Neural Network.

We propose a high-quality, three-dimensional display system based on a simplified light field image acquisition method, and a custom-trained full-connected deep neural network is proposed. The ultimate goal of the proposed system is to acquire and reconstruct the light field images with possibly the most elevated quality from the real-world objects in a general environment. A simplified light field image acquisition method acquires the three-dimensional information of natural objects in a simple way, with high-resolution/high-quality like multicamera-based methods. We trained a full-connected deep neural network model to output desired viewpoints of the object with the same quality. The custom-trained instant neural graphics primitives model with hash encoding output the overall desired viewpoints of the object within the acquired viewing angle in the same quality, based on the input perspectives, according to the pixel density of a display device and lens array specifications within the significantly short processing time. Finally, the elemental image array was rendered through the pixel re-arrangement from the entire viewpoints to visualize the entire field-of-view and re-constructed as a high-quality three-dimensional visualization on the integral imaging display. The system was implemented successfully, and the displayed visualizations and corresponding evaluated results confirmed that the proposed system offers a simple and effective way to acquire light field images from real objects with high-resolution and present high-quality three-dimensional visualization on the integral imaging display system.

High-quality 3D display system for an integral imaging microscope using a simplified direction-inversed computation based on user interaction.

We propose and implement a high-quality three-dimensional (3D) display system for an integral imaging microscope using a simplified direction-inversed computation method based on user interaction. A model of the specimen is generated from the estimated depth information (via the convolutional neural network-based algorithm), the quality of the model is defined by the high-resolution two-dimensional image. The new elemental image arrays are generated from the models via a simplified direction-inversed computation method according to the user interaction and directly displayed on the display device. A high-quality 3D visualization of the specimen is reconstructed and displayed while the lens array is placed in front of the display device. The user interaction enables more viewpoints of the specimen to be reconstructed by the proposed system, within the basic viewing zone. Remarkable quality improvement is confirmed through quantitative evaluations of the experimental results.

Three-dimensional see-through augmented-reality display system using a holographic micromirror array.

It is difficult to find the micromirror array with desired specifications for augmented-reality displays, and the custom fabricating methods are complicated and unstable. We propose a novel, to our knowledge, three-dimensional see-through augmented-reality display system using the holographic micromirror array. Unlike the conventional holographic waveguide-type augmented-reality displays, the proposed system utilizes the holographic micromirror array as an in-coupler, without any additional elements. The holographic micromirror array is fabricated through the simple, effective, and stable method of applying the total internal reflection-based hologram recording using a dual-prism. The optical mirror and microlens array are set as references, and the specifications can be customized. It reconstructs a three-dimensional image from a displayed elemental image set without using any additional device, and the user can observe a three-dimensional virtual image while viewing the real-world objects. Thus, the principal advantages of the existing holographic waveguide-type augmented-reality system are retained. An optical experiment confirmed that the proposed system displays three-dimensional images exploiting the augmented-reality system simply and effectively.

Simplified digital content generation based on an inverse-directed propagation algorithm for holographic stereogram printing.

Holographic stereogram (HS) printing requires extensive memory capacity and long computation time during perspective acquisition and implementation of the pixel re-arrangement algorithm. Hogels contain very weak depth information of the object. We propose a HS printing system that uses simplified digital content generation based on the inverse-directed propagation (IDP) algorithm for hogel generation. Specifically, the IDP algorithm generates an array of hogels using a simple process that acquires the full three-dimensional (3D) information of the object, including parallax, depth, color, and shading, via a computer-generated integral imaging technique. This technique requires a short computation time and is capable of accounting for occlusion and accommodation effects of the object points via the IDP algorithm. Parallel computing is utilized to produce a high-resolution hologram based on the properties of independent hogels. To demonstrate the proposed approach, optical experiments are conducted in which the natural 3D visualizations of real and virtual objects are printed on holographic material. Experimental results demonstrate the simplified computation involved in content generation using the proposed IDP-based HS printing system and the improved image quality of the holograms.

Super-Resolution Enhancement Method Based on Generative Adversarial Network for Integral Imaging Microscopy.

The integral imaging microscopy system provides a three-dimensional visualization of a microscopic object. However, it has a low-resolution problem due to the fundamental limitation of the F-number (the aperture stops) by using micro lens array (MLA) and a poor illumination environment. In this paper, a generative adversarial network (GAN)-based super-resolution algorithm is proposed to enhance the resolution where the directional view image is directly fed as input. In a GAN network, the generator regresses the high-resolution output from the low-resolution input image, whereas the discriminator distinguishes between the original and generated image. In the generator part, we use consecutive residual blocks with the content loss to retrieve the photo-realistic original image. It can restore the edges and enhance the resolution by ×2, ×4, and even ×8 times without seriously hampering the image quality. The model is tested with a variety of low-resolution microscopic sample images and successfully generates high-resolution directional view images with better illumination. The quantitative analysis shows that the proposed model performs better for microscopic images than the existing algorithms.

Improving the quality of full-color holographic three-dimensional displays using depth-related multiple wavefront recording planes with uniform active areas.

In this paper, a depth-related uniform multiple wavefront recording plane (UM-WRP) method is proposed for enhancing the image quality of point cloud-based holograms. Conventional multiple WRP methods, based on full-color computer-generated holograms, experience a color uniformity problem caused by intensity distributions. To solve this problem, the proposed method generates depth-related WRPs to enhance color uniformity, thereby accelerating hologram generation using a uniform active area. The aim is to calculate depth-related WRPs with designed active area sizes that then propagate to the hologram. Compared with conventional multiple WRP methods, reconstructed images have significantly improved quality, as confirmed by numerical simulations and optical experiments.

Implementation of full-color holographic system using non-uniformly sampled 2D images and compressed point cloud gridding.

A multiple-camera holographic system using non-uniformly sampled 2D images and compressed point cloud gridding (C-PCG) is suggested. High-quality, digital single-lens reflex cameras are used to acquire the depth and color information from real scenes; these are then virtually reconstructed by the uniform point cloud using a non-uniform sampling method. The C-PCG method is proposed to generate efficient depth grids by classifying groups of object points with the same depth values in the red, green, and blue channels. Holograms are obtained by applying fast Fourier transform diffraction calculations to the grids. Compared to wave-front recording plane methods, the quality of the reconstructed images is substantially better, and the computational complexity is dramatically reduced. The feasibility of our method is confirmed both numerically and optically.

Chromatic-dispersion-corrected full-color holographic display using directional-view image scaling method.

A novel directional-view image scaling method that corrects chromatic dispersion and enhances the quality of three-dimensional (3D) images reconstructed by a full-color holographic display system is proposed. When the 3D information of the real scene is acquired through the integral imaging pickup method, the orthographic projection image is reconstructed. Then, each directional-view image is separated and synthesized onto the computer-generated hologram. To correct the chromatic dispersion of the full-color holographic 3D display, each directional-view image is scaled depending on the relation between the different wavelengths of single-channel holograms and resolutions of the sub-holograms. According to the optical experimental results, it can be concluded that the proposed method is an effective way of producing full-color holographic images from an orthographic projection image through a simple process.

Multiple-camera holographic system featuring efficient depth grids for representation of real 3D objects.

Recently, computer-generated holograms (CGHs) of real three-dimensional (3D) objects have become widely used to support holographic displays. Here, a multiple-camera holographic system featuring an efficient depth grid is developed to provide the correct depth cue. Multidepth cameras are used to acquire depth and color information from real scenes, and then to virtually reconstruct point cloud models. Arranging the depth cameras in an inward-facing configuration allowed simultaneous capture of objects from different directions, facilitating rendering of the entire surface. The multiple relocated point cloud gridding method is proposed to generate efficient depth grids by classifying groups of object points with the same depth values in the red, green, and blue channels. CGHs are obtained by applying a fast Fourier transform diffraction calculation to the grids. Full-color reconstructed images were obtained flexibly and efficiently. The utility of our method was confirmed both numerically and optically.

Complex object wave extraction using time-multiplexing in off-axis digital holography.

In this paper, we present an algorithm for complex object wave extraction in off-axis digital holography using a time-multiplexing and frequency spectrum-shifting technique. The proposed approach utilizes the digital time-multiplexing technique, in which two 90-deg-rotated off-axis holograms are recorded in sequence, and corresponding spectra are subtracted in the computed Fourier domain to eliminate the DC term. Then, the two subtracted holograms are digitally multiplexed into one complex hologram in the same plane, and by shifting the spatial frequency spectra of the subtracted hologram in the spatial frequency domain, one of the two cross-correlations can be obtained in the center. This technique simply extracts the spectrum of the real image in the frequency domain than the spatial filtering method.

Quality enhancement and GPU acceleration for a full-color holographic system using a relocated point cloud gridding method.

The calculation of realistic full-color holographic displays is hindered by the high computational cost. Previously, we suggested a point cloud gridding (PCG) method to calculate monochrome holograms of real objects. In this research, a relocated point cloud gridding (R-PCG) method is proposed to enhance the reconstruction quality and accelerate the calculation speed in GPU for a full-color holographic system. We use a depth camera to acquire depth and color information from the real scene then reconstruct the point cloud model virtually. The R-PCG method allows us to classify groups of object points with the same depth values into grids in the red, green, and blue (RGB) channels. Computer-generated holograms (CGHs) are obtained by applying a fast Fourier transform (FFT) diffraction calculation to the grids. The feasibility of the R-PCG method is confirmed by numerical and optical reconstruction.

Enhancement of the depth-of-field of integral imaging microscope by using switchable bifocal liquid-crystalline polymer micro lens array.

An integral imaging microscopy (IIM) system with improved depth-of-field (DoF) using a custom-designed bifocal polarization-dependent liquid-crystalline polymer micro lens array (LCP-MLA) is proposed. The implemented MLA has improved electro-optical properties such as a small focal ratio, high fill factor, low driving voltage, and fast switching speed, utilizing a well-aligned reactive mesogen on the imprinted reverse shape of the lens and a polarization switching layer. A bifocal MLA switches its focal length according to the polarization angle and acquires different DoF information of the specimen. After two elemental image arrays are captured, the depth-slices are reconstructed and combined to provide a widened DoF. The fabricated bifocal MLA consists of two identical polarization-dependent LCP-MLAs with 1.6 mm and f/16 focal ratio. Our experimental results confirmed that the proposed system improves the DoF of IIM without the need for mechanical manipulation.

Enhanced depth-of-field of an integral imaging microscope using a bifocal holographic optical element-micro lens array.

We propose and implement an integral imaging microscope with extended depth-of-field (DoF) using a bifocal holographic micro lens array (MLA). The properties of the two MLAs are switched via peristrophic multiplexing, where different properties of the MLA are recorded onto the single holographic optical element (HOE). The recorded MLA properties are perpendicular to each other: after the first mode is recorded, the HOE is rotated by 90° clockwise, and the second mode is recorded. The experimental results confirm that the DoF of the integral imaging microscopy system is extended successfully by using the bifocal MLA.

Three-dimensional image acquisition and reconstruction system on a mobile device based on computer-generated integral imaging.

A mobile three-dimensional image acquisition and reconstruction system using a computer-generated integral imaging technique is proposed. A depth camera connected to the mobile device acquires the color and depth data of a real object simultaneously, and an elemental image array is generated based on the original three-dimensional information for the object, with lens array specifications input into the mobile device. The three-dimensional visualization of the real object is reconstructed on the mobile display through optical or digital reconstruction methods. The proposed system is implemented successfully and the experimental results certify that the system is an effective and interesting method of displaying real three-dimensional content on a mobile device.

Integral imaging microscopy with enhanced depth-of-field using a spatial multiplexing.

A depth-of-field enhancement method for integral imaging microscopy system using a spatial multiplexing structure consisting of a beamsplitter with dual video channels and micro lens arrays is proposed. A computational integral imaging reconstruction algorithm generates two sets of depth-sliced images for the acquired depth information of the captured elemental image arrays and the well-focused depth-slices of both image sets are combined where each is focused on a different depth plane of the specimen. A prototype is implemented, and the experimental results demonstrate that the depth-of-field of the reconstructed images in the proposed integral imaging microscopy is significantly increased compared with conventional integral imaging microscopy systems.

Integral imaging system using an adaptive lens array.

We produced an adaptive lens array composed of multiple flat lens arrays arranged in a curved shape with an adjustable radius of curvature, in order to overcome the hardware problem of the conventional flat or curved lens array-based systems. The manufactured adaptive lens array is applied to an integral imaging system. The gap mismatch that occurs when using a curved lens array is resolved by computing the exact display mapping position of element images through each lens. The results of the experiment demonstrate that the adaptive lens array-based integral imaging system successfully generated elemental images according to the curvature transformation of the adaptive lens array, and they were reconstructed as 3D images.

Vertical viewing angle enhancement for the 360 degree integral-floating display using an anamorphic optic system.

We propose a 360 degree integral-floating display with an enhanced vertical viewing angle. The system projects two-dimensional elemental image arrays via a high-speed digital micromirror device projector and reconstructs them into 3D perspectives with a lens array. Double floating lenses relate initial 3D perspectives to the center of a vertically curved convex mirror. The anamorphic optic system tailors the initial 3D perspectives horizontally and vertically disperse light rays more widely. By the proposed method, the entire 3D image provides both monocular and binocular depth cues, a full-parallax demonstration with high-angular ray density and an enhanced vertical viewing angle.

Real-time interactive display for integral imaging microscopy.

A real-time interactive orthographic-view image display of integral imaging (II) microscopy that includes the generation of intermediate-view elemental images (IVEIs) for resolution enhancement is proposed. Unlike the conventional II microscopes, parallel processing through a graphics processing unit is required for real-time display that generates the IVEIs and interactive orthographic-view images in high speed, according to the user interactive input. The real-time directional-view display for the specimen for which 3D information is acquired through II microscopy is successfully demonstrated by using resolution-enhanced elemental image arrays. A user interactive feature is also satisfied in the proposed real-time interactive display for II microscopy.

Real-time 3D display system based on computer-generated integral imaging technique using enhanced ISPP for hexagonal lens array.

This paper proposes an open computer language (OpenCL) parallel processing method to generate the elemental image arrays (EIAs) for hexagonal lens array from a three-dimensional (3D) object such as a volume data. Hexagonal lens array has a higher fill factor compared to the rectangular lens array case; however, each pixel of an elemental image should be determined to belong to the single hexagonal lens. Therefore, generation for the entire EIA requires very large computations. The proposed method reduces processing time for the EIAs for a given hexagonal lens array. By using the proposed image space parallel processing (ISPP) method, it can enhance the processing speed that generates the 3D display of real-time interactive integral imaging for hexagonal lens array. In our experiment, we implemented the EIAs for hexagonal lens array in real-time and obtained a good processing time for a large of volume data for multiple cases of lens arrays.