Three-dimensional integral imaging in photon-starved environments with high-sensitivity image sensors.

Imaging in poorly illuminated environments using three-dimensional (3D) imaging with passive imaging sensors that operate in the visible spectrum is a formidable task due to the low number of photons detected. 3D integral imaging, which integrates multiple two-dimensional perspectives, is expected to perform well in the presence of noise, as well as statistical fluctuation in the detected number of photons. In this paper, we present an investigation of 3D integral imaging in low-light-level conditions, where as low as a few photons and as high as several tens of photons are detected on average per pixel. In the experimental verification, we use an electron multiplying charge-coupled device (EM-CCD) and a scientific complementary metal-oxide-semiconductor (sCMOS) camera. For the EM-CCD, a theoretical model for the probability distribution of the pixel values is derived, then fitted with the experimental data to determine the camera parameters. Likewise, pixelwise calibration is performed on the sCMOS to determine the camera parameters for further analysis. Theoretical derivation of the expected signal-to-noise-ratio is provided for each image sensor and corroborated by the experimental findings. Further comparison between the cameras includes analysis of the contrast-to-noise ratio (CNR) as well as the perception-based image quality estimator (PIQE). Improvement of image quality metrics in the 3D reconstructed images is successfully confirmed compared with those of the 2D images. To the best of our knowledge, this is the first experimental report of low-light-level 3D integral imaging with as little as a few photons detected per pixel on average to improve scene visualization including occlusion removal from the scene.

Optical sensing and detection in turbid water using multidimensional integral imaging.

We present an approach for optical sensing and detection in turbid water using multidimensional spatial-temporal domain integral imaging and dedicated signal processing algorithms. An optical signal is encoded using pseudorandom sequences, and an image sensor array is used to capture elemental image video sequences of light propagating through turbid water. Using the captured information, multidimensional image reconstruction followed by multi-dimensional correlation to detect the source signal is performed. We experimentally demonstrate scenarios in which turbidity causes conventional signal detection to fail, while our proposed multidimensional approach enables successful detection under the same turbidity conditions. Statistical analysis is provided to support the experimental results. To the best of our knowledge, this is the first report of using multidimensional integral imaging signal detection in turbid water conditions.

Spatial-temporal human gesture recognition under degraded conditions using three-dimensional integral imaging.

We present spatial-temporal human gesture recognition in degraded conditions including low light levels and occlusions using passive sensing three-dimensional (3D) integral imaging (InIm) system and 3D correlation filters. The 4D (lateral, longitudinal, and temporal) reconstructed data is processed using a variety of algorithms including linear and non-linear distortion-invariant filters; and compared with previously reported space-time interest points (STIP) feature detector, 3D histogram of oriented gradients (3D HOG) feature descriptor, with a standard bag-of-features support vector machine (SVM) framework, etc. The gesture recognition results with different classification algorithms are compared using a variety of performance metrics such as receiver operating characteristic (ROC) curves, area under the curve (AUC), SNR, the probability of classification errors, and confusion matrix. Integral imaging video sequences of human gestures are captured under degraded conditions such as low light illumination and in the presence of partial occlusions. A four-dimensional (4D) reconstructed video sequence is computed that provides lateral and depth information of a scene over time i.e. (x, y, z, t). The total-variation denoising algorithm is applied to the signal to further reduce noise and preserve data in the video frames. We show that the 4D signal consists of decreased scene noise, partial occlusion removal, and improved SNR due to the computational InIm and/or denoising algorithms. Finally, gesture recognition is processed with classification algorithms, such as distortion-invariant correlation filters; and STIP, 3D HOG with SVM, which are applied to the reconstructed 4D gesture signal to classify the human gesture. Experiments are conducted using a synthetic aperture InIm system in ambient light. Our experiments indicate that the proposed approach is promising in detection of human gestures in degraded conditions such as low illumination conditions with partial occlusion. To the best of our knowledge, this is the first report on spatial-temporal human gesture recognition in degraded conditions using passive sensing 4D integral imaging with nonlinear correlation filters.

Sickle cell disease diagnosis based on spatio-temporal cell dynamics analysis using 3D printed shearing digital holographic microscopy.

We present a spatio-temporal analysis of cell membrane fluctuations to distinguish healthy patients from patients with sickle cell disease. A video hologram containing either healthy red blood cells (h-RBCs) or sickle cell disease red blood cells (SCD-RBCs) was recorded using a low-cost, compact, 3D printed shearing interferometer. Reconstructions were created for each hologram frame (time steps), forming a spatio-temporal data cube. Features were extracted by computing the standard deviations and the mean of the height fluctuations over time and for every location on the cell membrane, resulting in two-dimensional standard deviation and mean maps, followed by taking the standard deviations of these maps. The optical flow algorithm was used to estimate the apparent motion fields between subsequent frames (reconstructions). The standard deviation of the magnitude of the optical flow vectors across all frames was then computed. In addition, seven morphological cell (spatial) features based on optical path length were extracted from the cells to further improve the classification accuracy. A random forest classifier was trained to perform cell identification to distinguish between SCD-RBCs and h-RBCs. To the best of our knowledge, this is the first report of machine learning assisted cell identification and diagnosis of sickle cell disease based on cell membrane fluctuations and morphology using both spatio-temporal and spatial analysis.

Automatic multicell identification using a compact lensless single and double random phase encoding system.

We investigate the use of compact, lensless, single random phase encoding (SRPE) and double random phase encoding (DRPE) systems for automatic cell identification when multiple cells, either of the same or mixed classes, are in the field of view. A microscope glass slide containing the sample is inputted into the single or double random phase encoding system, which is then illuminated by a coherent or partially coherent light source generating a unique opto-biological signature (OBS) that is captured by an image sensor. Statistical features such as mean, standard deviation, skewness, kurtosis, entropy, and Pearson's correlation coefficient are extracted from the OBSs and used for cell identification with the random forest classifier. With the exception of the correlation coefficient, all features were extracted in both the spatial and frequency domains. Experiments are performed with single random phase encoding and double random phase encoding, and system analysis is presented to show the robustness and classification accuracy of the random phase encoding cell identification systems. The proposed systems are compact, as they are lensless and do not have spatial frequency bandwidth limitations due to the numerical aperture of a microscope objective lens. We demonstrate that cell identification is possible using both the SRPE and DRPE systems. While DRPE systems have been extensively used for image encryption, to the best of our knowledge, this is the first report on using DRPE for automated cell identification.

Automatic cell identification and visualization using digital holographic microscopy with head mounted augmented reality devices.

We propose a compact imaging system that integrates an augmented reality head mounted device with digital holographic microscopy for automated cell identification and visualization. A shearing interferometer is used to produce holograms of biological cells, which are recorded using customized smart glasses containing an external camera. After image acquisition, segmentation is performed to isolate regions of interest containing biological cells in the field-of-view, followed by digital reconstruction of the cells, which is used to generate a three-dimensional (3D) pseudocolor optical path length profile. Morphological features are extracted from the cell's optical path length map, including mean optical path length, coefficient of variation, optical volume, projected area, projected area to optical volume ratio, cell skewness, and cell kurtosis. Classification is performed using the random forest classifier, support vector machines, and K-nearest neighbor, and the results are compared. Finally, the augmented reality device displays the cell's pseudocolor 3D rendering of its optical path length profile, extracted features, and the identified cell's type or class. The proposed system could allow a healthcare worker to quickly visualize cells using augmented reality smart glasses and extract the relevant information for rapid diagnosis. To the best of our knowledge, this is the first report on the integration of digital holographic microscopy with augmented reality devices for automated cell identification and visualization.

Three-dimensional object visualization and detection in low light illumination using integral imaging.

Conventional two-dimensional (2D) imaging systems that operate in the visible spectrum may perform poorly in environments under low light illumination. In this work, we present the potential of passive three-dimensional (3D) integral imaging (II) to perform 3D imaging of a scene under low light conditions in the visible spectrum and without the need for a photon counting or cooled CCD camera. Using dedicated algorithms, we demonstrate that the reconstructed 3D integral image is naturally optimum in a maximum likelihood sense in low light levels and in the presence of detector noise enabling object visualization in the scene. The conventional 2D imaging fails due to the limited number of photons. Using 3D imaging, we demonstrate the potential for 3D detection of objects behind occlusion in a photon-starved scene. To the best of our knowledge, this is the first report of experimentally using II sensing under low illumination conditions for 3D visualization and 3D object detection in the presence of obscurations with a conventional image sensor.

Three-dimensional profilometric reconstruction using flexible sensing integral imaging and occlusion removal.

We present a method for three-dimensional (3D) profilometric reconstruction using flexible sensing integral imaging with object recognition and automatic occlusion removal. Two-dimensional images, known as elemental images (EIs), of a scene containing an object behind occlusion are captured by flexible sensing integral imaging using a moving camera randomly placed on a non-planar surface with unknown camera position and orientation. After 3D image acquisition, the unknown camera poses are estimated using the EIs and 3D reconstruction is performed based on flexible sensing integral imaging. Object recognition using the 3D reconstructed images is conducted to detect the object behind occlusion and estimate the object depth and position. Occlusion removal is then performed on the 2D EIs for the occluded object by computing variance maps of the scene. For each EI, occluded object pixels with low variance are replaced by object pixels from other perspectives using multi-view geometry. The new set of elemental images may be used to visualize the 3D profile of the scene containing the object without occlusion. Experiments are performed to validate the feasibility of the proposed method. To the best of our knowledge, this is the first report of applying flexible sensing integral imaging to profilometric reconstruction with object recognition and occlusion removal.

Three-dimensional integral imaging and object detection using long-wave infrared imaging.

We propose a passive three-dimensional (3D) imaging technique based on integral imaging using a long-wave infrared (LWIR) camera. 3D imaging can improve visualization and detection of objects in adverse environments, such as low light levels and the presence of partial occlusions, along with depth estimation by reconstructing the scene at the plane of the object. This is achieved by capturing multiple two-dimensional images, known as elemental images (EI), of a scene with each image having a unique perspective of the 3D objects. Moreover, LWIR imaging performs well in photon-limited environments due to detection of thermal radiation from an object rather than the reflected light. Once the EIs have been captured, image restoration is performed on the captured images. A 3D scene is then reconstructed and object detection using correlation filters and support vector machines is performed. Our experiments with human face detection show that 2D imaging may fail to detect occluded humans, whereas passive 3D imaging with LWIR could be successful. To the best of our knowledge, this is the first report of passive 3D integral imaging with LWIR for object detection, and in particular, in low light environments.

Compact and field-portable 3D printed shearing digital holographic microscope for automated cell identification.

We propose a low-cost, compact, and field-portable 3D printed holographic microscope for automated cell identification based on a common path shearing interferometer setup. Once a hologram is captured from the portable setup, a 3D reconstructed height profile of the cell is created. We extract several morphological cell features from the reconstructed 3D height profiles, including mean physical cell thickness, coefficient of variation, optical volume (OV) of the cell, projected area of the cell (PA), ratio of PA to OV, cell thickness kurtosis, cell thickness skewness, and the dry mass of the cell for identification using the random forest (RF) classifier. The 3D printed prototype can serve as a low-cost alternative for the developing world, where access to laboratory facilities for disease diagnosis are limited. Additionally, a cell phone sensor is used to capture the digital holograms. This enables the user to send the acquired holograms over the internet to a computational device located remotely for cellular identification and classification (analysis). The 3D printed system presented in this paper can be used as a low-cost, stable, and field-portable digital holographic microscope as well as an automated cell identification system. To the best of our knowledge, this is the first research paper presenting automatic cell identification using a low-cost 3D printed digital holographic microscopy setup based on common path shearing interferometry.

Double-random-phase encryption with photon counting for image authentication using only the amplitude of the encrypted image.

We present a photon-counting double-random-phase encryption technique that only requires the photon-limited amplitude of the encrypted image for decryption. The double-random-phase encryption is used to encrypt an image, generating a complex image. Photon counting is applied to the amplitude of the encrypted image, generating a sparse noise-like image; however, the phase information is not retained. By not using the phase information, the encryption process is simplified, allowing for intensity detection and also less information to be recorded. Using a phase numerically generated from the correct encryption keys together with the photon-limited amplitude of the encrypted image, we are able to decrypt the image. Moreover, nonlinear correlation algorithms can be used to authenticate the decrypted image. Both amplitude-based and full-phase encryption using the proposed method are investigated. Preliminary computational results and performance evaluation are presented.

Security authentication with a three-dimensional optical phase code using random forest classifier.

An object with a unique three-dimensional (3D) optical phase mask attached is analyzed for security and authentication. These 3D optical phase masks are more difficult to duplicate or to have a mathematical formulation compared with 2D masks and thus have improved security capabilities. A quick response code was modulated using a random 3D optical phase mask generating a 3D optical phase code (OPC). Due to the scattering of light through the 3D OPC, a unique speckle pattern based on the materials and structure in the 3D optical phase mask is generated and recorded on a CCD device. Feature extraction is performed by calculating the mean, variance, skewness, kurtosis, and entropy for each recorded speckle pattern. The random forest classifier is used for authentication. Optical experiments demonstrate the feasibility of the authentication scheme.

Cell identification using single beam lensless imaging with pseudo-random phase encoding.

In this Letter, we propose a novel compact optical system for automated cell identification. Our system employs pseudo-random encoding of the light modulated by the cells under inspection to capture the unique opto-biological signature of the micro-organisms by an image sensor and without using a microscope objective lens to magnify the object beam. The proposed instrument can be fabricated using a compact light source, a thin diffuser, and an image sensor connected to computational hardware; thus, it can be compact and cost effective. Experiments are presented using the proposed system to identify and classify various micro-objects and demonstrate proof of concept. The captured opto-biological signature pattern can be attributed to the micro-object's morphology, size, sub-cellular complex structure, index of refraction, internal material composition, etc. Using the captured signature of the micro-object, we extract statistical features such as mean, variance, skewness, kurtosis, entropy, and correlation coefficients for cell identification using the random forest classifier. For comparison, similar identification experiments were repeated with a digital shearing interferometer. To the best of our knowledge, this is the first report on automated cell identification using the proposed approach.

Integrated circuit authentication using photon-limited x-ray microscopy.

A counterfeit integrated circuit (IC) may contain subtle changes to its circuit configuration. These changes may be observed when imaged using an x-ray; however, the energy from the x-ray can potentially damage the IC. We have investigated a technique to authenticate ICs under photon-limited x-ray imaging. We modeled an x-ray image with lower energy by generating a photon-limited image from a real x-ray image using a weighted photon-counting method. We performed feature extraction on the image using the speeded-up robust features (SURF) algorithm. We then authenticated the IC by comparing the SURF features to a database of SURF features from authentic and counterfeit ICs. Our experimental results with real and counterfeit ICs using an x-ray microscope demonstrate that we can correctly authenticate an IC image captured using orders of magnitude lower energy x-rays. To the best of our knowledge, this Letter is the first one on using a photon-counting x-ray imaging model and relevant algorithms to authenticate ICs to prevent potential damage.

Augmented reality three-dimensional object visualization and recognition with axially distributed sensing.

An augmented reality (AR) smartglass display combines real-world scenes with digital information enabling the rapid growth of AR-based applications. We present an augmented reality-based approach for three-dimensional (3D) optical visualization and object recognition using axially distributed sensing (ADS). For object recognition, the 3D scene is reconstructed, and feature extraction is performed by calculating the histogram of oriented gradients (HOG) of a sliding window. A support vector machine (SVM) is then used for classification. Once an object has been identified, the 3D reconstructed scene with the detected object is optically displayed in the smartglasses allowing the user to see the object, remove partial occlusions of the object, and provide critical information about the object such as 3D coordinates, which are not possible with conventional AR devices. To the best of our knowledge, this is the first report on combining axially distributed sensing with 3D object visualization and recognition for applications to augmented reality. The proposed approach can have benefits for many applications, including medical, military, transportation, and manufacturing.

Full-phase photon-counting double-random-phase encryption.

We investigate a full-phase-based photon-counting double-random-phase encryption (PC-DRPE) method. A PC technique is applied during the encryption process, creating sparse images. The statistical distribution of the PC decrypted data for full-phase encoding and amplitude-phase encoding are derived, and their statistical parameters are used for authentication. The performance of the full-phase PC-DRPE is compared with the amplitude-based PC-DRPE method. The PC decrypted images make it difficult to visually authenticate the input image; however, advanced correlation filters can be used to authenticate the decrypted images given the correct keys. Initial computational simulations show that the full-phase PC-DRPE has the potential to require fewer photons for authentication than the amplitude-based PC-DRPE.