Simulating a turbulent video affected by spatiotemporally varying blur and tilt using temporal cross correlation of intermodal Zernike coefficients.

Videos captured in long-distance horizontal imaging through the atmosphere suffer from dynamic spatiotemporal movements and blur caused by the air turbulence. Simulations of atmospheric turbulence in such videos, which have been conducted in the past, are difficult to compute. Our goal in this research is to develop an effective simulation algorithm of videos affected by atmospheric turbulence characterized by spatiotemporally varying blur and tilt, when supplied with a given image. We accomplish this via extending an already established method that simulates atmospheric turbulence in a single image, by incorporating turbulence properties in the time domain that include both the tilts and blurring effects. This study also extends our previous work that simulated turbulence, but did not consider the space-varying property of the blur. This is done by employing the relationship between turbulence image distortions and the intermodal correlations of the Zernike coefficients in time and space, and also via analyzing the spatiotemporal matrix that represents the spatial correlation of movements between different frames. The proposed method can facilitate the production of simulations, given turbulence properties that include turbulence strength, object distance, and height. The simulation is applied to videos with low and high frame rates, and the differences between them are analyzed. The proposed method can prove useful when generating machine-learning algorithms that apply to videos affected by atmospheric turbulence, which require large labeled video datasets (with controlled turbulence and imaging parameters) for training.

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning.

Most human emotion recognition methods largely depend on classifying stereotypical facial expressions that represent emotions. However, such facial expressions do not necessarily correspond to actual emotional states and may correspond to communicative intentions. In other cases, emotions are hidden, cannot be expressed, or may have lower arousal manifested by less pronounced facial expressions, as may occur during passive video viewing. This study improves an emotion classification approach developed in a previous study, which classifies emotions remotely without relying on stereotypical facial expressions or contact-based methods, using short facial video data. In this approach, we desire to remotely sense transdermal cardiovascular spatiotemporal facial patterns associated with different emotional states and analyze this data via machine learning. In this paper, we propose several improvements, which include a better remote heart rate estimation via a preliminary skin segmentation, improvement of the heartbeat peaks and troughs detection process, and obtaining a better emotion classification accuracy by employing an appropriate deep learning classifier using an RGB camera input only with data. We used the dataset obtained in the previous study, which contains facial videos of 110 participants who passively viewed 150 short videos that elicited the following five emotion types: amusement, disgust, fear, sexual arousal, and no emotion, while three cameras with different wavelength sensitivities (visible spectrum, near-infrared, and longwave infrared) recorded them simultaneously. From the short facial videos, we extracted unique high-resolution spatiotemporal, physiologically affected features and examined them as input features with different deep-learning approaches. An EfficientNet-B0 model type was able to classify participants' emotional states with an overall average accuracy of 47.36% using a single input spatiotemporal feature map obtained from a regular RGB camera.

Turbulence video simulation using the spatiotemporal cross correlation of distortion fields.

Previous simulations of atmospheric turbulence in videos are computationally complex. The purpose of this study is to develop an efficient algorithm for simulating spatiotemporal videos affected by atmospheric turbulence, given a static image. We extend an existing method for the simulation of atmospheric turbulence in a single image by incorporating turbulence properties in the time domain and the blurring effect. We accomplish this through analysis of the correlation between turbulence image distortions in time and in space. The significance of this method is the ease with which it will be possible to produce a simulation, given properties of the turbulence (including turbulence strength, object distance, and height). We apply the simulation to low and high frame rate videos, and we show that the spatiotemporal cross correlation of the distortion fields in the simulated video matches the physical spatiotemporal cross correlation function. Such a simulation can be useful when developing algorithms that apply to videos degraded by atmospheric turbulence and require a large amount of imaging data for training.

3D Object Detection via 2D Segmentation-Based Computational Integral Imaging Applied to a Real Video.

This study aims to achieve accurate three-dimensional (3D) localization of multiple objects in a complicated scene using passive imaging. It is challenging, as it requires accurate localization of the objects in all three dimensions given recorded 2D images. An integral imaging system captures the scene from multiple angles and is able to computationally produce blur-based depth information about the objects in the scene. We propose a method to detect and segment objects in a 3D space using integral-imaging data obtained by a video camera array. Using objects' two-dimensional regions detected via deep learning, we employ local computational integral imaging in detected objects' depth tubes to estimate the depth positions of the objects along the viewing axis. This method analyzes object-based blurring characteristics in the 3D environment efficiently. Our camera array produces an array of multiple-view videos of the scene, called elemental videos. Thus, the proposed 3D object detection applied to the video frames allows for 3D tracking of the objects with knowledge of their depth positions along the video. Results show successful 3D object detection with depth localization in a real-life scene based on passive integral imaging. Such outcomes have not been obtained in previous studies using integral imaging; mainly, the proposed method outperforms them in its ability to detect the depth locations of objects that are in close proximity to each other, regardless of the object size. This study may contribute when robust 3D object localization is desired with passive imaging, but it requires a camera or lens array imaging apparatus.

Atmospheric Turbulence Degraded Video Restoration with Recurrent GAN (ATVR-GAN).

Atmospheric turbulence (AT) can change the path and direction of light during video capturing of a target in space due to the random motion of the turbulent medium, a phenomenon that is most noticeable when shooting videos at long ranges, resulting in severe video dynamic distortion and blur. To mitigate geometric distortion and reduce spatially and temporally varying blur, we propose a novel Atmospheric Turbulence Video Restoration Generative Adversarial Network (ATVR-GAN) with a specialized Recurrent Neural Network (RNN) generator, which is trained to predict the scene's turbulent optical flow (OF) field and utilizes a recurrent structure to catch both spatial and temporal dependencies. The new architecture is trained using a newly combined loss function that counts for the spatiotemporal distortions, specifically tailored to the AT problem. Our network was tested on synthetic and real imaging data and compared against leading algorithms in the field of AT mitigation and image restoration. The proposed method outperformed these methods for both synthetic and real data examined.

Fast and Enhanced MMW Imaging System Using a Simple Row Detector Circuit with GDDs as Sensor Elements and an FFT-Based Signal Acquisition System.

The relatively high atmospheric propagation of millimeter-waves (MMW) was found to be one of the most critical reasons for the development of reliable sensors for MMW detection. According to previous research works, it has been already shown that incident MMW radiation on a glow discharge detector (GDD) can increase the discharge current. Hence, the electrical mode of detection can be employed to detect the presence of MMW radiation. In this article, a new design of a row detector using GDDs as pixel elements, and the influence of MMW incidence on GDD's discharge current, were acquired using an elementary data acquisition (DAQ) platform. The DAQ system computes the averaged Fast Fourier Transform (FFT) spectrum of the time signal and returns the FFT results as magnitude based on the level of detection. An FFT-based signal acquisition proved to be a better alternative to the lock-in detection that was commonly used in MMW detection systems. This improved detection circuit provides enhanced noise filtering, thereby resulting in better MMW images within a short time. The overhead expense of the entire system is very low, as it can avoid lock-in amplifier stages that were previously used for signal enhancement. A scanning mechanism using a motorized translation stage (step motor) is involved to place and align the row detector in the image plane. The scanning can be carried out vertically to perform the imaging, by configuring the step motor after selecting the desired step size and position. A simplified version of the MMW detection circuit with a dedicated over-voltage protection facility is presented here. This made the detection system more stable and reliable during its operation. The MMW detection circuit demonstrated in this work was found to be a milestone to develop larger focal plane arrays (FPA) with very inexpensive sensor elements.

Ultra-wideband and inexpensive glow discharge detector for millimeter-wave wireless communication based on upconversion to visual light.

Data traffic is increasing rapidly, especially on wireless channels, pushing the carrier frequency to the X, K, and millimeter-wave (MMW) bands. This requires development of new technologies and communication components operating at those bands. The detectors and receivers for those bands are very expensive, have high sensitivity to electrostatic discharge, and can be damaged by high incident power. An ultra-wideband and inexpensive glow discharge detector (GDD) is presented here. The GDD was found to be an excellent microwave and MMW radiation detector. The detection mechanism presented here is based on upconversion of microwave and MMW radiation to visual light. The experimental results demonstrate ultra-wideband detection at X and at MMW bands. These results present a proof of concept for the ability of our system to be used as a detector in wireless communication for the 5th generation.

Pulse Oximetry with Two Infrared Wavelengths without Calibration in Extracted Arterial Blood.

Oxygen saturation in arterial blood (SaO2) provides information about the performance of the respiratory system. Non-invasive measurement of SaO2 by commercial pulse oximeters (SpO2) make use of photoplethysmographic pulses in the red and infrared regions and utilizes the different spectra of light absorption by oxygenated and de-oxygenated hemoglobin. Because light scattering and optical path-lengths differ between the two wavelengths, commercial pulse oximeters require empirical calibration which is based on SaO2 measurement in extracted arterial blood. They are still prone to error, because the path-lengths difference between the two wavelengths varies among different subjects. We have developed modified pulse oximetry, which makes use of two nearby infrared wavelengths that have relatively similar scattering constants and path-lengths and does not require an invasive calibration step. In measurements performed on adults during breath holding, the two-infrared pulse oximeter and a commercial pulse oximeter showed similar changes in SpO2. The two pulse oximeters showed similar accuracy when compared to SaO2 measurement in extracted arterial blood (the gold standard) performed in intensive care units on newborns and children with an arterial line. Errors in SpO2 because of variability in path-lengths difference between the two wavelengths are expected to be smaller in the two-infrared pulse oximeter.

Effects of elemental images' quantity on three-dimensional segmentation using computational integral imaging: publisher's note.

This publisher's note corrects the Funding section in Appl. Opt.56, 2132 (2017)APOPAI0003-693510.1364/AO.56.002132.

Effects of elemental images' quantity on three-dimensional segmentation using computational integral imaging.

3D object detection and isolation can be achieved algorithmically using computational integral-imaging data. The 3D scene is acquired by a multi-channel system, where each channel (elemental image) captures the scene from a shifted perspective angle. The number of these channels affects the weight, the cost, and the computational load of the segmentation process, while a lower number of channels may reduce the performance of the objects' separation in the 3D scene. This research examines the effect of the elemental images' quantity on the 3D object detection and segmentation, under both regular and noisy conditions. Moreover, based on our previous works, we perform an improvement of the 3D object segmentation quality using an adapted active-contour method.

Optimized depth of field methodology using annular liquid crystal modulator assisted by image processing.

An optical-digital tunable depth of field (DOF) methodology is presented. The suggested methodology forms a fused image based on the sharpest similar depth regions from a set of source images taken with different phase masks. Each phase mask contains a different degree of DOF extension and is implemented by using an annular liquid crystal spatial light modulator, which consists of 16-ring electrodes positioned in the pupil plane. A detailed description of the optical setup and characterization of selected pupil phase masks as well as optimization of the binary phase mask for maximal DOF extension is presented. Experimental results are investigated both qualitatively and quantitatively. In addition, the algorithm's results are compared with those of some well-known fusion algorithms and proved its supremacy.

Automatic 3D object localization and isolation using computational integral imaging.

Detecting objects in three-dimensional (3D) space may be useful for various applications. We present a method to detect the 3D locations of objects using computationally reconstructed images obtained by integral imaging. The new algorithm exploits the space-variant blurring properties of the reconstructed images to detect and isolate objects at their depth locations, while removing traces of objects from other depths. With regard to previous work, the proposed method is more efficient and more resistant to noise; it gives more information about the detected object's depth, and improves object isolation and presentation.

Spatial-phase-shift imaging interferometry using a spectrally modulated white light source.

An extension of the white light spatial-phase-shift (WLSPS) for object surface measurements is described. Using WLSPS, surface measurements can be obtained from any real object image without the need of a reference beam, thus achieving inherent vibration cancellation. The surface topography is obtained by acquiring multiple images of an object illuminated by a spectrally modulated white light source and using an appropriate algorithm. The modulation of the light source obviates the need for the continuous phase delay to obtain the interferograms.

Background modeling for moving object detection in long-distance imaging through turbulent medium.

A basic step in automatic moving objects detection is often modeling the background (i.e., the scene excluding the moving objects). The background model describes the temporal intensity distribution expected at different image locations. Long-distance imaging through atmospheric turbulent medium is affected mainly by blur and spatiotemporal movements in the image, which have contradicting effects on the temporal intensity distribution, mainly at edge locations. This paper addresses this modeling problem theoretically, and experimentally, for various long-distance imaging conditions. Results show that a unimodal distribution is usually a more appropriate model. However, if image deblurring is performed, a multimodal modeling might be more appropriate.

Detecting and tracking moving objects in long-distance imaging through turbulent medium.

The challenge of detecting and tracking moving objects in imaging throughout the atmosphere stems from the atmospheric turbulence effects that cause time-varying image shifts and blur. These phenomena significantly increase the miss and false detection rates in long-range horizontal imaging. An efficient method was developed, which is based on novel criteria for objects' spatio-temporal properties, to discriminate true from false detections, following an adaptive thresholding procedure for foreground detection and an activity-based false alarm likeliness masking. The method is demonstrated on significantly distorted videos and compared with state of the art methods, and shows better false alarm and miss detection rates.

Capability of long distance 100 GHz FMCW using a single GDD lamp sensor.

Millimeter wave (MMW)-based imaging systems are required for applications in medicine, homeland security, concealed weapon detection, and space technology. The lack of inexpensive room temperature imaging sensors makes it difficult to provide a suitable MMW system for many of the above applications. A 3D MMW imaging system based on chirp radar was studied previously using a scanning imaging system of a single detector. The radar system requires that the millimeter wave detector will be able to operate as a heterodyne detector. Since the source of radiation is a frequency modulated continuous wave (FMCW), the detected signal as a result of heterodyne detection gives the object's depth information according to value of difference frequency, in addition to the reflectance of the 2D image. New experiments show the capability of long distance FMCW detection by using a large scale Cassegrain projection system, described first (to our knowledge) in this paper. The system presents the capability to employ a long distance of at least 20 m with a low-cost plasma-based glow discharge detector (GDD) focal plane array (FPA). Each point on the object corresponds to a point in the image and includes the distance information. This will enable relatively inexpensive 3D MMW imaging.

Performance of visual search tasks from various types of contour information.

A recently proposed visual aid for patients with a restricted visual field (tunnel vision) combines a see-through head-mounted display and a simultaneous minified contour view of the wide-field image of the environment. Such a widening of the effective visual field is helpful for tasks, such as visual search, mobility, and orientation. The sufficiency of image contours for performing everyday visual tasks is of major importance for this application, as well as for other applications, and for basic understanding of human vision. This research aims is to examine and compare the use of different types of automatically created contours, and contour representations, for practical everyday visual operations using commonly observed images. The visual operations include visual searching for items, such as cutlery, housewares, etc. Considering different recognition levels, identification of an object is distinguished from mere detection (when the object is not necessarily identified). Some nonconventional visual-based contour representations were developed for this purpose. Experiments were performed with normal-vision subjects by superposing contours of the wide field of the scene over a narrow field (see-through) background. From the results, it appears that about 85% success is obtained for searched object identification when the best contour versions are employed. Pilot experiments with video simulations are reported at the end of the paper.

Performance quantification of a millimeter-wavelength imaging system based on inexpensive glow-discharge-detector focal-plane array.

Inexpensive millimeter-wavelength (MMW) optical digital imaging raises a challenge of evaluating the imaging performance and image quality because of the large electromagnetic wavelengths and pixel sensor sizes, which are 2 to 3 orders of magnitude larger than those of ordinary thermal or visual imaging systems, and also because of the noisiness of the inexpensive glow discharge detectors that compose the focal-plane array. This study quantifies the performances of this MMW imaging system. Its point-spread function and modulation transfer function were investigated. The experimental results and the analysis indicate that the image quality of this MMW imaging system is limited mostly by the noise, and the blur is dominated by the pixel sensor size. Therefore, the MMW image might be improved by oversampling, given that noise reduction is achieved. Demonstration of MMW image improvement through oversampling is presented.

Analysis of interferograms of multi-layered biological samples obtained from full field optical coherence tomography systems.

Several methods were developed in the past to analyze interferograms produced by optical coherence tomography, and successfully applied to simulated or animated samples. However, these techniques do not cope with noisy and distorted interferograms from biological tissues. In this paper, known techniques, including the fast Fourier transform and several variations of the continuous wavelet transform, were employed to analyze the interferogram data. However, to cope with the difficulties in biological data, pre- and post-processing procedures and adaptive thresholding were developed to provide stability and robustness. Additionally, three-dimensional structural models of the biological samples were constructed, and revealed information like the number and locations of interfaces, the layer thickness and pattern, and abnormalities.

Simulating the perceptual effects of electrode-retina distance in prosthetic vision.

Objective. Retinal prostheses aim to restore some vision in retinitis pigmentosa and age-related macular degeneration blind patients. Many spatial and temporal aspects have been found to affect prosthetic vision. Our objective is to study the impact of the space-variant distance between the stimulating electrodes and the surface of the retina on prosthetic vision and how to mitigate this impact.Approach. A prosthetic vision simulation was built to demonstrate the perceptual effects of the electrode-retina distance (ERD) with different random spatial variations, such as size, brightness, shape, dropout, and spatial shifts. Three approaches for reducing the ERD effects are demonstrated: electrode grouping (quads), ERD-based input-image enhancement, and object scanning with and without phosphene persistence. A quantitative assessment for the first two approaches was done based on experiments with 20 subjects and three vision-based computational image similarity metrics.Main results.The effects of various ERDs on phosphenes' size, brightness, and shape were simulated. Quads, chosen according to the ERDs, effectively elicit phosphenes without exceeding the safe charge density limit, whereas single electrodes with large ERD cannot do so. Input-image enhancement reduced the ERD effects effectively. These two approaches significantly improved ERD-affected prosthetic vision according to the experiment and image similarity metrics. A further reduction of the ERD effects was achieved by scanning an object while moving the head.Significance.ERD has multiple effects on perception with retinal prostheses. One of them is vision loss caused by the incapability of electrodes with large ERD to evoke phosphenes. The three approaches presented in this study can be used separately or together to mitigate the impact of ERD. A consideration of our approaches in reducing the perceptual effects of the ERD may help improve the perception with current prosthetic technology and influence the design of future prostheses.