Ultra-fast detail enhancement for a short-wave infrared image.

Image detail enhancement is critical to the performance of short-wave infrared (SWIR) imaging systems, especially to the long-range systems. However, the existing high-performance infrared (IR) image enhancement methods typically have difficulty in meeting the requirements of the imaging system with high resolution and high frame rate. In this paper, we propose an ultra-fast and simple SWIR image detail enhancement method based on the difference of Gaussian (DoG) filter and plateau equalization. Our method consists of efficient edge detail information extraction and histogram equalization. The experimental results demonstrated that the proposed method achieves outstanding enhancement performance with a frame rate around 50 fps for 1280×1024 SWIR images.

New Results on Small and Dim Infrared Target Detection.

Real-time small infrared (IR) target detection is critical to the performance of the situational awareness system in high-altitude aircraft. However, current IR target detection systems are generally hardware-unfriendly and have difficulty in achieving a robust performance in datasets with clouds occupying a large proportion of the image background. In this paper, we present new results by using an efficient method that extracts the candidate targets in the pre-processing stage and fuses the local scale, blob-based contrast map and gradient map in the detection stage. We also developed mid-wave infrared (MWIR) and long-wave infrared (LWIR) cameras for data collection experiments and algorithm evaluations. Experimental results using both publicly available datasets and image sequences acquired by our cameras clearly demonstrated that the proposed method achieves high detection accuracy with the mean AUC being at least 22.3% higher than comparable methods, and the computational cost beating the other methods by a large margin.

New Results on Radioactive Mixture Identification and Relative Count Contribution Estimation.

Detecting nuclear materials in mixtures is challenging due to low concentration, environmental factors, sensor noise, source-detector distance variations, and others. This paper presents new results on nuclear material identification and relative count contribution (also known as mixing ratio) estimation for mixtures of materials in which there are multiple isotopes present. Conventional and deep-learning-based machine learning algorithms were compared. Realistic simulated data using Gamma Detector Response and Analysis Software (GADRAS) were used in our comparative studies. It was observed that a deep learning approach is highly promising.

Real-time infrared image detail enhancement based on fast guided image filter and plateau equalization.

Image detail enhancement is critical to the performance of infrared imaging systems because the original images generally suffer from low contrast and a low signal-to-noise ratio. Although conventional decomposition-based methods have advantages in enhancing image details, they also have clear disadvantages, which include intensive computations, over-enhanced noise, and gradient reversal artifacts. In this paper, we propose to accelerate enhancement processing by using a fast guided filter and plateau equalization. Our method consists of image decomposition, base and detail layers processing, and projection of the enhanced image to an 8-bit dynamic range. Experimental results demonstrated that our proposed method achieves a good balance among detail enhancement performance, noise and gradient reversal artifacts suppression, and computational cost, with a frame rate around 30 fps for 640×512 infrared images.

Demosaicing of CFA 3.0 with Applications to Low Lighting Images.

Low lighting images usually contain Poisson noise, which is pixel amplitude-dependent. More panchromatic or white pixels in a color filter array (CFA) are believed to help the demosaicing performance in dark environments. In this paper, we first introduce a CFA pattern known as CFA 3.0 that has 75% white pixels, 12.5% green pixels, and 6.25% of red and blue pixels. We then present algorithms to demosaic this CFA, and demonstrate its performance for normal and low lighting images. In addition, a comparative study was performed to evaluate the demosaicing performance of three CFAs, namely the Bayer pattern (CFA 1.0), the Kodak CFA 2.0, and the proposed CFA 3.0. Using a clean Kodak dataset with 12 images, we emulated low lighting conditions by introducing Poisson noise into the clean images. In our experiments, normal and low lighting images were used. For the low lighting conditions, images with signal-to-noise (SNR) of 10 dBs and 20 dBs were studied. We observed that the demosaicing performance in low lighting conditions was improved when there are more white pixels. Moreover, denoising can further enhance the demosaicing performance for all CFAs. The most important finding is that CFA 3.0 performs better than CFA 1.0, but is slightly inferior to CFA 2.0, in low lighting images.

High-performance concealment of defective pixel clusters in infrared imagers.

Defective pixel concealment is a necessary procedure in infrared image processing and is widely used. However, current approaches are mainly focused on the concealment of isolated pixels and small defective pixel clusters. Consequently, these approaches cannot meet the requirements when applied to infrared detectors with large defective pixel clusters. In this paper, we present a novel and comprehensive approach to processing the image data acquired from infrared imagers with large and small defective pixel clusters. Our approach consists of preprocessing, coarse concealment, high dynamic range enhancement, and fine concealment by generative adversarial networks. Experiments using mid-wave infrared and long-wave infrared images demonstrated that the proposed approach achieves better results than the best conventional approach, known as transforming image completion, with the peak signal-to-noise ratio and structural similarity metrics improved by 2.7063 dB (16.3%) and 0.1951 dB (34.1%), respectively.

Detection and Confirmation of Multiple Human Targets Using Pixel-Wise Code Aperture Measurements.

Compressive video measurements can save bandwidth and data storage. However, conventional approaches to target detection require the compressive measurements to be reconstructed before any detectors are applied. This is not only time consuming but also may lose information in the reconstruction process. In this paper, we summarized the application of a recent approach to vehicle detection and classification directly in the compressive measurement domain to human targets. The raw videos were collected using a pixel-wise code exposure (PCE) camera, which condensed multiple frames into one frame. A combination of two deep learning-based algorithms (you only look once (YOLO) and residual network (ResNet)) was used for detection and confirmation. Optical and mid-wave infrared (MWIR) videos from a well-known database (SENSIAC) were used in our experiments. Extensive experiments demonstrated that the proposed framework was feasible for target detection up to 1500 m, but target confirmation needs more research.

Mastcam Image Resolution Enhancement with Application to Disparity Map Generation for Stereo Images with Different Resolutions.

In this paper, we introduce an in-depth application of high-resolution disparity map estimation using stereo images from Mars Curiosity rover's Mastcams, which have two imagers with different resolutions. The left Mastcam has three times lower resolution as that of the right. The left Mastcam image's resolution is first enhanced with three methods: Bicubic interpolation, pansharpening-based method, and a deep learning super resolution method. The enhanced left camera image and the right camera image are then used to estimate the disparity map. The impact of the left camera image enhancement is examined. The comparative performance analyses showed that the left camera enhancement results in getting more accurate disparity maps in comparison to using the original left Mastcam images for disparity map estimation. The deep learning-based method provided the best performance among the three for both image enhancement and disparity map estimation accuracy. A high-resolution disparity map, which is the result of the left camera image enhancement, is anticipated to improve the conducted science products in the Mastcam imagery such as 3D scene reconstructions, depth maps, and anaglyph images.

Deep Learning-Based Target Tracking and Classification for Low Quality Videos Using Coded Aperture Cameras.

Compressive sensing has seen many applications in recent years. One type of compressive sensing device is the Pixel-wise Code Exposure (PCE) camera, which has low power consumption and individual control of pixel exposure time. In order to use PCE cameras for practical applications, a time consuming and lossy process is needed to reconstruct the original frames. In this paper, we present a deep learning approach that directly performs target tracking and classification in the compressive measurement domain without any frame reconstruction. In particular, we propose to apply You Only Look Once (YOLO) to detect and track targets in the frames and we propose to apply Residual Network (ResNet) for classification. Extensive simulations using low quality optical and mid-wave infrared (MWIR) videos in the SENSIAC database demonstrated the efficacy of our proposed approach.

Preflight Contingency Planning Approach for Fixed Wing UAVs with Engine Failure in the Presence of Winds.

Preflight contingency planning that utilizes wind forecast in path planning can be highly beneficial to unmanned aerial vehicles (UAV) operators in preventing a possible mishap of the UAV. This especially becomes more important if the UAV has an engine failure resulting in the loss of all its thrust. Wind becomes a significant factor in determining reachability of the emergency landing site in a contingency like this. The preflight contingency plans can guide the UAV operators about how to glide the aircraft to the designated emergency landing site to make a safe landing. The need for a preflight or in-flight contingency plan is even more obvious in the case of a communication loss between the UAV operator and UAV since the UAV will then need to make the forced landing autonomously without the operator. In this paper, we introduce a preflight contingency planning approach that automates the forced landing path generation process for UAVs with engine failure. The contingency path generation aims true reachability to the emergency landing site by including the final approach part of the path in forecast wind conditions. In the contingency path generation, no-fly zones that could be in the area are accounted for and the contingency flight paths do not pass through them. If no plans can be found that fulfill reachability in the presence of no-fly zones, only then, as a last resort, the no-fly zone avoidance rule is relaxed. The contingency path generation utilizes hourly forecast wind data from National Oceanic and Atmospheric Administration for the geographical area of interest and time of the flight. Different from past works, we use trochoidal paths instead of Dubins curves and incorporate wind as a parameter in the contingency path design.

Further Improvement of Debayering Performance of RGBW Color Filter Arrays Using Deep Learning and Pansharpening Techniques.

The RGBW color filter arrays (CFA), also known as CFA2.0, contains R, G, B, and white (W) pixels. It is a 4 × 4 pattern that has 8 white pixels, 4 green pixels, 2 red pixels, and 2 blue pixels. The pattern repeats itself over the whole image. In an earlier conference paper, we cast the demosaicing process for CFA2.0 as a pansharpening problem. That formulation is modular and allows us to insert different pansharpening algorithms for demosaicing. New algorithms in interpolation and demosaicing can also be used. In this paper, we propose a new enhancement of our earlier approach by integrating a deep learning-based algorithm into the framework. Extensive experiments using IMAX and Kodak images clearly demonstrated that the new approach improved the demosaicing performance even further.

Remote Sensing Performance Enhancement in Hyperspectral Images.

Hyperspectral images with hundreds of spectral bands have been proven to yield high performance in material classification. However, despite intensive advancement in hardware, the spatial resolution is still somewhat low, as compared to that of color and multispectral (MS) imagers. In this paper, we aim at presenting some ideas that may further enhance the performance of some remote sensing applications such as border monitoring and Mars exploration using hyperspectral images. One popular approach to enhancing the spatial resolution of hyperspectral images is pansharpening. We present a brief review of recent image resolution enhancement algorithms, including single super-resolution and multi-image fusion algorithms, for hyperspectral images. Advantages and limitations of the enhancement algorithms are highlighted. Some limitations in the pansharpening process include the availability of high resolution (HR) panchromatic (pan) and/or MS images, the registration of images from multiple sources, the availability of point spread function (PSF), and reliable and consistent image quality assessment. We suggest some proactive ideas to alleviate the above issues in practice. In the event where hyperspectral images are not available, we suggest the use of band synthesis techniques to generate HR hyperspectral images from low resolution (LR) MS images. Several recent interesting applications in border monitoring and Mars exploration using hyperspectral images are presented. Finally, some future directions in this research area are highlighted.

Assessment of Spatiotemporal Fusion Algorithms for Planet and Worldview Images.

Although Worldview-2 (WV) images (non-pansharpened) have 2-m resolution, the re-visit times for the same areas may be seven days or more. In contrast, Planet images are collected using small satellites that can cover the whole Earth almost daily. However, the resolution of Planet images is 3.125 m. It would be ideal to fuse these two satellites images to generate high spatial resolution (2 m) and high temporal resolution (1 or 2 days) images for applications such as damage assessment, border monitoring, etc. that require quick decisions. In this paper, we evaluate three approaches to fusing Worldview (WV) and Planet images. These approaches are known as Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM), Flexible Spatiotemporal Data Fusion (FSDAF), and Hybrid Color Mapping (HCM), which have been applied to the fusion of MODIS and Landsat images in recent years. Experimental results using actual Planet and Worldview images demonstrated that the three aforementioned approaches have comparable performance and can all generate high quality prediction images.

Binary fingerprints at fluctuation-enhanced sensing.

We have developed a simple way to generate binary patterns based on spectral slopes in different frequency ranges at fluctuation-enhanced sensing. Such patterns can be considered as binary "fingerprints" of odors. The method has experimentally been demonstrated with a commercial semiconducting metal oxide (Taguchi) sensor exposed to bacterial odors (Escherichia coli and Anthrax-surrogate Bacillus subtilis) and processing their stochastic signals. With a single Taguchi sensor, the situations of empty chamber, tryptic soy agar (TSA) medium, or TSA with bacteria could be distinguished with 100% reproducibility. The bacterium numbers were in the range of 2.5 × 10(4)-10(6). To illustrate the relevance for ultra-low power consumption, we show that this new type of signal processing and pattern recognition task can be implemented by a simple analog circuitry and a few logic gates with total power consumption in the microWatts range.