Minimizing optical attribute errors for a lane departure warning system using an ultra-wide-angle camera.

Advanced driver assistance systems (ADAS) rely on lane departure warning (LDW) technology to enhance safety while driving. However, the current LDW method is limited to cameras with standard angles of view, such as mono cameras and black boxes. In recent times, more cameras with ultra-wide-angle lenses are being used to save money and improve accuracy. However, this has led to some challenges such as fixing optical distortion, making the camera process images faster, and ensuring its performance. To effectively implement LDW, we developed three technologies: (i) distortion correction using error functions based on the projection characteristics of optical lenses, (ii) automatic vanishing point estimation using geometric characteristics, and (iii) lane tracking and lane departure detection using constraints. The proposed technology improves system stability and convenience through automatic calculation and updating of parameters required for LDW function operation. By performing automatic distortion correction and vanishing point estimation, it has also been proven that fusion with other ADAS systems including front cameras is possible. Existing systems that use vanishing point information do not consider lens distortion and have slow and inaccurate vanishing point estimation, leading to a deterioration of system performance. The proposed method enables fast and accurate vanishing point estimation, allowing for adaptive responses to changes in the road environment.

Deep learning-based EEG analysis to classify normal, mild cognitive impairment, and dementia: Algorithms and dataset.

For automatic EEG diagnosis, this paper presents a new EEG data set with well-organized clinical annotations called Chung-Ang University Hospital EEG (CAUEEG), which has event history, patient's age, and corresponding diagnosis labels. We also designed two reliable evaluation tasks for the low-cost, non-invasive diagnosis to detect brain disorders: i) CAUEEG-Dementia with normal, mci, and dementia diagnostic labels and ii) CAUEEG-Abnormal with normal and abnormal. Based on the CAUEEG dataset, this paper proposes a new fully end-to-end deep learning model, called the CAUEEG End-to-end Deep neural Network (CEEDNet). CEEDNet pursues to bring all the functional elements for the EEG analysis in a seamless learnable fashion while restraining non-essential human intervention. Extensive experiments showed that our CEEDNet significantly improves the accuracy compared with existing methods, such as machine learning methods and Ieracitano-CNN (Ieracitano et al., 2019), due to taking full advantage of end-to-end learning. The high ROC-AUC scores of 0.9 on CAUEEG-Dementia and 0.86 on CAUEEG-Abnormal recorded by our CEEDNet models demonstrate that our method can lead potential patients to early diagnosis through automatic screening.

CODEN: combined optimization-based decomposition and learning-based enhancement network for Retinex-based brightness and contrast enhancement.

In this paper, we present a novel low-light image enhancement method by combining optimization-based decomposition and enhancement network for simultaneously enhancing brightness and contrast. The proposed method works in two steps including Retinex decomposition and illumination enhancement, and can be trained in an end-to-end manner. The first step separates the low-light image into illumination and reflectance components based on the Retinex model. Specifically, it performs model-based optimization followed by learning for edge-preserved illumination smoothing and detail-preserved reflectance denoising. In the second step, the illumination output from the first step, together with its gamma corrected and histogram equalized versions, serves as input to illumination enhancement network (IEN) including residual squeeze and excitation blocks (RSEBs). Extensive experiments prove that our method shows better performance compared with state-of-the-art low-light enhancement methods in the sense of both objective and subjective measures.

Photo-Realistic Image Dehazing and Verifying Networks via Complementary Adversarial Learning.

Physical model-based dehazing methods cannot, in general, avoid environmental variables and undesired artifacts such as non-collected illuminance, halo and saturation since it is difficult to accurately estimate the amount of the illuminance, light transmission and airlight. Furthermore, the haze model estimation process requires very high computational complexity. To solve this problem by directly estimating the radiance of the haze images, we present a novel dehazing and verifying network (DVNet). In the dehazing procedure, we enhanced the clean images by using a correction network (CNet), which uses the ground truth to learn the haze network. Haze images are then restored through a haze network (HNet). Furthermore, a verifying method verifies the error of both CNet and HNet using a self-supervised learning method. Finally, the proposed complementary adversarial learning method can produce results more naturally. Note that the proposed discriminator and generators (HNet & CNet) can be learned via an unpaired dataset. Overall, the proposed DVNet can generate a better dehazed result than state-of-the-art approaches under various hazy conditions. Experimental results show that the DVNet outperforms state-of-the-art dehazing methods in most cases.

Gated Dehazing Network via Least Square Adversarial Learning.

In a hazy environment, visibility is reduced and objects are difficult to identify. For this reason, many dehazing techniques have been proposed to remove the haze. Especially, in the case of the atmospheric scattering model estimation-based method, there is a problem of distortion when inaccurate models are estimated. We present a novel residual-based dehazing network model to overcome the performance limitation in an atmospheric scattering model-based method. More specifically, the proposed model adopted the gate fusion network that generates the dehazed results using a residual operator. To further reduce the divergence between the clean and dehazed images, the proposed discriminator distinguishes dehazed results and clean images, and then reduces the statistical difference via adversarial learning. To verify each element of the proposed model, we hierarchically performed the haze removal process in an ablation study. Experimental results show that the proposed method outperformed state-of-the-art approaches in terms of peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), international commission on illumination cie delta e 2000 (CIEDE2000), and mean squared error (MSE). It also gives subjectively high-quality images without color distortion or undesired artifacts for both synthetic and real-world hazy images.

Deep Binary Classification via Multi-Resolution Network and Stochastic Orthogonality for Subcompact Vehicle Recognition.

To encourage people to save energy, subcompact cars have several benefits of discount on parking or toll road charge. However, manual classification of the subcompact car is highly labor intensive. To solve this problem, automatic vehicle classification systems are good candidates. Since a general pattern-based classification technique can not successfully recognize the ambiguous features of a vehicle, we present a new multi-resolution convolutional neural network (CNN) and stochastic orthogonal learning method to train the network. We first extract the region of a bonnet in the vehicle image. Next, both extracted and input image are engaged to low and high resolution layers in the CNN model. The proposed network is then optimized based on stochastic orthogonality. We also built a novel subcompact vehicle dataset that will be open for a public use. Experimental results show that the proposed model outperforms state-of-the-art approaches in term of accuracy, which means that the proposed method can efficiently classify the ambiguous features between subcompact and non-subcompact vehicles.

SNS-CF: Siamese Network with Spatially Semantic Correlation Features for Object Tracking.

Recent advances in object tracking based on deep Siamese networks shifted the attention away from correlation filters. However, the Siamese network alone does not have as high accuracy as state-of-the-art correlation filter-based trackers, whereas correlation filter-based trackers alone have a frame update problem. In this paper, we present a Siamese network with spatially semantic correlation features (SNS-CF) for accurate, robust object tracking. To deal with various types of features spread in many regions of the input image frame, the proposed SNS-CF consists of-(1) a Siamese feature extractor, (2) a spatially semantic feature extractor, and (3) an adaptive correlation filter. To the best of authors knowledge, the proposed SNS-CF is the first attempt to fuse the Siamese network and the correlation filter to provide high frame rate, real-time visual tracking with a favorable tracking performance to the state-of-the-art methods in multiple benchmarks.

Uniformity Attentive Learning-Based Siamese Network for Person Re-Identification.

Person re-identification (Re-ID) has a problem that makes learning difficult such as misalignment and occlusion. To solve these problems, it is important to focus on robust features in intra-class variation. Existing attention-based Re-ID methods focus only on common features without considering distinctive features. In this paper, we present a novel attentive learning-based Siamese network for person Re-ID. Unlike existing methods, we designed an attention module and attention loss using the properties of the Siamese network to concentrate attention on common and distinctive features. The attention module consists of channel attention to select important channels and encoder-decoder attention to observe the whole body shape. We modified the triplet loss into an attention loss, called uniformity loss. The uniformity loss generates a unique attention map, which focuses on both common and discriminative features. Extensive experiments show that the proposed network compares favorably to the state-of-the-art methods on three large-scale benchmarks including Market-1501, CUHK03 and DukeMTMC-ReID datasets.

Enhanced Action Recognition Using Multiple Stream Deep Learning with Optical Flow and Weighted Sum.

Various action recognition approaches have recently been proposed with the aid of three-dimensional (3D) convolution and a multiple stream structure. However, existing methods are sensitive to background and optical flow noise, which prevents from learning the main object in a video frame. Furthermore, they cannot reflect the accuracy of each stream in the process of combining multiple streams. In this paper, we present a novel action recognition method that improves the existing method using optical flow and a multi-stream structure. The proposed method consists of two parts: (i) optical flow enhancement process using image segmentation and (ii) score fusion process by applying weighted sum of the accuracy. The enhancement process can help the network to efficiently analyze the flow information of the main object in the optical flow frame, thereby improving accuracy. A different accuracy of each stream can be reflected to the fused score while using the proposed score fusion method. We achieved an accuracy of 98.2% on UCF-101 and 82.4% on HMDB-51. The proposed method outperformed many state-of-the-art methods without changing the network structure and it is expected to be easily applied to other networks.

Camera orientation estimation using voting approach on the Gaussian sphere for in-vehicle camera.

Calibration of a vehicle camera is a key technology for advanced driver assistance systems (ADAS). This paper presents a novel estimation method to measure the orientation of a camera that is mounted on a driving vehicle. By considering the characteristics of vehicle cameras and driving environment, we detect three orthogonal vanishing points as a basis of the imaging geometry. The proposed method consists of three steps: i) detection of lines projected to the Gaussian sphere and extraction of the plane normal, ii) estimation of the vanishing point about the optical axis using linear Hough transform, and iii) voting for the rest two vanishing points using circular histogram. The proposed method increases both accuracy and stability by considering the practical driving situation using sequentially estimated three vanishing points. In addition, we can rapidly estimate the orientation by converting the voting space into a 2D plane at each stage. As a result, the proposed method can quickly and accurately estimate the orientation of the vehicle camera in a normal driving situation.

Illuminant-invariant stereo matching using cost volume and confidence-based disparity refinement.

In stereo-matching techniques for three-dimensional (3D) vision, illumination change is a major problem that degrades matching accuracy. When large intensity differences are observed between a pair of stereos, it is difficult to find the similarity in the matching process. In addition, inaccurately estimated disparities are obtained in textureless regions, since there are no distinguishable features in the region. To solve these problems, this paper presents a robust stereo-matching method using illuminant-invariant cost volume and confidence-based disparity refinement. In the step of matching a stereo pair, the proposed method combines two cost volumes using an invariant image and Weber local descriptor (WLD), which was originally motivated by human visual characteristics. The invariant image used in the matching step is insensitive to sudden brightness changes by shadow or light sources, and WLD reflects structural features of the invariant image with consideration of a gradual illumination change. After aggregating the cost using a guided filter, we refine the initially estimated disparity map based on the confidence map computed by the combined cost volume. Experimental results verify that the matching computation of the proposed method improves the accuracy of the disparity map under a radiometrically dynamic environment. Since the proposed disparity refinement method can also reduce the error of the initial disparity map in textureless areas, it can be applied to various 3D vision systems such as industrial robots and autonomous vehicles.

Real-Time Visual Tracking with Variational Structure Attention Network.

Online training framework based on discriminative correlation filters for visual tracking has recently shown significant improvement in both accuracy and speed. However, correlation filter-base discriminative approaches have a common problem of tracking performance degradation when the local structure of a target is distorted by the boundary effect problem. The shape distortion of the target is mainly caused by the circulant structure in the Fourier domain processing, and it makes the correlation filter learn distorted training samples. In this paper, we present a structure-attention network to preserve the target structure from the structure distortion caused by the boundary effect. More specifically, we adopt a variational auto-encoder as a structure-attention network to make various and representative target structures. We also proposed two denoising criteria using a novel reconstruction loss for variational auto-encoding framework to capture more robust structures even under the boundary condition. Through the proposed structure-attention framework, discriminative correlation filters can learn robust structure information of targets during online training with an enhanced discriminating performance and adaptability. Experimental results on major visual tracking benchmark datasets show that the proposed method produces a better or comparable performance compared with the state-of-the-art tracking methods with a real-time processing speed of more than 80 frames per second.

Intelligent Image-Based Railway Inspection System Using Deep Learning-Based Object Detection and Weber Contrast-Based Image Comparison.

For sustainable operation and maintenance of urban railway infrastructure, intelligent visual inspection of the railway infrastructure attracts increasing attention to avoid unreliable, manual observation by humans at night, while trains do not operate. Although various automatic approaches were proposed using image processing and computer vision techniques, most of them are focused only on railway tracks. In this paper, we present a novel railway inspection system using facility detection based on deep convolutional neural network and computer vision-based image comparison approach. The proposed system aims to automatically detect wears and cracks by comparing a pair of corresponding image sets acquired at different times. We installed line scan camera on the roof of the train. Unlike an area-based camera, the line scan camera quickly acquires images with a wide field of view. The proposed system consists of three main modules: (i) image reconstruction for registration of facility positions, (ii) facility detection using an improved single shot detector, and (iii) deformed region detection using image processing and computer vision techniques. In experiments, we demonstrate that the proposed system accurately finds facilities and detects their potential defects. For that reason, the proposed system can provide various advantages such as cost reduction for maintenance and accident prevention.

Disparity-selective stereo matching using correlation confidence measure.

Recently, the cost-volume filtering (CVF) methods for local stereo matching have provided fast and accurate results compared to those of the other method. However, CVF still causes incorrect results in the occlusion and texture-free regions. In particular, cost aggregation by pixel units involves complex computation because of its dependence on the image resolution and search range. This paper presents a robust stereo matching method for occluded regions. First, we generate cost volumes using the CENSUS transform and the scale-invariant feature transform (SIFT). Then, label-based cost volumes are aggregated using adaptive support weight and the simple linear iterative clustering (SLIC) scheme from two generated cost volumes. In order to obtain optimal disparity by two label-based cost volumes, we select the disparity corresponding to high confidence similarity of CENSUS or SIFT with minimum cost point. Experimental results show that our method estimates the optimal disparity in occlusion information, which exists only in the scene of one of the stereo pairs.

Segmentation-Based Color Channel Registration for Disparity Estimation of Dual Color-Filtered Aperture Camera.

Single-lens-based optical range finding systems were developed as an efficient, compact alternative for conventional stereo camera systems. Among various single-lens-based approaches, a multiple color-filtered aperture (MCA) system can generate disparity information among color channels, as well as normal color information. In this paper, we consider a dual color-filtered aperture (DCA) system as the most minimal version of the MCA system and present a novel inter-color image registration algorithm for disparity estimation. This proposed registration algorithm consists of three steps: (i) color channel independent feature extraction; (ii) feature-based adaptive weight disparity estimation; and (iii) color mapping matrix (CMM)-based cross-channel image registration. Experimental results show that the proposed method can not only generate an accurate disparity map, but also realize high quality cross-channel registration with a disparity prior for DCA-based range finding and color image enhancement.

Continuous digital zooming using local self-similarity-based super-resolution for an asymmetric dual camera system.

This paper presents a digital zooming method using a super-resolution (SR) algorithm based on the local self-similarity between the wide- and tele-view images acquired by an asymmetric dual camera system. The proposed SR algorithm consists of four steps: (i) registration of an optically zoomed image to the wide-view image, (ii) restoration of the central region of the zoomed wide-view image, (iii) restoration of the boundary region of the zoomed wide-view image, and (iv) fusion of the results from steps (ii) and (iii). Since an asymmetric dual camera system acquires different-resolution images on the same scene due to the different optical specifications, the proposed method can restore the low-resolution wide-view image using the ideal high-frequency component estimated from the optically zoomed image. Experimental results demonstrate that the proposed method can provide significantly improved high-resolution wide-view images compared to existing single-image-based SR methods.

Contrast-dependent saturation adjustment for outdoor image enhancement.

Outdoor images captured in bad-weather conditions usually have poor intensity contrast and color saturation since the light arriving at the camera is severely scattered or attenuated. The task of improving image quality in poor conditions remains a challenge. Existing methods of image quality improvement are usually effective for a small group of images but often fail to produce satisfactory results for a broader variety of images. In this paper, we propose an image enhancement method, which makes it applicable to enhance outdoor images by using content-adaptive contrast improvement as well as contrast-dependent saturation adjustment. The main contribution of this work is twofold: (1) we propose the content-adaptive histogram equalization based on the human visual system to improve the intensity contrast; and (2) we introduce a simple yet effective prior for adjusting the color saturation depending on the intensity contrast. The proposed method is tested with different kinds of images, compared with eight state-of-the-art methods: four enhancement methods and four haze removal methods. Experimental results show the proposed method can more effectively improve the visibility and preserve the naturalness of the images, as opposed to the compared methods.

Iterative Refinement of Transmission Map for Stereo Image Defogging Using a Dual Camera Sensor.

Recently, the stereo imaging-based image enhancement approach has attracted increasing attention in the field of video analysis. This paper presents a dual camera-based stereo image defogging algorithm. Optical flow is first estimated from the stereo foggy image pair, and the initial disparity map is generated from the estimated optical flow. Next, an initial transmission map is generated using the initial disparity map. Atmospheric light is then estimated using the color line theory. The defogged result is finally reconstructed using the estimated transmission map and atmospheric light. The proposed method can refine the transmission map iteratively. Experimental results show that the proposed method can successfully remove fog without color distortion. The proposed method can be used as a pre-processing step for an outdoor video analysis system and a high-end smartphone with a dual camera system.

Robust Video Stabilization Using Particle Keypoint Update and l1-Optimized Camera Path.

Acquisition of stabilized video is an important issue for various type of digital cameras. This paper presents an adaptive camera path estimation method using robust feature detection to remove shaky artifacts in a video. The proposed algorithm consists of three steps: (i) robust feature detection using particle keypoints between adjacent frames; (ii) camera path estimation and smoothing; and (iii) rendering to reconstruct a stabilized video. As a result, the proposed algorithm can estimate the optimal homography by redefining important feature points in the flat region using particle keypoints. In addition, stabilized frames with less holes can be generated from the optimal, adaptive camera path that minimizes a temporal total variation (TV). The proposed video stabilization method is suitable for enhancing the visual quality for various portable cameras and can be applied to robot vision, driving assistant systems, and visual surveillance systems.

Depth map generation using a single image sensor with phase masks.

Conventional stereo matching systems generate a depth map using two or more digital imaging sensors. It is difficult to use the small camera system because of their high costs and bulky sizes. In order to solve this problem, this paper presents a stereo matching system using a single image sensor with phase masks for the phase difference auto-focusing. A novel pattern of phase mask array is proposed to simultaneously acquire two pairs of stereo images. Furthermore, a noise-invariant depth map is generated from the raw format sensor output. The proposed method consists of four steps to compute the depth map: (i) acquisition of stereo images using the proposed mask array, (ii) variational segmentation using merging criteria to simplify the input image, (iii) disparity map generation using the hierarchical block matching for disparity measurement, and (iv) image matting to fill holes to generate the dense depth map. The proposed system can be used in small digital cameras without additional lenses or sensors.