U2PNet: An Unsupervised Underwater Image-Restoration Network Using Polarization.

This article presents U2PNet, a novel unsupervised underwater image restoration network using polarization for improving signal-to-noise ratio and image quality in underwater imaging environments. Traditional methods for underwater image restoration using polarization require specific cues or pairs of underwater polarization datasets, which limit their practical applications. Our proposed method requires only one mosaicked polarized image of the scene and does not require datasets for pretraining or specific cues. We design two subnetworks (T-net and B ∞ -net) to accurately estimate the transmission map and background light, and unique nonreference loss functions to ensure effective restoration. Our experiments are based on an indoor polarization simulated dataset and a real polarization image dataset constructed from our underwater robotic platform equipped with polarization cameras. Experiment results demonstrate that our proposed method achieves state-of-the-art performance on both simulated and real underwater polarization images. The code and datasets will be available at https://github.com/polwork/U-2Pnet.

Unsupervised Spectral Demosaicing With Lightweight Spectral Attention Networks.

This paper presents a deep learning-based spectral demosaicing technique trained in an unsupervised manner. Many existing deep learning-based techniques relying on supervised learning with synthetic images, often underperform on real-world images, especially as the number of spectral bands increases. This paper presents a comprehensive unsupervised spectral demosaicing (USD) framework based on the characteristics of spectral mosaic images. This framework encompasses a training method, model structure, transformation strategy, and a well-fitted model selection strategy. To enable the network to dynamically model spectral correlation while maintaining a compact parameter space, we reduce the complexity and parameters of the spectral attention module. This is achieved by dividing the spectral attention tensor into spectral attention matrices in the spatial dimension and spectral attention vector in the channel dimension. This paper also presents Mosaic 25 , a real 25-band hyperspectral mosaic image dataset featuring various objects, illuminations, and materials for benchmarking purposes. Extensive experiments on both synthetic and real-world datasets demonstrate that the proposed method outperforms conventional unsupervised methods in terms of spatial distortion suppression, spectral fidelity, robustness, and computational cost. Our code and dataset are publicly available at https://github.com/polwork/Unsupervised-Spectral-Demosaicing.

When Laplacian Scale Mixture Meets Three-Layer Transform: A Parametric Tensor Sparsity for Tensor Completion.

Recently, tensor sparsity modeling has achieved great success in the tensor completion (TC) problem. In real applications, the sparsity of a tensor can be rationally measured by low-rank tensor decomposition. However, existing methods either suffer from limited modeling power in estimating accurate rank or have difficulty in depicting hierarchical structure underlying such data ensembles. To address these issues, we propose a parametric tensor sparsity measure model, which encodes the sparsity for a general tensor by Laplacian scale mixture (LSM) modeling based on three-layer transform (TLT) for factor subspace prior with Tucker decomposition. Specifically, the sparsity of a tensor is first transformed into factor subspace, and then factor sparsity in the gradient domain is used to express the local similarity in within-mode. To further refine the sparsity, we adopt LSM by the transform learning scheme to self-adaptively depict deeper layer structured sparsity, in which the transformed sparse matrices in the sense of a statistical model can be modeled as the product of a Laplacian vector and a hidden positive scalar multiplier. We call the method as parametric tensor sparsity delivered by LSM-TLT. By a progressive transformation operator, we formulate the LSM-TLT model and use it to address the TC problem, and then the alternating direction method of multipliers-based optimization algorithm is designed to solve the problem. The experimental results on RGB images, hyperspectral images (HSIs), and videos demonstrate the proposed method outperforms state of the arts.

Multilayer Sparsity-Based Tensor Decomposition for Low-Rank Tensor Completion.

Existing methods for tensor completion (TC) have limited ability for characterizing low-rank (LR) structures. To depict the complex hierarchical knowledge with implicit sparsity attributes hidden in a tensor, we propose a new multilayer sparsity-based tensor decomposition (MLSTD) for the low-rank tensor completion (LRTC). The method encodes the structured sparsity of a tensor by the multiple-layer representation. Specifically, we use the CANDECOMP/PARAFAC (CP) model to decompose a tensor into an ensemble of the sum of rank-1 tensors, and the number of rank-1 components is easily interpreted as the first-layer sparsity measure. Presumably, the factor matrices are smooth since local piecewise property exists in within-mode correlation. In subspace, the local smoothness can be regarded as the second-layer sparsity. To describe the refined structures of factor/subspace sparsity, we introduce a new sparsity insight of subspace smoothness: a self-adaptive low-rank matrix factorization (LRMF) scheme, called the third-layer sparsity. By the progressive description of the sparsity structure, we formulate an MLSTD model and embed it into the LRTC problem. Then, an effective alternating direction method of multipliers (ADMM) algorithm is designed for the MLSTD minimization problem. Various experiments in RGB images, hyperspectral images (HSIs), and videos substantiate that the proposed LRTC methods are superior to state-of-the-art methods.

Structure-embedding network for predicting the transmission spectrum of a multilayer deep etched grating.

This Letter presents a structure-embedding network (SEmNet) to predict the transmission spectrum of a multilayer deep etched grating (MDEG). Spectral prediction is an important procedure in the MDEG design process. Existing approaches based on deep neural networks have been applied to spectral prediction to improve the design efficiency of similar devices, such as nanoparticles and metasurfaces. Due to a dimensionality mismatch between a structure parameter vector and the transmission spectrum vector, however, the prediction accuracy decreases. The proposed SEmNet can overcome the dimensionality mismatch problem of deep neural networks to increase the accuracy of predicting the transmission spectrum of an MDEG. SEmNet consists of a structure-embedding module and a deep neural network. The structure-embedding module increases the dimensionality of the structure parameter vector with a learnable matrix. The augmented structure parameter vector then becomes the input to the deep neural network to predict the transmission spectrum of the MDEG. Experiment results demonstrate that the proposed SEmNet improves the prediction accuracy of the transmission spectrum in comparison with the state-of-the-art approaches.

Polarization image demosaicking using polarization channel difference prior.

This paper presents a simple, yet effective demosaicking technique using polarization channel difference prior for polarization images captured by division of focal plane imaging sensors. The polarization channel difference prior embodies that high frequency energy of difference between orthogonal channels tends to be larger than that between non-orthogonal channels. This paper theoretically proves that this prior is physical valid. For each missing polarization channel at a pixel position, three initial predictions are recovered using different channel differences. The missing polarization channel is estimated by the weighted fusion of the three initial predictions, where the weights are determined by the proposed polarization channel difference prior. The prior helps recover polarization information of the edges, fast and effectively. Experiment results on the polarization dataset demonstrate the effectiveness of the polarization channel difference prior for polarization image demosaicking. The proposed polarization demosaicking method consists of only 16 convolution operations, which makes it fast and parallelizable for GPU acceleration. An image of size 1024×1024 can be processed in 0.33 sec on Ryzen 7 3700X CPU and approximately 60 times faster with RTX 2700 SUPER GPU.

Enhanced Sparsity Prior Model for Low-Rank Tensor Completion.

Conventional tensor completion (TC) methods generally assume that the sparsity of tensor-valued data lies in the global subspace. The so-called global sparsity prior is measured by the tensor nuclear norm. Such assumption is not reliable in recovering low-rank (LR) tensor data, especially when considerable elements of data are missing. To mitigate this weakness, this article presents an enhanced sparsity prior model for LRTC using both local and global sparsity information in a latent LR tensor. In specific, we adopt a doubly weighted strategy for nuclear norm along each mode to characterize global sparsity prior of tensor. Different from traditional tensor-based local sparsity description, the proposed factor gradient sparsity prior in the Tucker decomposition model describes the underlying subspace local smoothness in real-world tensor objects, which simultaneously characterizes local piecewise structure over all dimensions. Moreover, there is no need to minimize the rank of a tensor for the proposed local sparsity prior. Extensive experiments on synthetic data, real-world hyperspectral images, and face modeling data demonstrate that the proposed model outperforms state-of-the-art techniques in terms of prediction capability and efficiency.

Demosaicking DoFP images using Newton's polynomial interpolation and polarization difference model.

A demand for division of focal plane (DoFP) polarization imaging technology grows rapidly as nanofabrication technologies become mature. For real-time polarization imaging, a DoFP polarimeter often trades off its spatial resolution, which may cause instantaneous field of view (IFoV) errors. To deal with such problems, interpolation methods are often used to fill the missing polarization information. This paper presents an interpolation technique using Newton's polynomial for DoFP polarimeter demosaicking. The interpolation is performed in the polarization difference domain with the interpolation error taken into consideration. The proposed method uses an edge classifier based on polarization difference and a fusion scheme to recover more accurate boundary features. Experiments using both synthetic and real DoFP images in visible and long-wave infrared spectrum demonstrate that the proposed interpolation method outperforms the state-of-the-art techniques quantitatively as well as visually to reduce nonconformities caused by high-frequency energy.

Removal of reflections in LWIR image with polarization characteristics.

Long-wave infrared (LWIR) imaging has been successfully used in surveillance applications in low illumination conditions. However, infrared energy reflected from smooth surfaces such as floors and metallic objects may reduce object detection and tracking accuracies. In this paper, we present a novel reflection removal method using polarization properties of the reflection in LWIR imagery. Reflection can be distinguished from the scene by two unique characteristics of polarization: the difference of two orthogonal polarized components (OPC) and the uniformity of angle of polarization (AoP). The OPC difference helps locate the regions of reflection. The uniformity of AoP in the reflection region pose a strong constraint for reflection detection. The proposed joint reflection detection method combines the OPC difference and the uniformity of AoP can detect actual reflection region. Then the closed-form matting method improves the robustness of the method and removes the reflection from the scene. Experiment results demonstrate that the proposed scheme effectively removes the reflection in challenging situations where many existing techniques may fail.

Head pose estimation from a 2D face image using 3D face morphing with depth parameters.

This paper presents estimation of head pose angles from a single 2D face image using a 3D face model morphed from a reference face model. A reference model refers to a 3D face of a person of the same ethnicity and gender as the query subject. The proposed scheme minimizes the disparity between the two sets of prominent facial features on the query face image and the corresponding points on the 3D face model to estimate the head pose angles. The 3D face model used is morphed from a reference model to be more specific to the query face in terms of the depth error at the feature points. The morphing process produces a 3D face model more specific to the query image when multiple 2D face images of the query subject are available for training. The proposed morphing process is computationally efficient since the depth of a 3D face model is adjusted by a scalar depth parameter at feature points. Optimal depth parameters are found by minimizing the disparity between the 2D features of the query face image and the corresponding features on the morphed 3D model projected onto 2D space. The proposed head pose estimation technique was evaluated on two benchmarking databases: 1) the USF Human-ID database for depth estimation and 2) the Pointing'04 database for head pose estimation. Experiment results demonstrate that head pose estimation errors in nodding and shaking angles are as low as 7.93° and 4.65° on average for a single 2D input face image.

Invisible ink mark detection in the visible spectrum using absorption difference.

One of popular techniques in gambling fraud involves the use of invisible ink marks printed on the back surface of playing cards. Such covert patterns are transparent in the visible spectrum and therefore invisible to unaided human eyes. Invisible patterns can be made visible with ultraviolet (UV) illumination or a CCD camera installed with an infrared (IR) filter depending on the type of ink materials used. Cheating gamers often wear contact lenses or eyeglasses made of IR or UV filters to recognize the secret marks on the playing cards. This paper presents an image processing technique to reveal invisible ink patterns in the visible spectrum without the aid of special equipment such as UV lighting or IR filters. A printed invisible ink pattern leaves a thin coating on the surface with different refractive index for different wavelengths of light, which results in color dispersion or absorption difference. The proposed method finds the differences of color components caused by absorption difference to detect invisible ink patterns on the surface. Experiment results show that the proposed scheme is effective for both UV-active and IR-active invisible ink materials.

Sequence discrimination of heterogeneous crossing of seal impression and ink-printed text using adhesive tapes.

This paper presents a technique to discriminate the sequence of stamped seal impression and ink-printed text in a document to detect falsely signed documents. In many Asian countries where a seal is widely used to endorse documents, a possibly forged document involves a seal impressed before the text is printed. The proposed method uses adhesive tapes with peel adhesion strength of approximately 25 oz/in. to exfoliate the top layer of the overlapping region of seal impression and ink-printed text in the document. A pair of digital images of the overlapping region, captured using an infinite focus microscope, is compared for color changes before and after the exfoliation with adhesive tapes. The proposed sequence discrimination index (SDI) measures the amount of color changes before and after the exfoliation to determine the sequence of seal impression and printed text. Experiment results show that the SDI successfully discriminates the sequence of seal impression and printed text for different types of ink cartridges and seal inkpads under various storage conditions, enabling forensic investigation of falsely signed documents with a seal.

Frame-Based Recovery of Corrupted Video Files Using Video Codec Specifications.

In digital forensics, recovery of a damaged or altered video file plays a crucial role in searching for evidences to resolve a criminal case. This paper presents a frame-based recovery technique of a corrupted video file using the specifications of a codec used to encode the video data. A video frame is the minimum meaningful unit of video data. Many existing approaches attempt to recover a video file using file structure rather than frame structure. In case a target video file is severely fragmented or even has a portion of video overwritten by other video content, however, video file recovery of existing approaches may fail. The proposed approach addresses how to extract video frames from a portion of video to be restored as well as how to connect extracted video frames together according to the codec specifications. Experiment results show that the proposed technique successfully restores fragmented video files regardless of the amount of fragmentations. For a corrupted video file containing overwritten segments, the proposed technique can recover most of the video content in non-overwritten segments of the video file.

Detection of transcribed seal impressions using 3-D pressure traces.

Seals have been frequently used to certify that individuals or organizations have authorized or approved a document that bears these impressions. Much attention has been focused on the detection of forged seal impressions to expose and prevent seal-related frauds. This paper describes an image-processing technique that detects seal impressions transferred from a genuine document to a target document using transcription media. The proposed method utilizes a three-dimensional (3-D) scanner to generate a pressure trace map of the suspect seal impression. After utilizing a noise-reduction algorithm to improve image quality, the pressure map is aligned with a 2-D image of the same seal impression. The pressure ratio, determined by comparing the pressure map and inked impression of a suspect seal, can be used to determine whether the seal is genuine or was transferred to the target document. The results show that the proposed technique successfully identified transcribed seal impressions with an error rate of <1%.

Forged seal detection based on the seal overlay metric.

This paper describes a method for verifying the authenticity of a seal impression imprinted on a document based on the seal overlay metric, which refers to the ratio of an effective seal impression pattern and the noise in the neighborhood of the reference impression region. A reference seal pattern is obtained by taking the average of a number of high-quality impressions of a genuine seal. A target seal impression to be examined, often on paper with some background texts and lines, is segmented out from the background by an adaptive threshold applied to the histogram of color components. The segmented target seal impression is then spatially aligned with the reference by maximizing the count of matching pixels. Then the seal overlay metric is computed for the reference and the target. If the overlay metric of a target seal is below a predetermined limit for the similarity to the genuine, then the target is classified as a forged seal. To further reduce the misclassification rate, the seal overlay metric is adjusted by the filling rate, which reflects the quality of inked pattern of the target seal. Experiment results demonstrate that the proposed method can detect elaborate seal impressions created by advanced forgery techniques such as lithography and computer-aided manufacturing.

Atmospheric degradation correction of terahertz beams using multiscale signal restoration.

We present atmospheric degradation correction of terahertz (THz) beams based on multiscale signal decomposition and a combination of a Wiener deconvolution filter and artificial neural networks. THz beams suffer from strong attenuation by water molecules in the air. The proposed signal restoration approach finds the filter coefficients from a pair of reference signals previously measured from low-humidity conditions and current background air signals. Experimental results with two material samples of different chemical compositions demonstrate that the multiscale signal restoration technique is effective in correcting atmospheric degradation compared to individual and non-multiscale approaches.

Block-based neural networks for personalized ECG signal classification.

This paper presents evolvable block-based neural networks (BbNNs) for personalized ECG heartbeat pattern classification. A BbNN consists of a 2-D array of modular component NNs with flexible structures and internal configurations that can be implemented using reconfigurable digital hardware such as field-programmable gate arrays (FPGAs). Signal flow between the blocks determines the internal configuration of a block as well as the overall structure of the BbNN. Network structure and the weights are optimized using local gradient-based search and evolutionary operators with the rates changing adaptively according to their effectiveness in the previous evolution period. Such adaptive operator rate update scheme ensures higher fitness on average compared to predetermined fixed operator rates. The Hermite transform coefficients and the time interval between two neighboring R-peaks of ECG signals are used as inputs to the BbNN. A BbNN optimized with the proposed evolutionary algorithm (EA) makes a personalized heartbeat pattern classifier that copes with changing operating environments caused by individual difference and time-varying characteristics of ECG signals. Simulation results using the Massachusetts Institute of Technology/Beth Israel Hospital (MIT-BIH) arrhythmia database demonstrate high average detection accuracies of ventricular ectopic beats (98.1%) and supraventricular ectopic beats (96.6%) patterns for heartbeat monitoring, being a significant improvement over previously reported electrocardiogram (ECG) classification results.

Analysis of hyperspectral fluorescence images for poultry skin tumor inspection.

We present a hyperspectral fluorescence imaging system with a fuzzy inference scheme for detecting skin tumors on poultry carcasses. Hyperspectral images reveal spatial and spectral information useful for finding pathological lesions or contaminants on agricultural products. Skin tumors are not obvious because the visual signature appears as a shape distortion rather than a discoloration. Fluorescence imaging allows the visualization of poultry skin tumors more easily than reflectance. The hyperspectral image samples obtained for this poultry tumor inspection contain 65 spectral bands of fluorescence in the visible region of the spectrum at wavelengths ranging from 425 to 711 nm. The large amount of hyperspectral image data is compressed by use of a discrete wavelet transform in the spatial domain. Principal-component analysis provides an effective compressed representation of the spectral signal of each pixel in the spectral domain. A small number of significant features are extracted from two major spectral peaks of relative fluorescence intensity that have been identified as meaningful spectral bands for detecting tumors. A fuzzy inference scheme that uses a small number of fuzzy rules and Gaussian membership functions successfully detects skin tumors on poultry carcasses. Spatial-filtering techniques are used to significantly reduce false positives.