Unsupervised underwater imaging based on polarization and binocular depth estimation.

Scattering caused by suspended particles in the water severely reduces the radiance of the scene. This paper proposes an unsupervised underwater restoration method based on binocular estimation and polarization. Based on the correlation between the underwater transmission process and depth, this method combines the depth information and polarization information in the scene, uses the neural network to perform global optimization and the depth information is recalculated and updated in the network during the optimization process, and reduces the error generated by using the polarization image to calculate parameters, so that detailed parts of the image are restored. Furthermore, the method reduces the requirement for rigorous pairing of data compared to previous approaches for underwater imaging using neural networks. Experimental results show that this method can effectively reduce the noise in the original image and effectively preserve the detailed information in the scene.

Super-resolution non-line-of-sight imaging based on temporal encoding.

Non-line-of-sight (NLOS) imaging techniques have the ability to reconstruct objects beyond line-of-sight view, which would be useful in a variety of applications. In transient NLOS techniques, a fundamental problem is that the time resolution of imaging depends on the single-photon timing resolution (SPTR) of a detector. In this paper, a temporal super-resolution method named temporal encoding non-line-of-sight (TE-NLOS) is proposed. Specifically, by exploiting the spatial-temporal correlation among transient images, high-resolution transient images can be reconstructed through modulator encoding. We have demonstrated that the proposed method is capable of reconstructing transient images with a time resolution of 20 picoseconds from a detector with a limited SPTR of approximately nanoseconds. In systems with low time jitter, this method exhibits superior accuracy in reconstructing objects compared to direct detection, and it also demonstrates robustness against miscoding. Utilizing high-frequency modulation, our framework can reconstruct accurate objects with coarse-SPTR detectors, which provides an enlightening reference for solving the problem of hardware defects.

High-throughput imaging through dynamic scattering media based on speckle de-blurring.

Effectively imaging through dynamic scattering media is of great importance and challenge. Some imaging methods based on physical or learning models have been designed for object reconstruction. However, with an increase in exposure time or more drastic changes in the scattering medium, the speckle pattern superimposed during camera integration time undergoes more significant changes, resulting in a modification of the collected speckle structure and increased blurring, which brings significant challenges to the reconstruction. Here, the clearer structural information of blurred speckles is unearthed with a presented speckle de-blurring algorithm, and a high-throughput imaging method through rapidly changing scattering media is proposed for reconstruction under long exposure. For the problem of varying blur degrees in different regions of the speckle, a block-based method is proposed to divide the speckle into distinct sub-speckles, which can realize the reconstruction of hidden objects. The imaging of hidden objects with different complexity through dynamic scattering media is demonstrated, and the reconstruction results are improved significantly for speckles with different blur degrees, which verifies the effectiveness of the method. This method is a high-throughput approach that enables non-invasive imaging solely through the collection of a single speckle. It directly operates on blurred speckles, making it suitable for traditional speckle-correlation methods and deep learning (DL) methods. This provides a new way of thinking about solving practical scattering imaging challenges.

Single image relighting based on illumination field reconstruction.

Relighting a single low-light image is a crucial and challenging task. Previous works primarily focused on brightness enhancement but neglected the differences in light and shadow variations, which leads to unsatisfactory results. Herein, an illumination field reconstruction (IFR) algorithm is proposed to address this issue by leveraging physical mechanism guidance, physical-based supervision, and data-based modeling. Firstly, we derived the Illumination field modulation equation as a physical prior to guide the network design. Next, we constructed a physical-based dataset consisting of image sequences with diverse illumination levels as supervision. Finally, we proposed the IFR neural network (IFRNet) to model the relighting progress and reconstruct photorealistic images. Extensive experiments demonstrate the effectiveness of our method on both simulated and real-world datasets, showing its generalization ability in real-world scenarios, even training solely from simulation.

Task-Decoupled Knowledge Transfer for Cross-Modality Object Detection.

In harsh weather conditions, the infrared modality can supplement or even replace the visible modality. However, the lack of a large-scale dataset for infrared features hinders the generation of a robust pre-training model. Most existing infrared object-detection algorithms rely on pre-training models from the visible modality, which can accelerate network convergence but also limit performance due to modality differences. In order to provide more reliable feature representation for cross-modality object detection and enhance its performance, this paper investigates the impact of various task-relevant features on cross-modality object detection and proposes a knowledge transfer algorithm based on classification and localization decoupling analysis. A task-decoupled pre-training method is introduced to adjust the attributes of various tasks learned by the pre-training model. For the training phase, a task-relevant hyperparameter evolution method is proposed to increase the network's adaptability to attribute changes in pre-training weights. Our proposed method improves the accuracy of multiple modalities in multiple datasets, with experimental results on the FLIR ADAS dataset reaching a state-of-the-art level and surpassing most multi-spectral object-detection methods.

Deep speckle reassignment: towards bootstrapped imaging in complex scattering states with limited speckle grains.

Optical imaging through scattering media is a practical challenge with crucial applications in many fields. Many computational imaging methods have been designed for object reconstruction through opaque scattering layers, and remarkable recovery results have been demonstrated in the physical models or learning models. However, most of the imaging approaches are dependent on relatively ideal states with a sufficient number of speckle grains and adequate data volume. Here, the in-depth information with limited speckle grains has been unearthed with speckle reassignment and a bootstrapped imaging method is proposed for reconstruction in complex scattering states. Benefiting from the bootstrap priors-informed data augmentation strategy with a limited training dataset, the validity of the physics-aware learning method has been demonstrated and the high-fidelity reconstruction results through unknown diffusers are obtained. This bootstrapped imaging method with limited speckle grains broadens the way to highly scalable imaging in complex scattering scenes and gives a heuristic reference to practical imaging problems.

Structured light 3-D sensing for scenes with discontinuous reflectivity: error removal based on scene reconstruction and normalization.

Structured light-based 3-D sensing technique reconstructs the 3-D shape from the disparity given by pixel correspondence of two sensors. However, for scene surface containing discontinuous reflectivity (DR), the captured intensity deviates from its actual value caused by the non-ideal camera point spread function (PSF), thus generating 3-D measurement error. First, we construct the error model of fringe projection profilometry (FPP). From which, we conclude that the DR error of FPP is related to both the camera PSF and the scene reflectivity. The DR error of FPP is hard to be alleviated because of unknown scene reflectivity. Second, we introduce single-pixel imaging (SI) to reconstruct the scene reflectivity and normalize the scene with scene reflectivity "captured" by the projector. From the normalized scene reflectivity, pixel correspondence with error opposite to the original reflectivity is calculated for the DR error removal. Third, we propose an accurate 3-D reconstruction method under discontinuous reflectivity. In this method, pixel correspondence is first established by using FPP, and then refined by using SI with reflectivity normalization. Both the analysis and the measurement accuracy are verified under scenes with different reflectivity distributions in the experiments. As a result, the DR error is effectively alleviated while taking an acceptable measurement time.

Prior-free imaging unknown target through unknown scattering medium.

Imaging through scattering medium based on deep learning has been extensively studied. However, existing methods mainly utilize paired data-prior and lack physical-process fusion, and it is difficult to reconstruct hidden targets without the trained networks. This paper proposes an unsupervised neural network that integrates the universal physical process. The reconstruction process of the network is irrelevant to the system and only requires one frame speckle pattern and unpaired targets. The proposed network enables online optimization by using physical process instead of fitting data. Thus, large-scale paired data no longer need to be obtained to train the network in advance, and the proposed method does not need prior information. The optimization of the network is a physical-based process rather than a data mapping process, and the proposed method also increases the insufficient generalization ability of the learning-based method in scattering medium and targets. The universal applicability of the proposed method to different optical systems increases the likelihood that the method will be used in practice.

Non-invasive imaging through scattering medium and around corners beyond 3D memory effect.

The three-dimensional (3D) memory effect (ME) has been shown to exist in a variety of scattering scenes. Limited by the scope of ME, speckle correlation technology only can be applied in a small imaging field of view (FOV) with a small depth of field (DOF). In this Letter, an untrained neural network is constructed and used as an optimization tool to restore the targets beyond the 3D ME range. The autocorrelation consistency relationship and the generative adversarial strategy are combined. Only single frame speckle and unaligned real targets are needed for online optimization; therefore, the neural network does not need to train in advance. Furthermore, the proposed method does not need to conduct additional modulation for the system. This method can reconstruct not only hidden targets behind the scattering medium, but also targets around corners. The combination strategy of the generative adversarial framework with physical priors used to decouple the aliasing information and reconstruct the target will provide inspiration for the field of computational imaging.

Road-Scene Parsing Based on Attentional Prototype-Matching.

Road-scene parsing is complex and changeable; the interferences in the background destroy the visual structure in the image data, increasing the difficulty of target detection. The key to addressing road-scene parsing is to amplify the feature differences between the targets, as well as those between the targets and the background. This paper proposes a novel scene-parsing network, Attentional Prototype-Matching Network (APMNet), to segment targets by matching candidate features with target prototypes regressed from labeled road-scene data. To obtain reliable target prototypes, we designed the Sample-Selection and the Class-Repellence Algorithm in the prototype-regression progress. Also, we built the class-to-class and target-to-background attention mechanisms to increase feature recognizability based on the target's visual characteristics and spatial-target distribution. Experiments conducted on two road-scene datasets, CamVid and Cityscapes, demonstrate that our approach effectively improves the representation of targets and achieves impressive results compared with other approaches.

Locating through dynamic scattering media based on speckle correlations.

In complex imaging settings, optical scattering often prohibits the formation of a clear target image, and instead, only a speckle without the original spatial structure information is obtained. Scattering seriously interferes with the locating of targets; especially, when the scattering medium is dynamic, the dynamic nature leads to rapid decorrelation of optical information in time, and the challenge increases. Here, a locating method is proposed to detect the target hidden behind a dynamic scattering medium, which uses the a priori information of a known reference object in the neighborhood of the target. The research further designs an automatic calibration method to simplify the locating process, and analyzes the factors affecting positioning accuracy. The proposed method enables us to predict the position of a target from the autocorrelation of the captured speckle pattern; the angle and distance deviations of the target are all within 2.5%. This approach can locate a target using only a single-shot speckle pattern, and it is beneficial for target localization in dynamic scattering conditions.

Infrared-Visible Image Fusion Based on Semantic Guidance and Visual Perception.

Infrared-visible fusion has great potential in night-vision enhancement for intelligent vehicles. The fusion performance depends on fusion rules that balance target saliency and visual perception. However, most existing methods do not have explicit and effective rules, which leads to the poor contrast and saliency of the target. In this paper, we propose the SGVPGAN, an adversarial framework for high-quality infrared-visible image fusion, which consists of an infrared-visible image fusion network based on Adversarial Semantic Guidance (ASG) and Adversarial Visual Perception (AVP) modules. Specifically, the ASG module transfers the semantics of the target and background to the fusion process for target highlighting. The AVP module analyzes the visual features from the global structure and local details of the visible and fusion images and then guides the fusion network to adaptively generate a weight map of signal completion so that the resulting fusion images possess a natural and visible appearance. We construct a joint distribution function between the fusion images and the corresponding semantics and use the discriminator to improve the fusion performance in terms of natural appearance and target saliency. Experimental results demonstrate that our proposed ASG and AVP modules can effectively guide the image-fusion process by selectively preserving the details in visible images and the salient information of targets in infrared images. The SGVPGAN exhibits significant improvements over other fusion methods.

Efficient color imaging through unknown opaque scattering layers via physics-aware learning.

Color imaging with scattered light is crucial to many practical applications and becomes one of the focuses in optical imaging fields. More physics theories have been introduced in the deep learning (DL) approach for the optical tasks and improve the imaging capability a lot. Here, an efficient color imaging method is proposed in reconstructing complex objects hidden behind unknown opaque scattering layers, which can obtain high reconstruction fidelity in spatial structure and accurate restoration in color information by training with only one diffuser. More information is excavated by utilizing the scattering redundancy and promotes the physics-aware DL approach to reconstruct the color objects hidden behind unknown opaque scattering layers with robust generalization capability by an efficient means. This approach gives impetus to color imaging through dynamic scattering media and provides an enlightening reference for solving complex inverse problems based on physics-aware DL methods.

De-noising imaging through diffusers with autocorrelation.

Recovering targets through diffusers is an important topic as well as a general problem in optical imaging. The difficulty of recovering is increased due to the noise interference caused by an imperfect imaging environment. Existing approaches generally require a high-signal-to-noise-ratio (SNR) speckle pattern to recover the target, but still have limitations in de-noising or generalizability. Here, featuring information of high-SNR autocorrelation as a physical constraint, we propose a two-stage (de-noising and reconstructing) method to improve robustness based on data driving. Specifically, a two-stage convolutional neural network (CNN) called autocorrelation reconstruction (ACR) CNN is designed to de-noise and reconstruct targets from low-SNR speckle patterns. We experimentally demonstrate the robustness through various diffusers with different levels of noise, from simulative Gaussian noise to the detector and photon noise captured by the actual optical system. The de-noising stage improves the peak SNR from 20 to 38 dB in the system data, and the reconstructing stage, compared with the unconstrained method, successfully recovers targets hidden in unknown diffusers with the detector and photon noise. With the help of the physical constraint to optimize the learning process, our two-stage method is realized to improve generalizability and has potential in various fields such as imaging in low illumination.

High Sensitivity Snapshot Spectrometer Based on Deep Network Unmixing.

Spectral detection provides rich spectral-temporal information with wide applications. In our previous work, we proposed a dual-path sub-Hadamard-s snapshot Hadamard transform spectrometer (Sub-s HTS). In order to reduce the complexity of the system and improve its performance, we present a convolution neural network-based method to recover the light intensity distribution from the overlapped dispersive spectra, rather than adding an extra light path to capture it directly. In this paper, we construct a network-based single-path snapshot Hadamard transform spectrometer (net-based HTS). First, we designed a light intensity recovery neural network (LIRNet) with an unmixing module (UM) and an enhanced module (EM) to recover the light intensity from the dispersive image. Then, we used the reconstructed light intensity as the original light intensity to recover high signal-to-noise ratio spectra successfully. Compared with Sub-s HTS, the net-based HTS has a more compact structure and high sensitivity. A large number of simulations and experimental results have demonstrated that the proposed net-based HTS can obtain a better-reconstructed signal-to-noise ratio spectrum than the Sub-s HTS because of its higher light throughput.

Locating and Imaging through Scattering Medium in a Large Depth.

Scattering medium brings great difficulties to locate and reconstruct objects especially when the objects are distributed in different positions. In this paper, a novel physics and learning-heuristic method is presented to locate and image the object through a strong scattering medium. A novel physics-informed framework, named DINet, is constructed to predict the depth and the image of the hidden object from the captured speckle pattern. With the phase-space constraint and the efficient network structure, the proposed method enables to locate the object with a depth mean error less than 0.05 mm, and image the object with an average peak signal-to-noise ratio (PSNR) above 24 dB, ranging from 350 mm to 1150 mm. The constructed DINet firstly solves the problem of quantitative locating and imaging via a single speckle pattern in a large depth. Comparing with the traditional methods, it paves the way to the practical applications requiring multi-physics through scattering media.

Spatial pattern-shifting method for complete two-wavelength fringe projection profilometry: erratum.

We present an erratum and generalization to our Letter [Opt. Lett.45, 3115 (2020)OPLEDP0146-959210.1364/OL.392102]. This erratum corrects an error in Eq. (12), and the generalization converts Rh to kh for more general situations of wavelengths. Neither has any influence on the conclusions of the original Letter.

Spatial pattern-shifting method for complete two-wavelength fringe projection profilometry.

Two-wavelength fringe projection profilometry (FPP) unwraps a phase with the unambiguous phase range (UPR) of the least common multiple (LCM) of the two wavelengths. It is accurate, convenient, and robust, and thus plays an important role in shape measurement. However, when two non-coprime wavelengths are used, only a small UPR can be generated, and the unwrapping performance is compromised. In this Letter, a spatial pattern-shifting method (SPSM) is proposed to generate the maximum UPR (i.e., the product of the two wavelengths) from two non-coprime wavelengths. For the first time, to the best of our knowledge, the SPSM breaks the constraint of wavelength selection and enables a complete (i.e., either coprime or non-coprime) two-wavelength FPP. The SPSM, on the other hand, only requires spatially shift of the low-frequency pattern with the designed amounts and accordingly adjusting the fringe order determination, which is extremely convenient in implementation. Both numerical and experimental analyses verify its flexibility and correctness.

Learning-based method to reconstruct complex targets through scattering medium beyond the memory effect.

Strong scattering medium brings great difficulties to image objects. Optical memory effect makes it possible to image through strong random scattering medium in a limited angle field-of-view (FOV). The limitation of FOV results in a limited optical memory effect range, which prevents the optical memory effect to be applied to real imaging applications. In this paper, a kind of practical convolutional neural network called PDSNet (Pragmatic De-scatter ConvNet) is constructed to image objects hidden behind different scattering media. The proposed method can expand at least 40 times of the optical memory effect range with a average PSNR above 24dB, and enable to image complex objects in real time, even for objects with untrained scales. The provided experiments can verify its accurateness and efficiency.

Local structure preserving sparse coding for infrared target recognition.

Sparse coding performs well in image classification. However, robust target recognition requires a lot of comprehensive template images and the sparse learning process is complex. We incorporate sparsity into a template matching concept to construct a local sparse structure matching (LSSM) model for general infrared target recognition. A local structure preserving sparse coding (LSPSc) formulation is proposed to simultaneously preserve the local sparse and structural information of objects. By adding a spatial local structure constraint into the classical sparse coding algorithm, LSPSc can improve the stability of sparse representation for targets and inhibit background interference in infrared images. Furthermore, a kernel LSPSc (K-LSPSc) formulation is proposed, which extends LSPSc to the kernel space to weaken the influence of the linear structure constraint in nonlinear natural data. Because of the anti-interference and fault-tolerant capabilities, both LSPSc- and K-LSPSc-based LSSM can implement target identification based on a simple template set, which just needs several images containing enough local sparse structures to learn a sufficient sparse structure dictionary of a target class. Specifically, this LSSM approach has stable performance in the target detection with scene, shape and occlusions variations. High performance is demonstrated on several datasets, indicating robust infrared target recognition in diverse environments and imaging conditions.