ABDGAN: Arbitrary Time Blur Decomposition Using Critic-Guided TripleGAN.

Recent studies have proposed methods for extracting latent sharp frames from a single blurred image. However, these methods still suffer from limitations in restoring satisfactory images. In addition, most existing methods are limited to decomposing a blurred image into sharp frames with a fixed frame rate. To address these problems, we present an Arbitrary Time Blur Decomposition Triple Generative Adversarial Network (ABDGAN) that restores sharp frames with flexible frame rates. Our framework plays a min-max game consisting of a generator, a discriminator, and a time-code predictor. The generator serves as a time-conditional deblurring network, while the discriminator and the label predictor provide feedback to the generator on producing realistic and sharp image depending on given time code. To provide adequate feedback for the generator, we propose a critic-guided (CG) loss by collaboration of the discriminator and time-code predictor. We also propose a pairwise order-consistency (POC) loss to ensure that each pixel in a predicted image consistently corresponds to the same ground-truth frame. Extensive experiments show that our method outperforms previously reported methods in both qualitative and quantitative evaluations. Compared to the best competitor, the proposed ABDGAN improves PSNR, SSIM, and LPIPS on the GoPro test set by 16.67%, 9.16%, and 36.61%, respectively. For the B-Aist++ test set, our method shows improvements of 6.99%, 2.38%, and 17.05% in PSNR, SSIM, and LPIPS, respectively, compared to the best competitive method.

Edge Detection of Motion-Blurred Images Aided by Inertial Sensors.

Edge detection serves as the foundation for advanced image processing tasks. The accuracy of edge detection is significantly reduced when applied to motion-blurred images. In this paper, we propose an effective deblurring method adapted to the edge detection task, utilizing inertial sensors to aid in the deblurring process. To account for measurement errors of the inertial sensors, we transform them into blur kernel errors and apply a total-least-squares (TLS) based iterative optimization scheme to handle the image deblurring problem involving blur kernel errors, whose relating priors are learned by neural networks. We apply the Canny edge detection algorithm to each intermediate output of the iterative process and use all the edge detection results to calculate the network's total loss function, enabling a closer coupling between the edge detection task and the deblurring iterative process. Based on the BSDS500 edge detection dataset and an independent inertial sensor dataset, we have constructed a synthetic dataset for training and evaluating the network. Results on the synthetic dataset indicate that, compared to existing representative deblurring methods, our proposed approach demonstrates higher accuracy and robustness in edge detection of motion-blurred images.

Nonlinear Deblurring for Low-Light Saturated Image.

Single image deblurring has achieved significant progress for natural daytime images. Saturation is a common phenomenon in blurry images, due to the low light conditions and long exposure times. However, conventional linear deblurring methods usually deal with natural blurry images well but result in severe ringing artifacts when recovering low-light saturated blurry images. To solve this problem, we formulate the saturation deblurring problem as a nonlinear model, in which all the saturated and unsaturated pixels are modeled adaptively. Specifically, we additionally introduce a nonlinear function to the convolution operator to accommodate the procedure of the saturation in the presence of the blurring. The proposed method has two advantages over previous methods. On the one hand, the proposed method achieves the same high quality of restoring the natural image as seen in conventional deblurring methods, while also reducing the estimation errors in saturated areas and suppressing ringing artifacts. On the other hand, compared with the recent saturated-based deblurring methods, the proposed method captures the formation of unsaturated and saturated degradations straightforwardly rather than with cumbersome and error-prone detection steps. Note that, this nonlinear degradation model can be naturally formulated into a maximum-a posterioriframework, and can be efficiently decoupled into several solvable sub-problems via the alternating direction method of multipliers (ADMM). Experimental results on both synthetic and real-world images demonstrate that the proposed deblurring algorithm outperforms the state-of-the-art low-light saturation-based deblurring methods.

An Efficient Image Deblurring Network with a Hybrid Architecture.

Blurring is one of the main degradation factors in image degradation, so image deblurring is of great interest as a fundamental problem in low-level computer vision. Because of the limited receptive field, traditional CNNs lack global fuzzy region modeling, and do not make full use of rich context information between features. Recently, a transformer-based neural network structure has performed well in natural language tasks, inspiring rapid development in the field of defuzzification. Therefore, in this paper, a hybrid architecture based on CNN and transformers is used for image deblurring. Specifically, we first extract the shallow features of the blurred images using a cross-layer feature fusion block that emphasizes the contextual information of each feature extraction layer. Secondly, an efficient transformer module for extracting deep features is designed, which fully aggregates feature information at medium and long distances using vertical and horizontal intra- and inter-strip attention layers, and a dual gating mechanism is used as a feedforward neural network, which effectively reduces redundant features. Finally, the cross-layer feature fusion block is used to complement the feature information to obtain the deblurred image. Extensive experimental results on publicly available benchmark datasets GoPro, HIDE, and the real dataset RealBlur show that the proposed method outperforms the current mainstream deblurring algorithms and recovers the edge contours and texture details of the images more clearly.

Vision Transformers in Image Restoration: A Survey.

The Vision Transformer (ViT) architecture has been remarkably successful in image restoration. For a while, Convolutional Neural Networks (CNN) predominated in most computer vision tasks. Now, both CNN and ViT are efficient approaches that demonstrate powerful capabilities to restore a better version of an image given in a low-quality format. In this study, the efficiency of ViT in image restoration is studied extensively. The ViT architectures are classified for every task of image restoration. Seven image restoration tasks are considered: Image Super-Resolution, Image Denoising, General Image Enhancement, JPEG Compression Artifact Reduction, Image Deblurring, Removing Adverse Weather Conditions, and Image Dehazing. The outcomes, the advantages, the limitations, and the possible areas for future research are detailed. Overall, it is noted that incorporating ViT in the new architectures for image restoration is becoming a rule. This is due to some advantages compared to CNN, such as better efficiency, especially when more data are fed to the network, robustness in feature extraction, and a better feature learning approach that sees better the variances and characteristics of the input. Nevertheless, some drawbacks exist, such as the need for more data to show the benefits of ViT over CNN, the increased computational cost due to the complexity of the self-attention block, a more challenging training process, and the lack of interpretability. These drawbacks represent the future research direction that should be targeted to increase the efficiency of ViT in the image restoration domain.

Correction of motion artifact in CL based on MAFusNet.

Computed laminography (CL) is one of the best methods for nondestructive testing of plate-like objects. If the object and the detector move continually while the scanning is being done, the data acquisition efficiency of CL will be significantly increased. However, the projection images will contain motion artifact as a result. A multi-angle fusion network (MAFusNet) is presented in order to correct the motion artifact of CL projection images considering the properties of CL projection images. The multi-angle fusion module significantly increases the ability of MAFusNet to deblur by using data from nearby projection images, and the feature fusion module lessens information loss brought on by data flow between the encoders. In contrast to conventional deblurring networks, the MAFusNet network employs synthetic datasets for training and performed well on realistic data, proving the network's outstanding generalization. The multi-angle fusion-based network has a significant improvement in the correction effect of CL motion artifact through ablation study and comparison with existing classical deblurring networks, and the synthetic training dataset can also significantly lower the training cost, which can effectively improve the quality and efficiency of CL imaging in industrial nondestructive testing.

A New Framework of Designing Iterative Techniques for Image Deblurring.

In this work we present a framework of designing iterative techniques for image deblurring in inverse problem. The new framework is based on two observations about existing methods. We used Landweber method as the basis to develop and present the new framework but note that the framework is applicable to other iterative techniques. First, we observed that the iterative steps of Landweber method consist of a constant term, which is a low-pass filtered version of the already blurry observation. We proposed a modification to use the observed image directly. Second, we observed that Landweber method uses an estimate of the true image as the starting point. This estimate, however, does not get updated over iterations. We proposed a modification that updates this estimate as the iterative process progresses. We integrated the two modifications into one framework of iteratively deblurring images. Finally, we tested the new method and compared its performance with several existing techniques, including Landweber method, Van Cittert method, GMRES (generalized minimal residual method), and LSQR (least square), to demonstrate its superior performance in image deblurring.

Blind Remote Sensing Image Deblurring Based on Overlapped Patches' Non-Linear Prior.

The remote sensing imaging environment is complex, in which many factors cause image blur. Thus, without prior knowledge, the restoration model established to obtain clear images can only rely on the observed blurry images. We still build the prior with extreme pixels but no longer traverse all pixels, such as the extreme channels. The features are extracted in units of patches, which are segmented from an image and partially overlap with each other. In this paper, we design a new prior, i.e., overlapped patches' non-linear (OPNL) prior, derived from the ratio of extreme pixels affected by blurring in patches. The analysis of more than 5000 remote sensing images confirms that OPNL prior prefers clear images rather than blurry images in the restoration process. The complexity of the optimization problem is increased due to the introduction of OPNL prior, which makes it impossible to solve it directly. A related solving algorithm is established based on the projected alternating minimization (PAM) algorithm combined with the half-quadratic splitting method, the fast iterative shrinkage-thresholding algorithm (FISTA), fast Fourier transform (FFT), etc. Numerous experiments prove that this algorithm has excellent stability and effectiveness and has obtained competitive processing results in restoring remote sensing images.

Blind Deblurring of Remote-Sensing Single Images Based on Feature Alignment.

Motion blur recovery is a common method in the field of remote sensing image processing that can effectively improve the accuracy of detection and recognition. Among the existing motion blur recovery methods, the algorithms based on deep learning do not rely on a priori knowledge and, thus, have better generalizability. However, the existing deep learning algorithms usually suffer from feature misalignment, resulting in a high probability of missing details or errors in the recovered images. This paper proposes an end-to-end generative adversarial network (SDD-GAN) for single-image motion deblurring to address this problem and to optimize the recovery of blurred remote sensing images. Firstly, this paper applies a feature alignment module (FAFM) in the generator to learn the offset between feature maps to adjust the position of each sample in the convolution kernel and to align the feature maps according to the context; secondly, a feature importance selection module is introduced in the generator to adaptively filter the feature maps in the spatial and channel domains, preserving reliable details in the feature maps and improving the performance of the algorithm. In addition, this paper constructs a self-constructed remote sensing dataset (RSDATA) based on the mechanism of image blurring caused by the high-speed orbital motion of satellites. Comparative experiments are conducted on self-built remote sensing datasets and public datasets as well as on real remote sensing blurred images taken by an in-orbit satellite (CX-6(02)). The results show that the algorithm in this paper outperforms the comparison algorithm in terms of both quantitative evaluation and visual effects.

Dual-Branch Discrimination Network Using Multiple Sparse Priors for Image Deblurring.

Blind image deblurring is a challenging problem in computer vision, aiming to restore the sharp image from blurred observation. Due to the incompatibility between the complex unknown degradation and the simple synthetic model, directly training a deep convolutional neural network (CNN) usually cannot sufficiently handle real-world blurry images. An existed generative adversarial network (GAN) can generate more detailed and realistic images, but the game between generator and discriminator is unbalancing, which leads to the training parameters not being able to converge to the ideal Nash equilibrium points. In this paper, we propose a GAN with a dual-branch discriminator using multiple sparse priors for image deblurring (DBSGAN) to overcome this limitation. By adding the multiple sparse priors into the other branch of the discriminator, the task of the discriminator is more complex. It can balance the game between the generator and the discriminator. Extensive experimental results on both synthetic and real-world blurry image datasets demonstrate the superior performance of our method over the state of the art in terms of quantitative metrics and visual quality. Especially for the GOPRO dataset, the averaged PSNR improves 1.7% over others.

Motion Blur Kernel Rendering Using an Inertial Sensor: Interpreting the Mechanism of a Thermal Detector.

Various types of motion blur are frequently observed in the images captured by sensors based on thermal and photon detectors. The difference in mechanisms between thermal and photon detectors directly results in different patterns of motion blur. Motivated by this observation, we propose a novel method to synthesize blurry images from sharp images by analyzing the mechanisms of the thermal detector. Further, we propose a novel blur kernel rendering method, which combines our proposed motion blur model with the inertial sensor in the thermal image domain. The accuracy of the blur kernel rendering method is evaluated by the task of thermal image deblurring. We construct a synthetic blurry image dataset based on acquired thermal images using an infrared camera for evaluation. This dataset is the first blurry thermal image dataset with ground-truth images in the thermal image domain. Qualitative and quantitative experiments are extensively carried out on our dataset, which show that our proposed method outperforms state-of-the-art methods.

Blind Deblurring Based on Sigmoid Function.

Blind image deblurring, also known as blind image deconvolution, is a long-standing challenge in the field of image processing and low-level vision. To restore a clear version of a severely degraded image, this paper proposes a blind deblurring algorithm based on the sigmoid function, which constructs novel blind deblurring estimators for both the original image and the degradation process by exploring the excellent property of sigmoid function and considering image derivative constraints. Owing to these symmetric and non-linear estimators of low computation complexity, high-quality images can be obtained by the algorithm. The algorithm is also extended to image sequences. The sigmoid function enables the proposed algorithm to achieve state-of-the-art performance in various scenarios, including natural, text, face, and low-illumination images. Furthermore, the method can be extended naturally to non-uniform deblurring. Quantitative and qualitative experimental evaluations indicate that the algorithm can remove the blur effect and improve the image quality of actual and simulated images. Finally, the use of sigmoid function provides a new approach to algorithm performance optimization in the field of image restoration.

SharpGAN: Dynamic Scene Deblurring Method for Smart Ship Based on Receptive Field Block and Generative Adversarial Networks.

Complex marine environment has an adverse effect on the object detection algorithm based on the vision sensor for the smart ship sailing at sea. In order to eliminate the motion blur in the images during the navigation of the smart ship and ensure safety, we propose SharpGAN, a new image deblurring method based on the generative adversarial network (GAN). First of all, we introduce the receptive field block net (RFBNet) to the deblurring network to enhance the network's ability to extract blurred image features. Secondly, we propose a feature loss that combines different levels of image features to guide the network to perform higher-quality deblurring and improve the feature similarity between the restored images and the sharp images. Besides, we use the lightweight RFB-s module to significantly improve the real-time performance of the deblurring network. Compared with the existing deblurring methods, the proposed method not only has better deblurring performance in subjective visual effects and objective evaluation criteria, but also has higher deblurring efficiency. Finally, the experimental results reveal that the SharpGAN has a high correlation with the deblurring methods based on the physical model.

Frequency Disentanglement Distillation Image Deblurring Network.

Due to the blur information and content information entanglement in the blind deblurring task, it is very challenging to directly recover the sharp latent image from the blurred image. Considering that in the high-dimensional feature map, blur information mainly exists in the low-frequency region, and content information exists in the high-frequency region. In this paper, we propose a encoder-decoder model to realize disentanglement from the perspective of frequency, and we named it as frequency disentanglement distillation image deblurring network (FDDN). First, we modified the traditional distillation block by embedding the frequency split block (FSB) in the distillation block to separate the low-frequency and high-frequency region. Second, the modified distillation block, we named frequency distillation block (FDB), can recursively distill the low-frequency feature to disentangle the blurry information from the content information, so as to improve the restored image quality. Furthermore, to reduce the complexity of the network and ensure the high-dimension of the feature map, the frequency distillation block (FDB) is placed on the end of encoder to edit the feature map on the latent space. Quantitative and qualitative experimental evaluations indicate that the FDDN can remove the blur effect and improve the image quality of actual and simulated images.

A Lightweight Fusion Distillation Network for Image Deblurring and Deraining.

Recently, deep learning-based image deblurring and deraining have been well developed. However, most of these methods fail to distill the useful features. What is more, exploiting the detailed image features in a deep learning framework always requires a mass of parameters, which inevitably makes the network suffer from a high computational burden. We propose a lightweight fusion distillation network (LFDN) for image deblurring and deraining to solve the above problems. The proposed LFDN is designed as an encoder-decoder architecture. In the encoding stage, the image feature is reduced to various small-scale spaces for multi-scale information extraction and fusion without much information loss. Then, a feature distillation normalization block is designed at the beginning of the decoding stage, which enables the network to distill and screen valuable channel information of feature maps continuously. Besides, an information fusion strategy between distillation modules and feature channels is also carried out by the attention mechanism. By fusing different information in the proposed approach, our network can achieve state-of-the-art image deblurring and deraining results with a smaller number of parameters and outperform the existing methods in model complexity.

Image Deblurring Using Multi-Stream Bottom-Top-Bottom Attention Network and Global Information-Based Fusion and Reconstruction Network.

Image deblurring has been a challenging ill-posed problem in computer vision. Gaussian blur is a common model for image and signal degradation. The deep learning-based deblurring methods have attracted much attention due to their advantages over the traditional methods relying on hand-designed features. However, the existing deep learning-based deblurring techniques still cannot perform well in restoring the fine details and reconstructing the sharp edges. To address this issue, we have designed an effective end-to-end deep learning-based non-blind image deblurring algorithm. In the proposed method, a multi-stream bottom-top-bottom attention network (MBANet) with the encoder-to-decoder structure is designed to integrate low-level cues and high-level semantic information, which can facilitate extracting image features more effectively and improve the computational efficiency of the network. Moreover, the MBANet adopts a coarse-to-fine multi-scale strategy to process the input images to improve image deblurring performance. Furthermore, the global information-based fusion and reconstruction network is proposed to fuse multi-scale output maps to improve the global spatial information and recurrently refine the output deblurred image. The experiments were done on the public GoPro dataset and the realistic and dynamic scenes (REDS) dataset to evaluate the effectiveness and robustness of the proposed method. The experimental results show that the proposed method generally outperforms some traditional deburring methods and deep learning-based state-of-the-art deblurring methods such as scale-recurrent network (SRN) and denoising prior driven deep neural network (DPDNN) in terms of such quantitative indexes as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) and human vision.

Intelligent Image De-Blurring for Imaging Flow Cytometry.

By virtue of the combined merits of optical microscopy and flow cytometry, imaging flow cytometry is a powerful tool for rapid, high-content analysis of single cells in large heterogeneous populations. However, its efficiency (defined by the ratio of the number of clearly imaged cells to the total cell population) is not high (typically 50-80%), due to out-of-focus image blurring caused by imperfect fluidic focusing of cells, a common drawback that not only reduces the number of cell images useable for high-content analysis but also increases the probability of false events and missed rare cells. To address this challenge and expand the efficacy of imaging flow cytometry, here, we propose and demonstrate intelligent deblurring of out-of-focus cell images in imaging flow cytometry. Specifically, by using our machine learning algorithms, we show an 11% increase in variance and a 95% increase in first-order gradient summation of cell images taken with an optofluidic time-stretch microscope. Without strict hardware requirements, our intelligent de-blurring method provides a promising solution to the out-of-focus blurring problem of imaging flow cytometers and holds promise for significantly improving their performance. © 2019 International Society for Advancement of Cytometry.

A New Design in Iterative Image Deblurring for Improved Robustness and Performance.

In many applications, image deblurring is a pre-requisite to improve the sharpness of an image before it can be further processed. Iterative methods are widely used for deblurring images but care must be taken to ensure that the iterative process is robust, meaning that the process does not diverge and reaches the solution reasonably fast, two goals that sometimes compete against each other. In practice, it remains challenging to choose parameters for the iterative process to be robust. We propose a new approach consisting of relaxed initialization and pixel-wise updates of the step size for iterative methods to achieve robustness. The first novel design of the approach is to modify the initialization of existing iterative methods to stop a noise term from being propagated throughout the iterative process. The second novel design is the introduction of a vectorized step size that is adaptively determined through the iteration to achieve higher stability and accuracy in the whole iterative process. The vectorized step size aims to update each pixel of an image individually, instead of updating all the pixels by the same factor. In this work, we implemented the above designs based on the Landweber method to test and demonstrate the new approach. Test results showed that the new approach can deblur images from noisy observations and achieve a low mean squared error with a more robust performance.

Ensemble Dictionary Learning for Single Image Deblurring via Low-Rank Regularization.

Sparse representation is a powerful statistical technique that has been widely utilized in image restoration applications. In this paper, an improved sparse representation model regularized by a low-rank constraint is proposed for single image deblurring. The key motivation for the proposed model lies in the observation that natural images are full of self-repetitive structures and they can be represented by similar patterns. However, as input images contain noise, blur, and other visual artifacts, extracting nonlocal similarities only with patch clustering algorithms is insufficient. In this paper, we first propose an ensemble dictionary learning method to represent different similar patterns. Then, low-rank embedded regularization is directly imposed on inputs to regularize the desired solution space which favors natural and sharp structures. The proposed method can be optimized by alternatively solving nuclear norm minimization and l 1 norm minimization problems to achieve higher restoration quality. Experimental comparisons validate the superior results of the proposed method compared with other deblurring algorithms in terms of visual quality and quantitative metrics.

A Compressed-Sensing Based Blind Deconvolution Method for Image Deblurring in Dental Cone-Beam Computed Tomography.

In cone-beam computed tomography (CBCT), reconstructed images are inherently degraded, restricting its image performance, due mainly to imperfections in the imaging process resulting from detector resolution, noise, X-ray tube's focal spot, and reconstruction procedure as well. Thus, the recovery of CBCT images from their degraded version is essential for improving image quality. In this study, we investigated a compressed-sensing (CS)-based blind deconvolution method to solve the blurring problem in CBCT where both the image to be recovered and the blur kernel (or point-spread function) of the imaging system are simultaneously recursively identified. We implemented the proposed algorithm and performed a systematic simulation and experiment to demonstrate the feasibility of using the algorithm for image deblurring in dental CBCT. In the experiment, we used a commercially available dental CBCT system that consisted of an X-ray tube, which was operated at 90 kVp and 5 mA, and a CMOS flat-panel detector with a 200-µm pixel size. The image characteristics were quantitatively investigated in terms of the image intensity, the root-mean-square error, the contrast-to-noise ratio, and the noise power spectrum. The results indicate that our proposed method effectively reduced the image blur in dental CBCT, excluding repetitious measurement of the system's blur kernel.