Video reconstruction from a single motion blurred image using learned dynamic phase coding.

Video reconstruction from a single motion-blurred image is a challenging problem, which can enhance the capabilities of existing cameras. Recently, several works addressed this task using conventional imaging and deep learning. Yet, such purely digital methods are inherently limited, due to direction ambiguity and noise sensitivity. Some works attempt to address these limitations with non-conventional image sensors, however, such sensors are extremely rare and expensive. To circumvent these limitations by simpler means, we propose a hybrid optical-digital method for video reconstruction that requires only simple modifications to existing optical systems. We use learned dynamic phase-coding in the lens aperture during image acquisition to encode motion trajectories, which serve as prior information for the video reconstruction process. The proposed computational camera generates a sharp frame burst of the scene at various frame rates from a single coded motion-blurred image, using an image-to-video convolutional neural network. We present advantages and improved performance compared to existing methods, with both simulations and a real-world camera prototype. We extend our optical coding to video frame interpolation and present robust and improved results for noisy videos.

A Function Space Analysis of Finite Neural Networks With Insights From Sampling Theory.

This work suggests using sampling theory to analyze the function space represented by interpolating mappings. While the analysis in this paper is general, we focus it on neural networks with bounded weights that are known for their ability to interpolate (fit) the training data. First, we show, under the assumption of a finite input domain, which is the common case in training neural networks, that the function space generated by multi-layer networks with bounded weights, and non-expansive activation functions are smooth. This extends over previous works that show results for the case of infinite width ReLU networks. Then, under the assumption that the input is band-limited, we provide novel error bounds for univariate neural networks. We analyze both deterministic uniform and random sampling showing the advantage of the former.

An artificial neural network explains how bats might use vision for navigation.

Animals navigate using various sensory information to guide their movement. Miniature tracking devices now allow documenting animals' routes with high accuracy. Despite this detailed description of animal movement, how animals translate sensory information to movement is poorly understood. Recent machine learning advances now allow addressing this question with unprecedented statistical learning tools. We harnessed this power to address visual-based navigation in fruit bats. We used machine learning and trained a convolutional neural network to navigate along a bat's route using visual information that would have been available to the real bat, which we collected using a drone. We show that a simple feed-forward network can learn to guide the agent towards a goal based on sensory input, and can generalize its learning both in time and in space. Our analysis suggests how animals could potentially use visual input for navigation and which features might be useful for this purpose.

Use of machine learning to achieve keratoconus detection skills of a corneal expert.

PURPOSE: To construct an automatic machine-learning derived algorithm discriminating between normal corneas and suspect irregular or keratoconic corneas. METHODS: A total of 8526 corneal tomography images of 4904 eyes obtained between November 2010 and July 2017 using a combined Scheimpflug/Placido tomographer were retrospectively evaluated. Each image was evaluated for acquisition quality and was labeled as normal, suspect irregular or keratoconic by a cornea specialist. Two algorithms were built. The first was based on 94 instrument-derived output parameters, and the second integrated keratoconus prediction indices of the device with the 94 instrument-derived output parameters. Both models were compared with the tomographer's keratoconus detection algorithms. Out of the 8526 images evaluated, 7104 images of 3787 eyes had sufficient acquisition quality. Of those, 5904 examinations were randomly chosen for construction of the models using the random forest algorithm. The models were then validated using the remaining 1200 examinations. RESULTS: Both RF algorithms had a larger AUC compared with any of the tomographer's KC detection algorithms (p < 10-9). The first constructed model had 90.2% accuracy, sensitivity of 94.2%, and specificity of 89.6% (Youden 0.838). Calculated AUC was 0.964. The second model had 91.5% accuracy, sensitivity of 94.7%, and specificity of 89.8% (Youden 0.846). Calculated AUC was 0.969. CONCLUSION: Using the RF machine-learning algorithm, accuracy of discrimination between normal, suspect irregular and keratoconic corneas approximates that of an experienced corneal expert. Applying machine learning to corneal tomography can facilitate keratoconus screening in large populations as well as off-site screening of refractive surgery candidates.

A Self Supervised StyleGAN for Image Annotation and Classification With Extremely Limited Labels.

The recent success of learning-based algorithms can be greatly attributed to the immense amount of annotated data used for training. Yet, many datasets lack annotations due to the high costs associated with labeling, resulting in degraded performances of deep learning methods. Self-supervised learning is frequently adopted to mitigate the reliance on massive labeled datasets since it exploits unlabeled data to learn relevant feature representations. In this work, we propose SS-StyleGAN, a self-supervised approach for image annotation and classification suitable for extremely small annotated datasets. This novel framework adds self-supervision to the StyleGAN architecture by integrating an encoder that learns the embedding to the StyleGAN latent space, which is well-known for its disentangled properties. The learned latent space enables the smart selection of representatives from the data to be labeled for improved classification performance. We show that the proposed method attains strong classification results using small labeled datasets of sizes 50 and even 10. We demonstrate the superiority of our approach for the tasks of COVID-19 and liver tumor pathology identification.

Back-Projection based Fidelity Term for Ill-Posed Linear Inverse Problems.

Ill-posed linear inverse problems appear in many image processing applications, such as deblurring, superresolution and compressed sensing. Many restoration strategies involve minimizing a cost function, which is composed of fidelity and prior terms, balanced by a regularization parameter. While a vast amount of research has been focused on different prior models, the fidelity term is almost always chosen to be the least squares (LS) objective, that encourages fitting the linearly transformed optimization variable to the observations. In this paper, we examine a different fidelity term, which has been implicitly used by the recently proposed iterative denoising and backward projections (IDBP) framework. This term encourages agreement between the projection of the optimization variable onto the row space of the linear operator and the pseudoinverse of the linear operator ("back-projection") applied on the observations. We analytically examine the difference between the two fidelity terms for Tikhonov regularization and identify cases (such as a badly conditioned linear operator) where the new term has an advantage over the standard LS one. Moreover, we demonstrate empirically that the behavior of the two induced cost functions for sophisticated convex and non-convex priors, such as total-variation, BM3D, and deep generative models, correlates with the obtained theoretical analysis.

Fiber-optic distributed seismic sensing data generator and its application for training classification nets.

Distributed acoustic sensing (DAS) is a powerful tool thanks to its ease of use, high spatial and temporal resolution, and sensitivity. Growing demand for long-distance distributed seismic sensing (DSeiS) measurements, in conjunction with the development of efficient algorithms for data processing, has led to an increased interest in the technology from industry and academia. Machine-learning-based data processing, however, necessitates tedious in situ calibration experiments that require significant effort and resources. In this Letter, a geophysics-driven approach for generating synthetic DSeiS data is described, analyzed, and tested. The generated synthetic data are used to train DSeiS classification algorithms. The approach is validated by training an artificial neural-network-based classifier using synthetic data and testing its performance on experimental DSeiS records. Accuracy is greatly improved thanks to the incorporation of a geophysical model when generating training data.

TOP-GAN: Stain-free cancer cell classification using deep learning with a small training set.

We propose a new deep learning approach for medical imaging that copes with the problem of a small training set, the main bottleneck of deep learning, and apply it for classification of healthy and cancer cell lines acquired by quantitative phase imaging. The proposed method, called transferring of pre-trained generative adversarial network (TOP-GAN), is hybridization between transfer learning and generative adversarial networks (GANs). Healthy cells and cancer cells of different metastatic potential have been imaged by low-coherence off-axis holography. After the acquisition, the optical path delay maps of the cells are extracted and directly used as inputs to the networks. In order to cope with the small number of classified images, we use GANs to train a large number of unclassified images from another cell type (sperm cells). After this preliminary training, we change the last layers of the network and design automatic classifiers for the correct cell type (healthy/primary cancer/metastatic cancer) with 90-99% accuracies, although small training sets of down to several images are used. These results are better in comparison to other classic methods that aim at coping with the same problem of a small training set. We believe that our approach makes the combination of holographic microscopy and deep learning networks more accessible to the medical field by enabling a rapid, automatic and accurate classification in stain-free imaging flow cytometry. Furthermore, our approach is expected to be applicable to many other medical image classification tasks, suffering from a small training set.

Image Restoration by Iterative Denoising and Backward Projections.

Inverse problems appear in many applications, such as image deblurring and inpainting. The common approach to address them is to design a specific algorithm for each problem. The Plug-and-Play (P&P) framework, which has been recently introduced, allows solving general inverse problems by leveraging the impressive capabilities of existing denoising algorithms. While this fresh strategy has found many applications, a burdensome parameter tuning is often required in order to obtain high-quality results. In this paper, we propose an alternative method for solving inverse problems using off-the-shelf denoisers, which requires less parameter tuning. First, we transform a typical cost function, composed of fidelity and prior terms, into a closely related, novel optimization problem. Then, we propose an efficient minimization scheme with a P&P property, i.e., the prior term is handled solely by a denoising operation. Finally, we present an automatic tuning mechanism to set the method's parameters. We provide a theoretical analysis of the method and empirically demonstrate its competitiveness with task-specific techniques and the P&P approach for image inpainting and deblurring.

DeepISP: Towards Learning an End-to-End Image Processing Pipeline.

We present DeepISP, a full end-to-end deep neural model of the camera image signal processing (ISP) pipeline. Our model learns a mapping from the raw low-light mosaiced image to the final visually compelling image and encompasses low-level tasks such as demosaicing and denoising as well as higher-level tasks such as color correction and image adjustment. The training and evaluation of the pipeline were performed on a dedicated dataset containing pairs of low-light and well-lit images captured by a Samsung S7 smartphone camera in both raw and processed JPEG formats. The proposed solution achieves state-of-the-art performance in objective evaluation of PSNR on the subtask of joint denoising and demosaicing. For the full end-to-end pipeline, it achieves better visual quality compared to the manufacturer ISP, in both a subjective human assessment and when rated by a deep model trained for assessing image quality.

Learned phase coded aperture for the benefit of depth of field extension.

Modern consumer electronics market dictates the need for small-scale and high-performance cameras. Such designs involve trade-offs between various system parameters. In such trade-offs, Depth Of Field (DOF) is a significant issue very often. We propose a computational imaging-based technique to overcome DOF limitations. Our approach is based on the synergy between a simple phase aperture coding element and a convolutional neural network (CNN). The phase element, designed for DOF extension using color diversity in the imaging system response, causes chromatic variations by creating a different defocus blur for each color channel of the image. The phase-mask is designed such that the CNN model is able to restore from the coded image an all-in-focus image easily. This is achieved by using a joint end-to-end training of both the phase element and the CNN parameters using backpropagation. The proposed approach provides superior performance to other methods in simulations as well as in real-world scenes.

Class-Aware Fully-Convolutional Gaussian and Poisson Denoising.

We propose a fully-convolutional neural-network architecture for image denoising which is simple yet powerful. Its structure allows to exploit the gradual nature of the denoising process, in which shallow layers handle local noise statistics, while deeper layers recover edges and enhance textures. Our method advances the state-of-the-art when trained for different noise levels and distributions (both Gaussian and Poisson). In addition, we show that making the denoiser class-aware by exploiting semantic class information boosts performance, enhances textures and reduces artifacts.

Postprocessing of Compressed Images via Sequential Denoising.

In this paper, we propose a novel postprocessing technique for compression-artifact reduction. Our approach is based on posing this task as an inverse problem, with a regularization that leverages on existing state-of-the-art image denoising algorithms. We rely on the recently proposed Plug-and-Play Prior framework, suggesting the solution of general inverse problems via alternating direction method of multipliers, leading to a sequence of Gaussian denoising steps. A key feature in our scheme is a linearization of the compression-decompression process, so as to get a formulation that can be optimized. In addition, we supply a thorough analysis of this linear approximation for several basic compression procedures. The proposed method is suitable for diverse compression techniques that rely on transform coding. In particular, we demonstrate impressive gains in image quality for several leading compression methods-JPEG, JPEG2000, and HEVC.

Sparsity-based Poisson denoising with dictionary learning.

The problem of Poisson denoising appears in various imaging applications, such as low-light photography, medical imaging, and microscopy. In cases of high SNR, several transformations exist so as to convert the Poisson noise into an additive-independent identically distributed. Gaussian noise, for which many effective algorithms are available. However, in a low-SNR regime, these transformations are significantly less accurate, and a strategy that relies directly on the true noise statistics is required. Salmon et al took this route, proposing a patch-based exponential image representation model based on Gaussian mixture model, leading to state-of-the-art results. In this paper, we propose to harness sparse-representation modeling to the image patches, adopting the same exponential idea. Our scheme uses a greedy pursuit with boot-strapping-based stopping condition and dictionary learning within the denoising process. The reconstruction performance of the proposed scheme is competitive with leading methods in high SNR and achieving state-of-the-art results in cases of low SNR.