Adaptive imaging through dense dynamic scattering media using transfer learning.

Imaging through scattering media is a long-standing challenge in optical imaging, holding substantial importance in fields like biology, transportation, and remote sensing. Recent advancements in learning-based methods allow accurate and rapid imaging through optically thick scattering media. However, the practical application of data-driven deep learning faces substantial hurdles due to its inherent limitations in generalization, especially in scenarios such as imaging through highly non-static scattering media. Here we utilize the concept of transfer learning toward adaptive imaging through dense dynamic scattering media. Our approach specifically involves using a known segment of the imaging target to fine-tune the pre-trained de-scattering model. Since the training data of downstream tasks used for transfer learning can be acquired simultaneously with the current test data, our method can achieve clear imaging under varying scattering conditions. Experiment results show that the proposed approach (with transfer learning) is capable of providing more than 5dB improvements when optical thickness varies from 11.6 to 13.1 compared with the conventional deep learning approach (without transfer learning). Our method holds promise for applications in video surveillance and beacon guidance under dense dynamic scattering conditions.

Multifunctional Metasurfaces Enabled by Multifold Geometric Phase Interference.

Geometric phase is frequently used in artificially designed metasurfaces; it is typically used only once in reported works, leading to conjugate responses of two spins. Supercells containing multiple nanoantennas can break this limitation by introducing more degrees of freedom to generate new modulation capabilities. Here, we provide a method for constructing supercells for geometric phases using triple rotations, each of which achieves a specific modulation function. The physical meaning of each rotation is revealed by stepwise superposition. Based on this idea, spin-selective holography, nanoprinting, and their hybrid displays are demonstrated. As a typical application, we have designed a metalens that enables spin-selective transmission, allowing for high-quality imaging with only one spin state, which can serve as a plug-and-play chiral detection device. Finally, we analyzed how the size of supercells and the phase distribution inside it can affect the higher order diffraction, which may help in designing supercells for different scenarios.

DeepSCI: scalable speckle correlation imaging using physics-enhanced deep learning.

In this Letter we present a physics-enhanced deep learning approach for speckle correlation imaging (SCI), i.e., DeepSCI. DeepSCI incorporates the theoretical model of SCI into both the training and test stages of a neural network to achieve interpretable data preprocessing and model-driven fine-tuning, allowing the full use of data and physics priors. It can accurately reconstruct the image from the speckle pattern and is highly scalable to both medium perturbations and domain shifts. Our experimental results demonstrate the suitability and effectiveness of DeepSCI for solving the problem of limited generalization generally encountered in data-driven approaches.

Dammann gratings-based truly parallel optical matrix multiplication accelerator.

Matrix multiplication (MM) is a fundamental operation in various scientific and engineering computations, as well as in artificial intelligence algorithms. Efficient implementation of MM is crucial for speeding up numerous applications. Photonics presents an opportunity for efficient acceleration of dense matrix computation, owing to its intrinsic advantages, such as huge parallelism, low latency, and low power consumption. However, most optical matrix computing architectures have been limited to realizing single-channel vector-matrix multiplication or using complex configurations to expand the number of channels, which does not fully exploit the parallelism of optics. In this study, we propose a novel, to the best of our knowledge, scheme for the implementation of large-scale two-dimensional optical MM with truly massive parallelism based on a specially designed Dammann grating. We demonstrate a sequence of MMs of 50 pairs of randomly generated 4 × 8 and 8 × 4 matrices in our proof-of-principle experiment. The results indicate that the mean relative error is approximately 0.048, thereby demonstrating optical robustness and high accuracy.

Learning-based adaptive under-sampling for Fourier single-pixel imaging.

In this Letter, we present a learning-based method for efficient Fourier single-pixel imaging (FSI). Based on the auto-encoder, the proposed adaptive under-sampling technique (AuSamNet) manages to optimize a sampling mask and a deep neural network at the same time to achieve both under-sampling of the object image's Fourier spectrum and high-quality reconstruction from the under-sampled measurements. It is thus helpful in determining the best encoding and decoding scheme for FSI. Simulation and experiments demonstrate that AuSamNet can reconstruct high-quality natural color images even when the sampling ratio is as low as 7.5%. The proposed adaptive under-sampling strategy can be used for other computational imaging modalities, such as tomography and ptychography. We have released our source code.

Optical random micro-phase-shift DropConnect in a diffractive deep neural network.

The formulation and training of unitary neural networks is the basis of an active modulation diffractive deep neural network. In this Letter, an optical random phase DropConnect is implemented on an optical weight to manipulate a jillion of optical connections in the form of massively parallel sub-networks, in which a micro-phase assumed as an essential ingredient is drilled into Bernoulli holes to enable training convergence, and malposed deflections of the geometrical phase ray are reformulated constantly in epochs, allowing for enhancement of statistical inference. Optically, the random micro-phase-shift acts like a random phase sparse griddle with respect to values and positions, and is operated in the optical path of a projective imaging system. We investigate the performance of the full-drilling and part-drilling phenomena. In general, random micro-phase-shift part-drilling outperforms its full-drilling counterpart both in the training and inference since there are more possible recombinations of geometrical ray deflections induced by random phase DropConnect.

Single-shot noninvasive imaging through scattering medium under white-light illumination.

We experimentally investigate image reconstruction through a scattering medium under white-light illumination. To solve the inverse problem of noninvasive scattering imaging, a modified iterative algorithm is employed with an interpretable constraint on the optical transfer function (OTF). As a result, a sparse and real object can be retrieved whether it is illuminated with a narrowband or broadband light. Compared with the well-known speckle correlation technique (SCT), the proposed method requires no restrictions on the speckle autocorrelation and shows a potential advantage in scattering imaging.

Non-line-of-sight imaging under white-light illumination: a two-step deep learning approach.

Non-line-of-sight (NLOS) imaging has received considerable attentions for its ability to recover occluded objects from an indirect view. Various NLOS imaging techniques have been demonstrated recently. Here, we propose a white-light NLOS imaging method that is equipped only with an ordinary camera, and not necessary to operate under active coherent illumination as in other existing NLOS systems. The central idea is to incorporate speckle correlation-based model into a deep neural network (DNN), and form a two-step DNN strategy that endeavors to learn the optimization of the scattered pattern autocorrelation and object image reconstruction, respectively. Optical experiments are carried out to demonstrate the proposed method.

Two-step training deep learning framework for computational imaging without physics priors.

Deep learning (DL) is a powerful tool in computational imaging for many applications. A common strategy is to use a preprocessor to reconstruct a preliminary image as the input to a neural network to achieve an optimized image. Usually, the preprocessor incorporates knowledge of the physics priors in the imaging model. One outstanding challenge, however, is errors that arise from imperfections in the assumed model. Model mismatches degrade the quality of the preliminary image and therefore affect the DL predictions. Another main challenge is that many imaging inverse problems are ill-posed and the networks are over-parameterized; DL networks have flexibility to extract features from the data that are not directly related to the imaging model. This can lead to suboptimal training and poorer image reconstruction results. To solve these challenges, a two-step training DL (TST-DL) framework is proposed for computational imaging without physics priors. First, a single fully-connected layer (FCL) is trained to directly learn the inverse model with the raw measurement data as the inputs and the images as the outputs. Then, this pre-trained FCL is fixed and concatenated with an un-trained deep convolutional network with a U-Net architecture for a second-step training to optimize the output image. This approach has the advantage that does not rely on an accurate representation of the imaging physics since the first-step training directly learns the inverse model. Furthermore, the TST-DL approach mitigates network over-parameterization by separately training the FCL and U-Net. We demonstrate this framework using a linear single-pixel camera imaging model. The results are quantitatively compared with those from other frameworks. The TST-DL approach is shown to perform comparable to approaches which incorporate perfect knowledge of the imaging model, to be robust to noise and model ill-posedness, and to be more robust to model mismatch than approaches which incorporate imperfect knowledge of the imaging model. Furthermore, TST-DL yields better results than end-to-end training while suffering from less overfitting. Overall, this TST-DL framework is a flexible approach for image reconstruction without physics priors, applicable to diverse computational imaging systems.

Optical random phase dropout in a diffractive deep neural network.

Unitary learning is a backpropagation (BP) method that serves to update unitary weights in fully connected deep complex-valued neural networks, meeting a prior unitary in an active modulation diffractive deep neural network. However, the square matrix characteristic of unitary weights in each layer results in its learning belonging to a small-sample training, which produces an almost useless network that has a fairly poor generalization capability. To alleviate such a serious over-fitting problem, in this Letter, optical random phase dropout is formulated and designed. The equivalence between unitary forward and diffractive networks deduces a synthetic mask that is seamlessly compounded with a computational modulation and a random sampling comb called dropout. The dropout is filled with random phases in its zero positions that satisfy the Bernoulli distribution, which could slightly deflect parts of transmitted optical rays in each output end to generate statistical inference networks. The enhancement of generalization benefits from the fact that massively parallel full connection with different optical links is involved in the training. The random phase comb introduced into unitary BP is in the form of conjugation, which indicates the significance of optical BP.

Recovery of the topological charge of a vortex beam propagated through a scattering layer.

Coherent vortex beams have shown great potential in many applications including information transmission under non-ideal conditions, as information can be encoded in the orbital angular momentum. However, inhomogeneity of atmosphere tends to scramble the vortex structure and give rise to speckle. It is therefore of great interest to reconstruct the topological charge of a vortex beam after it propagates through a scattering medium. Here, we propose a feasible solution for this. The proposed method measures holographically the scattered field and reconstructs the spiral phase from it by taking advantage of both the deterministic nature and the ergodicity of the scattering process. Our preliminary experiments show promising results and suggest that the proposed method can have great potential in information transmission under non-ideal conditions.

Sparse regularization-based reconstruction for 3D flame chemiluminescence tomography.

Flame chemiluminescence tomography (FCT) is a non-intrusive method that is based on using cameras to measure projections, and it plays a crucial role in combustion diagnostics and measurement. Mathematically, the inversion problem is ill-posed, and in the case of limited optical accessibility in practical applications, it is rank deficient. Therefore, the solution process should ideally be supported by prior information, which can be based on the known physics. In this work, the total variation (TV) regularization has been combined with the well-known algebraic reconstruction technique (ART) for practical FCT applications. The TV method endorses smoothness while also preserving typical flame features such as the flame front. Split Bregman iteration has been adopted for TV minimization. Five different noise conditions and the chosen regularization parameter have been tested in numerical studies. Additionally, for the 12 perspectives, an experimental FCT system is demonstrated, which is utilized to recover the three-dimensional (3D) chemiluminescence distribution of candle flames. Both the numerical and experimental studies show that the typical line artifacts that appear with the conventional ART algorithm when recovering the continuous chemiluminescence field of the flames are significantly reduced with the proposed algorithm.

Single-shot three-dimensional imaging with a scattering layer [Invited].

In this paper, we propose a single-shot three-dimensional imaging technique. This is achieved by simply placing a normal thin scattering layer in front of a two-dimensional image sensor, making it a light-field-like camera. The working principle of the proposed technique is based on the statistical independence and spatial ergodicity of the speckle produced by the scattering layer. Thus, the local point responses of the scattering layer should be measured in advance and are used for image reconstruction. We demonstrate the proposed method with proof-of-concept experiments and analyze the factors that affect its performance.

Chosen-plaintext attack on the double random polarization encryption.

Images can be optically encrypted by random encoding in the phase, the polarization, or even the coherence of a light field. It is important for these optical encryption methods to undergo rigorous cryptanalysis. However, only phase-encoding-based encryption has been rigorously analyzed to date. In this manuscript, we demonstrate that the double random polarization encryption (DRPolE) is vulnerable to chosen-plaintext attack (CPA). We show that the keys can be retrieved if one can choose the polarization states of two plaintext images and collect the corresponding cyphertext images. Our study reveals a serious concern regarding the DRPolE that should be addressed in the design of polarization-based optical encryption methods.

Learning from simulation: An end-to-end deep-learning approach for computational ghost imaging.

Artificial intelligence (AI) techniques such as deep learning (DL) for computational imaging usually require to experimentally collect a large set of labeled data to train a neural network. Here we demonstrate that a practically usable neural network for computational imaging can be trained by using simulation data. We take computational ghost imaging (CGI) as an example to demonstrate this method. We develop a one-step end-to-end neural network, trained with simulation data, to reconstruct two-dimensional images directly from experimentally acquired one-dimensional bucket signals, without the need of the sequence of illumination patterns. This is in particular useful for image transmission through quasi-static scattering media as little care is needed to take to simulate the scattering process when generating the training data. We believe that the concept of training using simulation data can be used in various DL-based solvers for general computational imaging.

Digital holography and 3D imaging: introduction to the Applied Optics and Journal of the Optical Society of America A joint feature issue.

The OSA Topical Meeting on Digital Holography and 3D Imaging (DH) was held June 25-28, 2018, in Orlando, Florida, USA. Feature issues based on the DH meeting series have been released by Applied Optics (AO) since 2007. This year, AO and the Journal of the Optical Society of America A (JOSA A) jointly decided to have one such feature issue in each journal. This feature issue includes thirty-eight papers in AO and nine in JOSA A, and covers a large range of topics, reflecting the rapidly expanding techniques and applications of digital holography and 3D imaging. The upcoming DH Conference (DH 2019) will be held May 19-23 in Bordeaux, France.

Cyphertext-only attack on the joint-transform-correlator-based optical encryption: experimental demonstration.

A cyphertext-only attack (COA) on a joint transform correlator encryption system is proposed. Under the proposed COA scheme, the energy spectral density of plaintext can be calculated with cyphertext, and then plaintext information can be recovered by using a phase retrieval algorithm such as the hybrid input-output algorithm. We also numerically and experimentally demonstrate the COA to verify feasibility of the proposed technique.

Digital holography and 3D imaging: introduction to the Applied Optics and Journal of the Optical Society of America A joint feature issue.

The OSA Topical Meeting on Digital Holography and 3D Imaging (DH) was held June 25-June 28, 2018, in Orlando, Florida, USA. Feature issues based on the DH meeting series have been released by Applied Optics (AO) since 2007. This year, AO and the Journal of the Optical Society of America A (JOSA A) jointly decided to have one such feature issue in each journal. This feature issue includes thirty-eight papers in AO and nine in JOSA A, and covers a large range of topics, reflecting the rapidly expanding techniques and applications of digital holography and 3D imaging. The upcoming DH Conference (DH 2019) will be held May 19-May 23 in Bordeaux, France.

eHoloNet: a learning-based end-to-end approach for in-line digital holographic reconstruction.

It is well known that in-line digital holography (DH) makes use of the full pixel count in forming the holographic imaging. But it usually requires phase-shifting or phase retrieval techniques to remove the zero-order and twin-image terms, resulting in the so-called two-step reconstruction process, i.e., phase recovery and focusing. Here, we propose a one-step end-to-end learning-based method for in-line holography reconstruction, namely, the eHoloNet, which can reconstruct the object wavefront directly from a single-shot in-line digital hologram. In addition, the proposed learning-based DH technique has strong robustness to the change of optical path difference between reference beam and object light and does not require the reference beam to be a plane or spherical wave.

Digital Holography and 3D Imaging: introduction to the joint feature issue in Applied Optics and Journal of the Optical Society of America A.

The OSA Topical Meeting on Digital Holography and 3D Imaging (DH) was held 29 May to 1 June 2017 on Jeju Island, South Korea. Feature issues based on the DH meeting series have been released by Applied Optics (AO) since 2007. This year, AO and the Journal of the Optical Society of America A (JOSA A) jointly decided to have one such feature issue in each journal. This feature issue includes 33 papers in AO and 9 in JOSA A and covers a large range of topics, reflecting the rapidly expanding techniques and applications of digital holography and 3D imaging. The upcoming DH meeting (DH 2018) will be held 25-28 June 2018 in Orlando, Florida, as part of the OSA Imaging and Applied Optics Congress.