Low-Light Phase Retrieval With Implicit Generative Priors.

Phase retrieval (PR) is fundamentally important in scientific imaging and is crucial for nanoscale techniques like coherent diffractive imaging (CDI). Low radiation dose imaging is essential for applications involving radiation-sensitive samples. However, most PR methods struggle in low-dose scenarios due to high shot noise. Recent advancements in optical data acquisition setups, such as in-situ CDI, have shown promise for low-dose imaging, but they rely on a time series of measurements, making them unsuitable for single-image applications. Similarly, data-driven phase retrieval techniques are not easily adaptable to data-scarce situations. Zero-shot deep learning methods based on pre-trained and implicit generative priors have been effective in various imaging tasks but have shown limited success in PR. In this work, we propose low-dose deep image prior (LoDIP), which combines in-situ CDI with the power of implicit generative priors to address single-image low-dose phase retrieval. Quantitative evaluations demonstrate LoDIP's superior performance in this task and its applicability to real experimental scenarios.

Real space iterative reconstruction for vector tomography (RESIRE-V).

Tomography has had an important impact on the physical, biological, and medical sciences. To date, most tomographic applications have been focused on 3D scalar reconstructions. However, in some crucial applications, vector tomography is required to reconstruct 3D vector fields such as the electric and magnetic fields. Over the years, several vector tomography methods have been developed. Here, we present the mathematical foundation and algorithmic implementation of REal Space Iterative REconstruction for Vector tomography, termed RESIRE-V. RESIRE-V uses multiple tilt series of projections and iterates between the projections and a 3D reconstruction. Each iteration consists of a forward step using the Radon transform and a backward step using its transpose, then updates the object via gradient descent. Incorporating with a 3D support constraint, the algorithm iteratively minimizes an error metric, defined as the difference between the measured and calculated projections. The algorithm can also be used to refine the tilt angles and further improve the 3D reconstruction. To validate RESIRE-V, we first apply it to a simulated data set of the 3D magnetization vector field, consisting of two orthogonal tilt series, each with a missing wedge. Our quantitative analysis shows that the three components of the reconstructed magnetization vector field agree well with the ground-truth counterparts. We then use RESIRE-V to reconstruct the 3D magnetization vector field of a ferromagnetic meta-lattice consisting of three tilt series. Our 3D vector reconstruction reveals the existence of topological magnetic defects with positive and negative charges. We expect that RESIRE-V can be incorporated into different imaging modalities as a general vector tomography method. To make the algorithm accessible to a broad user community, we have made our RESIRE-V MATLAB source codes and the data freely available at https://github.com/minhpham0309/RESIRE-V .

In-context operator learning with data prompts for differential equation problems.

This paper introduces the paradigm of "in-context operator learning" and the corresponding model "In-Context Operator Networks" to simultaneously learn operators from the prompted data and apply it to new questions during the inference stage, without any weight update. Existing methods are limited to using a neural network to approximate a specific equation solution or a specific operator, requiring retraining when switching to a new problem with different equations. By training a single neural network as an operator learner, rather than a solution/operator approximator, we can not only get rid of retraining (even fine-tuning) the neural network for new problems but also leverage the commonalities shared across operators so that only a few examples in the prompt are needed when learning a new operator. Our numerical results show the capability of a single neural network as a few-shot operator learner for a diversified type of differential equation problems, including forward and inverse problems of ordinary differential equations, partial differential equations, and mean-field control problems, and also show that it can generalize its learning capability to operators beyond the training distribution.

Accurate real space iterative reconstruction (RESIRE) algorithm for tomography.

Tomography has made a revolutionary impact on the physical, biological and medical sciences. The mathematical foundation of tomography is to reconstruct a three-dimensional (3D) object from a set of two-dimensional (2D) projections. As the number of projections that can be measured from a sample is usually limited by the tolerable radiation dose and/or the geometric constraint on the tilt range, a main challenge in tomography is to achieve the best possible 3D reconstruction from a limited number of projections with noise. Over the years, a number of tomographic reconstruction methods have been developed including direct inversion, real-space, and Fourier-based iterative algorithms. Here, we report the development of a real-space iterative reconstruction (RESIRE) algorithm for accurate tomographic reconstruction. RESIRE iterates between the update of a reconstructed 3D object and the measured projections using a forward and back projection step. The forward projection step is implemented by the Fourier slice theorem or the Radon transform, and the back projection step by a linear transformation. Our numerical and experimental results demonstrate that RESIRE performs more accurate 3D reconstructions than other existing tomographic algorithms, when there are a limited number of projections with noise. Furthermore, RESIRE can be used to reconstruct the 3D structure of extended objects as demonstrated by the determination of the 3D atomic structure of an amorphous Ta thin film. We expect that RESIRE can be widely employed in the tomography applications in different fields. Finally, to make the method accessible to the general user community, the MATLAB source code of RESIRE and all the simulated and experimental data are available at  https://zenodo.org/record/7273314 .

A Hamilton-Jacobi-based proximal operator.

First-order optimization algorithms are widely used today. Two standard building blocks in these algorithms are proximal operators (proximals) and gradients. Although gradients can be computed for a wide array of functions, explicit proximal formulas are known for only limited classes of functions. We provide an algorithm, HJ-Prox, for accurately approximating such proximals. This is derived from a collection of relations between proximals, Moreau envelopes, Hamilton-Jacobi (HJ) equations, heat equations, and Monte Carlo sampling. In particular, HJ-Prox smoothly approximates the Moreau envelope and its gradient. The smoothness can be adjusted to act as a denoiser. Our approach applies even when functions are accessible only by (possibly noisy) black box samples. We show that HJ-Prox is effective numerically via several examples.

Taming hyperparameter tuning in continuous normalizing flows using the JKO scheme.

A normalizing flow (NF) is a mapping that transforms a chosen probability distribution to a normal distribution. Such flows are a common technique used for data generation and density estimation in machine learning and data science. The density estimate obtained with a NF requires a change of variables formula that involves the computation of the Jacobian determinant of the NF transformation. In order to tractably compute this determinant, continuous normalizing flows (CNF) estimate the mapping and its Jacobian determinant using a neural ODE. Optimal transport (OT) theory has been successfully used to assist in finding CNFs by formulating them as OT problems with a soft penalty for enforcing the standard normal distribution as a target measure. A drawback of OT-based CNFs is the addition of a hyperparameter, [Formula: see text], that controls the strength of the soft penalty and requires significant tuning. We present JKO-Flow, an algorithm to solve OT-based CNF without the need of tuning [Formula: see text]. This is achieved by integrating the OT CNF framework into a Wasserstein gradient flow framework, also known as the JKO scheme. Instead of tuning [Formula: see text], we repeatedly solve the optimization problem for a fixed [Formula: see text] effectively performing a JKO update with a time-step [Formula: see text]. Hence we obtain a "divide and conquer" algorithm by repeatedly solving simpler problems instead of solving a potentially harder problem with large [Formula: see text].

Three-dimensional topological magnetic monopoles and their interactions in a ferromagnetic meta-lattice.

Topological magnetic monopoles (TMMs), also known as hedgehogs or Bloch points, are three-dimensional (3D) non-local spin textures that are robust to thermal and quantum fluctuations due to the topology protection1-4. Although TMMs have been observed in skyrmion lattices1,5, spinor Bose-Einstein condensates6,7, chiral magnets8, vortex rings2,9 and vortex cores10, it has been difficult to directly measure the 3D magnetization vector field of TMMs and probe their interactions at the nanoscale. Here we report the creation of 138 stable TMMs at the specific sites of a ferromagnetic meta-lattice at room temperature. We further develop soft X-ray vector ptycho-tomography to determine the magnetization vector and emergent magnetic field of the TMMs with a 3D spatial resolution of 10 nm. This spatial resolution is comparable to the magnetic exchange length of transition metals11, enabling us to probe monopole-monopole interactions. We find that the TMM and anti-TMM pairs are separated by 18.3 ± 1.6 nm, while the TMM and TMM, and anti-TMM and anti-TMM pairs are stabilized at comparatively longer distances of 36.1 ± 2.4 nm and 43.1 ± 2.0 nm, respectively. We also observe virtual TMMs created by magnetic voids in the meta-lattice. This work demonstrates that ferromagnetic meta-lattices could be used as a platform to create and investigate the interactions and dynamics of TMMs. Furthermore, we expect that soft X-ray vector ptycho-tomography can be broadly applied to quantitatively image 3D vector fields in magnetic and anisotropic materials at the nanoscale.

Mean field control problems for vaccine distribution.

With the invention of the COVID-19 vaccine, shipping and distributing are crucial in controlling the pandemic. In this paper, we build a mean-field variational problem in a spatial domain, which controls the propagation of pandemics by the optimal transportation strategy of vaccine distribution. Here, we integrate the vaccine distribution into the mean-field SIR model designed in Lee W, Liu S, Tembine H, Li W, Osher S (2020) Controlling propagation of epidemics via mean-field games. arXiv preprint arXiv:2006.01249. Numerical examples demonstrate that the proposed model provides practical strategies for vaccine distribution in a spatial domain.

Extraction of Hidden Science from Nanoscale Images.

Scanning probe microscopies and spectroscopies enable investigation of surfaces and even buried interfaces down to the scale of chemical-bonding interactions, and this capability has been enhanced with the support of computational algorithms for data acquisition and image processing to explore physical, chemical, and biological phenomena. Here, we describe how scanning probe techniques have been enhanced by some of these recent algorithmic improvements. One improvement to the data acquisition algorithm is to advance beyond a simple rastering framework by using spirals at constant angular velocity then switching to constant linear velocity, which limits the piezo creep and hysteresis issues seen in traditional acquisition methods. One can also use image-processing techniques to model the distortions that appear from tip motion effects and to make corrections to these images. Another image-processing algorithm we discuss enables researchers to segment images by domains and subdomains, thereby highlighting reactive and interesting disordered sites at domain boundaries. Lastly, we discuss algorithms used to examine the dipole direction of individual molecules and surface domains, hydrogen bonding interactions, and molecular tilt. The computational algorithms used for scanning probe techniques are still improving rapidly and are incorporating machine learning at the next level of iteration. That said, the algorithms are not yet able to perform live adjustments during data recording that could enhance the microscopy and spectroscopic imaging methods significantly.

Three-dimensional atomic packing in amorphous solids with liquid-like structure.

Liquids and solids are two fundamental states of matter. However, our understanding of their three-dimensional atomic structure is mostly based on physical models. Here we use atomic electron tomography to experimentally determine the three-dimensional atomic positions of monatomic amorphous solids, namely a Ta thin film and two Pd nanoparticles. We observe that pentagonal bipyramids are the most abundant atomic motifs in these amorphous materials. Instead of forming icosahedra, the majority of pentagonal bipyramids arrange into pentagonal bipyramid networks with medium-range order. Molecular dynamics simulations further reveal that pentagonal bipyramid networks are prevalent in monatomic metallic liquids, which rapidly grow in size and form more icosahedra during the quench from the liquid to the glass state. These results expand our understanding of the atomic structures of amorphous solids and will encourage future studies on amorphous-crystalline phase and glass transitions in non-crystalline materials with three-dimensional atomic resolution.

Alternating the population and control neural networks to solve high-dimensional stochastic mean-field games.

We present APAC-Net, an alternating population and agent control neural network for solving stochastic mean-field games (MFGs). Our algorithm is geared toward high-dimensional instances of MFGs that are not approachable with existing solution methods. We achieve this in two steps. First, we take advantage of the underlying variational primal-dual structure that MFGs exhibit and phrase it as a convex-concave saddle-point problem. Second, we parameterize the value and density functions by two neural networks, respectively. By phrasing the problem in this manner, solving the MFG can be interpreted as a special case of training a generative adversarial network (GAN). We show the potential of our method on up to 100-dimensional MFG problems.

Determining the three-dimensional atomic structure of an amorphous solid.

Amorphous solids such as glass, plastics and amorphous thin films are ubiquitous in our daily life and have broad applications ranging from telecommunications to electronics and solar cells1-4. However, owing to the lack of long-range order, the three-dimensional (3D) atomic structure of amorphous solids has so far eluded direct experimental determination5-15. Here we develop an atomic electron tomography reconstruction method to experimentally determine the 3D atomic positions of an amorphous solid. Using a multi-component glass-forming alloy as proof of principle, we quantitatively characterize the short- and medium-range order of the 3D atomic arrangement. We observe that, although the 3D atomic packing of the short-range order is geometrically disordered, some short-range-order structures connect with each other to form crystal-like superclusters and give rise to medium-range order. We identify four types of crystal-like medium-range order-face-centred cubic, hexagonal close-packed, body-centred cubic and simple cubic-coexisting in the amorphous sample, showing translational but not orientational order. These observations provide direct experimental evidence to support the general framework of the efficient cluster packing model for metallic glasses10,12-14,16. We expect that this work will pave the way for the determination of the 3D structure of a wide range of amorphous solids, which could transform our fundamental understanding of non-crystalline materials and related phenomena.

Potential of Attosecond Coherent Diffractive Imaging.

Attosecond science has been transforming our understanding of electron dynamics in atoms, molecules, and solids. However, to date almost all of the attoscience experiments have been based on spectroscopic measurements because attosecond pulses have intrinsically very broad spectra due to the uncertainty principle and are incompatible with conventional imaging systems. Here we report an important advance towards achieving attosecond coherent diffractive imaging. Using simulated attosecond pulses, we simultaneously reconstruct the spectrum, 17 probes, and 17 spectral images of extended objects from a set of ptychographic diffraction patterns. We further confirm the principle and feasibility of this method by successfully performing a ptychographic coherent diffractive imaging experiment using a light-emitting diode with a broad spectrum. We believe this work clears the way to an unexplored domain of attosecond imaging science, which could have a far-reaching impact across different disciplines.

A machine learning framework for solving high-dimensional mean field game and mean field control problems.

Mean field games (MFG) and mean field control (MFC) are critical classes of multiagent models for the efficient analysis of massive populations of interacting agents. Their areas of application span topics in economics, finance, game theory, industrial engineering, crowd motion, and more. In this paper, we provide a flexible machine learning framework for the numerical solution of potential MFG and MFC models. State-of-the-art numerical methods for solving such problems utilize spatial discretization that leads to a curse of dimensionality. We approximately solve high-dimensional problems by combining Lagrangian and Eulerian viewpoints and leveraging recent advances from machine learning. More precisely, we work with a Lagrangian formulation of the problem and enforce the underlying Hamilton-Jacobi-Bellman (HJB) equation that is derived from the Eulerian formulation. Finally, a tailored neural network parameterization of the MFG/MFC solution helps us avoid any spatial discretization. Our numerical results include the approximate solution of 100-dimensional instances of optimal transport and crowd motion problems on a standard work station and a validation using a Eulerian solver in two dimensions. These results open the door to much-anticipated applications of MFG and MFC models that are beyond reach with existing numerical methods.

Semi-implicit relaxed Douglas-Rachford algorithm (sDR) for ptychography.

Alternating projection based methods, such as ePIE and rPIE, have been used widely in ptychography. However, they only work well if there are adequate measurements (diffraction patterns); in the case of sparse data (i.e. fewer measurements) alternating projection underperforms and might not even converge. In this paper, we propose semi-implicit relaxed Douglas-Rachford (sDR), an accelerated iterative method, to solve the classical ptychography problem. Using both simulated and experimental data, we show that sDR improves the convergence speed and the reconstruction quality relative to extended ptychographic iterative engine (ePIE) and regularized ptychographic iterative engine (rPIE). Furthermore, in certain cases when sparsity is high, sDR converges while ePIE and rPIE fail or encounter slow convergence. To facilitate others to use the algorithm, we post the Matlab source code of sDR on a public website (www.physics.ucla.edu/research/imaging/sDR/index.html). We anticipate that this algorithm can be generally applied to the ptychographic reconstruction of a wide range of samples in the physical and biological sciences.

Generalized proximal smoothing (GPS) for phase retrieval.

In this paper, we report the development of the generalized proximal smoothing (GPS) algorithm for phase retrieval of noisy data. GPS is a optimization-based algorithm, in which we relax both the Fourier magnitudes and support constraint. We relax the support constraint by incorporating the Moreau-Yosida regularization and heat kernel smoothing, and derive the associated proximal mapping. We also relax the magnitude constraint into a least squares fidelity term, whose proximal mapping is available as well. GPS alternatively iterates between the two proximal mappings in primal and dual spaces, respectively. Using both numerical simulation and experimental data, we show that GPS algorithm consistently outperforms the classical phase retrieval algorithms such as hybrid input-output (HIO) and oversampling smoothness (OSS), in terms of the convergence speed, consistency of the phase retrieval, and robustness to noise.

s-SMOOTH: Sparsity and Smoothness Enhanced EEG Brain Tomography.

EEG source imaging enables us to reconstruct current density in the brain from the electrical measurements with excellent temporal resolution (~ ms). The corresponding EEG inverse problem is an ill-posed one that has infinitely many solutions. This is due to the fact that the number of EEG sensors is usually much smaller than that of the potential dipole locations, as well as noise contamination in the recorded signals. To obtain a unique solution, regularizations can be incorporated to impose additional constraints on the solution. An appropriate choice of regularization is critically important for the reconstruction accuracy of a brain image. In this paper, we propose a novel Sparsity and SMOOthness enhanced brain TomograpHy (s-SMOOTH) method to improve the reconstruction accuracy by integrating two recently proposed regularization techniques: Total Generalized Variation (TGV) regularization and ℓ1-2 regularization. TGV is able to preserve the source edge and recover the spatial distribution of the source intensity with high accuracy. Compared to the relevant total variation (TV) regularization, TGV enhances the smoothness of the image and reduces staircasing artifacts. The traditional TGV defined on a 2D image has been widely used in the image processing field. In order to handle 3D EEG source images, we propose a voxel-based Total Generalized Variation (vTGV) regularization that extends the definition of second-order TGV from 2D planar images to 3D irregular surfaces such as cortex surface. In addition, the ℓ1-2 regularization is utilized to promote sparsity on the current density itself. We demonstrate that ℓ1-2 regularization is able to enhance sparsity and accelerate computations than ℓ1 regularization. The proposed model is solved by an efficient and robust algorithm based on the difference of convex functions algorithm (DCA) and the alternating direction method of multipliers (ADMM). Numerical experiments using synthetic data demonstrate the advantages of the proposed method over other state-of-the-art methods in terms of total reconstruction accuracy, localization accuracy and focalization degree. The application to the source localization of event-related potential data further demonstrates the performance of the proposed method in real-world scenarios.

Mapping Buried Hydrogen-Bonding Networks.

We map buried hydrogen-bonding networks within self-assembled monolayers of 3-mercapto-N-nonylpropionamide on Au{111}. The contributing interactions include the buried S-Au bonds at the substrate surface and the buried plane of linear networks of hydrogen bonds. Both are simultaneously mapped with submolecular resolution, in addition to the exposed interface, to determine the orientations of molecular segments and directional bonding. Two-dimensional mode-decomposition techniques are used to elucidate the directionality of these networks. We find that amide-based hydrogen bonds cross molecular domain boundaries and areas of local disorder.

Graph fractional-order total variation EEG source reconstruction.

EEG source imaging is able to reconstruct sources in the brain from scalp measurements with high temporal resolution. Due to the limited number of sensors, it is very challenging to locate the source accurately with high spatial resolution. Recently, several total variation (TV) based methods have been proposed to explore sparsity of the source spatial gradients, which is based on the assumption that the source is constant at each subregion. However, since the sources have more complex structures in practice, these methods have difficulty in recovering the current density variation and locating source peaks. To overcome this limitation, we propose a graph Fractional-Order Total Variation (gFOTV) based method, which provides the freedom to choose the smoothness order by imposing sparsity of the spatial fractional derivatives so that it locates source peaks accurately. The performance of gFOTV and various state-of-the-art methods is compared using a large amount of simulations and evaluated with several quantitative criteria. The results demonstrate the superior performance of gFOTV not only in spatial resolution but also in localization accuracy and total reconstruction accuracy.

Normal Estimation of a Transparent Object Using a Video.

Reconstructing transparent objects is a challenging problem. While producing reasonable results for quite complex objects, existing approaches require custom calibration or somewhat expensive labor to achieve high precision. When an overall shape preserving salient and fine details is sufficient, we show in this paper a significant step toward solving the problem when the object's silhouette is available and simple user interaction is allowed, by using a video of a transparent object shot under varying illumination. Specifically, we estimate the normal map of the exterior surface of a given solid transparent object, from which the surface depth can be integrated. Our technical contribution lies in relating this normal estimation problem to one of graph-cut segmentation. Unlike conventional formulations, however, our graph is dual-layered, since we can see a transparent object's foreground as well as the background behind it. Quantitative and qualitative evaluation are performed to verify the efficacy of this practical solution.