Imaging extended single crystal lattice distortion fields with multi-peak Bragg ptychography.

Recent advances in phase-retrieval-based x-ray imaging methods have demonstrated the ability to reconstruct 3D distortion vector fields within a nanocrystal by using coherent diffraction information from multiple crystal Bragg reflections. However, these works do not provide a solution to the challenges encountered in imaging lattice distortions in crystals with significant defect content that result in phase wrapping. Moreover, these methods only apply to isolated crystals smaller than the x-ray illumination, and therefore cannot be used for imaging of distortions in extended crystals. We introduce multi-peak Bragg ptychography which addresses both challenges via an optimization framework that combines stochastic gradient descent and phase unwrapping methods for robust image reconstruction of lattice distortions and defects in extended crystals. Our work uses modern automatic differentiation toolsets so that the method is easy to extend to other settings and easy to implement in high-performance computers. This work is particularly timely given the broad interest in using the increased coherent flux in fourth-generation synchrotrons for innovative material research.

Broadband X-ray ptychography using multi-wavelength algorithm.

Ptychography is a rapidly developing scanning microscopy which is able to view the internal structures of samples at a high resolution beyond the illumination size. The achieved spatial resolution is theoretically dose-limited. A broadband source can provide much higher flux compared with a monochromatic source; however, it conflicts with the necessary coherence requirements of this coherent diffraction imaging technique. In this paper, a multi-wavelength reconstruction algorithm has been developed to deal with the broad bandwidth in ptychography. Compared with the latest development of mixed-state reconstruction approach, this multi-wavelength approach is more accurate in the physical model, and also considers the spot size variation as a function of energy due to the chromatic focusing optics. Therefore, this method has been proved in both simulation and experiment to significantly improve the reconstruction when the source bandwidth, illumination size and scan step size increase. It is worth mentioning that the accurate and detailed information of the energy spectrum for the incident beam is not required in advance for the proposed method. Further, we combine multi-wavelength and mixed-state approaches to jointly solve temporal and spatial partial coherence in ptychography so that it can handle various disadvantageous experimental effects. The significant relaxation in coherence requirements by our approaches allows the use of high-flux broadband X-ray sources for high-efficient and high-resolution ptychographic imaging.

Efficient ptychographic phase retrieval via a matrix-free Levenberg-Marquardt algorithm.

The phase retrieval problem, where one aims to recover a complex-valued image from far-field intensity measurements, is a classic problem encountered in a range of imaging applications. Modern phase retrieval approaches usually rely on gradient descent methods in a nonlinear minimization framework. Calculating closed-form gradients for use in these methods is tedious work, and formulating second order derivatives is even more laborious. Additionally, second order techniques often require the storage and inversion of large matrices of partial derivatives, with memory requirements that can be prohibitive for data-rich imaging modalities. We use a reverse-mode automatic differentiation (AD) framework to implement an efficient matrix-free version of the Levenberg-Marquardt (LM) algorithm, a longstanding method that finds popular use in nonlinear least-square minimization problems but which has seen little use in phase retrieval. Furthermore, we extend the basic LM algorithm so that it can be applied for more general constrained optimization problems (including phase retrieval problems) beyond just the least-square applications. Since we use AD, we only need to specify the physics-based forward model for a specific imaging application; the first and second-order derivative terms are calculated automatically through matrix-vector products, without explicitly forming the large Jacobian or Gauss-Newton matrices typically required for the LM method. We demonstrate that this algorithm can be used to solve both the unconstrained ptychographic object retrieval problem and the constrained "blind" ptychographic object and probe retrieval problems, under the popular Gaussian noise model as well as the Poisson noise model. We compare this algorithm to state-of-the-art first order ptychographic reconstruction methods to demonstrate empirically that this method outperforms best-in-class first-order methods: it provides excellent convergence guarantees with (in many cases) a superlinear rate of convergence, all with a computational cost comparable to, or lower than, the tested first-order algorithms.

Using automatic differentiation as a general framework for ptychographic reconstruction.

Coherent diffraction imaging methods enable imaging beyond lens-imposed resolution limits. In these methods, the object can be recovered by minimizing an error metric that quantifies the difference between diffraction patterns as observed, and those calculated from a present guess of the object. Efficient minimization methods require analytical calculation of the derivatives of the error metric, which is not always straightforward. This limits our ability to explore variations of basic imaging approaches. In this paper, we propose to substitute analytical derivative expressions with the automatic differentiation method, whereby we can achieve object reconstruction by specifying only the physics-based experimental forward model. We demonstrate the generality of the proposed method through straightforward object reconstruction for a variety of complex ptychographic experimental models.

Simultaneous cryo X-ray ptychographic and fluorescence microscopy of green algae.

Trace metals play important roles in normal and in disease-causing biological functions. X-ray fluorescence microscopy reveals trace elements with no dependence on binding affinities (unlike with visible light fluorophores) and with improved sensitivity relative to electron probes. However, X-ray fluorescence is not very sensitive for showing the light elements that comprise the majority of cellular material. Here we show that X-ray ptychography can be combined with fluorescence to image both cellular structure and trace element distribution in frozen-hydrated cells at cryogenic temperatures, with high structural and chemical fidelity. Ptychographic reconstruction algorithms deliver phase and absorption contrast images at a resolution beyond that of the illuminating lens or beam size. Using 5.2-keV X-rays, we have obtained sub-30-nm resolution structural images and ∼90-nm-resolution fluorescence images of several elements in frozen-hydrated green algae. This combined approach offers a way to study the role of trace elements in their structural context.

Strategies for high-throughput focused-beam ptychography.

X-ray ptychography is being utilized for a wide range of imaging experiments with a resolution beyond the limit of the X-ray optics used. Introducing a parameter for the ptychographic resolution gain Gp (the ratio of the beam size over the achieved pixel size in the reconstructed image), strategies for data sampling and for increasing imaging throughput when the specimen is at the focus of an X-ray beam are considered. The tradeoffs between large and small illumination spots are examined.

Simultaneous X-ray fluorescence and scanning X-ray diffraction microscopy at the Australian Synchrotron XFM beamline.

Owing to its extreme sensitivity, quantitative mapping of elemental distributions via X-ray fluorescence microscopy (XFM) has become a key microanalytical technique. The recent realisation of scanning X-ray diffraction microscopy (SXDM) meanwhile provides an avenue for quantitative super-resolved ultra-structural visualization. The similarity of their experimental geometries indicates excellent prospects for simultaneous acquisition. Here, in both step- and fly-scanning modes, robust, simultaneous XFM-SXDM is demonstrated.

Continuous motion scan ptychography: characterization for increased speed in coherent x-ray imaging.

Ptychography is a coherent diffraction imaging (CDI) method for extended objects in which diffraction patterns are acquired sequentially from overlapping coherent illumination spots. The object's complex transmission function can be reconstructed from those diffraction patterns at a spatial resolution limited only by the scattering strength of the object and the detector geometry. Most experiments to date have positioned the illumination spots on the sample using a move-settle-measure sequence in which the move and settle steps can take longer to complete than the measure step. We describe here the use of a continuous "fly-scan" mode for ptychographic data collection in which the sample is moved continuously, so that the experiment resembles one of integrating the diffraction patterns from multiple probe positions. This allows one to use multiple probe mode reconstruction methods to obtain an image of the object and also of the illumination function. We show in simulations, and in x-ray imaging experiments, some of the characteristics of fly-scan ptychography, including a factor of 25 reduction in the data acquisition time. This approach will become increasingly important as brighter x-ray sources are developed, such as diffraction limited storage rings.

Parallel ptychographic reconstruction.

Ptychography is an imaging method whereby a coherent beam is scanned across an object, and an image is obtained by iterative phasing of the set of diffraction patterns. It is able to be used to image extended objects at a resolution limited by scattering strength of the object and detector geometry, rather than at an optics-imposed limit. As technical advances allow larger fields to be imaged, computational challenges arise for reconstructing the correspondingly larger data volumes, yet at the same time there is also a need to deliver reconstructed images immediately so that one can evaluate the next steps to take in an experiment. Here we present a parallel method for real-time ptychographic phase retrieval. It uses a hybrid parallel strategy to divide the computation between multiple graphics processing units (GPUs) and then employs novel techniques to merge sub-datasets into a single complex phase and amplitude image. Results are shown on a simulated specimen and a real dataset from an X-ray experiment conducted at a synchrotron light source.

Capturing dynamical correlations using implicit neural representations.

Understanding the nature and origin of collective excitations in materials is of fundamental importance for unraveling the underlying physics of a many-body system. Excitation spectra are usually obtained by measuring the dynamical structure factor, S(Q, ω), using inelastic neutron or x-ray scattering techniques and are analyzed by comparing the experimental results against calculated predictions. We introduce a data-driven analysis tool which leverages 'neural implicit representations' that are specifically tailored for handling spectrographic measurements and are able to efficiently obtain unknown parameters from experimental data via automatic differentiation. In this work, we employ linear spin wave theory simulations to train a machine learning platform, enabling precise exchange parameter extraction from inelastic neutron scattering data on the square-lattice spin-1 antiferromagnet La2NiO4, showcasing a viable pathway towards automatic refinement of advanced models for ordered magnetic systems.

CryoAI: Amortized Inference of Poses for Ab Initio Reconstruction of 3D Molecular Volumes from Real Cryo-EM Images.

Cryo-electron microscopy (cryo-EM) has become a tool of fundamental importance in structural biology, helping us understand the basic building blocks of life. The algorithmic challenge of cryo-EM is to jointly estimate the unknown 3D poses and the 3D electron scattering potential of a biomolecule from millions of extremely noisy 2D images. Existing reconstruction algorithms, however, cannot easily keep pace with the rapidly growing size of cryo-EM datasets due to their high computational and memory cost. We introduce cryoAI, an ab initio reconstruction algorithm for homogeneous conformations that uses direct gradient-based optimization of particle poses and the electron scattering potential from single-particle cryo-EM data. CryoAI combines a learned encoder that predicts the poses of each particle image with a physics-based decoder to aggregate each particle image into an implicit representation of the scattering potential volume. This volume is stored in the Fourier domain for computational efficiency and leverages a modern coordinate network architecture for memory efficiency. Combined with a symmetrized loss function, this framework achieves results of a quality on par with state-of-the-art cryo-EM solvers for both simulated and experimental data, one order of magnitude faster for large datasets and with significantly lower memory requirements than existing methods.

A machine learning photon detection algorithm for coherent x-ray ultrafast fluctuation analysis.

X-ray free electron laser experiments have brought unique capabilities and opened new directions in research, such as creating new states of matter or directly measuring atomic motion. One such area is the ability to use finely spaced sets of coherent x-ray pulses to be compared after scattering from a dynamic system at different times. This enables the study of fluctuations in many-body quantum systems at the level of the ultrafast pulse durations, but this method has been limited to a select number of examples and required complex and advanced analytical tools. By applying a new methodology to this problem, we have made qualitative advances in three separate areas that will likely also find application to new fields. As compared to the "droplet-type" models, which typically are used to estimate the photon distributions on pixelated detectors to obtain the coherent x-ray speckle patterns, our algorithm achieves an order of magnitude speedup on CPU hardware and two orders of magnitude improvement on GPU hardware. We also find that it retains accuracy in low-contrast conditions, which is the typical regime for many experiments in structural dynamics. Finally, it can predict photon distributions in high average-intensity applications, a regime which up until now has not been accessible. Our artificial intelligence-assisted algorithm will enable a wider adoption of x-ray coherence spectroscopies, by both automating previously challenging analyses and enabling new experiments that were not otherwise feasible without the developments described in this work.

3D Modeling of craniofacial ontogeny and sexual dimorphism in children.

BACKGROUND: The range of normal variation of growth and development of the craniofacial region is of direct clinical interest but incompletely understood. Here we develop a statistical model of craniofacial growth and development to compare craniofacial ontogeny between age groups and sexes and pilot an approach to modeling that is relatively straightforward to apply in the context of clinical research and assessment. METHODS: The sample comprises head surface meshes captured using a 3dMD five-camera system from 65 males and 47 females (range 3-20 years) from the Headspace project, Liverpool, UK. The surface meshes were parameterized using 16 anatomical landmarks and 59 semilandmarks on curves and surfaces. Modes and degrees of growth and development were assessed and compared among ages and sexes using Procrustes based geometric morphometric methods. RESULTS: Regression analyses indicate that 3-10 year olds undergo greater changes than 11-20 year olds and that craniofacial growth and development differs between these age groups. The analyses indicate that males extend growth allometrically into larger size ranges, contributing substantially to adult dimorphism. Comparisons of ontogenetic trajectories between sexes find no significant differences, yet when hypermorphosis is accounted for in the older age group there is a significant residual sexual dimorphism. CONCLUSIONS: The study adds to knowledge of how adult craniofacial form and sexual dimorphism develop. It was carried out using readily available software which facilitates replication of this work in diverse populations to underpin clinical assessment of deformity and the outcomes of corrective interventions.

Three dimensions, two microscopes, one code: Automatic differentiation for x-ray nanotomography beyond the depth of focus limit.

Conventional tomographic reconstruction algorithms assume that one has obtained pure projection images, involving no within-specimen diffraction effects nor multiple scattering. Advances in x-ray nanotomography are leading toward the violation of these assumptions, by combining the high penetration power of x-rays, which enables thick specimens to be imaged, with improved spatial resolution that decreases the depth of focus of the imaging system. We describe a reconstruction method where multiple scattering and diffraction effects in thick samples are modeled by multislice propagation and the 3D object function is retrieved through iterative optimization. We show that the same proposed method works for both full-field microscopy and for coherent scanning techniques like ptychography. Our implementation uses the optimization toolbox and the automatic differentiation capability of the open-source deep learning package TensorFlow, demonstrating a straightforward way to solve optimization problems in computational imaging with flexibility and portability.

InSituNet: Deep Image Synthesis for Parameter Space Exploration of Ensemble Simulations.

We propose InSituNet, a deep learning based surrogate model to support parameter space exploration for ensemble simulations that are visualized in situ. In situ visualization, generating visualizations at simulation time, is becoming prevalent in handling large-scale simulations because of the I/O and storage constraints. However, in situ visualization approaches limit the flexibility of post-hoc exploration because the raw simulation data are no longer available. Although multiple image-based approaches have been proposed to mitigate this limitation, those approaches lack the ability to explore the simulation parameters. Our approach allows flexible exploration of parameter space for large-scale ensemble simulations by taking advantage of the recent advances in deep learning. Specifically, we design InSituNet as a convolutional regression model to learn the mapping from the simulation and visualization parameters to the visualization results. With the trained model, users can generate new images for different simulation parameters under various visualization settings, which enables in-depth analysis of the underlying ensemble simulations. We demonstrate the effectiveness of InSituNet in combustion, cosmology, and ocean simulations through quantitative and qualitative evaluations.

Real-time coherent diffraction inversion using deep generative networks.

Phase retrieval, or the process of recovering phase information in reciprocal space to reconstruct images from measured intensity alone, is the underlying basis to a variety of imaging applications including coherent diffraction imaging (CDI). Typical phase retrieval algorithms are iterative in nature, and hence, are time-consuming and computationally expensive, making real-time imaging a challenge. Furthermore, iterative phase retrieval algorithms struggle to converge to the correct solution especially in the presence of strong phase structures. In this work, we demonstrate the training and testing of CDI NN, a pair of deep deconvolutional networks trained to predict structure and phase in real space of a 2D object from its corresponding far-field diffraction intensities alone. Once trained, CDI NN can invert a diffraction pattern to an image within a few milliseconds of compute time on a standard desktop machine, opening the door to real-time imaging.

Correlative 3D x-ray fluorescence and ptychographic tomography of frozen-hydrated green algae.

Accurate knowledge of elemental distributions within biological organisms is critical for understanding their cellular roles. The ability to couple this knowledge with overall cellular architecture in three dimensions (3D) deepens our understanding of cellular chemistry. Using a whole, frozen-hydrated Chlamydomonas reinhardtii cell as an example, we report the development of 3D correlative microscopy through a combination of simultaneous cryogenic x-ray ptychography and x-ray fluorescence microscopy. By taking advantage of a recently developed tomographic reconstruction algorithm, termed GENeralized Fourier Iterative REconstruction (GENFIRE), we produce high-quality 3D maps of the unlabeled alga's cellular ultrastructure and elemental distributions within the cell. We demonstrate GENFIRE's ability to outperform conventional tomography algorithms and to further improve the reconstruction quality by refining the experimentally intended tomographic angles. As this method continues to advance with brighter coherent light sources and more efficient data handling, we expect correlative 3D x-ray fluorescence and ptychographic tomography to be a powerful tool for probing a wide range of frozen-hydrated biological specimens, ranging from small prokaryotes such as bacteria, algae, and parasites to large eukaryotes such as mammalian cells, with applications that include understanding cellular responses to environmental stimuli and cell-to-cell interactions.

Nanoscale x-ray imaging of circuit features without wafer etching.

Modern integrated circuits (ICs) employ a myriad of materials organized at nanoscale dimensions, and certain critical tolerances must be met for them to function. To understand departures from intended functionality, it is essential to examine ICs as manufactured so as to adjust design rules, ideally in a non-destructive way so that imaged structures can be correlated with electrical performance. Electron microscopes can do this on thin regions, or on exposed surfaces, but the required processing alters or even destroys functionality. Microscopy with multi-keV x-rays provides an alternative approach with greater penetration, but the spatial resolution of x-ray imaging lenses has not allowed one to see the required detail in the latest generation of ICs. X-ray ptychography provides a way to obtain images of ICs without lens-imposed resolution limits, with past work delivering 20-40 nm resolution on thinned ICs. We describe a simple model for estimating the required exposure, and use it to estimate the future potential for this technique. Here we show for the first time that this approach can be used to image circuit detail through an unprocessed 300 µm thick silicon wafer, with sub-20 nm detail clearly resolved after mechanical polishing to 240 µm thickness was used to eliminate image contrast caused by Si wafer surface scratches. By using continuous x-ray scanning, massively parallel computation, and a new generation of synchrotron light sources, this should enable entire non-etched ICs to be imaged to 10 nm resolution or better while maintaining their ability to function in electrical tests.

X-ray ptychographic and fluorescence microscopy of frozen-hydrated cells using continuous scanning.

X-ray microscopy can be used to image whole, unsectioned cells in their native hydrated state. It complements the higher resolution of electron microscopy for submicrometer thick specimens, and the molecule-specific imaging capabilites of fluorescence light microscopy. We describe here the first use of fast, continuous x-ray scanning of frozen hydrated cells for simultaneous sub-20 nm resolution ptychographic transmission imaging with high contrast, and sub-100 nm resolution deconvolved x-ray fluorescence imaging of diffusible and bound ions at native concentrations, without the need to add specific labels. By working with cells that have been rapidly frozen without the use of chemical fixatives, and imaging them under cryogenic conditions, we are able to obtain images with well preserved structural and chemical composition, and sufficient stability against radiation damage to allow for multiple images to be obtained with no observable change.

Opportunities and limitations for combined fly-scan ptychography and fluorescence microscopy.

X-ray fluorescence offers unparalleled sensitivity for imaging the nanoscale distribution of trace elements in micrometer thick samples, while x-ray ptychography offers an approach to image light element containing structures at a resolution beyond that of the x-ray lens used. These methods can be used in combination, and in continuous scan mode for rapid data acquisition when using multiple probe mode reconstruction methods. We discuss here the opportunities and limitations of making use of additional information provided by ptychography to improve x-ray fluorescence images in two ways: by using position-error-correction algorithms to correct for scan distortions in fluorescence scans, and by considering the signal-to-noise limits on previously-demonstrated ptychographic probe deconvolution methods. This highlights the advantages of using a combined approach.