Shot noise-mitigated secondary electron imaging with ion count-aided microscopy.

Modern science is dependent on imaging on the nanoscale, often achieved through processes that detect secondary electrons created by a highly focused incident charged particle beam. Multiple types of measurement noise limit the ultimate trade-off between the image quality and the incident particle dose, which can preclude useful imaging of dose-sensitive samples. Existing methods to improve image quality do not fundamentally mitigate the noise sources. Furthermore, barriers to assigning a physically meaningful scale make the images qualitative. Here we introduce ion count-aided microscopy (ICAM), which is a quantitative imaging technique that uses statistically principled estimation of the secondary electron yield. With a readily implemented change in data collection, ICAM substantially reduces source shot noise. In helium ion microscopy, we demonstrate 3× dose reduction and a good match between these empirical results and theoretical performance predictions. ICAM facilitates imaging of fragile samples and may make imaging with heavier particles more attractive.

Shot noise-mitigated secondary electron imaging with ion count-aided microscopy.

Modern science is dependent on imaging on the nanoscale, often achieved through processes that detect secondary electrons created by a highly focused incident charged particle beam. Multiple types of measurement noise limit the ultimate trade-off between the image quality and the incident particle dose, which can preclude useful imaging of dose-sensitive samples. Existing methods to improve image quality do not fundamentally mitigate the noise sources. Furthermore, barriers to assigning a physically meaningful scale make the images qualitative. Here, we introduce ion count-aided microscopy (ICAM), which is a quantitative imaging technique that uses statistically principled estimation of the secondary electron yield. With a readily implemented change in data collection, ICAM substantially reduces source shot noise. In helium ion microscopy, we demonstrate 3[Formula: see text] dose reduction and a good match between these empirical results and theoretical performance predictions. ICAM facilitates imaging of fragile samples and may make imaging with heavier particles more attractive.

Absorption-based hyperspectral thermal ranging: performance analyses, optimization, and simulations.

The wavelength dependence of atmospheric absorption creates range cues in hyperspectral measurements that can be exploited for passive ranging using only thermal emissions. In this work, we present fundamental limits on absorption-based ranging under a model of known air temperature and wavelength-dependent attenuation coefficient, with object temperature and emissivity unknown; reflected solar and environmental radiance is omitted from our analysis. Fisher information computations illustrate how performance limits depend on atmospheric conditions such as air temperature and humidity; temperature contrast in the scene; spectral resolution of measurement; and distance. These results should prove valuable in sensor system design.

Non-line-of-sight snapshots and background mapping with an active corner camera.

The ability to form reconstructions beyond line-of-sight view could be transformative in a variety of fields, including search and rescue, autonomous vehicle navigation, and reconnaissance. Most existing active non-line-of-sight (NLOS) imaging methods use data collection steps in which a pulsed laser is directed at several points on a relay surface, one at a time. The prevailing approaches include raster scanning of a rectangular grid on a vertical wall opposite the volume of interest to generate a collection of confocal measurements. These and a recent method that uses a horizontal relay surface are inherently limited by the need for laser scanning. Methods that avoid laser scanning to operate in a snapshot mode are limited to treating the hidden scene of interest as one or two point targets. In this work, based on more complete optical response modeling yet still without multiple illumination positions, we demonstrate accurate reconstructions of foreground objects while also introducing the capability of mapping the stationary scenery behind moving objects. The ability to count, localize, and characterize the sizes of hidden objects, combined with mapping of the stationary hidden scene, could greatly improve indoor situational awareness in a variety of applications.

Secondary electron count imaging in SEM.

Scanning electron microscopy (SEM) is a versatile technique used to image samples at the nanoscale. Conventional imaging by this technique relies on finding the average intensity of the signal generated on a detector by secondary electrons (SEs) emitted from the sample and is subject to noise due to variations in the voltage signal from the detector. This noise can result in degradation of the SEM image quality for a given imaging dose. SE count imaging, which uses the direct count of SEs detected from the sample instead of the average signal intensity, would overcome this limitation and lead to improvement in SEM image quality. In this paper, we implement an SE count imaging scheme by synchronously outcoupling the detector and beam scan signals from the microscope and using custom code to count detected SEs. We demonstrate a ∼30% increase in the image signal-to-noise-ratio due to SE counting compared to conventional imaging. The only external hardware requirement for this imaging scheme is an oscilloscope fast enough to accurately sample the detector signal for SE counting, making the scheme easily implementable on any SEM.

Compressively sampling the optical transmission matrix of a multimode fibre.

The measurement of the optical transmission matrix (TM) of an opaque material is an advanced form of space-variant aberration correction. Beyond imaging, TM-based methods are emerging in a range of fields, including optical communications, micro-manipulation, and computing. In many cases, the TM is very sensitive to perturbations in the configuration of the scattering medium it represents. Therefore, applications often require an up-to-the-minute characterisation of the fragile TM, typically entailing hundreds to thousands of probe measurements. Here, we explore how these measurement requirements can be relaxed using the framework of compressive sensing, in which the incorporation of prior information enables accurate estimation from fewer measurements than the dimensionality of the TM we aim to reconstruct. Examples of such priors include knowledge of a memory effect linking the input and output fields, an approximate model of the optical system, or a recent but degraded TM measurement. We demonstrate this concept by reconstructing the full-size TM of a multimode fibre supporting 754 modes at compression ratios down to ∼5% with good fidelity. We show that in this case, imaging is still possible using TMs reconstructed at compression ratios down to ∼1% (eight probe measurements). This compressive TM sampling strategy is quite general and may be applied to a variety of other scattering samples, including diffusers, thin layers of tissue, fibre optics of any refractive profile, and reflections from opaque walls. These approaches offer a route towards the measurement of high-dimensional TMs either quickly or with access to limited numbers of measurements.

Non-line-of-sight imaging over 1.43 km.

Non-line-of-sight (NLOS) imaging has the ability to reconstruct hidden objects from indirect light paths that scatter multiple times in the surrounding environment, which is of considerable interest in a wide range of applications. Whereas conventional imaging involves direct line-of-sight light transport to recover the visible objects, NLOS imaging aims to reconstruct the hidden objects from the indirect light paths that scatter multiple times, typically using the information encoded in the time-of-flight of scattered photons. Despite recent advances, NLOS imaging has remained at short-range realizations, limited by the heavy loss and the spatial mixing due to the multiple diffuse reflections. Here, both experimental and conceptual innovations yield hardware and software solutions to increase the standoff distance of NLOS imaging from meter to kilometer range, which is about three orders of magnitude longer than previous experiments. In hardware, we develop a high-efficiency, low-noise NLOS imaging system at near-infrared wavelength based on a dual-telescope confocal optical design. In software, we adopt a convex optimizer, equipped with a tailored spatial-temporal kernel expressed using three-dimensional matrix, to mitigate the effect of the spatial-temporal broadening over long standoffs. Together, these enable our demonstration of NLOS imaging and real-time tracking of hidden objects over a distance of 1.43 km. The results will open venues for the development of NLOS imaging techniques and relevant applications to real-world conditions.

Seeing around corners with edge-resolved transient imaging.

Non-line-of-sight (NLOS) imaging is a rapidly growing field seeking to form images of objects outside the field of view, with potential applications in autonomous navigation, reconnaissance, and even medical imaging. The critical challenge of NLOS imaging is that diffuse reflections scatter light in all directions, resulting in weak signals and a loss of directional information. To address this problem, we propose a method for seeing around corners that derives angular resolution from vertical edges and longitudinal resolution from the temporal response to a pulsed light source. We introduce an acquisition strategy, scene response model, and reconstruction algorithm that enable the formation of 2.5-dimensional representations-a plan view plus heights-and a 180∘ field of view for large-scale scenes. Our experiments demonstrate accurate reconstructions of hidden rooms up to 3 meters in each dimension despite a small scan aperture (1.5-centimeter radius) and only 45 measurement locations.

Dithered depth imaging.

Single-photon lidar (SPL) is a promising technology for depth measurement at long range or from weak reflectors because of the sensitivity to extremely low light levels. However, constraints on the timing resolution of existing arrays of single-photon avalanche diode (SPAD) detectors limit the precision of resulting depth estimates. In this work, we describe an implementation of subtractively-dithered SPL that can recover high-resolution depth estimates despite the coarse resolution of the detector. Subtractively-dithered measurement is achieved by adding programmable delays into the photon timing circuitry that introduce relative time shifts between the illumination and detection that are shorter than the time bin duration. Careful modeling of the temporal instrument response function leads to an estimator that outperforms the sample mean and results in depth estimates with up to 13 times lower root mean-squared error than if dither were not used. The simple implementation and estimation suggest that globally dithered SPAD arrays could be used for high spatial- and temporal-resolution depth sensing.

Source shot noise mitigation in focused ion beam microscopy by time-resolved measurement.

Focused ion beam microscopy suffers from source shot noise - random variation in the number of incident ions in any fixed dwell time - along with random variation in the number of detected secondary electrons per incident ion. This multiplicity of sources of randomness increases the variance of the measurements and thus worsens the trade-off between incident ion dose and image accuracy. Repeated measurement with low dwell time, without changing the total ion dose, is a way to introduce time resolution to this form of microscopy. Through theoretical analyses and Monte Carlo simulations, we show that three ways to process time-resolved measurements result in mean-squared error (MSE) improvements compared to the conventional method of having no time resolution. In particular, maximum likelihood estimation provides reduction in MSE or reduction in required dose by a multiplicative factor approximately equal to the secondary electron yield. This improvement factor is similar to complete mitigation of source shot noise. Experiments with a helium ion microscope are consistent with the analyses and suggest accuracy improvement for a fixed source dose by a factor of about 4.

Computational periscopy with an ordinary digital camera.

Computing the amounts of light arriving from different directions enables a diffusely reflecting surface to play the part of a mirror in a periscope-that is, perform non-line-of-sight imaging around an obstruction. Because computational periscopy has so far depended on light-travel distances being proportional to the times of flight, it has mostly been performed with expensive, specialized ultrafast optical systems1-12. Here we introduce a two-dimensional computational periscopy technique that requires only a single photograph captured with an ordinary digital camera. Our technique recovers the position of an opaque object and the scene behind (but not completely obscured by) the object, when both the object and scene are outside the line of sight of the camera, without requiring controlled or time-varying illumination. Such recovery is based on the visible penumbra of the opaque object having a linear dependence on the hidden scene that can be modelled through ray optics. Non-line-of-sight imaging using inexpensive, ubiquitous equipment may have considerable value in monitoring hazardous environments, navigation and detecting hidden adversaries.

Quantum-inspired computational imaging.

Computational imaging combines measurement and computational methods with the aim of forming images even when the measurement conditions are weak, few in number, or highly indirect. The recent surge in quantum-inspired imaging sensors, together with a new wave of algorithms allowing on-chip, scalable and robust data processing, has induced an increase of activity with notable results in the domain of low-light flux imaging and sensing. We provide an overview of the major challenges encountered in low-illumination (e.g., ultrafast) imaging and how these problems have recently been addressed for imaging applications in extreme conditions. These methods provide examples of the future imaging solutions to be developed, for which the best results are expected to arise from an efficient codesign of the sensors and data analysis tools.

Photon-efficient imaging with a single-photon camera.

Reconstructing a scene's 3D structure and reflectivity accurately with an active imaging system operating in low-light-level conditions has wide-ranging applications, spanning biological imaging to remote sensing. Here we propose and experimentally demonstrate a depth and reflectivity imaging system with a single-photon camera that generates high-quality images from ∼1 detected signal photon per pixel. Previous achievements of similar photon efficiency have been with conventional raster-scanning data collection using single-pixel photon counters capable of ∼10-ps time tagging. In contrast, our camera's detector array requires highly parallelized time-to-digital conversions with photon time-tagging accuracy limited to ∼ns. Thus, we develop an array-specific algorithm that converts coarsely time-binned photon detections to highly accurate scene depth and reflectivity by exploiting both the transverse smoothness and longitudinal sparsity of natural scenes. By overcoming the coarse time resolution of the array, our framework uniquely achieves high photon efficiency in a relatively short acquisition time.

Computational multi-depth single-photon imaging.

We present an imaging framework that is able to accurately reconstruct multiple depths at individual pixels from single-photon observations. Our active imaging method models the single-photon detection statistics from multiple reflectors within a pixel, and it also exploits the fact that a multi-depth profile at each pixel can be expressed as a sparse signal. We interpret the multi-depth reconstruction problem as a sparse deconvolution problem using single-photon observations, create a convex problem through discretization and relaxation, and use a modified iterative shrinkage-thresholding algorithm to efficiently solve for the optimal multi-depth solution. We experimentally demonstrate that the proposed framework is able to accurately reconstruct the depth features of an object that is behind a partially-reflecting scatterer and 4 m away from the imager with root mean-square error of 11 cm, using only 19 signal photon detections per pixel in the presence of moderate background light. In terms of root mean-square error, this is a factor of 4.2 improvement over the conventional method of Gaussian-mixture fitting for multi-depth recovery.

First-photon imaging.

Imagers that use their own illumination can capture three-dimensional (3D) structure and reflectivity information. With photon-counting detectors, images can be acquired at extremely low photon fluxes. To suppress the Poisson noise inherent in low-flux operation, such imagers typically require hundreds of detected photons per pixel for accurate range and reflectivity determination. We introduce a low-flux imaging technique, called first-photon imaging, which is a computational imager that exploits spatial correlations found in real-world scenes and the physics of low-flux measurements. Our technique recovers 3D structure and reflectivity from the first detected photon at each pixel. We demonstrate simultaneous acquisition of sub-pulse duration range and 4-bit reflectivity information in the presence of high background noise. First-photon imaging may be of considerable value to both microscopy and remote sensing.

Sparsity-promoting calibration for GRAPPA accelerated parallel MRI reconstruction.

The amount of calibration data needed to produce images of adequate quality can prevent auto-calibrating parallel imaging reconstruction methods like generalized autocalibrating partially parallel acquisitions (GRAPPA) from achieving a high total acceleration factor. To improve the quality of calibration when the number of auto-calibration signal (ACS) lines is restricted, we propose a sparsity-promoting regularized calibration method that finds a GRAPPA kernel consistent with the ACS fit equations that yields jointly sparse reconstructed coil channel images. Several experiments evaluate the performance of the proposed method relative to unregularized and existing regularized calibration methods for both low-quality and underdetermined fits from the ACS lines. These experiments demonstrate that the proposed method, like other regularization methods, is capable of mitigating noise amplification, and in addition, the proposed method is particularly effective at minimizing coherent aliasing artifacts caused by poor kernel calibration in real data. Using the proposed method, we can increase the total achievable acceleration while reducing degradation of the reconstructed image better than existing regularized calibration methods.

Denoising sparse images from GRAPPA using the nullspace method.

To accelerate magnetic resonance imaging using uniformly undersampled (nonrandom) parallel imaging beyond what is achievable with generalized autocalibrating partially parallel acquisitions (GRAPPA) alone, the DEnoising of Sparse Images from GRAPPA using the Nullspace method is developed. The trade-off between denoising and smoothing the GRAPPA solution is studied for different levels of acceleration. Several brain images reconstructed from uniformly undersampled k-space data using DEnoising of Sparse Images from GRAPPA using the Nullspace method are compared against reconstructions using existing methods in terms of difference images (a qualitative measure), peak-signal-to-noise ratio, and noise amplification (g-factors) as measured using the pseudo-multiple replica method. Effects of smoothing, including contrast loss, are studied in synthetic phantom data. In the experiments presented, the contrast loss and spatial resolution are competitive with existing methods. Results for several brain images demonstrate significant improvements over GRAPPA at high acceleration factors in denoising performance with limited blurring or smoothing artifacts. In addition, the measured g-factors suggest that DEnoising of Sparse Images from GRAPPA using the Nullspace method mitigates noise amplification better than both GRAPPA and L1 iterative self-consistent parallel imaging reconstruction (the latter limited here by uniform undersampling).

Exploiting sparsity in time-of-flight range acquisition using a single time-resolved sensor.

Range acquisition systems such as light detection and ranging (LIDAR) and time-of-flight (TOF) cameras operate by measuring the time difference of arrival between a transmitted pulse and the scene reflection. We introduce the design of a range acquisition system for acquiring depth maps of piecewise-planar scenes with high spatial resolution using a single, omnidirectional, time-resolved photodetector and no scanning components. In our experiment, we reconstructed 64 × 64-pixel depth maps of scenes comprising two to four planar shapes using only 205 spatially-patterned, femtosecond illuminations of the scene. The reconstruction uses parametric signal modeling to recover a set of depths present in the scene. Then, a convex optimization that exploits sparsity of the Laplacian of the depth map of a typical scene determines correspondences between spatial positions and depths. In contrast with 2D laser scanning used in LIDAR systems and low-resolution 2D sensor arrays used in TOF cameras, our experiment demonstrates that it is possible to build a non-scanning range acquisition system with high spatial resolution using only a standard, low-cost photodetector and a spatial light modulator.

Multi-contrast reconstruction with Bayesian compressed sensing.

Clinical imaging with structural MRI routinely relies on multiple acquisitions of the same region of interest under several different contrast preparations. This work presents a reconstruction algorithm based on Bayesian compressed sensing to jointly reconstruct a set of images from undersampled k-space data with higher fidelity than when the images are reconstructed either individually or jointly by a previously proposed algorithm, M-FOCUSS. The joint inference problem is formulated in a hierarchical Bayesian setting, wherein solving each of the inverse problems corresponds to finding the parameters (here, image gradient coefficients) associated with each of the images. The variance of image gradients across contrasts for a single volumetric spatial position is a single hyperparameter. All of the images from the same anatomical region, but with different contrast properties, contribute to the estimation of the hyperparameters, and once they are found, the k-space data belonging to each image are used independently to infer the image gradients. Thus, commonality of image spatial structure across contrasts is exploited without the problematic assumption of correlation across contrasts. Examples demonstrate improved reconstruction quality (up to a factor of 4 in root-mean-square error) compared with previous compressed sensing algorithms and show the benefit of joint inversion under a hierarchical Bayesian model.

SIMULTANEOUSLY SPARSE SOLUTIONS TO LINEAR INVERSE PROBLEMS WITH MULTIPLE SYSTEM MATRICES AND A SINGLE OBSERVATION VECTOR.

A problem that arises in slice-selective magnetic resonance imaging (MRI) radio-frequency (RF) excitation pulse design is abstracted as a novel linear inverse problem with a simultaneous sparsity constraint. Multiple unknown signal vectors are to be determined, where each passes through a different system matrix and the results are added to yield a single observation vector. Given the matrices and lone observation, the objective is to find a simultaneously sparse set of unknown vectors that approximately solves the system. We refer to this as the multiple-system single-output (MSSO) simultaneous sparse approximation problem. This manuscript contrasts the MSSO problem with other simultaneous sparsity problems and conducts an initial exploration of algorithms with which to solve it. Greedy algorithms and techniques based on convex relaxation are derived and compared empirically. Experiments involve sparsity pattern recovery in noiseless and noisy settings and MRI RF pulse design.