Roadmap on nanoscale magnetic resonance imaging.

The field of nanoscale magnetic resonance imaging (NanoMRI) was started 30 years ago. It was motivated by the desire to image single molecules and molecular assemblies, such as proteins and virus particles, with near-atomic spatial resolution and on a length scale of 100 nm. Over the years, the NanoMRI field has also expanded to include the goal of useful high-resolution nuclear magnetic resonance (NMR) spectroscopy of molecules under ambient conditions, including samples up to the micron-scale. The realization of these goals requires the development of spin detection techniques that are many orders of magnitude more sensitive than conventional NMR and MRI, capable of detecting and controlling nanoscale ensembles of spins. Over the years, a number of different technical approaches to NanoMRI have emerged, each possessing a distinct set of capabilities for basic and applied areas of science. The goal of this roadmap article is to report the current state of the art in NanoMRI technologies, outline the areas where they are poised to have impact, identify the challenges that lie ahead, and propose methods to meet these challenges. This roadmap also shows how developments in NanoMRI techniques can lead to breakthroughs in emerging quantum science and technology applications. .

Submerged single-photon LiDAR imaging sensor used for real-time 3D scene reconstruction in scattering underwater environments.

We demonstrate a fully submerged underwater LiDAR transceiver system based on single-photon detection technologies. The LiDAR imaging system used a silicon single-photon avalanche diode (SPAD) detector array fabricated in complementary metal-oxide semiconductor (CMOS) technology to measure photon time-of-flight using picosecond resolution time-correlated single-photon counting. The SPAD detector array was directly interfaced to a Graphics Processing Unit (GPU) for real-time image reconstruction capability. Experiments were performed with the transceiver system and target objects immersed in a water tank at a depth of 1.8 meters, with the targets placed at a stand-off distance of approximately 3 meters. The transceiver used a picosecond pulsed laser source with a central wavelength of 532 nm, operating at a repetition rate of 20 MHz and average optical power of up to 52 mW, dependent on scattering conditions. Three-dimensional imaging was demonstrated by implementing a joint surface detection and distance estimation algorithm for real-time processing and visualization, which achieved images of stationary targets with up to 7.5 attenuation lengths between the transceiver and the target. The average processing time per frame was approximately 33 ms, allowing real-time three-dimensional video demonstrations of moving targets at ten frames per second at up to 5.5 attenuation lengths between transceiver and target.

Rapid single-photon color imaging of moving objects.

This paper outlines an experimental demonstration of a Bayesian image reconstruction approach to achieve rapid single-photon color imaging of moving objects. The capacity to extract the color of objects is important in a variety of target identification and computer vision applications. Nonetheless, it remains challenging to achieve high-speed color imaging of moving objects in low-photon flux environments. The low-photon regime presents particular challenges for efficient spectral separation and identification, while unsupervised image reconstruction algorithms are often slow and computationally expensive. In this paper, we address both of these difficulties using a combination of hardware and computational solutions. We demonstrate color imaging using a Single-Photon Avalanche Diode (SPAD) detector array for rapid, low-light-level data acquisition, with an integrated color filter array (CFA) for efficient spectral unmixing. High-speed image reconstruction is achieved using a bespoke Bayesian algorithm to produce high-fidelity color videos. The analysis is conducted first on simulated data allowing different pixel formats and photon flux scenarios to be investigated. Experiments are then performed using a plasmonic metasurface-based CFA, integrated with a 64 × 64 pixel format SPAD array. Passive imaging is conducted using white-light illumination of multi-colored, moving targets. Intensity information is recorded in a series of 2D photon-counting SPAD frames, from which accurate color information is extracted using the fast Bayesian method introduced herein. The per-frame reconstruction rate proves to be hundreds of times faster than the previous computational method. Furthermore, this approach yields additional information in the form of uncertainty measures, which can be used to assist with imaging system optimization and decision-making in real-world applications. The techniques demonstrated point the way towards rapid video-rate single-photon color imaging. The developed Bayesian algorithm, along with more advanced SPAD technology and utilization of time-correlated single-photon counting (TCSPC) will permit live 3D, color videography in extremely low-photon flux environments.

Compressive hyperspectral imaging in the molecular fingerprint band.

Spectrally-resolved imaging provides a spectrum for each pixel of an image that, in the mid-infrared, can enable its chemical composition to be mapped by exploiting the correlation between spectroscopic features and specific molecular groups. The compatibility of Fourier-transform interferometry with full-field imaging makes it the spectroscopic method of choice, but Nyquist-limited fringe sampling restricts the increments of the interferometer arm length to no more than a few microns, making the acquisition time-consuming. Here, we demonstrate a compressive hyperspectral imaging strategy that combines non-uniform sampling and a smoothness-promoting prior to acquire data at 15% of the Nyquist rate, providing a significant acquisition-rate improvement over state-of-the-art techniques. By illuminating test objects with a sequence of suitably designed light spectra, we demonstrate compressive hyperspectral imaging across the 700-1400 cm-1 region in transmission mode. A post-processing analysis of the resulting hyperspectral images shows the potential of the method for efficient non-destructive classification of different materials on painted cultural heritage.

Spectral classification of sparse photon depth images.

By illumination of target scenes using a set of different wavelengths, we demonstrate color classification of scenes, as well as depth estimation, in photon-starved images. The spectral signatures are classified with a new advanced statistical image processing method from measurements of the same scene, in this case using combinations of 33, 16, 8 or 4 different wavelengths in the range 500 - 820 nm. This approach makes it possible to perform color classification and depth estimation on images containing as few as one photon per pixel, on average. Compared to single wavelength imaging, this approach improves target discrimination by extracting more spectral information, which, in turn, improves the depth estimation since this approach is robust to changes in target reflectivity. We demonstrate color classification and depth profiling of complex targets at average signal levels as low as 1.0 photons per pixel from as few as 4 different wavelength measurements.

Wavelength-time coding for multispectral 3D imaging using single-photon LiDAR.

Single-photon multispectral light detection and ranging (LiDAR) approaches have emerged as a route to color reconstruction and enhanced target identification in photon-starved imaging scenarios. In this paper, we present a three-dimensional imaging system based on a time-of-flight approach which is capable of simultaneous multispectral measurements using only one single-photon detector. Unlike other techniques, this approach does not require a wavelength router in the receiver channel. By observing multiple wavelengths at each spatial location, or per pixel (four discrete visible wavelengths are used in this work), we can obtain a single waveform with wavelength-to-time mapped peaks. The time-mapped peaks are created by the known chromatic group delay dispersion in the laser source's optical fiber, resulting in temporal separations between these peaks being in the region of 200 to 1000 ps, in this case. A multispectral single waveform algorithm was proposed to fit these multiple peaked LiDAR waveforms, and then reconstruct the color (spectral response) and depth profiles for the entire image. To the best of our knowledge, this is the first dedicated computational method operating in the photon-starved regime capable of discriminating multiple peaks associated with different wavelengths in a single pixel waveform and reconstructing spectral responses and depth.

High-resolution depth profiling using a range-gated CMOS SPAD quanta image sensor.

A CMOS single-photon avalanche diode (SPAD) quanta image sensor is used to reconstruct depth and intensity profiles when operating in a range-gated mode used in conjunction with pulsed laser illumination. By designing the CMOS SPAD array to acquire photons within a pre-determined temporal gate, the need for timing circuitry was avoided and it was therefore possible to have an enhanced fill factor (61% in this case) and a frame rate (100,000 frames per second) that is more difficult to achieve in a SPAD array which uses time-correlated single-photon counting. When coupled with appropriate image reconstruction algorithms, millimeter resolution depth profiles were achieved by iterating through a sequence of temporal delay steps in synchronization with laser illumination pulses. For photon data with high signal-to-noise ratios, depth images with millimeter scale depth uncertainty can be estimated using a standard cross-correlation approach. To enhance the estimation of depth and intensity images in the sparse photon regime, we used a bespoke clustering-based image restoration strategy, taking into account the binomial statistics of the photon data and non-local spatial correlations within the scene. For sparse photon data with total exposure times of 75 ms or less, the bespoke algorithm can reconstruct depth images with millimeter scale depth uncertainty at a stand-off distance of approximately 2 meters. We demonstrate a new approach to single-photon depth and intensity profiling using different target scenes, taking full advantage of the high fill-factor, high frame rate and large array format of this range-gated CMOS SPAD array.

Characterization and modelling of inter-core coupling in coherent fiber bundles.

Recent developments in optical endomicroscopy (OEM) and associated fluorescent SmartProbes present a need for sensitive imaging with high detection performance. Inter-core coupling within coherent fiber bundles is a well recognized limitation, affecting the technology's imaging capabilities. Fiber cross coupling has been studied both experimentally and within a theoretical framework (coupled mode theory), providing (i) insights on the factors affecting cross talk, and (ii) recommendations for optimal fiber bundle design. However, due to physical limitations, such as the tradeoff between cross coupling and core density, cross coupling can be suppressed yet not eliminated through optimal fiber design. This study introduces a novel approach for measuring, analyzing and quantifying cross coupling within coherent fiber bundles, in a format that can be integrated into a linear model, which in turn can enable computational compensation of the associated blurring introduced to OEM images.

Robust real-time imaging through flexible multimode fibers.

Conventional endoscopes comprise a bundle of optical fibers, associating one fiber for each pixel in the image. In principle, this can be reduced to a single multimode optical fiber (MMF), the width of a human hair, with one fiber spatial-mode per image pixel. However, images transmitted through a MMF emerge as unrecognizable speckle patterns due to dispersion and coupling between the spatial modes of the fiber. Furthermore, speckle patterns change as the fiber undergoes bending, making the use of MMFs in flexible imaging applications even more complicated. In this paper, we propose a real-time imaging system using flexible MMFs, but which is robust to bending. Our approach does not require access or feedback signal from the distal end of the fiber during imaging. We leverage a variational autoencoder to reconstruct and classify images from the speckles and show that these images can still be recovered when the bend configuration of the fiber is changed to one that was not part of the training set. We utilize a MMF 300 mm long with a 62.5 µm core for imaging [Formula: see text] cm objects placed approximately at 20 cm from the fiber and the system can deal with a change in fiber bend of 50[Formula: see text] and range of movement of 8 cm.

Fast Scalable Image Restoration Using Total Variation Priors and Expectation Propagation.

This paper presents a scalable approximate Bayesian method for image restoration using Total Variation (TV) priors, with the ability to offer uncertainty quantification. In contrast to most optimization methods based on maximum a posteriori estimation, we use the Expectation Propagation (EP) framework to approximate minimum mean squared error (MMSE) estimates and marginal (pixel-wise) variances, without resorting to Monte Carlo sampling. For the classical anisotropic TV-based prior, we also propose an iterative scheme to automatically adjust the regularization parameter via Expectation Maximization (EM). Using Gaussian approximating densities with diagonal covariance matrices, the resulting method allows highly parallelizable steps and can scale to large images for denoising, deconvolution, and compressive sensing (CS) problems. The simulation results illustrate that such EP methods can provide a posteriori estimates on par with those obtained via sampling methods but at a fraction of the computational cost. Moreover, EP does not exhibit strong underestimation of posteriori variances, in contrast to variational Bayes alternatives.

Quantitative imaging and automated fuel pin identification for passive gamma emission tomography.

Compliance of member States to the Treaty on the Non-Proliferation of Nuclear Weapons is monitored through nuclear safeguards. The Passive Gamma Emission Tomography (PGET) system is a novel instrument developed within the framework of the International Atomic Energy Agency (IAEA) project JNT 1510, which included the European Commission, Finland, Hungary and Sweden. The PGET is used for the verification of spent nuclear fuel stored in water pools. Advanced image reconstruction techniques are crucial for obtaining high-quality cross-sectional images of the spent-fuel bundle to allow inspectors of the IAEA to monitor nuclear material and promptly identify its diversion. In this work, we have developed a software suite to accurately reconstruct the spent-fuel cross sectional image, automatically identify present fuel rods, and estimate their activity. Unique image reconstruction challenges are posed by the measurement of spent fuel, due to its high activity and the self-attenuation. While the former is mitigated by detector physical collimation, we implemented a linear forward model to model the detector responses to the fuel rods inside the PGET, to account for the latter. The image reconstruction is performed by solving a regularized linear inverse problem using the fast-iterative shrinkage-thresholding algorithm. We have also implemented the traditional filtered back projection (FBP) method based on the inverse Radon transform for comparison and applied both methods to reconstruct images of simulated mockup fuel assemblies. Higher image resolution and fewer reconstruction artifacts were obtained with the inverse-problem approach, with the mean-square-error reduced by 50%, and the structural-similarity improved by 200%. We then used a convolutional neural network (CNN) to automatically identify the bundle type and extract the pin locations from the images; the estimated activity levels finally being compared with the ground truth. The proposed computational methods accurately estimated the activity levels of the present pins, with an associated uncertainty of approximately 5%.

Robust 3D Reconstruction of Dynamic Scenes From Single-Photon Lidar Using Beta-Divergences.

In this article, we present a new algorithm for fast, online 3D reconstruction of dynamic scenes using times of arrival of photons recorded by single-photon detector arrays. One of the main challenges in 3D imaging using single-photon lidar in practical applications is the presence of strong ambient illumination which corrupts the data and can jeopardize the detection of peaks/surface in the signals. This background noise not only complicates the observation model classically used for 3D reconstruction but also the estimation procedure which requires iterative methods. In this work, we consider a new similarity measure for robust depth estimation, which allows us to use a simple observation model and a non-iterative estimation procedure while being robust to mis-specification of the background illumination model. This choice leads to a computationally attractive depth estimation procedure without significant degradation of the reconstruction performance. This new depth estimation procedure is coupled with a spatio-temporal model to capture the natural correlation between neighboring pixels and successive frames for dynamic scene analysis. The resulting online inference process is scalable and well suited for parallel implementation. The benefits of the proposed method are demonstrated through a series of experiments conducted with simulated and real single-photon lidar videos, allowing the analysis of dynamic scenes at 325 m observed under extreme ambient illumination conditions.

Bayesian Activity Estimation and Uncertainty Quantification of Spent Nuclear Fuel Using Passive Gamma Emission Tomography.

In this paper, we address the problem of activity estimation in passive gamma emission tomography (PGET) of spent nuclear fuel. Two different noise models are considered and compared, namely, the isotropic Gaussian and the Poisson noise models. The problem is formulated within a Bayesian framework as a linear inverse problem and prior distributions are assigned to the unknown model parameters. In particular, a Bernoulli-truncated Gaussian prior model is considered to promote sparse pin configurations. A Markov chain Monte Carlo (MCMC) method, based on a split and augmented Gibbs sampler, is then used to sample the posterior distribution of the unknown parameters. The proposed algorithm is first validated by simulations conducted using synthetic data, generated using the nominal models. We then consider more realistic data simulated using a bespoke simulator, whose forward model is non-linear and not available analytically. In that case, the linear models used are mis-specified and we analyse their robustness for activity estimation. The results demonstrate superior performance of the proposed approach in estimating the pin activities in different assembly patterns, in addition to being able to quantify their uncertainty measures, in comparison with existing methods.

Simultaneous multi-spectral, single-photon fluorescence imaging using a plasmonic colour filter array.

We present the first realisation of simultaneous multi-spectral fluorescence imaging using a single-photon avalanche diode (SPAD) array, where the spectral unmixing is facilitated by a plasmonic metasurface mosaic colour filter array (CFA). A 64 × 64 pixel format silicon SPAD array is used to record widefield fluorescence and brightfield data from four biological samples. A plasmonic metasurface composed of an arrangement of circular and elliptical nanoholes etched into an aluminium thin film deposited on a glass substrate provides the high transmission efficiency CFA, enabling a bespoke spectral unmixing algorithm to reconstruct high fidelity, full colour images from as few as ∼3 photons per pixel. This approach points the way toward real-time, single-photon sensitive multi-spectral fluorescence imaging. Furthermore, this is possible without additional bulky components such as a filter wheel, prism or diffraction grating, nor the need for multiple sample exposures or multiple detectors.

High-speed particle detection and tracking in microfluidic devices using event-based sensing.

Visualising fluids and particles within channels is a key element of microfluidic work. Current imaging methods for particle image velocimetry often require expensive high-speed cameras with powerful illuminating sources, thus potentially limiting accessibility. This study explores for the first time the potential of an event-based camera for particle and fluid behaviour characterisation in a microfluidic system. Event-based cameras have the unique capacity to detect light intensity changes asynchronously and to record spatial and temporal information with low latency, low power and high dynamic range. Event-based cameras could consequently be relevant for detecting light intensity changes due to moving particles, chemical reactions or intake of fluorescent dyes by cells to mention a few. As a proof-of-principle, event-based sensing was tested in this work to detect 1 µm and 10 µm diameter particles flowing in a microfluidic channel for average fluid velocities of up to 1.54 m s-1. Importantly, experiments were performed by directly connecting the camera to a standard fluorescence microscope, only relying on the microscope arc lamp for illumination. We present a data processing strategy that allows particle detection and tracking in both bright-field and fluorescence imaging. Detection was achieved up to a fluid velocity of 1.54 m s-1 and tracking up to 0.4 m s-1 suggesting that event-based cameras could be a new paradigm shift in microscopic imaging.

Seeing around corners with edge-resolved transient imaging.

Non-line-of-sight (NLOS) imaging is a rapidly growing field seeking to form images of objects outside the field of view, with potential applications in autonomous navigation, reconnaissance, and even medical imaging. The critical challenge of NLOS imaging is that diffuse reflections scatter light in all directions, resulting in weak signals and a loss of directional information. To address this problem, we propose a method for seeing around corners that derives angular resolution from vertical edges and longitudinal resolution from the temporal response to a pulsed light source. We introduce an acquisition strategy, scene response model, and reconstruction algorithm that enable the formation of 2.5-dimensional representations-a plan view plus heights-and a 180∘ field of view for large-scale scenes. Our experiments demonstrate accurate reconstructions of hidden rooms up to 3 meters in each dimension despite a small scan aperture (1.5-centimeter radius) and only 45 measurement locations.

Expectation-propagation for weak radionuclide identification at radiation portal monitors.

We propose a sparsity-promoting Bayesian algorithm capable of identifying radionuclide signatures from weak sources in the presence of a high radiation background. The proposed method is relevant to radiation identification for security applications. In such scenarios, the background typically consists of terrestrial, cosmic, and cosmogenic radiation that may cause false positive responses. We evaluate the new Bayesian approach using gamma-ray data and are able to identify weapons-grade plutonium, masked by naturally-occurring radioactive material (NORM), in a measurement time of a few seconds. We demonstrate this identification capability using organic scintillators (stilbene crystals and EJ-309 liquid scintillators), which do not provide direct, high-resolution, source spectroscopic information. Compared to the EJ-309 detector, the stilbene-based detector exhibits a lower identification error, on average, owing to its better energy resolution. Organic scintillators are used within radiation portal monitors to detect gamma rays emitted from conveyances crossing ports of entry. The described method is therefore applicable to radiation portal monitors deployed in the field and could improve their threat discrimination capability by minimizing "nuisance" alarms produced either by NORM-bearing materials found in shipped cargoes, such as ceramics and fertilizers, or radionuclides in recently treated nuclear medicine patients.

Fast online 3D reconstruction of dynamic scenes from individual single-photon detection events.

In this paper, we present an algorithm for online 3D reconstruction of dynamic scenes using individual times of arrival (ToA) of photons recorded by single-photon detector arrays. One of the main challenges in 3D imaging using single-photon Lidar is the integration time required to build ToA histograms and reconstruct reliably 3D profiles in the presence of non-negligible ambient illumination. This long integration time also prevents the analysis of rapid dynamic scenes using existing techniques. We propose a new method which does not rely on the construction of ToA histograms but allows, for the first time, individual detection events to be processed online, in a parallel manner in different pixels, while accounting for the intrinsic spatiotemporal structure of dynamic scenes. Adopting a Bayesian approach, a Bayesian model is constructed to capture the dynamics of the 3D profile and an approximate inference scheme based on assumed density filtering is proposed, yielding a fast and robust reconstruction algorithm able to process efficiently thousands to millions of frames, as usually recorded using single-photon detectors. The performance of the proposed method, able to process hundreds of frames per second, is assessed using a series of experiments conducted with static and dynamic 3D scenes and the results obtained pave the way to a new family of real-time 3D reconstruction solutions.

Real-time 3D reconstruction from single-photon lidar data using plug-and-play point cloud denoisers.

Single-photon lidar has emerged as a prime candidate technology for depth imaging through challenging environments. Until now, a major limitation has been the significant amount of time required for the analysis of the recorded data. Here we show a new computational framework for real-time three-dimensional (3D) scene reconstruction from single-photon data. By combining statistical models with highly scalable computational tools from the computer graphics community, we demonstrate 3D reconstruction of complex outdoor scenes with processing times of the order of 20 ms, where the lidar data was acquired in broad daylight from distances up to 320 metres. The proposed method can handle an unknown number of surfaces in each pixel, allowing for target detection and imaging through cluttered scenes. This enables robust, real-time target reconstruction of complex moving scenes, paving the way for single-photon lidar at video rates for practical 3D imaging applications.

Bayesian bacterial detection using irregularly sampled optical endomicroscopy images.

Pneumonia is a major cause of morbidity and mortality of patients in intensive care. Rapid determination of the presence and gram status of the pathogenic bacteria in the distal lung may enable a more tailored treatment regime. Optical Endomicroscopy (OEM) is an emerging medical imaging platform with preclinical and clinical utility. Pulmonary OEM via multi-core fibre bundles has the potential to provide in vivo, in situ, fluorescent molecular signatures of the causes of infection and inflammation. This paper presents a Bayesian approach for bacterial detection in OEM images. The model considered assumes that the observed pixel fluorescence is a linear combination of the actual intensity value associated with tissues or background, corrupted by additive Gaussian noise and potentially by an additional sparse outlier term modelling anomalies (bacteria). The bacteria detection problem is formulated in a Bayesian framework and prior distributions are assigned to the unknown model parameters. A Markov chain Monte Carlo algorithm based on a partially collapsed Gibbs sampler is used to sample the posterior distribution of the unknown parameters. The proposed algorithm is first validated by simulations conducted using synthetic datasets for which good performance is obtained. Analysis is then conducted using two ex vivo lung datasets in which fluorescently labelled bacteria are present in the distal lung. A good correlation between bacteria counts identified by a trained clinician and those of the proposed method, which detects most of the manually annotated regions, is observed.