Submerged single-photon LiDAR imaging sensor used for real-time 3D scene reconstruction in scattering underwater environments.

We demonstrate a fully submerged underwater LiDAR transceiver system based on single-photon detection technologies. The LiDAR imaging system used a silicon single-photon avalanche diode (SPAD) detector array fabricated in complementary metal-oxide semiconductor (CMOS) technology to measure photon time-of-flight using picosecond resolution time-correlated single-photon counting. The SPAD detector array was directly interfaced to a Graphics Processing Unit (GPU) for real-time image reconstruction capability. Experiments were performed with the transceiver system and target objects immersed in a water tank at a depth of 1.8 meters, with the targets placed at a stand-off distance of approximately 3 meters. The transceiver used a picosecond pulsed laser source with a central wavelength of 532 nm, operating at a repetition rate of 20 MHz and average optical power of up to 52 mW, dependent on scattering conditions. Three-dimensional imaging was demonstrated by implementing a joint surface detection and distance estimation algorithm for real-time processing and visualization, which achieved images of stationary targets with up to 7.5 attenuation lengths between the transceiver and the target. The average processing time per frame was approximately 33 ms, allowing real-time three-dimensional video demonstrations of moving targets at ten frames per second at up to 5.5 attenuation lengths between transceiver and target.

Robust real-time 3D imaging of moving scenes through atmospheric obscurant using single-photon LiDAR.

Recently, time-of-flight LiDAR using the single-photon detection approach has emerged as a potential solution for three-dimensional imaging in challenging measurement scenarios, such as over distances of many kilometres. The high sensitivity and picosecond timing resolution afforded by single-photon detection offers high-resolution depth profiling of remote, complex scenes while maintaining low power optical illumination. These properties are ideal for imaging in highly scattering environments such as through atmospheric obscurants, for example fog and smoke. In this paper we present the reconstruction of depth profiles of moving objects through high levels of obscurant equivalent to five attenuation lengths between transceiver and target at stand-off distances up to 150 m. We used a robust statistically based processing algorithm designed for the real time reconstruction of single-photon data obtained in the presence of atmospheric obscurant, including providing uncertainty estimates in the depth reconstruction. This demonstration of real-time 3D reconstruction of moving scenes points a way forward for high-resolution imaging from mobile platforms in degraded visual environments.

Robust 3D Reconstruction of Dynamic Scenes From Single-Photon Lidar Using Beta-Divergences.

In this article, we present a new algorithm for fast, online 3D reconstruction of dynamic scenes using times of arrival of photons recorded by single-photon detector arrays. One of the main challenges in 3D imaging using single-photon lidar in practical applications is the presence of strong ambient illumination which corrupts the data and can jeopardize the detection of peaks/surface in the signals. This background noise not only complicates the observation model classically used for 3D reconstruction but also the estimation procedure which requires iterative methods. In this work, we consider a new similarity measure for robust depth estimation, which allows us to use a simple observation model and a non-iterative estimation procedure while being robust to mis-specification of the background illumination model. This choice leads to a computationally attractive depth estimation procedure without significant degradation of the reconstruction performance. This new depth estimation procedure is coupled with a spatio-temporal model to capture the natural correlation between neighboring pixels and successive frames for dynamic scene analysis. The resulting online inference process is scalable and well suited for parallel implementation. The benefits of the proposed method are demonstrated through a series of experiments conducted with simulated and real single-photon lidar videos, allowing the analysis of dynamic scenes at 325 m observed under extreme ambient illumination conditions.

High concentration factor diffractive microlenses integrated with CMOS single-photon avalanche diode detector arrays for fill-factor improvement.

Large-format single-photon avalanche diode (SPAD) arrays often suffer from low fill-factors-the ratio of the active area to the overall pixel area. The detection efficiency of these detector arrays can be vastly increased with the integration of microlens arrays designed to concentrate incident light onto the active areas and may be refractive or diffractive in nature. The ability of diffractive optical elements (DOEs) to efficiently cover a square or rectangular pixel, combined with their capability of working as fast lenses (i.e., ∼f/3) makes them versatile and practical lens designs for use in sparse photon applications using microscale, large-format detector arrays. Binary-mask-based photolithography was employed to fabricate fast diffractive microlenses for two designs of 32×32 SPAD detector arrays, each design having a different pixel pitch and fill-factor. A spectral characterization of the lenses is performed, as well as analysis of performance under different illumination conditions from wide- to narrow-angle illumination (i.e., f/2 to f/22 optics). The performance of the microlenses presented exceeds previous designs in terms of both concentration factor (i.e., increase in light collection capability) and lens speed. Concentration factors greater than 33× are achieved for focal lengths in the substrate material as short as 190µm, representing a microlens f-number of 3.8 and providing a focal spot diameter of <4µm. These results were achieved while retaining an extremely high degree of performance uniformity across the 1024 devices in each case, which demonstrates the significant benefits to be gained by the implementation of DOEs as part of an integrated detector system using SPAD arrays with very small active areas.

3D LIDAR imaging using Ge-on-Si single-photon avalanche diode detectors.

We present a scanning light detection and ranging (LIDAR) system incorporating an individual Ge-on-Si single-photon avalanche diode (SPAD) detector for depth and intensity imaging in the short-wavelength infrared region. The time-correlated single-photon counting technique was used to determine the return photon time-of-flight for target depth information. In laboratory demonstrations, depth and intensity reconstructions were made of targets at short range, using advanced image processing algorithms tailored for the analysis of single-photon time-of-flight data. These laboratory measurements were used to predict the performance of the single-photon LIDAR system at longer ranges, providing estimations that sub-milliwatt average power levels would be required for kilometer range depth measurements.

Learning Non-Local Spatial Correlations To Restore Sparse 3D Single-Photon Data.

This paper presents a new algorithm for the learning of spatial correlation and non-local restoration of single-photon 3-Dimensional Lidar images acquired in the photon starved regime (fewer or less than one photon per pixel) or with a reduced number of scanned spatial points (pixels). The algorithm alternates between three steps: (i) extract multi-scale information, (ii) build a robust graph of non-local spatial correlations between pixels, and (iii) the restoration of depth and reflectivity images. A non-uniform sampling approach, which assigns larger patches to homogeneous regions and smaller ones to heterogeneous regions, is adopted to reduce the computational cost associated with the graph. The restoration of the 3D images is achieved by minimizing a cost function accounting for the multi-scale information and the non-local spatial correlation between patches. This minimization problem is efficiently solved using the alternating direction method of multipliers (ADMM) that presents fast convergence properties. Various results based on simulated and real Lidar data show the benefits of the proposed algorithm that improves the quality of the estimated depth and reflectivity images, especially in the photon-starved regime or when containing a reduced number of spatial points.

Three-dimensional imaging of stationary and moving targets in turbid underwater environments using a single-photon detector array.

Three-dimensional imaging in underwater environments was investigated using a picosecond resolution silicon single-photon avalanche diode (SPAD) detector array fabricated in complementary metal-oxide semiconductor (CMOS) technology. Each detector in the 192  × 128 SPAD array had an individual time-to-digital converter allowing rapid, simultaneous acquisition of data for the entire array using the time-correlated single-photon counting approach. A picosecond pulsed laser diode source operating at a wavelength of 670 nm was used to illuminate the underwater scenes, emitting an average optical power up to 8 mW. Both stationary and moving targets were imaged under a variety of underwater scattering conditions. The acquisition of depth and intensity videos of moving targets was demonstrated in dark laboratory conditions through scattering water, equivalent to having up to 6.7 attenuation lengths between the transceiver and target. Data were analyzed using a pixel-wise approach, as well as an image processing algorithm based on a median filter and polynomial approximation.

Real-time 3D reconstruction from single-photon lidar data using plug-and-play point cloud denoisers.

Single-photon lidar has emerged as a prime candidate technology for depth imaging through challenging environments. Until now, a major limitation has been the significant amount of time required for the analysis of the recorded data. Here we show a new computational framework for real-time three-dimensional (3D) scene reconstruction from single-photon data. By combining statistical models with highly scalable computational tools from the computer graphics community, we demonstrate 3D reconstruction of complex outdoor scenes with processing times of the order of 20 ms, where the lidar data was acquired in broad daylight from distances up to 320 metres. The proposed method can handle an unknown number of surfaces in each pixel, allowing for target detection and imaging through cluttered scenes. This enables robust, real-time target reconstruction of complex moving scenes, paving the way for single-photon lidar at video rates for practical 3D imaging applications.

Three-dimensional single-photon imaging through obscurants.

We investigate the depth imaging of objects through various densities of different obscurants (water fog, glycol-based vapor, and incendiary smoke) using a time-correlated single-photon detection system which had an operating wavelength of 1550 nm and an average optical output power of approximately 1.5 mW. It consisted of a monostatic scanning transceiver unit used in conjunction with a picosecond laser source and an individual Peltier-cooled InGaAs/InP single-photon avalanche diode (SPAD) detector. We acquired depth and intensity data of targets imaged through distances of up to 24 meters for the different obscurants. We compare several statistical algorithms which reconstruct both the depth and intensity images for short data acquisition times, including very low signal returns in the photon-starved regime.

Wavelength-time coding for multispectral 3D imaging using single-photon LiDAR.

Single-photon multispectral light detection and ranging (LiDAR) approaches have emerged as a route to color reconstruction and enhanced target identification in photon-starved imaging scenarios. In this paper, we present a three-dimensional imaging system based on a time-of-flight approach which is capable of simultaneous multispectral measurements using only one single-photon detector. Unlike other techniques, this approach does not require a wavelength router in the receiver channel. By observing multiple wavelengths at each spatial location, or per pixel (four discrete visible wavelengths are used in this work), we can obtain a single waveform with wavelength-to-time mapped peaks. The time-mapped peaks are created by the known chromatic group delay dispersion in the laser source's optical fiber, resulting in temporal separations between these peaks being in the region of 200 to 1000 ps, in this case. A multispectral single waveform algorithm was proposed to fit these multiple peaked LiDAR waveforms, and then reconstruct the color (spectral response) and depth profiles for the entire image. To the best of our knowledge, this is the first dedicated computational method operating in the photon-starved regime capable of discriminating multiple peaks associated with different wavelengths in a single pixel waveform and reconstructing spectral responses and depth.

Spectral classification of sparse photon depth images.

By illumination of target scenes using a set of different wavelengths, we demonstrate color classification of scenes, as well as depth estimation, in photon-starved images. The spectral signatures are classified with a new advanced statistical image processing method from measurements of the same scene, in this case using combinations of 33, 16, 8 or 4 different wavelengths in the range 500 - 820 nm. This approach makes it possible to perform color classification and depth estimation on images containing as few as one photon per pixel, on average. Compared to single wavelength imaging, this approach improves target discrimination by extracting more spectral information, which, in turn, improves the depth estimation since this approach is robust to changes in target reflectivity. We demonstrate color classification and depth profiling of complex targets at average signal levels as low as 1.0 photons per pixel from as few as 4 different wavelength measurements.

A compact fiber-optic probe-based singlet oxygen luminescence detection system.

This paper presents a novel compact fiberoptic based singlet oxygen near-infrared luminescence probe coupled to an InGaAs/InP single photon avalanche diode (SPAD) detector. Patterned time gating of the single-photon detector is used to limit unwanted dark counts and eliminate the strong photosensitizer luminescence background. Singlet oxygen luminescence detection at 1270 nm is confirmed through spectral filtering and lifetime fitting for Rose Bengal in water, and Photofrin in methanol as model photosensitizers. The overall performance, measured by the signal-to-noise ratio, improves by a factor of 50 over a previous system that used a fiberoptic-coupled superconducting nanowire single-photon detector. The effect of adding light scattering to the photosensitizer is also examined as a first step towards applications in tissue in vivo.

A Comparison of Singlet Oxygen Explicit Dosimetry (SOED) and Singlet Oxygen Luminescence Dosimetry (SOLD) for Photofrin-Mediated Photodynamic Therapy.

Accurate photodynamic therapy (PDT) dosimetry is critical for the use of PDT in the treatment of malignant and nonmalignant localized diseases. A singlet oxygen explicit dosimetry (SOED) model has been developed for in vivo purposes. It involves the measurement of the key components in PDT-light fluence (rate), photosensitizer concentration, and ground-state oxygen concentration ([³O2])-to calculate the amount of reacted singlet oxygen ([¹O2]rx), the main cytotoxic component in type II PDT. Experiments were performed in phantoms with the photosensitizer Photofrin and in solution using phosphorescence-based singlet oxygen luminescence dosimetry (SOLD) to validate the SOED model. Oxygen concentration and photosensitizer photobleaching versus time were measured during PDT, along with direct SOLD measurements of singlet oxygen and triplet state lifetime (τΔ and τt), for various photosensitizer concentrations to determine necessary photophysical parameters. SOLD-determined cumulative [¹O2]rx was compared to SOED-calculated [¹O2]rx for various photosensitizer concentrations to show a clear correlation between the two methods. This illustrates that explicit dosimetry can be used when phosphorescence-based dosimetry is not feasible. Using SOED modeling, we have also shown evidence that SOLD-measured [¹O2]rx using a 523 nm pulsed laser can be used to correlate to singlet oxygen generated by a 630 nm laser during a clinical malignant pleural mesothelioma (MPM) PDT protocol by using a conversion formula.

A feasibility study of singlet oxygen explicit dosmietry (SOED) of PDT by intercomparison with a singlet oxygen luminescence dosimetry (SOLD) system.

An explicit dosimetry model has been developed to calculate the apparent reacted 1O2 concentration ([1O2]rx) in an in-vivo model. In the model, a macroscopic quantity, g, is introduced to account for oxygen perfusion to the medium during PDT. In this study, the SOED model is extended for PDT treatment in phantom conditions where vasculature is not present; the oxygen perfusion is achieved through the air-phantom interface instead. The solution of the SOED model is obtained by solving the coupled photochemical rate equations incorporating oxygen perfusion through the air-liquid interface. Experiments were performed for two photosensitizers (PS), Rose Bengal (RB) and Photofrin (PH), in solution, using SOED and SOLD measurements to determine both the instantaneous [1O2] as well as cumulative [1O2]rx concentrations, where [1O2] rx = (1/τΔ) · ∫[1O2]dt. The PS concentrations varied between 10 and 100 mM for RB and ~200 mM for Photofrin. The resulting magnitudes of [1O2] were compared between SOED and SOLD.

Lidar Waveform-Based Analysis of Depth Images Constructed Using Sparse Single-Photon Data.

This paper presents a new Bayesian model and algorithm used for depth and reflectivity profiling using full waveforms from the time-correlated single-photon counting measurement in the limit of very low photon counts. The proposed model represents each Lidar waveform as a combination of a known impulse response, weighted by the target reflectivity, and an unknown constant background, corrupted by Poisson noise. Prior knowledge about the problem is embedded through prior distributions that account for the different parameter constraints and their spatial correlation among the image pixels. In particular, a gamma Markov random field (MRF) is used to model the joint distribution of the target reflectivity, and a second MRF is used to model the distribution of the target depth, which are both expected to exhibit significant spatial correlations. An adaptive Markov chain Monte Carlo algorithm is then proposed to perform Bayesian inference. This algorithm is equipped with a stochastic optimization adaptation mechanism that automatically adjusts the parameters of the MRFs by maximum marginal likelihood estimation. Finally, the benefits of the proposed methodology are demonstrated through a series of experiments using real data.

Fill-factor improvement of Si CMOS single-photon avalanche diode detector arrays by integration of diffractive microlens arrays.

Single-photon avalanche diode (SPAD) detector arrays generally suffer from having a low fill-factor, in which the photo-sensitive area of each pixel is small compared to the overall area of the pixel. This paper describes the integration of different configurations of high efficiency diffractive optical microlens arrays onto a 32 × 32 SPAD array, fabricated using a 0.35 µm CMOS technology process. The characterization of SPAD arrays with integrated microlens arrays is reported over the spectral range of 500-900 nm, and a range of f-numbers from f/2 to f/22. We report an average concentration factor of 15 measured for the entire SPAD array with integrated microlens array. The integrated SPAD and microlens array demonstrated a very high uniformity in overall efficiency.

Underwater depth imaging using time-correlated single-photon counting.

A depth imaging system, based on the time-of-flight approach and the time-correlated single-photon counting (TCSPC) technique, was investigated for use in highly scattering underwater environments. The system comprised a pulsed supercontinuum laser source, a monostatic scanning transceiver, with a silicon single-photon avalanche diode (SPAD) used for detection of the returned optical signal. Depth images were acquired in the laboratory at stand-off distances of up to 8 attenuation lengths, using per-pixel acquisition times in the range 0.5 to 100 ms, at average optical powers in the range 0.8 nW to 950 µW. In parallel, a LiDAR model was developed and validated using experimental data. The model can be used to estimate the performance of the system under a variety of scattering conditions and system parameters.

Influence of magnetic fields on magneto-aerotaxis.

The response of cells to changes in their physico-chemical micro-environment is essential to their survival. For example, bacterial magnetotaxis uses the Earth's magnetic field together with chemical sensing to help microorganisms move towards favoured habitats. The studies of such complex responses are lacking a method that permits the simultaneous mapping of the chemical environment and the response of the organisms, and the ability to generate a controlled physiological magnetic field. We have thus developed a multi-modal microscopy platform that fulfils these requirements. Using simultaneous fluorescence and high-speed imaging in conjunction with diffusion and aerotactic models, we characterized the magneto-aerotaxis of Magnetospirillum gryphiswaldense. We assessed the influence of the magnetic field (orientation; strength) on the formation and the dynamic of a micro-aerotactic band (size, dynamic, position). As previously described by models of magnetotaxis, the application of a magnetic field pointing towards the anoxic zone of an oxygen gradient results in an enhanced aerotaxis even down to Earth's magnetic field strength. We found that neither a ten-fold increase of the field strength nor a tilt of 45° resulted in a significant change of the aerotactic efficiency. However, when the field strength is zeroed or when the field angle is tilted to 90°, the magneto-aerotaxis efficiency is drastically reduced. The classical model of magneto-aerotaxis assumes a response proportional to the cosine of the angle difference between the directions of the oxygen gradient and that of the magnetic field. Our experimental evidence however shows that this behaviour is more complex than assumed in this model, thus opening up new avenues for research.

Kilometer-range depth imaging at 1,550 nm wavelength using an InGaAs/InP single-photon avalanche diode detector.

We have used an InGaAs/InP single-photon avalanche diode detector module in conjunction with a time-of-flight depth imager operating at a wavelength of 1550 nm, to acquire centimeter resolution depth images of low signature objects at stand-off distances of up to one kilometer. The scenes of interest were scanned by the transceiver system using pulsed laser illumination with an average optical power of less than 600 µW and per-pixel acquisition times of between 0.5 ms and 20 ms. The fiber-pigtailed InGaAs/InP detector was Peltier-cooled and operated at a temperature of 230 K. This detector was used in electrically gated mode with a single-photon detection efficiency of about 26% at a dark count rate of 16 kilocounts per second. The system's overall instrumental temporal response was 144 ps full width at half maximum. Measurements made in daylight on a number of target types at ranges of 325 m, 910 m, and 4.5 km are presented, along with an analysis of the depth resolution achieved.

Kilometer-range, high resolution depth imaging via 1560 nm wavelength single-photon detection.

This paper highlights a significant advance in time-of-flight depth imaging: by using a scanning transceiver which incorporated a free-running, low noise superconducting nanowire single-photon detector, we were able to obtain centimeter resolution depth images of low-signature objects in daylight at stand-off distances of the order of one kilometer at the relatively eye-safe wavelength of 1560 nm. The detector used had an efficiency of 18% at 1 kHz dark count rate, and the overall system jitter was ~100 ps. The depth images were acquired by illuminating the scene with an optical output power level of less than 250 µW average, and using per-pixel dwell times in the millisecond regime.