Human activity recognition using a single-photon direct time-of-flight sensor.

Single-Photon Avalanche Diode (SPAD) direct Time-of-Flight (dToF) sensors provide depth imaging over long distances, enabling the detection of objects even in the absence of contrast in colour or texture. However, distant objects are represented by just a few pixels and are subject to noise from solar interference, limiting the applicability of existing computer vision techniques for high-level scene interpretation. We present a new SPAD-based vision system for human activity recognition, based on convolutional and recurrent neural networks, which is trained entirely on synthetic data. In tests using real data from a 64×32 pixel SPAD, captured over a distance of 40 m, the scheme successfully overcomes the limited transverse resolution (in which human limbs are approximately one pixel across), achieving an average accuracy of 89% in distinguishing between seven different activities. The approach analyses continuous streams of video-rate depth data at a maximal rate of 66 FPS when executed on a GPU, making it well-suited for real-time applications such as surveillance or situational awareness in autonomous systems.

Handheld wide-field fluorescence lifetime imaging system based on a distally mounted SPAD array.

In this work a handheld Fluorescent Lifetime IMaging (FLIM) system based on a distally mounted < 2 mm2 128 × 120 single photon avalanche diode (SPAD) array operating over a > 1 m long wired interface is demonstrated. The head of the system is ∼4.5 cm x 4.5 cm x 4.5 cm making it suitable for hand-held ex vivo applications. This is, to the best of the authors' knowledge, the first example of a SPAD array mounted on the distal end of a handheld FLIM system in this manner. All existing systems to date use a fibre to collect and relay fluorescent light to detectors at the proximal end of the system. This has clear potential biological and biomedical applications. To demonstrate this, the system is used to provide contrast between regions of differing tissue composition in ovine kidney samples, and between healthy and stressed or damaged plant leaves. Additionally, FLIM videos are provided showing that frame rates of > 1 Hz are achievable. It is thus an important step in realising an in vivo miniaturized chip-on-tip FLIM endoscopy system.

Submerged single-photon LiDAR imaging sensor used for real-time 3D scene reconstruction in scattering underwater environments.

We demonstrate a fully submerged underwater LiDAR transceiver system based on single-photon detection technologies. The LiDAR imaging system used a silicon single-photon avalanche diode (SPAD) detector array fabricated in complementary metal-oxide semiconductor (CMOS) technology to measure photon time-of-flight using picosecond resolution time-correlated single-photon counting. The SPAD detector array was directly interfaced to a Graphics Processing Unit (GPU) for real-time image reconstruction capability. Experiments were performed with the transceiver system and target objects immersed in a water tank at a depth of 1.8 meters, with the targets placed at a stand-off distance of approximately 3 meters. The transceiver used a picosecond pulsed laser source with a central wavelength of 532 nm, operating at a repetition rate of 20 MHz and average optical power of up to 52 mW, dependent on scattering conditions. Three-dimensional imaging was demonstrated by implementing a joint surface detection and distance estimation algorithm for real-time processing and visualization, which achieved images of stationary targets with up to 7.5 attenuation lengths between the transceiver and the target. The average processing time per frame was approximately 33 ms, allowing real-time three-dimensional video demonstrations of moving targets at ten frames per second at up to 5.5 attenuation lengths between transceiver and target.

Development of a high-speed line-scanning fluorescence lifetime imaging microscope for biological imaging.

We report the development of a novel line-scanning microscope capable of acquiring high-speed time-correlated single-photon counting (TCSPC)-based fluorescence lifetime imaging microscopy (FLIM) imaging. The system consists of a laser-line focus, which is optically conjugated to a 1024 × 8 single-photon avalanche diode (SPAD)-based line-imaging complementary metal-oxide semiconductor (CMOS), with 23.78 µm pixel pitch at 49.31% fill factor. Incorporation of on-chip histogramming on the line-sensor enables acquisition rates 33 times faster than our previously reported bespoke high-speed FLIM platforms. We demonstrate the imaging capability of the high-speed FLIM platform in a number of biological applications.

Guided Direct Time-of-Flight Lidar Using Stereo Cameras for Enhanced Laser Power Efficiency.

Self-driving vehicles demand efficient and reliable depth-sensing technologies. Lidar, with its capability for long-distance, high-precision measurement, is a crucial component in this pursuit. However, conventional mechanical scanning implementations suffer from reliability, cost, and frame rate limitations. Solid-state lidar solutions have emerged as a promising alternative, but the vast amount of photon data processed and stored using conventional direct time-of-flight (dToF) prevents long-distance sensing unless power-intensive partial histogram approaches are used. In this paper, we introduce a groundbreaking 'guided' dToF approach, harnessing external guidance from other onboard sensors to narrow down the depth search space for a power and data-efficient solution. This approach centers around a dToF sensor in which the exposed time window of independent pixels can be dynamically adjusted. We utilize a 64-by-32 macropixel dToF sensor and a pair of vision cameras to provide the guiding depth estimates. Our demonstrator captures a dynamic outdoor scene at 3 fps with distances up to 75 m. Compared to a conventional full histogram approach, on-chip data is reduced by over twenty times, while the total laser cycles in each frame are reduced by at least six times compared to any partial histogram approach. The capability of guided dToF to mitigate multipath reflections is also demonstrated. For self-driving vehicles where a wealth of sensor data is already available, guided dToF opens new possibilities for efficient solid-state lidar.

High-sensitivity inter-satellite optical communications using chip-scale LED and single-photon detector hardware.

Small satellites have challenging size weight and power requirements for communications modules, which we address here by using chip-scale light-emitting diode (LED) transmitters and single-photon avalanche diode receivers. Data rates of 100 Mb/s have been demonstrated at a sensitivity of -55.2 dBm, and simulations with supporting experimental work indicate ranges in excess of 1 km are feasible with a directional gain of up to 52 dBi and comparatively modest pointing requirements. A 750 m, 20 Mb/s link using a single micro-LED has been demonstrated experimentally. The low electrical power requirements and compact, semiconductor nature of these devices offer high data rate, high sensitivity communications for small satellite platforms.

Robust super-resolution depth imaging via a multi-feature fusion deep network.

The number of applications that use depth imaging is increasing rapidly, e.g. self-driving autonomous vehicles and auto-focus assist on smartphone cameras. Light detection and ranging (LIDAR) via single-photon sensitive detector (SPAD) arrays is an emerging technology that enables the acquisition of depth images at high frame rates. However, the spatial resolution of this technology is typically low in comparison to the intensity images recorded by conventional cameras. To increase the native resolution of depth images from a SPAD camera, we develop a deep network built to take advantage of the multiple features that can be extracted from a camera's histogram data. The network is designed for a SPAD camera operating in a dual-mode such that it captures alternate low resolution depth and high resolution intensity images at high frame rates, thus the system does not require any additional sensor to provide intensity images. The network then uses the intensity images and multiple features extracted from down-sampled histograms to guide the up-sampling of the depth. Our network provides significant image resolution enhancement and image denoising across a wide range of signal-to-noise ratios and photon levels. Additionally, we show that the network can be applied to other data types of SPAD data, demonstrating the generality of the algorithm.

Time-correlated single photon Raman spectroscopy at megahertz repetition rates.

Significant improvements in time-correlated single photon counting (TCSPC) Raman spectroscopy acquisition times can be achieved through exploitation of megahertz (MHz) laser repetition rates. We have developed a TCSPC Raman spectroscopy system based on a high peak power (>40W) pulsed laser, a high pulse repetition rate (40 MHz), a custom f/1.5 spectrometer, and a 512 spectral channel × 16 time bin single photon avalanche diode line sensor. We report millisecond Raman spectrum acquisition times, a peak Raman count rate of 104 kcps, and a linewidth aggregated count rate of 440 kcps with a diamond sample. This represents a three-order-of-magnitude increase in measured Raman count rate in comparison with a 104 kHz pulsed laser operating at 300 W and a four-order-of-magnitude increase over a 0.1 W pulsed laser operating at 40 MHz. A Raman-to-fluorescence ratio of 4.76 is achieved with a sesame oil sample at a 20 MHz repetition rate. Achieving high count rates and Raman-to-fluorescence ratios unlocks the potential of combined Raman/fluorescence lifetime spectroscopy for imaging and other short acquisition time applications.

Combining time of flight and photometric stereo imaging for 3D reconstruction of discontinuous scenes.

Time of flight and photometric stereo are two three-dimensional (3D) imaging techniques with complementary properties, where the former can achieve depth accuracy in discontinuous scenes, and the latter can reconstruct surfaces of objects with fine depth details and high spatial resolution. In this work, we demonstrate the surface reconstruction of complex 3D fields with discontinuity between objects by combining the two imaging methods. Using commercial LEDs, a single-photon avalanche diode camera, and a mobile phone device, high resolution of surface reconstruction is achieved with a RMS error of 6% for an object auto-selected from a scene imaged at a distance of 50 cm.

Cleansing and separation procedures reflect resource concerns.

We propose that procedures of separation have two functions, namely first, to establish the integrity of individual parts, and second, to make previously joint entities discreet and therefore countable. This allows taking stock of available resources, including evaluating the use of individual objects, a process that is especially adaptive under conditions of threat of contagious disease and resource scarcity.

Distortion losses of high-speed single-photon avalanche diode optical receivers approaching quantum sensitivity.

The high internal gain of single-photon avalanche diodes (SPADs) operating in Geiger mode allows the quantum limit of detection to be approached. This offers a significantly improved sensitivity for optical communication over existing photodiodes. A fully integrated CMOS SPAD array receiver (RX) is presented which achieves 500 Mb s-1 four-level pulse amplitude modulation in a visible light communication link within 15.2 dB of the quantum limit. However, SPAD dead time induces around 5.7 dB of transient distortion which restricts error performance and data rate. We propose a model describing a discrete photon counting system which exhibits this nonlinear behaviour and compare it to practical measurements with the RX. A unipolar intensity modulated optical signal is considered, as opposed to bipolar electric fields in conventional radio frequency wireless systems. Intermodulation between the DC and harmonic components of the data-carrying waveform is investigated, and the resulting degradation of signal-to-noise-and-distortion ratio and bit error rate is evaluated. The model is developed as a tool for understanding distortion to ultimately allow rectification through RX architecture, modulation scheme, coding and equalization techniques. This article is part of the theme issue 'Optical wireless communication'.

High concentration factor diffractive microlenses integrated with CMOS single-photon avalanche diode detector arrays for fill-factor improvement.

Large-format single-photon avalanche diode (SPAD) arrays often suffer from low fill-factors-the ratio of the active area to the overall pixel area. The detection efficiency of these detector arrays can be vastly increased with the integration of microlens arrays designed to concentrate incident light onto the active areas and may be refractive or diffractive in nature. The ability of diffractive optical elements (DOEs) to efficiently cover a square or rectangular pixel, combined with their capability of working as fast lenses (i.e., ∼f/3) makes them versatile and practical lens designs for use in sparse photon applications using microscale, large-format detector arrays. Binary-mask-based photolithography was employed to fabricate fast diffractive microlenses for two designs of 32×32 SPAD detector arrays, each design having a different pixel pitch and fill-factor. A spectral characterization of the lenses is performed, as well as analysis of performance under different illumination conditions from wide- to narrow-angle illumination (i.e., f/2 to f/22 optics). The performance of the microlenses presented exceeds previous designs in terms of both concentration factor (i.e., increase in light collection capability) and lens speed. Concentration factors greater than 33× are achieved for focal lengths in the substrate material as short as 190µm, representing a microlens f-number of 3.8 and providing a focal spot diameter of <4µm. These results were achieved while retaining an extremely high degree of performance uniformity across the 1024 devices in each case, which demonstrates the significant benefits to be gained by the implementation of DOEs as part of an integrated detector system using SPAD arrays with very small active areas.

Multispectral time-of-flight imaging using light-emitting diodes.

Multispectral and 3-D imaging are useful for a wide variety of applications, adding valuable spectral and depth information for image analysis. Single-photon avalanche diode (SPAD) based imaging systems provide photon time-of-arrival information, and can be used for imaging with time-correlated single photon counting techniques. Here we demonstrate an LED based synchronised illumination system, where temporally structured light can be used to relate time-of-arrival to specific wavelengths, thus recovering reflectance information. Cross-correlation of the received multi-peak histogram with a reference measurement yields a time delay, allowing depth information to be determined with cm-scale resolution despite the long sequence of relatively wide (∼10 ns) pulses. Using commercial LEDs and a SPAD imaging array, multispectral 3-D imaging is demonstrated across 9 visible wavelength bands.

High-resolution depth profiling using a range-gated CMOS SPAD quanta image sensor.

A CMOS single-photon avalanche diode (SPAD) quanta image sensor is used to reconstruct depth and intensity profiles when operating in a range-gated mode used in conjunction with pulsed laser illumination. By designing the CMOS SPAD array to acquire photons within a pre-determined temporal gate, the need for timing circuitry was avoided and it was therefore possible to have an enhanced fill factor (61% in this case) and a frame rate (100,000 frames per second) that is more difficult to achieve in a SPAD array which uses time-correlated single-photon counting. When coupled with appropriate image reconstruction algorithms, millimeter resolution depth profiles were achieved by iterating through a sequence of temporal delay steps in synchronization with laser illumination pulses. For photon data with high signal-to-noise ratios, depth images with millimeter scale depth uncertainty can be estimated using a standard cross-correlation approach. To enhance the estimation of depth and intensity images in the sparse photon regime, we used a bespoke clustering-based image restoration strategy, taking into account the binomial statistics of the photon data and non-local spatial correlations within the scene. For sparse photon data with total exposure times of 75 ms or less, the bespoke algorithm can reconstruct depth images with millimeter scale depth uncertainty at a stand-off distance of approximately 2 meters. We demonstrate a new approach to single-photon depth and intensity profiling using different target scenes, taking full advantage of the high fill-factor, high frame rate and large array format of this range-gated CMOS SPAD array.

Cylindrical microlensing for enhanced collection efficiency of small pixel SPAD arrays in single-molecule localisation microscopy.

Single-photon avalanche photodiode (SPAD) image sensors offer time-gated photon counting, at high binary frame rates of >100 kFPS and with no readout noise. This makes them well-suited to a range of scientific applications, including microscopy, sensing and quantum optics. However, due to the complex electronics required, the fill factor tends to be significantly lower (< 10%) than that of EMCCD and sCMOS cameras (>90%), whilst the pixel size is typically larger, impacting the sensitivity and practicalities of the SPAD devices. This paper presents the first characterisation of a cylindrical-shaped microlens array applied to a small, 8 micron, pixel SPAD imager. The enhanced fill factor, ≈50% for collimated light, is the highest reported value amongst SPAD sensors with comparable resolution and pixel pitch. We demonstrate the impact of the increased sensitivity in single-molecule localisation microscopy, obtaining a resolution of below 40nm, the best reported figure for a SPAD sensor.

Multifocal multiphoton volumetric imaging approach for high-speed time-resolved Förster resonance energy transfer imaging in vivo.

In this Letter, we will discuss the development of a multifocal multiphoton fluorescent lifetime imaging system where four individual fluorescent intensity and lifetime planes are acquired simultaneously, allowing us to obtain volumetric data without the need for sequential scanning at different axial depths. Using a phase-only spatial light modulator (SLM) with an appropriate algorithm to generate a holographic pattern, we project a beamlet array within a sample volume of a size, which can be preprogrammed by the user. We demonstrate the capabilities of the system to image live-cell interactions. While only four planes are shown, this technique can be rescaled to a large number of focal planes, enabling full 3D acquisition and reconstruction.

Single-Photon Tracking for High-Speed Vision.

Quanta Imager Sensors provide photon detections at high frame rates, with negligible read-out noise, making them ideal for high-speed optical tracking. At the basic level of bit-planes or binary maps of photon detections, objects may present limited detail. However, through motion estimation and spatial reassignment of photon detections, the objects can be reconstructed with minimal motion artefacts. We here present the first demonstration of high-speed two-dimensional (2D) tracking and reconstruction of rigid, planar objects with a Quanta Image Sensor, including a demonstration of depth-resolved tracking.

High Dynamic Range Imaging at the Quantum Limit with Single Photon Avalanche Diode-Based Image Sensors.

This paper examines methods to best exploit the High Dynamic Range (HDR) of the single photon avalanche diode (SPAD) in a high fill-factor HDR photon counting pixel that is scalable to megapixel arrays. The proposed method combines multi-exposure HDR with temporal oversampling in-pixel. We present a silicon demonstration IC with 96 × 40 array of 8.25 µm pitch 66% fill-factor SPAD-based pixels achieving >100 dB dynamic range with 3 back-to-back exposures (short, mid, long). Each pixel sums 15 bit-planes or binary field images internally to constitute one frame providing 3.75× data compression, hence the 1k frames per second (FPS) output off-chip represents 45,000 individual field images per second on chip. Two future projections of this work are described: scaling SPAD-based image sensors to HDR 1 MPixel formats and shrinking the pixel pitch to 1-3 µm.

New high-speed centre of mass method incorporating background subtraction for accurate determination of fluorescence lifetime.

We demonstrate an implementation of a centre-of-mass method (CMM) incorporating background subtraction for use in multifocal fluorescence lifetime imaging microscopy to accurately determine fluorescence lifetime in live cell imaging using the Megaframe camera. The inclusion of background subtraction solves one of the major issues associated with centre-of-mass approaches, namely the sensitivity of the algorithm to background signal. The algorithm, which is predominantly implemented in hardware, provides real-time lifetime output and allows the user to effectively condense large amounts of photon data. Instead of requiring the transfer of thousands of photon arrival times, the lifetime is simply represented by one value which allows the system to collect data up to limit of pulse pile-up without any limitations on data transfer rates. In order to evaluate the performance of this new CMM algorithm with existing techniques (i.e. rapid lifetime determination and Levenburg-Marquardt), we imaged live MCF-7 human breast carcinoma cells transiently transfected with FRET standards. We show that, it offers significant advantages in terms of lifetime accuracy and insensitivity to variability in dark count rate (DCR) between Megaframe camera pixels. Unlike other algorithms no prior knowledge of the expected lifetime is required to perform lifetime determination. The ability of this technique to provide real-time lifetime readout makes it extremely useful for a number of applications.

Real-time fluorescence lifetime actuation for cell sorting using a CMOS SPAD silicon photomultiplier.

Time-correlated single photon counting (TCSPC) is a fundamental fluorescence lifetime measurement technique offering high signal to noise ratio (SNR). However, its requirement for complex software algorithms for histogram processing restricts throughput in flow cytometers and prevents on-the-fly sorting of cells. We present a single-point digital silicon photomultiplier (SiPM) detector accomplishing real-time fluorescence lifetime-activated actuation targeting cell sorting applications in flow cytometry. The sensor also achieves burst-integrated fluorescence lifetime (BIFL) detection by TCSPC. The SiPM is a single-chip complementary metal-oxide-semiconductor (CMOS) sensor employing a 32×32 single-photon avalanche diode (SPAD) array and eight pairs of time-interleaved time to digital converters (TI-TDCs) with a 50 ps minimum timing resolution. The sensor's pile-up resistant embedded center of mass method (CMM) processor accomplishes low-latency measurement and thresholding of fluorescence lifetime. A digital control signal is generated with a 16.6 µs latency for cell sorter actuation allowing a maximum cell throughput of 60,000 cells per second and an error rate of 0.6%.