Human activity recognition using a single-photon direct time-of-flight sensor.

Single-Photon Avalanche Diode (SPAD) direct Time-of-Flight (dToF) sensors provide depth imaging over long distances, enabling the detection of objects even in the absence of contrast in colour or texture. However, distant objects are represented by just a few pixels and are subject to noise from solar interference, limiting the applicability of existing computer vision techniques for high-level scene interpretation. We present a new SPAD-based vision system for human activity recognition, based on convolutional and recurrent neural networks, which is trained entirely on synthetic data. In tests using real data from a 64×32 pixel SPAD, captured over a distance of 40 m, the scheme successfully overcomes the limited transverse resolution (in which human limbs are approximately one pixel across), achieving an average accuracy of 89% in distinguishing between seven different activities. The approach analyses continuous streams of video-rate depth data at a maximal rate of 66 FPS when executed on a GPU, making it well-suited for real-time applications such as surveillance or situational awareness in autonomous systems.

Fluorescence lifetime imaging with distance and ranging using a miniaturised SPAD system.

In this work we demonstrate a miniaturised imaging system based around a time-gated SPAD array operating in a "chip-on-tip" manner. Two versions of the system are demonstrated, each measuring 23 mm × 23 mm × 28 mm with differing fields of view and working distances. Initial tests demonstrate contrast between materials in widefield fluorescence imaging (WFLIm) mode, with frame rates of > 2 Hz achievable. Following this, WFLIm images of autofluorescence in ovine lung tissue are obtained at frame rates of ~ 1 Hz. Finally, the ability of the second system to perform simultaneous WFLIm and time of flight (aka Flourescence Lifetime Imaging Distance and Ranging, FLImDAR) is also tested. This shows that the system is capable of 4 mm resolution of object separation when tested on 3D printed samples. It is further demonstrated as being able to perform scene reconstruction on autofluorescent lung tissue. This system is, to date, the smallest chip on tip WFLIm system published, and is the first demonstration of the FLImDAR technique in a compact, portable system.

Combined fluorescence lifetime and surface topographical imaging of biological tissue.

In this work a combined fluorescence lifetime and surface topographical imaging system is demonstrated. Based around a 126 × 192 time resolved single photon avalanche diode (SPAD) array operating in time correlated single-photon counting (TCSPC) mode, both the fluorescence lifetime and time of flight (ToF) can be calculated on a pixel by pixel basis. Initial tests on fluorescent samples show it is able to provide 4 mm resolution in distance and 0.4 ns resolution in lifetime. This combined modality has potential biomedical applications such as surgical guidance, endoscopy, and diagnostic imaging. The system is demonstrated on both ovine and human pulmonary tissue samples, where it offers excellent fluorescence lifetime contrast whilst also giving a measure of the distance to the sample surface.

A time-correlated single photon counting SPAD array camera with a bespoke data-processing algorithm for lightsheet fluorescence lifetime imaging (FLIM) and FLIM videos.

A wide-field microscope with epi-fluorescence and selective plane illumination was combined with a single-photon avalanche diode (SPAD) array camera to enable live-cell fluorescence lifetime imaging (FLIM) using time-correlated single-photon counting (TCSPC). The camera sensor comprised of 192 × 128 pixels, each integrating a single SPAD and a time-to-digital converter. Jointly, they produced a stream of single-photon images of photon arrival times with ≈ 38 ps accuracy. The photon arrival times were subject to systematic delays and nonlinearities, which were corrected by a Monte-Carlo algorithm. The SPAD camera was then applied to FLIM where histogramming the resulting photon arrival times in each pixel resulted in decays compatible with common data processing pipelines for fluorescence lifetime analysis. The capabilities of the TCSPC camera-based FLIM microscope were demonstrated by imaging living unicellular photosynthetic algae and artificial lipid vesicles. Epi-fluorescence illumination enabled rapid fluorescence lifetime imaging of living cells and selective-plane illumination enabled 3-dimensional FLIM of stationary samples.