Human activity recognition using a single-photon direct time-of-flight sensor.

Single-Photon Avalanche Diode (SPAD) direct Time-of-Flight (dToF) sensors provide depth imaging over long distances, enabling the detection of objects even in the absence of contrast in colour or texture. However, distant objects are represented by just a few pixels and are subject to noise from solar interference, limiting the applicability of existing computer vision techniques for high-level scene interpretation. We present a new SPAD-based vision system for human activity recognition, based on convolutional and recurrent neural networks, which is trained entirely on synthetic data. In tests using real data from a 64×32 pixel SPAD, captured over a distance of 40 m, the scheme successfully overcomes the limited transverse resolution (in which human limbs are approximately one pixel across), achieving an average accuracy of 89% in distinguishing between seven different activities. The approach analyses continuous streams of video-rate depth data at a maximal rate of 66 FPS when executed on a GPU, making it well-suited for real-time applications such as surveillance or situational awareness in autonomous systems.

Fluorescence lifetime imaging with distance and ranging using a miniaturised SPAD system.

In this work we demonstrate a miniaturised imaging system based around a time-gated SPAD array operating in a "chip-on-tip" manner. Two versions of the system are demonstrated, each measuring 23 mm × 23 mm × 28 mm with differing fields of view and working distances. Initial tests demonstrate contrast between materials in widefield fluorescence imaging (WFLIm) mode, with frame rates of > 2 Hz achievable. Following this, WFLIm images of autofluorescence in ovine lung tissue are obtained at frame rates of ~ 1 Hz. Finally, the ability of the second system to perform simultaneous WFLIm and time of flight (aka Flourescence Lifetime Imaging Distance and Ranging, FLImDAR) is also tested. This shows that the system is capable of 4 mm resolution of object separation when tested on 3D printed samples. It is further demonstrated as being able to perform scene reconstruction on autofluorescent lung tissue. This system is, to date, the smallest chip on tip WFLIm system published, and is the first demonstration of the FLImDAR technique in a compact, portable system.

A time-correlated single photon counting SPAD array camera with a bespoke data-processing algorithm for lightsheet fluorescence lifetime imaging (FLIM) and FLIM videos.

A wide-field microscope with epi-fluorescence and selective plane illumination was combined with a single-photon avalanche diode (SPAD) array camera to enable live-cell fluorescence lifetime imaging (FLIM) using time-correlated single-photon counting (TCSPC). The camera sensor comprised of 192 × 128 pixels, each integrating a single SPAD and a time-to-digital converter. Jointly, they produced a stream of single-photon images of photon arrival times with ≈ 38 ps accuracy. The photon arrival times were subject to systematic delays and nonlinearities, which were corrected by a Monte-Carlo algorithm. The SPAD camera was then applied to FLIM where histogramming the resulting photon arrival times in each pixel resulted in decays compatible with common data processing pipelines for fluorescence lifetime analysis. The capabilities of the TCSPC camera-based FLIM microscope were demonstrated by imaging living unicellular photosynthetic algae and artificial lipid vesicles. Epi-fluorescence illumination enabled rapid fluorescence lifetime imaging of living cells and selective-plane illumination enabled 3-dimensional FLIM of stationary samples.

Combined fluorescence lifetime and surface topographical imaging of biological tissue.

In this work a combined fluorescence lifetime and surface topographical imaging system is demonstrated. Based around a 126 × 192 time resolved single photon avalanche diode (SPAD) array operating in time correlated single-photon counting (TCSPC) mode, both the fluorescence lifetime and time of flight (ToF) can be calculated on a pixel by pixel basis. Initial tests on fluorescent samples show it is able to provide 4 mm resolution in distance and 0.4 ns resolution in lifetime. This combined modality has potential biomedical applications such as surgical guidance, endoscopy, and diagnostic imaging. The system is demonstrated on both ovine and human pulmonary tissue samples, where it offers excellent fluorescence lifetime contrast whilst also giving a measure of the distance to the sample surface.

Guided Direct Time-of-Flight Lidar Using Stereo Cameras for Enhanced Laser Power Efficiency.

Self-driving vehicles demand efficient and reliable depth-sensing technologies. Lidar, with its capability for long-distance, high-precision measurement, is a crucial component in this pursuit. However, conventional mechanical scanning implementations suffer from reliability, cost, and frame rate limitations. Solid-state lidar solutions have emerged as a promising alternative, but the vast amount of photon data processed and stored using conventional direct time-of-flight (dToF) prevents long-distance sensing unless power-intensive partial histogram approaches are used. In this paper, we introduce a groundbreaking 'guided' dToF approach, harnessing external guidance from other onboard sensors to narrow down the depth search space for a power and data-efficient solution. This approach centers around a dToF sensor in which the exposed time window of independent pixels can be dynamically adjusted. We utilize a 64-by-32 macropixel dToF sensor and a pair of vision cameras to provide the guiding depth estimates. Our demonstrator captures a dynamic outdoor scene at 3 fps with distances up to 75 m. Compared to a conventional full histogram approach, on-chip data is reduced by over twenty times, while the total laser cycles in each frame are reduced by at least six times compared to any partial histogram approach. The capability of guided dToF to mitigate multipath reflections is also demonstrated. For self-driving vehicles where a wealth of sensor data is already available, guided dToF opens new possibilities for efficient solid-state lidar.

Handheld wide-field fluorescence lifetime imaging system based on a distally mounted SPAD array.

In this work a handheld Fluorescent Lifetime IMaging (FLIM) system based on a distally mounted < 2 mm2 128 × 120 single photon avalanche diode (SPAD) array operating over a > 1 m long wired interface is demonstrated. The head of the system is ∼4.5 cm x 4.5 cm x 4.5 cm making it suitable for hand-held ex vivo applications. This is, to the best of the authors' knowledge, the first example of a SPAD array mounted on the distal end of a handheld FLIM system in this manner. All existing systems to date use a fibre to collect and relay fluorescent light to detectors at the proximal end of the system. This has clear potential biological and biomedical applications. To demonstrate this, the system is used to provide contrast between regions of differing tissue composition in ovine kidney samples, and between healthy and stressed or damaged plant leaves. Additionally, FLIM videos are provided showing that frame rates of > 1 Hz are achievable. It is thus an important step in realising an in vivo miniaturized chip-on-tip FLIM endoscopy system.

Visualising varnish removal for conservation of paintings by fluorescence lifetime imaging (FLIM).

The removal of varnish from the surface is a key step in painting conservation. Varnish removal is traditionally monitored by examining the painting surface under ultraviolet illumination. We show here that by imaging the fluorescence lifetime instead, much better contrast, sensitivity, and specificity can be achieved. For this purpose, we developed a lightweight (4.8 kg) portable instrument for macroscopic fluorescence lifetime imaging (FLIM). It is based on a time-correlated single-photon avalanche diode (SPAD) camera to acquire the FLIM images and a pulsed 440 nm diode laser to excite the varnish fluorescence. A historical model painting was examined to demonstrate the capabilities of the system. We found that the FLIM images provided information on the distribution of the varnish on the painting surface with greater sensitivity, specificity, and contrast compared to the traditional ultraviolet illumination photography. The distribution of the varnish and other painting materials was assessed using FLIM during and after varnish removal with different solvent application methods. Monitoring of the varnish removal process between successive solvent applications by a swab revealed an evolving image contrast as a function of the cleaning progress. FLIM of dammar and mastic resin varnishes identified characteristic changes to their fluorescence lifetimes depending on their ageing conditions. Thus, FLIM has a potential to become a powerful and versatile tool to visualise varnish removal from paintings.

Field programmable gate array compression for large array multispeckle diffuse correlation spectroscopy.

Significance: Diffuse correlation spectroscopy (DCS) is an indispensable tool for quantifying cerebral blood flow noninvasively by measuring the autocorrelation function (ACF) of the diffused light. Recently, a multispeckle DCS approach was proposed to scale up the sensitivity with the number of independent speckle measurements, leveraging the rapid development of single-photon avalanche diode (SPAD) cameras. However, the extremely high data rate from advanced SPAD cameras is beyond the data transfer rate commonly available and requires specialized high-performance computation to calculate large number of autocorrelators (ACs) for real-time measurements. Aim: We aim to demonstrate a data compression scheme in the readout field-programmable gate array (FPGA) of a large-pixel-count SPAD camera. On-FPGA, data compression should democratize SPAD cameras and streamline system integration for multispeckle DCS. Approach: We present a 192×128 SPAD array with 128 linear ACs embedded on an FPGA to calculate 12,288 ACFs in real time. Results: We achieved a signal-to-noise ratio (SNR) gain of 110 over a single-pixel DCS system and more than threefold increase in SNR with respect to the state-of-the-art multispeckle DCS. Conclusions: The FPGA-embedded autocorrelation algorithm offers a scalable data compression method to large SPAD array, which can improve the sensitivity and usability of multispeckle DCS instruments.

Submerged single-photon LiDAR imaging sensor used for real-time 3D scene reconstruction in scattering underwater environments.

We demonstrate a fully submerged underwater LiDAR transceiver system based on single-photon detection technologies. The LiDAR imaging system used a silicon single-photon avalanche diode (SPAD) detector array fabricated in complementary metal-oxide semiconductor (CMOS) technology to measure photon time-of-flight using picosecond resolution time-correlated single-photon counting. The SPAD detector array was directly interfaced to a Graphics Processing Unit (GPU) for real-time image reconstruction capability. Experiments were performed with the transceiver system and target objects immersed in a water tank at a depth of 1.8 meters, with the targets placed at a stand-off distance of approximately 3 meters. The transceiver used a picosecond pulsed laser source with a central wavelength of 532 nm, operating at a repetition rate of 20 MHz and average optical power of up to 52 mW, dependent on scattering conditions. Three-dimensional imaging was demonstrated by implementing a joint surface detection and distance estimation algorithm for real-time processing and visualization, which achieved images of stationary targets with up to 7.5 attenuation lengths between the transceiver and the target. The average processing time per frame was approximately 33 ms, allowing real-time three-dimensional video demonstrations of moving targets at ten frames per second at up to 5.5 attenuation lengths between transceiver and target.

Development of a high-speed line-scanning fluorescence lifetime imaging microscope for biological imaging.

We report the development of a novel line-scanning microscope capable of acquiring high-speed time-correlated single-photon counting (TCSPC)-based fluorescence lifetime imaging microscopy (FLIM) imaging. The system consists of a laser-line focus, which is optically conjugated to a 1024 × 8 single-photon avalanche diode (SPAD)-based line-imaging complementary metal-oxide semiconductor (CMOS), with 23.78 µm pixel pitch at 49.31% fill factor. Incorporation of on-chip histogramming on the line-sensor enables acquisition rates 33 times faster than our previously reported bespoke high-speed FLIM platforms. We demonstrate the imaging capability of the high-speed FLIM platform in a number of biological applications.

Social threat indirectly increases moral condemnation via thwarting fundamental social needs.

Individuals who experience threats to their social needs may attempt to avert further harm by condemning wrongdoers more severely. Three pre-registered studies tested whether threatened social esteem is associated with increased moral condemnation. In Study 1 (N = 381) participants played a game in which they were socially included or excluded and then evaluated the actions of moral wrongdoers. We observed an indirect effect: Exclusion increased social needs-threat, which in turn increased moral condemnation. Study 2 (N = 428) was a direct replication, and also showed this indirect effect. Both studies demonstrated the effect across five moral foundations, and was most pronounced for harm violations. Study 3 (N = 102) examined dispositional concerns about social needs threat, namely social anxiety, and showed a positive correlation between this trait and moral judgments. Overall, results suggest threatened social standing is linked to moral condemnation, presumably because moral wrongdoers pose a further threat when one's ability to cope is already compromised.

Full spectrum fluorescence lifetime imaging with 0.5 nm spectral and 50 ps temporal resolution.

The use of optical techniques to interrogate wide ranging samples from semiconductors to biological tissue for rapid analysis and diagnostics has gained wide adoption over the past decades. The desire to collect ever more spatially, spectrally and temporally detailed optical signatures for sample characterization has specifically driven a sharp rise in new optical microscopy technologies. Here we present a high-speed optical scanning microscope capable of capturing time resolved images across 512 spectral and 32 time channels in a single acquisition with the potential for ~0.2 frames per second (256 × 256 image pixels). Each pixel in the resulting images contains a detailed data cube for the study of diverse time resolved light driven phenomena. This is enabled by integration of system control electronics and on-chip processing which overcomes the challenges presented by high data volume and low imaging speed, often bottlenecks in previous systems.

Time-correlated single photon Raman spectroscopy at megahertz repetition rates.

Significant improvements in time-correlated single photon counting (TCSPC) Raman spectroscopy acquisition times can be achieved through exploitation of megahertz (MHz) laser repetition rates. We have developed a TCSPC Raman spectroscopy system based on a high peak power (>40W) pulsed laser, a high pulse repetition rate (40 MHz), a custom f/1.5 spectrometer, and a 512 spectral channel × 16 time bin single photon avalanche diode line sensor. We report millisecond Raman spectrum acquisition times, a peak Raman count rate of 104 kcps, and a linewidth aggregated count rate of 440 kcps with a diamond sample. This represents a three-order-of-magnitude increase in measured Raman count rate in comparison with a 104 kHz pulsed laser operating at 300 W and a four-order-of-magnitude increase over a 0.1 W pulsed laser operating at 40 MHz. A Raman-to-fluorescence ratio of 4.76 is achieved with a sesame oil sample at a 20 MHz repetition rate. Achieving high count rates and Raman-to-fluorescence ratios unlocks the potential of combined Raman/fluorescence lifetime spectroscopy for imaging and other short acquisition time applications.

Combining time of flight and photometric stereo imaging for 3D reconstruction of discontinuous scenes.

Time of flight and photometric stereo are two three-dimensional (3D) imaging techniques with complementary properties, where the former can achieve depth accuracy in discontinuous scenes, and the latter can reconstruct surfaces of objects with fine depth details and high spatial resolution. In this work, we demonstrate the surface reconstruction of complex 3D fields with discontinuity between objects by combining the two imaging methods. Using commercial LEDs, a single-photon avalanche diode camera, and a mobile phone device, high resolution of surface reconstruction is achieved with a RMS error of 6% for an object auto-selected from a scene imaged at a distance of 50 cm.

Disease and Disapproval: COVID-19 Concern is Related to Greater Moral Condemnation.

Prior research has indicated that disease threat and disgust are associated with harsher moral condemnation. We investigated the role of a specific, highly salient health concern, namely the spread of the coronavirus, and associated COVID-19 disease, on moral disapproval. We hypothesized that individuals who report greater subjective worry about COVID-19 would be more sensitive to moral transgressions. Across three studies (N = 913), conducted March-May 2020 as the pandemic started to unfold in the United States, we found that individuals who were worried about contracting the infectious disease made harsher moral judgments than those who were relatively less worried. This effect was not restricted to transgressions involving purity, but extended to transgressions involving harm, fairness, authority, and loyalty, and remained when controlling for political orientation. Furthermore, for Studies 1 and 2 the effect also was robust when taking into account the contamination subscale of the Disgust Scale-Revised. These findings add to the growing literature that concrete threats to health can play a role in abstract moral considerations, supporting the notion that judgments of wrongdoing are not based on rational thought alone.

Robust super-resolution depth imaging via a multi-feature fusion deep network.

The number of applications that use depth imaging is increasing rapidly, e.g. self-driving autonomous vehicles and auto-focus assist on smartphone cameras. Light detection and ranging (LIDAR) via single-photon sensitive detector (SPAD) arrays is an emerging technology that enables the acquisition of depth images at high frame rates. However, the spatial resolution of this technology is typically low in comparison to the intensity images recorded by conventional cameras. To increase the native resolution of depth images from a SPAD camera, we develop a deep network built to take advantage of the multiple features that can be extracted from a camera's histogram data. The network is designed for a SPAD camera operating in a dual-mode such that it captures alternate low resolution depth and high resolution intensity images at high frame rates, thus the system does not require any additional sensor to provide intensity images. The network then uses the intensity images and multiple features extracted from down-sampled histograms to guide the up-sampling of the depth. Our network provides significant image resolution enhancement and image denoising across a wide range of signal-to-noise ratios and photon levels. Additionally, we show that the network can be applied to other data types of SPAD data, demonstrating the generality of the algorithm.

High-sensitivity inter-satellite optical communications using chip-scale LED and single-photon detector hardware.

Small satellites have challenging size weight and power requirements for communications modules, which we address here by using chip-scale light-emitting diode (LED) transmitters and single-photon avalanche diode receivers. Data rates of 100 Mb/s have been demonstrated at a sensitivity of -55.2 dBm, and simulations with supporting experimental work indicate ranges in excess of 1 km are feasible with a directional gain of up to 52 dBi and comparatively modest pointing requirements. A 750 m, 20 Mb/s link using a single micro-LED has been demonstrated experimentally. The low electrical power requirements and compact, semiconductor nature of these devices offer high data rate, high sensitivity communications for small satellite platforms.

Cleansing and separation procedures reflect resource concerns.

We propose that procedures of separation have two functions, namely first, to establish the integrity of individual parts, and second, to make previously joint entities discreet and therefore countable. This allows taking stock of available resources, including evaluating the use of individual objects, a process that is especially adaptive under conditions of threat of contagious disease and resource scarcity.

High concentration factor diffractive microlenses integrated with CMOS single-photon avalanche diode detector arrays for fill-factor improvement.

Large-format single-photon avalanche diode (SPAD) arrays often suffer from low fill-factors-the ratio of the active area to the overall pixel area. The detection efficiency of these detector arrays can be vastly increased with the integration of microlens arrays designed to concentrate incident light onto the active areas and may be refractive or diffractive in nature. The ability of diffractive optical elements (DOEs) to efficiently cover a square or rectangular pixel, combined with their capability of working as fast lenses (i.e., ∼f/3) makes them versatile and practical lens designs for use in sparse photon applications using microscale, large-format detector arrays. Binary-mask-based photolithography was employed to fabricate fast diffractive microlenses for two designs of 32×32 SPAD detector arrays, each design having a different pixel pitch and fill-factor. A spectral characterization of the lenses is performed, as well as analysis of performance under different illumination conditions from wide- to narrow-angle illumination (i.e., f/2 to f/22 optics). The performance of the microlenses presented exceeds previous designs in terms of both concentration factor (i.e., increase in light collection capability) and lens speed. Concentration factors greater than 33× are achieved for focal lengths in the substrate material as short as 190µm, representing a microlens f-number of 3.8 and providing a focal spot diameter of <4µm. These results were achieved while retaining an extremely high degree of performance uniformity across the 1024 devices in each case, which demonstrates the significant benefits to be gained by the implementation of DOEs as part of an integrated detector system using SPAD arrays with very small active areas.

Quantitative real-time imaging of intracellular FRET biosensor dynamics using rapid multi-beam confocal FLIM.

Fluorescence lifetime imaging (FLIM) is a quantitative, intensity-independent microscopical method for measurement of diverse biochemical and physical properties in cell biology. It is a highly effective method for measurements of Förster resonance energy transfer (FRET), and for quantification of protein-protein interactions in cells. Time-domain FLIM-FRET measurements of these dynamic interactions are particularly challenging, since the technique requires excellent photon statistics to derive experimental parameters from the complex decay kinetics often observed from fluorophores in living cells. Here we present a new time-domain multi-confocal FLIM instrument with an array of 64 visible beamlets to achieve parallelised excitation and detection with average excitation powers of ~ 1-2 µW per beamlet. We exemplify this instrument with up to 0.5 frames per second time-lapse FLIM measurements of cAMP levels using an Epac-based fluorescent biosensor in live HeLa cells with nanometer spatial and picosecond temporal resolution. We demonstrate the use of time-dependent phasor plots to determine parameterisation for multi-exponential decay fitting to monitor the fractional contribution of the activated conformation of the biosensor. Our parallelised confocal approach avoids having to compromise on speed, noise, accuracy in lifetime measurements and provides powerful means to quantify biochemical dynamics in living cells.