Coupling a recurrent neural network to SPAD TCSPC systems for real-time fluorescence lifetime imaging.

Fluorescence lifetime imaging (FLI) has been receiving increased attention in recent years as a powerful diagnostic technique in biological and medical research. However, existing FLI systems often suffer from a tradeoff between processing speed, accuracy, and robustness. Inspired by the concept of Edge Artificial Intelligence (Edge AI), we propose a robust approach that enables fast FLI with no degradation of accuracy. This approach couples a recurrent neural network (RNN), which is trained to estimate the fluorescence lifetime directly from raw timestamps without building histograms, to SPAD TCSPC systems, thereby drastically reducing transfer data volumes and hardware resource utilization, and enabling real-time FLI acquisition. We train two variants of the RNN on a synthetic dataset and compare the results to those obtained using center-of-mass method (CMM) and least squares fitting (LS fitting). Results demonstrate that two RNN variants, gated recurrent unit (GRU) and long short-term memory (LSTM), are comparable to CMM and LS fitting in terms of accuracy, while outperforming them in the presence of background noise by a large margin. To explore the ultimate limits of the approach, we derive the Cramer-Rao lower bound of the measurement, showing that RNN yields lifetime estimations with near-optimal precision. To demonstrate real-time operation, we build a FLI microscope based on an existing SPAD TCSPC system comprising a 32[Formula: see text]32 SPAD sensor named Piccolo. Four quantized GRU cores, capable of processing up to 4 million photons per second, are deployed on the Xilinx Kintex-7 FPGA that controls the Piccolo. Powered by the GRU, the FLI setup can retrieve real-time fluorescence lifetime images at up to 10 frames per second. The proposed FLI system is promising and ideally suited for biomedical applications, including biological imaging, biomedical diagnostics, and fluorescence-assisted surgery, etc.

Subsurface fluorescence time-of-flight imaging using a large-format single-photon avalanche diode sensor for tumor depth assessment.

Significance: Fluorescence guidance is used clinically by surgeons to visualize anatomical and/or physiological phenomena in the surgical field that are difficult or impossible to detect by the naked eye. Such phenomena include tissue perfusion or molecular phenotypic information about the disease being resected. Conventional fluorescence-guided surgery relies on long, microsecond scale laser pulses to excite fluorescent probes. However, this technique only provides two-dimensional information; crucial depth information, such as the location of malignancy below the tissue surface, is not provided. Aim: We developed a depth sensing imaging technique using light detection and ranging (LiDAR) time-of-flight (TOF) technology to sense the depth of target tissue while overcoming the influence of tissue optical properties and fluorescent probe concentration. Approach: The technology is based on a large-format (512×512 pixel), binary, gated, single-photon avalanche diode (SPAD) sensor with an 18 ps time-gate step, synchronized with a picosecond pulsed laser. The fast response of the sensor was developed and tested for its ability to quantify fluorescent inclusions at depth and optical properties in tissue-like phantoms through analytical model fitting of the fast temporal remission data. Results: After calibration and algorithmic extraction of the data, the SPAD LiDAR technique allowed for sub-mm resolution depth sensing of fluorescent inclusions embedded in tissue-like phantoms, up to a maximum of 5 mm in depth. The approach provides robust depth sensing even in the presence of variable tissue optical properties and separates the effects of fluorescence depth from absorption and scattering variations. Conclusions: LiDAR TOF fluorescence imaging using an SPAD camera provides both fluorescence intensity images and the temporal profile of fluorescence, which can be used to determine the depth at which the signal is emitted over a wide field of view. The proposed tool enables fluorescence imaging at a higher depth in tissue and with higher spatial precision than standard, steady-state fluorescence imaging tools, such as intensity-based near-infrared fluorescence imaging, optical coherence tomography, Raman spectroscopy, or confocal microscopy. Integration of this technique into a standard surgical tool could enable rapid, more accurate estimation of resection boundaries, thereby improving the surgeon's efficacy and efficiency, and ultimately improving patient outcomes.

Highly sensitive single-molecule detection of macromolecule ion beams.

The analysis of proteins in the gas phase benefits from detectors that exhibit high efficiency and precise spatial resolution. Although modern secondary electron multipliers already address numerous analytical requirements, additional methods are desired for macromolecules at energies lower than currently used in post-acceleration detection. Previous studies have proven the sensitivity of superconducting detectors to high-energy particles in time-of-flight mass spectrometry. Here, we demonstrate that superconducting nanowire detectors are exceptionally well suited for quadrupole mass spectrometry and exhibit an outstanding quantum yield at low-impact energies. At energies as low as 100 eV, the sensitivity of these detectors surpasses conventional ion detectors by three orders of magnitude, and they offer the possibility to discriminate molecules by their impact energy and charge. We demonstrate three developments with these compact and sensitive devices, the recording of 2D ion beam profiles, photochemistry experiments in the gas phase, and advanced cryogenic electronics to pave the way toward highly integrated detectors.

NIR Fluorescence lifetime macroscopic imaging with a novel time-gated SPAD camera.

SwissSPAD3 is the latest of a family of widefield time-gated SPAD imagers developed for fluorescence lifetime imaging (FLI) applications. Its distinctive features are (i) the ability to define shorter gates than its predecessors (width W < 1 ns), (ii) support for laser repetition rates up to at least 80 MHz and (iii) a dual-gate architecture providing an effective duty cycle of 100%. We present widefield macroscopic FLI measurements of short lifetime NIR dyes, analyzed using the phasor approach. The results are compared with those previously obtained with SwissSPAD2 and to theoretical predictions.

Correlated-photon imaging at 10 volumetric images per second.

The correlation properties of light provide an outstanding tool to overcome the limitations of traditional imaging techniques. A relevant case is represented by correlation plenoptic imaging (CPI), a quantum-inspired volumetric imaging protocol employing spatio-temporally correlated photons from either entangled or chaotic sources to address the main limitations of conventional light-field imaging, namely, the poor spatial resolution and the reduced change of perspective for 3D imaging. However, the application potential of high-resolution imaging modalities relying on photon correlations is limited, in practice, by the need to collect a large number of frames. This creates a gap, unacceptable for many relevant tasks, between the time performance of correlated-light imaging and that of traditional imaging methods. In this article, we address this issue by exploiting the photon number correlations intrinsic in chaotic light, combined with a cutting-edge ultrafast sensor made of a large array of single-photon avalanche diodes (SPADs). This combination of source and sensor is embedded within a novel single-lens CPI scheme enabling to acquire 10 volumetric images per second. Our results place correlated-photon imaging at a competitive edge and prove its potential in practical applications.

Challenges and prospects for multi-chip microlens imprints on front-side illuminated SPAD imagers.

The overall sensitivity of frontside-illuminated, silicon single-photon avalanche diode (SPAD) arrays has often suffered from fill factor limitations. The fill factor loss can however be recovered by employing microlenses, whereby the challenges specific to SPAD arrays are represented by large pixel pitch (> 10 µm), low native fill factor (as low as ∼10%), and large size (up to 10 mm). In this work we report on the implementation of refractive microlenses by means of photoresist masters, used to fabricate molds for imprints of UV curable hybrid polymers deposited on SPAD arrays. Replications were successfully carried out for the first time, to the best of our knowledge, at wafer reticle level on different designs in the same technology and on single large SPAD arrays with very thin residual layers (∼10 µm), as needed for better efficiency at higher numerical aperture (NA > 0.25). In general, concentration factors within 15-20% of the simulation results were obtained for the smaller arrays (32×32 and 512×1), achieving for example an effective fill factor of 75.6-83.2% for a 28.5 µm pixel pitch with a native fill factor of 28%. A concentration factor up to 4.2 was measured on large 512×512 arrays with a pixel pitch of 16.38 µm and a native fill factor of 10.5%, whereas improved simulation tools could give a better estimate of the actual concentration factor. Spectral measurements were also carried out, resulting in good and uniform transmission in the visible and NIR.

Single-photon avalanche diode fabricated in standard 55†nm bipolar-CMOS-DMOS technology with sub-20†V breakdown voltage.

This paper presents a single-photon avalanche diode (SPAD) in 55 nm bipolar-CMOS-DMOS (BCD) technology. In order to realize a SPAD having sub-20 V breakdown voltage for mobile applications while preventing high tunneling noise, a high-voltage N-well available in BCD is utilized to implement the avalanche multiplication region. The resulting SPAD has a breakdown voltage of 18.4 V while achieving an excellent dark count rate of 4.4 cps/µm2 at the excess bias voltage of 7 V in spite of the advanced technology node. At the same time, the device achieves a high peak photon detection probability (PDP) of 70.1% at 450 nm thanks to the high and uniform E-field. Its PDP values at 850 and 940 nm, wavelengths of interest for 3D ranging applications reach 7.2 and 3.1%, respectively, with the use of deep N-well. The timing jitter of the SPAD, full width at half maximum (FWHM), is 91 ps at 850 nm. It is expected that the presented SPAD enables cost-effective time-of-flight and LiDAR sensors with the advanced standard technology for many mobile applications.

Light-field tomographic fluorescence lifetime imaging microscopy.

Fluorescence lifetime imaging microscopy (FLIM) is a powerful imaging technique that enables the visualization of biological samples at the molecular level by measuring the fluorescence decay rate of fluorescent probes. This provides critical information about molecular interactions, environmental changes, and localization within biological systems. However, creating high-resolution lifetime maps using conventional FLIM systems can be challenging, as it often requires extensive scanning that can significantly lengthen acquisition times. This issue is further compounded in three-dimensional (3D) imaging because it demands additional scanning along the depth axis. To tackle this challenge, we developed a novel computational imaging technique called light field tomographic FLIM (LIFT-FLIM). Our approach allows for the acquisition of volumetric fluorescence lifetime images in a highly data-efficient manner, significantly reducing the number of scanning steps required compared to conventional point-scanning or line-scanning FLIM imagers. Moreover, LIFT-FLIM enables the measurement of high-dimensional data using low-dimensional detectors, which are typically low-cost and feature a higher temporal bandwidth. We demonstrated LIFT-FLIM using a linear single-photon avalanche diode array on various biological systems, showcasing unparalleled single-photon detection sensitivity. Additionally, we expanded the functionality of our method to spectral FLIM and demonstrated its application in high-content multiplexed imaging of lung organoids. LIFT-FLIM has the potential to open up new avenues in both basic and translational biomedical research.

Massively parallel, real-time multispeckle diffuse correlation spectroscopy using a 500 × 500 SPAD camera.

Diffuse correlation spectroscopy (DCS) is a promising noninvasive technique for monitoring cerebral blood flow and measuring cortex functional activation tasks. Taking multiple parallel measurements has been shown to increase sensitivity, but is not easily scalable with discrete optical detectors. Here we show that with a large 500 × 500 SPAD array and an advanced FPGA design, we achieve an SNR gain of almost 500 over single-pixel mDCS performance. The system can also be reconfigured to sacrifice SNR to decrease correlation bin width, with 400 ns resolution being demonstrated over 8000 pixels.

LinoSPAD2: an FPGA-based, hardware-reconfigurable 512×1 single-photon camera system.

We report on LinoSPAD2, a single-photon camera system, comprising a 512×1 single-photon avalanche diode (SPAD) front-end and one or two FPGA-based back-ends. Digital signals generated by the SPADs are processed by the FPGA in real time, whereas the FPGA offers full reconfigurability at a very high level of granularity both in time and space domains. The LinoSPAD2 camera system can process 512 SPADs simultaneously through 256 channels, duplicated on each FPGA-based back-end, with a bank of 64 time-to-digital converters (TDCs) operating at 133 MSa/s, whereas each TDC has a time resolution of 20 ps (LSB). To the best of our knowledge, LinoSPAD2 is the first fully reconfigurable SPAD camera system of large format. The SPAD front-end features a pitch of 26.2 µm, a native fill factor of 25.1%, and a microlens array achieving 2.3× concentration factor. At room temperature, the median dark count rate (DCR) is 80 cps at 7 V excess bias, the peak photon detection probability (PDP) is 53% at 520 nm wavelength, and the single-photon timing resolution (SPTR) is 50 ps FWHM. The instrument response function (IRF) is around 100 ps FWHM at system level. The LinoSPAD2 camera system is suitable for numerous applications, including LiDAR imaging, heralded spectroscopy, compressive Raman sensing, and other computational imaging techniques.

In vitro and in vivo NIR fluorescence lifetime imaging with a time-gated SPAD camera.

Near-infrared (NIR) fluorescence lifetime imaging (FLI) provides a unique contrast mechanism to monitor biological parameters and molecular events in vivo. Single-photon avalanche diode (SPAD) cameras have been recently demonstrated in FLI microscopy (FLIM) applications, but their suitability for in vivo macroscopic FLI (MFLI) in deep tissues remains to be demonstrated. Herein, we report in vivo NIR MFLI measurement with SwissSPAD2, a large time-gated SPAD camera. We first benchmark its performance in well-controlled in vitro experiments, ranging from monitoring environmental effects on fluorescence lifetime, to quantifying Förster resonant energy transfer (FRET) between dyes. Next, we use it for in vivo studies of target-drug engagement in live and intact tumor xenografts using FRET. Information obtained with SwissSPAD2 was successfully compared to that obtained with a gated intensified charge-coupled device (ICCD) camera, using two different approaches. Our results demonstrate that SPAD cameras offer a powerful technology for in vivo preclinical applications in the NIR window.

Resolving the Controversy in Biexciton Binding Energy of Cesium Lead Halide Perovskite Nanocrystals through Heralded Single-Particle Spectroscopy.

Understanding exciton-exciton interaction in multiply excited nanocrystals is crucial to their utilization as functional materials. Yet, for lead halide perovskite nanocrystals, which are promising candidates for nanocrystal-based technologies, numerous contradicting values have been reported for the strength and sign of their exciton-exciton interaction. In this work, we unambiguously determine the biexciton binding energy in single cesium lead halide perovskite nanocrystals at room temperature. This is enabled by the recently introduced single-photon avalanche diode array spectrometer, capable of temporally isolating biexciton-exciton emission cascades while retaining spectral resolution. We demonstrate that CsPbBr3 nanocrystals feature an attractive exciton-exciton interaction, with a mean biexciton binding energy of 10 meV. For CsPbI3 nanocrystals, we observe a mean biexciton binding energy that is close to zero, and individual nanocrystals show either weakly attractive or weakly repulsive exciton-exciton interaction. We further show that, within ensembles of both materials, single-nanocrystal biexciton binding energies are correlated with the degree of charge-carrier confinement.

Theoretical minimum uncertainty of single-molecule localizations using a single-photon avalanche diode array.

Single-photon avalanche diode (SPAD) arrays can be used for single-molecule localization microscopy (SMLM) because of their high frame rate and lack of readout noise. SPAD arrays have a binary frame output, which means photon arrivals should be described as a binomial process rather than a Poissonian process. Consequentially, the theoretical minimum uncertainty of the localizations is not accurately predicted by the Poissonian Cramér-Rao lower bound (CRLB). Here, we derive a binomial CRLB and benchmark it using simulated and experimental data. We show that if the expected photon count is larger than one for all pixels within one standard deviation of a Gaussian point spread function, the binomial CRLB gives a 46% higher theoretical uncertainty than the Poissonian CRLB. For typical SMLM photon fluxes, where no saturation occurs, the binomial CRLB predicts the same uncertainty as the Poissonian CRLB. Therefore, the binomial CRLB can be used to predict and benchmark localization uncertainty for SMLM with SPAD arrays for all practical emitter intensities.

Heralded Spectroscopy Reveals Exciton-Exciton Correlations in Single Colloidal Quantum Dots.

Multiply excited states in semiconductor quantum dots feature intriguing physics and play a crucial role in nanocrystal-based technologies. While photoluminescence provides a natural probe to investigate these states, room-temperature single-particle spectroscopy of their emission has proved elusive due to the temporal and spectral overlap with emission from the singly excited and charged states. Here, we introduce biexciton heralded spectroscopy enabled by a single-photon avalanche diode array based spectrometer. This allows us to directly observe biexciton-exciton emission cascades and measure the biexciton binding energy of single quantum dots at room temperature, even though it is well below the scale of thermal broadening and spectral diffusion. Furthermore, we uncover correlations hitherto masked in ensembles of the biexciton binding energy with both charge-carrier confinement and fluctuations of the local electrostatic potential. Heralded spectroscopy has the potential of greatly extending our understanding of charge-carrier dynamics in multielectron systems and of parallelization of quantum optics protocols.

Fluorescence lifetime imaging with a megapixel SPAD camera and neural network lifetime estimation.

Fluorescence lifetime imaging microscopy (FLIM) is a key technology that provides direct insight into cell metabolism, cell dynamics and protein activity. However, determining the lifetimes of different fluorescent proteins requires the detection of a relatively large number of photons, hence slowing down total acquisition times. Moreover, there are many cases, for example in studies of cell collectives, where wide-field imaging is desired. We report scan-less wide-field FLIM based on a 0.5 MP resolution, time-gated Single Photon Avalanche Diode (SPAD) camera, with acquisition rates up to 1 Hz. Fluorescence lifetime estimation is performed via a pre-trained artificial neural network with 1000-fold improvement in processing times compared to standard least squares fitting techniques. We utilised our system to image HT1080-human fibrosarcoma cell line as well as Convallaria. The results show promise for real-time FLIM and a viable route towards multi-megapixel fluorescence lifetime images, with a proof-of-principle mosaic image shown with 3.6 MP.

Erratum: Author Correction: Single-photon avalanche diode imagers in biophotonics: review and outlook.

[This corrects the article DOI: 10.1038/s41377-019-0191-5.].

Wide-field time-gated SPAD imager for phasor-based FLIM applications.

We describe the performance of a new wide area time-gated single-photon avalanche diode (SPAD) array for phasor-FLIM, exploring the effect of gate length, gate number and signal intensity on the measured lifetime accuracy and precision. We conclude that the detector functions essentially as an ideal shot noise limited sensor and is capable of video rate FLIM measurement. The phasor approach used in this work appears ideally suited to handle the large amount of data generated by this type of very large sensor (512 × 512 pixels), even in the case of small number of gates and limited photon budget.

Single-Photon, Time-Gated, Phasor-Based Fluorescence Lifetime Imaging through Highly Scattering Medium.

Fluorescence lifetime imaging (FLI) is increasingly recognized as a powerful tool for biochemical and cellular investigations, including in vivo applications. Fluorescence lifetime is an intrinsic characteristic of any fluorescent dye which, to a large extent, does not depend on excitation intensity and signal level. In particular, it allows distinguishing dyes with similar emission spectra, offering additional multiplexing capabilities. However, in vivo FLI in the visible range is complicated by the contamination by (i) tissue autofluorescence, which decreases contrast, and by (ii) light scattering and absorption in tissues, which significantly reduce fluorescence intensity and modify the temporal profile of the signal. Here, we demonstrate how these issues can be accounted for and overcome, using a new time-gated single-photon avalanche diode array camera, SwissSPAD2, combined with phasor analysis to provide a simple and fast visual method for lifetime imaging. In particular, we show how phasor dispersion increases with increasing scattering and/or decreasing fluorescence intensity. Next, we show that as long as the fluorescence signal of interest is larger than the phantom autofluorescence, the presence of a distinct lifetime can be clearly identified with appropriate background correction. We use these results to demonstrate the detection of A459 cells expressing the fluorescent protein mCyRFP1 through highly scattering and autofluorescent phantom layers. These results showcase the possibility to perform FLI in challenging conditions, using standard, bright, visible fluorophore or fluorescence proteins.

Quantum correlation measurement with single photon avalanche diode arrays.

Temporal photon correlation measurement, instrumental to probing the quantum properties of light, typically requires multiple single photon detectors. Progress in single photon avalanche diode (SPAD) array technology highlights their potential as high-performance detector arrays for quantum imaging and photon number-resolving (PNR) experiments. Here, we demonstrate this potential by incorporating a novel on-chip SPAD array with 42% peak photon detection efficiency, low dark count rate and crosstalk probability of 0.14% per detection in a confocal microscope. This enables reliable measurements of second and third order photon correlations from a single quantum dot emitter. Our analysis overcomes the inter-detector optical crosstalk background even though it is over an order of magnitude larger than our faint signal. To showcase the vast application space of such an approach, we implement a recently introduced super-resolution imaging method, quantum image scanning microscopy (Q-ISM).

Single-photon avalanche diode imagers in biophotonics: review and outlook.

Single-photon avalanche diode (SPAD) arrays are solid-state detectors that offer imaging capabilities at the level of individual photons, with unparalleled photon counting and time-resolved performance. This fascinating technology has progressed at a very fast pace in the past 15 years, since its inception in standard CMOS technology in 2003. A host of architectures have been investigated, ranging from simpler implementations, based solely on off-chip data processing, to progressively "smarter" sensors including on-chip, or even pixel level, time-stamping and processing capabilities. As the technology has matured, a range of biophotonics applications have been explored, including (endoscopic) FLIM, (multibeam multiphoton) FLIM-FRET, SPIM-FCS, super-resolution microscopy, time-resolved Raman spectroscopy, NIROT and PET. We will review some representative sensors and their corresponding applications, including the most relevant challenges faced by chip designers and end-users. Finally, we will provide an outlook on the future of this fascinating technology.