LinoSPAD2: an FPGA-based, hardware-reconfigurable 512×1 single-photon camera system.

We report on LinoSPAD2, a single-photon camera system, comprising a 512×1 single-photon avalanche diode (SPAD) front-end and one or two FPGA-based back-ends. Digital signals generated by the SPADs are processed by the FPGA in real time, whereas the FPGA offers full reconfigurability at a very high level of granularity both in time and space domains. The LinoSPAD2 camera system can process 512 SPADs simultaneously through 256 channels, duplicated on each FPGA-based back-end, with a bank of 64 time-to-digital converters (TDCs) operating at 133 MSa/s, whereas each TDC has a time resolution of 20 ps (LSB). To the best of our knowledge, LinoSPAD2 is the first fully reconfigurable SPAD camera system of large format. The SPAD front-end features a pitch of 26.2 µm, a native fill factor of 25.1%, and a microlens array achieving 2.3× concentration factor. At room temperature, the median dark count rate (DCR) is 80 cps at 7 V excess bias, the peak photon detection probability (PDP) is 53% at 520 nm wavelength, and the single-photon timing resolution (SPTR) is 50 ps FWHM. The instrument response function (IRF) is around 100 ps FWHM at system level. The LinoSPAD2 camera system is suitable for numerous applications, including LiDAR imaging, heralded spectroscopy, compressive Raman sensing, and other computational imaging techniques.

Challenges and prospects for multi-chip microlens imprints on front-side illuminated SPAD imagers.

The overall sensitivity of frontside-illuminated, silicon single-photon avalanche diode (SPAD) arrays has often suffered from fill factor limitations. The fill factor loss can however be recovered by employing microlenses, whereby the challenges specific to SPAD arrays are represented by large pixel pitch (> 10 µm), low native fill factor (as low as ∼10%), and large size (up to 10 mm). In this work we report on the implementation of refractive microlenses by means of photoresist masters, used to fabricate molds for imprints of UV curable hybrid polymers deposited on SPAD arrays. Replications were successfully carried out for the first time, to the best of our knowledge, at wafer reticle level on different designs in the same technology and on single large SPAD arrays with very thin residual layers (∼10 µm), as needed for better efficiency at higher numerical aperture (NA > 0.25). In general, concentration factors within 15-20% of the simulation results were obtained for the smaller arrays (32×32 and 512×1), achieving for example an effective fill factor of 75.6-83.2% for a 28.5 µm pixel pitch with a native fill factor of 28%. A concentration factor up to 4.2 was measured on large 512×512 arrays with a pixel pitch of 16.38 µm and a native fill factor of 10.5%, whereas improved simulation tools could give a better estimate of the actual concentration factor. Spectral measurements were also carried out, resulting in good and uniform transmission in the visible and NIR.

Single-photon avalanche diode fabricated in standard 55†nm bipolar-CMOS-DMOS technology with sub-20†V breakdown voltage.

This paper presents a single-photon avalanche diode (SPAD) in 55 nm bipolar-CMOS-DMOS (BCD) technology. In order to realize a SPAD having sub-20 V breakdown voltage for mobile applications while preventing high tunneling noise, a high-voltage N-well available in BCD is utilized to implement the avalanche multiplication region. The resulting SPAD has a breakdown voltage of 18.4 V while achieving an excellent dark count rate of 4.4 cps/µm2 at the excess bias voltage of 7 V in spite of the advanced technology node. At the same time, the device achieves a high peak photon detection probability (PDP) of 70.1% at 450 nm thanks to the high and uniform E-field. Its PDP values at 850 and 940 nm, wavelengths of interest for 3D ranging applications reach 7.2 and 3.1%, respectively, with the use of deep N-well. The timing jitter of the SPAD, full width at half maximum (FWHM), is 91 ps at 850 nm. It is expected that the presented SPAD enables cost-effective time-of-flight and LiDAR sensors with the advanced standard technology for many mobile applications.

Heralded Spectroscopy Reveals Exciton-Exciton Correlations in Single Colloidal Quantum Dots.

Multiply excited states in semiconductor quantum dots feature intriguing physics and play a crucial role in nanocrystal-based technologies. While photoluminescence provides a natural probe to investigate these states, room-temperature single-particle spectroscopy of their emission has proved elusive due to the temporal and spectral overlap with emission from the singly excited and charged states. Here, we introduce biexciton heralded spectroscopy enabled by a single-photon avalanche diode array based spectrometer. This allows us to directly observe biexciton-exciton emission cascades and measure the biexciton binding energy of single quantum dots at room temperature, even though it is well below the scale of thermal broadening and spectral diffusion. Furthermore, we uncover correlations hitherto masked in ensembles of the biexciton binding energy with both charge-carrier confinement and fluctuations of the local electrostatic potential. Heralded spectroscopy has the potential of greatly extending our understanding of charge-carrier dynamics in multielectron systems and of parallelization of quantum optics protocols.

Theoretical minimum uncertainty of single-molecule localizations using a single-photon avalanche diode array.

Single-photon avalanche diode (SPAD) arrays can be used for single-molecule localization microscopy (SMLM) because of their high frame rate and lack of readout noise. SPAD arrays have a binary frame output, which means photon arrivals should be described as a binomial process rather than a Poissonian process. Consequentially, the theoretical minimum uncertainty of the localizations is not accurately predicted by the Poissonian Cramér-Rao lower bound (CRLB). Here, we derive a binomial CRLB and benchmark it using simulated and experimental data. We show that if the expected photon count is larger than one for all pixels within one standard deviation of a Gaussian point spread function, the binomial CRLB gives a 46% higher theoretical uncertainty than the Poissonian CRLB. For typical SMLM photon fluxes, where no saturation occurs, the binomial CRLB predicts the same uncertainty as the Poissonian CRLB. Therefore, the binomial CRLB can be used to predict and benchmark localization uncertainty for SMLM with SPAD arrays for all practical emitter intensities.

Quantum correlation measurement with single photon avalanche diode arrays.

Temporal photon correlation measurement, instrumental to probing the quantum properties of light, typically requires multiple single photon detectors. Progress in single photon avalanche diode (SPAD) array technology highlights their potential as high-performance detector arrays for quantum imaging and photon number-resolving (PNR) experiments. Here, we demonstrate this potential by incorporating a novel on-chip SPAD array with 42% peak photon detection efficiency, low dark count rate and crosstalk probability of 0.14% per detection in a confocal microscope. This enables reliable measurements of second and third order photon correlations from a single quantum dot emitter. Our analysis overcomes the inter-detector optical crosstalk background even though it is over an order of magnitude larger than our faint signal. To showcase the vast application space of such an approach, we implement a recently introduced super-resolution imaging method, quantum image scanning microscopy (Q-ISM).

A 512×512 SPAD Image Sensor with Integrated Gating for Widefield FLIM.

We report on SwissSPAD2, an image sensor with 512×512 photon-counting pixels, each comprising a single-photon avalanche diode (SPAD), a 1-bit memory, and a gating mechanism capable of turning the SPAD on and off, with a skew of 250ps and 344ps, respectively, for a minimum duration of 5.75ns. The sensor is designed to achieve a frame rate of up to 97,700 binary frames per second and sub-40ps gate shifts. By synchronizing it with a pulsed laser and using multiple successive overlapping gates, one can reconstruct a molecule's fluorescent response with picosecond temporal resolution. Thanks to the sensor's number of pixels (the largest to date) and the fully integrated gated operation, SwissSPAD2 enables widefield FLIM with an all-solid-state solution and at relatively high frame rates. This was demonstrated with preliminary results on organic dyes and semiconductor quantum dots using both decay fitting and phasor analysis. Furthermore, pixels with an exceptionally low dark count rate and high photon detection probability enable uniform and high quality imaging of biologically relevant fluorescent samples stained with multiple dyes. While future versions will feature the addition of microlenses and optimize firmware speed, our results open the way to low-cost alternatives to commercially available scientific time-resolved imagers.

Widefield High Frame Rate Single-Photon SPAD Imagers for SPIM-FCS.

Photon-counting sensors based on standard complementary metal-oxide-semiconductor single-photon avalanche diodes (SPADs) represent an emerging class of imagers that enable the counting and/or timing of single photons at zero readout noise (better than high-speed electron-multiplying charge-coupling devices) and over large arrays. They have seen substantial progress over the last 15 years, increasing their spatial resolution, timing accuracy, and sensitivity while reducing spurious signals such as afterpulsing and dark counts. They are increasingly being applied for time-resolved applications with the added advantage of enabling real-time options such as autocorrelation. We report in this study on the use of such a state-of-the-art 512 × 128 SPAD array, capable of a time resolution of 10-5-10-6 s for full frames while retaining acceptable photosensitivity thanks to the use of dedicated microlenses, in a selective plane illumination-fluorescence correlation spectroscopy setup. The latter allows us to perform thousands of fluorescence-correlation spectroscopy measurements simultaneously in a two-dimensional slice of the sample. This high-speed SPAD imager enables the measurement of molecular motion of small fluorescent particles such as single chemical dye molecules. Inhomogeneities in the molecular detection efficiency were compensated for by means of a global fit of the auto- and cross-correlation curves, which also made a calibration-free measurement of various samples possible. The afterpulsing effect could also be mitigated, making the measurement of the diffusion of Alexa-488 possible, and the overall result quality was further improved by spatial binning. The particle concentrations in the focus tend to be overestimated by a factor of 1.7 compared to a confocal setup; a calibration is thus required if absolute concentrations need to be measured. The first high-speed selective plane illumination-fluorescence correlation spectroscopy in vivo measurements to our knowledge were also recorded: although two-component fit models could not be employed because of noise, the diffusion of eGFP oligomers in HeLa cells could be measured. Sensitivity and noise will be further improved in the next generation of SPAD-based widefield sensors, which are currently under testing.

Dynamic range extension for photon counting arrays.

Confocal microscopes use photomultiplier tubes and hybrid detectors due to their large dynamic range, which typically exceeds the one of single-photon avalanche diodes (SPADs). The latter, due to their photon counting operation, are usually limited to an output count rate to 1/Tdead. In this paper, we present a thorough analysis, which can actually be applied to any photon counting detector, on how to extend the SPAD dynamic range by exploiting the nonlinear photon response at high count rates and for different recharge mechanisms. We applied passive, active event-driven and clock-driven (i.e. clocked, following quanta image sensor response) recharge directly to the SPADs. The photon response, photon count standard deviation, signal-to-noise ratio and dynamic range were measured and compared to models. Measurements were performed with a CMOS SPAD array targeted for image scanning microscopy, featuring best-in-class 11 V excess bias, 55% peak photon detection probability at 520 nm and >40% from 440 to 640 nm. The array features an extremely low median dark count rate below 0.05 cps/µm2 at 9 V of excess bias and 0°C. We show that active event-driven recharge provides ×75 dynamic range extension and offers novel ways for high dynamic range imaging. When compared to the clock-driven recharge and the quanta image sensor approach, the dynamic range is extended by a factor of ×12.7-26.4. Additionally, for the first time, we evaluate the influence of clock-driven recharge on the SPAD afterpulsing.

Automatic hand phantom map generation and detection using decomposition support vector machines.

BACKGROUND: There is a need for providing sensory feedback for myoelectric prosthesis users. Providing tactile feedback can improve object manipulation abilities, enhance the perceptual embodiment of myoelectric prostheses and help reduce phantom limb pain. Many amputees have referred sensation from their missing hand on their residual limbs (phantom maps). This skin area can serve as a target for providing amputees with non-invasive tactile sensory feedback. One of the challenges of providing sensory feedback on the phantom map is to define the accurate boundary of each phantom digit because the phantom map distribution varies from person to person. METHODS: In this paper, automatic phantom map detection methods based on four decomposition support vector machine algorithms and three sampling methods are proposed, complemented by fuzzy logic and active learning strategies. The algorithms and methods are tested on two databases: the first one includes 400 generated phantom maps, whereby the phantom map generation algorithm was based on our observation of the phantom maps to ensure smooth phantom digit edges, variety, and representativeness. The second database includes five reported phantom map images and transformations thereof. The accuracy and training/ classification time of each algorithm using a dense stimulation array (with 100 [Formula: see text] 100 actuators) and two coarse stimulation arrays (with 3 [Formula: see text] 5 and 4 [Formula: see text] 6 actuators) are presented and compared. RESULTS: Both generated and reported phantom map images share the same trends. Majority-pooling sampling effectively increases the training size, albeit introducing some noise, and thus produces the smallest error rates among the three proposed sampling methods. For different decomposition architectures, one-vs-one reduces unclassified regions and in general has higher classification accuracy than the other architectures. By introducing fuzzy logic to bias the penalty parameter, the influence of pooling-induced noise is reduced. Moreover, active learning with different strategies was also tested and shown to improve the accuracy by introducing more representative training samples. Overall, dense arrays employing one-vs-one fuzzy support vector machines with majority-pooling sampling have the smallest average absolute error rate (8.78% for generated phantom maps and 11.5% for reported and transformed phantom map images). The detection accuracy of coarse arrays was found to be significantly lower than for dense array. CONCLUSIONS: The results demonstrate the effectiveness of support vector machines using a dense array in detecting refined phantom map shapes, whereas coarse arrays are unsuitable for this task. We therefore propose a two-step approach, using first a non-wearable dense array to detect an accurate phantom map shape, then to apply a wearable coarse stimulation array customized according to the detection results. The proposed methodology can be used as a tool for helping haptic feedback designers and for tracking the evolvement of phantom maps.

Photon-Counting Arrays for Time-Resolved Imaging.

The paper presents a camera comprising 512 × 128 pixels capable of single-photon detection and gating with a maximum frame rate of 156 kfps. The photon capture is performed through a gated single-photon avalanche diode that generates a digital pulse upon photon detection and through a digital one-bit counter. Gray levels are obtained through multiple counting and accumulation, while time-resolved imaging is achieved through a 4-ns gating window controlled with subnanosecond accuracy by a field-programmable gate array. The sensor, which is equipped with microlenses to enhance its effective fill factor, was electro-optically characterized in terms of sensitivity and uniformity. Several examples of capture of fast events are shown to demonstrate the suitability of the approach.

Architecture and applications of a high resolution gated SPAD image sensor.

We present the architecture and three applications of the largest resolution image sensor based on single-photon avalanche diodes (SPADs) published to date. The sensor, fabricated in a high-voltage CMOS process, has a resolution of 512 × 128 pixels and a pitch of 24 µm. The fill-factor of 5% can be increased to 30% with the use of microlenses. For precise control of the exposure and for time-resolved imaging, we use fast global gating signals to define exposure windows as small as 4 ns. The uniformity of the gate edges location is ∼140 ps (FWHM) over the whole array, while in-pixel digital counting enables frame rates as high as 156 kfps. Currently, our camera is used as a highly sensitive sensor with high temporal resolution, for applications ranging from fluorescence lifetime measurements to fluorescence correlation spectroscopy and generation of true random numbers.

Coupling a recurrent neural network to SPAD TCSPC systems for real-time fluorescence lifetime imaging.

Fluorescence lifetime imaging (FLI) has been receiving increased attention in recent years as a powerful diagnostic technique in biological and medical research. However, existing FLI systems often suffer from a tradeoff between processing speed, accuracy, and robustness. Inspired by the concept of Edge Artificial Intelligence (Edge AI), we propose a robust approach that enables fast FLI with no degradation of accuracy. This approach couples a recurrent neural network (RNN), which is trained to estimate the fluorescence lifetime directly from raw timestamps without building histograms, to SPAD TCSPC systems, thereby drastically reducing transfer data volumes and hardware resource utilization, and enabling real-time FLI acquisition. We train two variants of the RNN on a synthetic dataset and compare the results to those obtained using center-of-mass method (CMM) and least squares fitting (LS fitting). Results demonstrate that two RNN variants, gated recurrent unit (GRU) and long short-term memory (LSTM), are comparable to CMM and LS fitting in terms of accuracy, while outperforming them in the presence of background noise by a large margin. To explore the ultimate limits of the approach, we derive the Cramer-Rao lower bound of the measurement, showing that RNN yields lifetime estimations with near-optimal precision. To demonstrate real-time operation, we build a FLI microscope based on an existing SPAD TCSPC system comprising a 32[Formula: see text]32 SPAD sensor named Piccolo. Four quantized GRU cores, capable of processing up to 4 million photons per second, are deployed on the Xilinx Kintex-7 FPGA that controls the Piccolo. Powered by the GRU, the FLI setup can retrieve real-time fluorescence lifetime images at up to 10 frames per second. The proposed FLI system is promising and ideally suited for biomedical applications, including biological imaging, biomedical diagnostics, and fluorescence-assisted surgery, etc.

Subsurface fluorescence time-of-flight imaging using a large-format single-photon avalanche diode sensor for tumor depth assessment.

Significance: Fluorescence guidance is used clinically by surgeons to visualize anatomical and/or physiological phenomena in the surgical field that are difficult or impossible to detect by the naked eye. Such phenomena include tissue perfusion or molecular phenotypic information about the disease being resected. Conventional fluorescence-guided surgery relies on long, microsecond scale laser pulses to excite fluorescent probes. However, this technique only provides two-dimensional information; crucial depth information, such as the location of malignancy below the tissue surface, is not provided. Aim: We developed a depth sensing imaging technique using light detection and ranging (LiDAR) time-of-flight (TOF) technology to sense the depth of target tissue while overcoming the influence of tissue optical properties and fluorescent probe concentration. Approach: The technology is based on a large-format (512×512 pixel), binary, gated, single-photon avalanche diode (SPAD) sensor with an 18 ps time-gate step, synchronized with a picosecond pulsed laser. The fast response of the sensor was developed and tested for its ability to quantify fluorescent inclusions at depth and optical properties in tissue-like phantoms through analytical model fitting of the fast temporal remission data. Results: After calibration and algorithmic extraction of the data, the SPAD LiDAR technique allowed for sub-mm resolution depth sensing of fluorescent inclusions embedded in tissue-like phantoms, up to a maximum of 5 mm in depth. The approach provides robust depth sensing even in the presence of variable tissue optical properties and separates the effects of fluorescence depth from absorption and scattering variations. Conclusions: LiDAR TOF fluorescence imaging using an SPAD camera provides both fluorescence intensity images and the temporal profile of fluorescence, which can be used to determine the depth at which the signal is emitted over a wide field of view. The proposed tool enables fluorescence imaging at a higher depth in tissue and with higher spatial precision than standard, steady-state fluorescence imaging tools, such as intensity-based near-infrared fluorescence imaging, optical coherence tomography, Raman spectroscopy, or confocal microscopy. Integration of this technique into a standard surgical tool could enable rapid, more accurate estimation of resection boundaries, thereby improving the surgeon's efficacy and efficiency, and ultimately improving patient outcomes.

NIR Fluorescence lifetime macroscopic imaging with a novel time-gated SPAD camera.

SwissSPAD3 is the latest of a family of widefield time-gated SPAD imagers developed for fluorescence lifetime imaging (FLI) applications. Its distinctive features are (i) the ability to define shorter gates than its predecessors (width W < 1 ns), (ii) support for laser repetition rates up to at least 80 MHz and (iii) a dual-gate architecture providing an effective duty cycle of 100%. We present widefield macroscopic FLI measurements of short lifetime NIR dyes, analyzed using the phasor approach. The results are compared with those previously obtained with SwissSPAD2 and to theoretical predictions.

Massively parallel, real-time multispeckle diffuse correlation spectroscopy using a 500 × 500 SPAD camera.

Diffuse correlation spectroscopy (DCS) is a promising noninvasive technique for monitoring cerebral blood flow and measuring cortex functional activation tasks. Taking multiple parallel measurements has been shown to increase sensitivity, but is not easily scalable with discrete optical detectors. Here we show that with a large 500 × 500 SPAD array and an advanced FPGA design, we achieve an SNR gain of almost 500 over single-pixel mDCS performance. The system can also be reconfigured to sacrifice SNR to decrease correlation bin width, with 400 ns resolution being demonstrated over 8000 pixels.

Correlated-photon imaging at 10 volumetric images per second.

The correlation properties of light provide an outstanding tool to overcome the limitations of traditional imaging techniques. A relevant case is represented by correlation plenoptic imaging (CPI), a quantum-inspired volumetric imaging protocol employing spatio-temporally correlated photons from either entangled or chaotic sources to address the main limitations of conventional light-field imaging, namely, the poor spatial resolution and the reduced change of perspective for 3D imaging. However, the application potential of high-resolution imaging modalities relying on photon correlations is limited, in practice, by the need to collect a large number of frames. This creates a gap, unacceptable for many relevant tasks, between the time performance of correlated-light imaging and that of traditional imaging methods. In this article, we address this issue by exploiting the photon number correlations intrinsic in chaotic light, combined with a cutting-edge ultrafast sensor made of a large array of single-photon avalanche diodes (SPADs). This combination of source and sensor is embedded within a novel single-lens CPI scheme enabling to acquire 10 volumetric images per second. Our results place correlated-photon imaging at a competitive edge and prove its potential in practical applications.

Light-field tomographic fluorescence lifetime imaging microscopy.

Fluorescence lifetime imaging microscopy (FLIM) is a powerful imaging technique that enables the visualization of biological samples at the molecular level by measuring the fluorescence decay rate of fluorescent probes. This provides critical information about molecular interactions, environmental changes, and localization within biological systems. However, creating high-resolution lifetime maps using conventional FLIM systems can be challenging, as it often requires extensive scanning that can significantly lengthen acquisition times. This issue is further compounded in three-dimensional (3D) imaging because it demands additional scanning along the depth axis. To tackle this challenge, we developed a novel computational imaging technique called light field tomographic FLIM (LIFT-FLIM). Our approach allows for the acquisition of volumetric fluorescence lifetime images in a highly data-efficient manner, significantly reducing the number of scanning steps required compared to conventional point-scanning or line-scanning FLIM imagers. Moreover, LIFT-FLIM enables the measurement of high-dimensional data using low-dimensional detectors, which are typically low-cost and feature a higher temporal bandwidth. We demonstrated LIFT-FLIM using a linear single-photon avalanche diode array on various biological systems, showcasing unparalleled single-photon detection sensitivity. Additionally, we expanded the functionality of our method to spectral FLIM and demonstrated its application in high-content multiplexed imaging of lung organoids. LIFT-FLIM has the potential to open up new avenues in both basic and translational biomedical research.

Highly sensitive single-molecule detection of macromolecule ion beams.

The analysis of proteins in the gas phase benefits from detectors that exhibit high efficiency and precise spatial resolution. Although modern secondary electron multipliers already address numerous analytical requirements, additional methods are desired for macromolecules at energies lower than currently used in post-acceleration detection. Previous studies have proven the sensitivity of superconducting detectors to high-energy particles in time-of-flight mass spectrometry. Here, we demonstrate that superconducting nanowire detectors are exceptionally well suited for quadrupole mass spectrometry and exhibit an outstanding quantum yield at low-impact energies. At energies as low as 100 eV, the sensitivity of these detectors surpasses conventional ion detectors by three orders of magnitude, and they offer the possibility to discriminate molecules by their impact energy and charge. We demonstrate three developments with these compact and sensitive devices, the recording of 2D ion beam profiles, photochemistry experiments in the gas phase, and advanced cryogenic electronics to pave the way toward highly integrated detectors.

In vitro and in vivo NIR fluorescence lifetime imaging with a time-gated SPAD camera.

Near-infrared (NIR) fluorescence lifetime imaging (FLI) provides a unique contrast mechanism to monitor biological parameters and molecular events in vivo. Single-photon avalanche diode (SPAD) cameras have been recently demonstrated in FLI microscopy (FLIM) applications, but their suitability for in vivo macroscopic FLI (MFLI) in deep tissues remains to be demonstrated. Herein, we report in vivo NIR MFLI measurement with SwissSPAD2, a large time-gated SPAD camera. We first benchmark its performance in well-controlled in vitro experiments, ranging from monitoring environmental effects on fluorescence lifetime, to quantifying Förster resonant energy transfer (FRET) between dyes. Next, we use it for in vivo studies of target-drug engagement in live and intact tumor xenografts using FRET. Information obtained with SwissSPAD2 was successfully compared to that obtained with a gated intensified charge-coupled device (ICCD) camera, using two different approaches. Our results demonstrate that SPAD cameras offer a powerful technology for in vivo preclinical applications in the NIR window.