Smartphone-based optical sectioning (SOS) microscopy with a telecentric design for fluorescence imaging.

We propose a smartphone-based optical sectioning (SOS) microscope based on the HiLo technique, with a single smartphone replacing a high-cost illumination source and a camera sensor. We built our SOS with off-the-shelf optical, mechanical cage systems with 3D-printed adapters to seamlessly integrate the smartphone with the SOS main body. The liquid light guide can be integrated with the adapter, guiding the smartphone's LED light to the digital mirror device (DMD) with neglectable loss. We used an electrically tuneable lens (ETL) instead of a mechanical translation stage to realise low-cost axial scanning. The ETL was conjugated to the objective lens's back pupil plane (BPP) to construct a telecentric design by a 4f configuration to maintain the lateral magnification for different axial positions. SOS has a 571.5 µm telecentric scanning range and an 11.7 µm axial resolution. The broadband smartphone LED torch can effectively excite fluorescent polystyrene (PS) beads. We successfully used SOS for high-contrast fluorescent PS beads imaging with different wavelengths and optical sectioning imaging of multilayer fluorescent PS beads. To our knowledge, the proposed SOS is the first smartphone-based HiLo optical sectioning microscopy (£1965), which can save around £7035 compared with a traditional HiLo system (£9000). It is a powerful tool for biomedical research in resource-limited areas.

High-performance reconstruction method combining total variation with a video denoiser for compressed ultrafast imaging.

Compressed ultrafast photography (CUP) is a novel two-dimensional (2D) imaging technique to capture ultrafast dynamic scenes. Effective image reconstruction is essential in CUP systems. However, existing reconstruction algorithms mostly rely on image priors and complex parameter spaces. Therefore, in general, they are time-consuming and result in poor imaging quality, which limits their practical applications. In this paper, we propose a novel reconstruction algorithm, to the best of our knowledge, named plug-in-plug-fast deep video denoising net-total variation (PnP-TV-FastDVDnet), which exploits an image's spatial features and correlation features in the temporal dimension. Therefore, it offers higher-quality images than those in previously reported methods. First, we built a forward mathematical model of the CUP, and the closed-form solution of the three suboptimization problems was derived according to plug-in and plug-out frames. Secondly, we used an advanced video denoising algorithm based on a neural network named FastDVDnet to solve the denoising problem. The peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) are improved on actual CUP data compared with traditional algorithms. On benchmark and real CUP datasets, the proposed method shows the comparable visual results while reducing the running time by 96% over state-of-the-art algorithms.

Fast Analysis of Time-Domain Fluorescence Lifetime Imaging via Extreme Learning Machine.

We present a fast and accurate analytical method for fluorescence lifetime imaging microscopy (FLIM), using the extreme learning machine (ELM). We used extensive metrics to evaluate ELM and existing algorithms. First, we compared these algorithms using synthetic datasets. The results indicate that ELM can obtain higher fidelity, even in low-photon conditions. Afterwards, we used ELM to retrieve lifetime components from human prostate cancer cells loaded with gold nanosensors, showing that ELM also outperforms the iterative fitting and non-fitting algorithms. By comparing ELM with a computational efficient neural network, ELM achieves comparable accuracy with less training and inference time. As there is no back-propagation process for ELM during the training phase, the training speed is much higher than existing neural network approaches. The proposed strategy is promising for edge computing with online training.

Simple and Robust Deep Learning Approach for Fast Fluorescence Lifetime Imaging.

Fluorescence lifetime imaging (FLIM) is a powerful tool that provides unique quantitative information for biomedical research. In this study, we propose a multi-layer-perceptron-based mixer (MLP-Mixer) deep learning (DL) algorithm named FLIM-MLP-Mixer for fast and robust FLIM analysis. The FLIM-MLP-Mixer has a simple network architecture yet a powerful learning ability from data. Compared with the traditional fitting and previously reported DL methods, the FLIM-MLP-Mixer shows superior performance in terms of accuracy and calculation speed, which has been validated using both synthetic and experimental data. All results indicate that our proposed method is well suited for accurately estimating lifetime parameters from measured fluorescence histograms, and it has great potential in various real-time FLIM applications.

Combining time of flight and photometric stereo imaging for 3D reconstruction of discontinuous scenes.

Time of flight and photometric stereo are two three-dimensional (3D) imaging techniques with complementary properties, where the former can achieve depth accuracy in discontinuous scenes, and the latter can reconstruct surfaces of objects with fine depth details and high spatial resolution. In this work, we demonstrate the surface reconstruction of complex 3D fields with discontinuity between objects by combining the two imaging methods. Using commercial LEDs, a single-photon avalanche diode camera, and a mobile phone device, high resolution of surface reconstruction is achieved with a RMS error of 6% for an object auto-selected from a scene imaged at a distance of 50 cm.

Theoretical investigations of a modified compressed ultrafast photography method suitable for single-shot fluorescence lifetime imaging.

A single-shot fluorescence lifetime imaging (FLIM) method based on the compressed ultrafast photography (CUP) is proposed, named space-restricted CUP (srCUP). srCUP is suitable for imaging objects moving slowly (<∼150/Mmm/s, M is the magnification of the objective lens) in the field of view with the intensity changing within nanoseconds in a measurement window around 10 ns. We used synthetic datasets to explore the performances of srCUP compared with CUP and TCUP (a variant of CUP). srCUP not only provides superior reconstruction performances, but its reconstruction speed is also twofold and threefold faster than CUP and TCUP, respectively. The lifetime determination performances were assessed by estimating lifetime components, amplitude- and intensity-weighted average lifetimes (τA and τI), with the reconstructed scenes using the least squares method based on a bi-exponential model. srCUP has the best accuracy and precision for lifetime determinations with a relative bias less than 7% and a coefficient of variation less than 7% for τA, and a relative bias less than 10% and a coefficient of variation less than 11% for τI.

Single-shot compressed ultrafast photography based on U-net network.

The compressive ultrafast photography (CUP) has achieved real-time femtosecond imaging based on the compressive-sensing methods. However, the reconstruction performance usually suffers from artifacts brought by strong noise, aberration, and distortion, which prevents its applications. We propose a deep compressive ultrafast photography (DeepCUP) method. Various numerical simulations have been demonstrated on both the MNIST and UCF-101 datasets and compared with other state-of-the-art algorithms. The result shows that our DeepCUP has a superior performance in both PSNR and SSIM compared to previous compressed-sensing methods. We also illustrate the outstanding performance of the proposed method under system errors and noise in comparison to other methods.

Multispectral time-of-flight imaging using light-emitting diodes.

Multispectral and 3-D imaging are useful for a wide variety of applications, adding valuable spectral and depth information for image analysis. Single-photon avalanche diode (SPAD) based imaging systems provide photon time-of-arrival information, and can be used for imaging with time-correlated single photon counting techniques. Here we demonstrate an LED based synchronised illumination system, where temporally structured light can be used to relate time-of-arrival to specific wavelengths, thus recovering reflectance information. Cross-correlation of the received multi-peak histogram with a reference measurement yields a time delay, allowing depth information to be determined with cm-scale resolution despite the long sequence of relatively wide (∼10 ns) pulses. Using commercial LEDs and a SPAD imaging array, multispectral 3-D imaging is demonstrated across 9 visible wavelength bands.

Single-shot time-gated fluorescence lifetime imaging using three-frame images.

Qualitative and quantitative measurements of complex flows demand for fast single-shot fluorescence lifetime imaging (FLI) technology with high precision. A method, single-shot time-gated fluorescence lifetime imaging using three-frame images (TFI-TGFLI), is presented. To our knowledge, it is the first work to combine a three-gate rapid lifetime determination (RLD) scheme and a four-channel framing camera to achieve this goal. Different from previously proposed two-gate RLD schemes, TFI-TGFLI can provide a wider lifetime range 0.6 ~ 13ns with reasonable precision. The performances of the proposed approach have been examined by both Monte-Carlo simulations and toluene seeded gas mixing jet diagnosis experiments. The measured average lifetimes of the whole excited areas agree well with the results obtained by the streak camera, and they are 7.6ns (N2 = 7L/min; O2 < 0.1L/min) and 2.6ns (N2 = 19L/min; O2 = 1L/min) with the standard deviations of 1.7ns and 0.8ns among the lifetime image pixels, respectively. The concentration distributions of the quenchers and fluorescent species were further analyzed, and they are consistent with the experimental settings.

Optimizing Laguerre expansion based deconvolution methods for analysing bi-exponential fluorescence lifetime images.

Fast deconvolution is an essential step to calibrate instrument responses in big fluorescence lifetime imaging microscopy (FLIM) image analysis. This paper examined a computationally effective least squares deconvolution method based on Laguerre expansion (LSD-LE), recently developed for clinical diagnosis applications, and proposed new criteria for selecting Laguerre basis functions (LBFs) without considering the mutual orthonormalities between LBFs. Compared with the previously reported LSD-LE, the improved LSD-LE allows to use a higher laser repetition rate, reducing the acquisition time per measurement. Moreover, we extended it, for the first time, to analyze bi-exponential fluorescence decays for more general FLIM-FRET applications. The proposed method was tested on both synthesized bi-exponential and realistic FLIM data for studying the endocytosis of gold nanorods in Hek293 cells. Compared with the previously reported constrained LSD-LE, it shows promising results.

Towards unsupervised fluorescence lifetime imaging using low dimensional variable projection.

Analyzing large fluorescence lifetime imaging (FLIM) data is becoming overwhelming; the latest FLIM systems easily produce massive amounts of data, making an efficient analysis more challenging than ever. In this paper we propose the combination of a custom-fit variable projection method, with a Laguerre expansion based deconvolution, to analyze bi-exponential data obtained from time-domain FLIM systems. Unlike nonlinear least squares methods, which require a suitable initial guess from an experienced researcher, the new method is free from manual interventions and hence can support automated analysis. Monte Carlo simulations are carried out on synthesized FLIM data to demonstrate the performance compared to other approaches. The performance is also illustrated on real-life FLIM data obtained from the study of autofluorescence of daisy pollen and the endocytosis of gold nanorods (GNRs) in living cells. In the latter, the fluorescence lifetimes of the GNRs are much shorter than the full width at half maximum of the instrument response function. Overall, our proposed method contains simple steps and shows great promise in realising automated FLIM analysis of large data sets.

New high-speed centre of mass method incorporating background subtraction for accurate determination of fluorescence lifetime.

We demonstrate an implementation of a centre-of-mass method (CMM) incorporating background subtraction for use in multifocal fluorescence lifetime imaging microscopy to accurately determine fluorescence lifetime in live cell imaging using the Megaframe camera. The inclusion of background subtraction solves one of the major issues associated with centre-of-mass approaches, namely the sensitivity of the algorithm to background signal. The algorithm, which is predominantly implemented in hardware, provides real-time lifetime output and allows the user to effectively condense large amounts of photon data. Instead of requiring the transfer of thousands of photon arrival times, the lifetime is simply represented by one value which allows the system to collect data up to limit of pulse pile-up without any limitations on data transfer rates. In order to evaluate the performance of this new CMM algorithm with existing techniques (i.e. rapid lifetime determination and Levenburg-Marquardt), we imaged live MCF-7 human breast carcinoma cells transiently transfected with FRET standards. We show that, it offers significant advantages in terms of lifetime accuracy and insensitivity to variability in dark count rate (DCR) between Megaframe camera pixels. Unlike other algorithms no prior knowledge of the expected lifetime is required to perform lifetime determination. The ability of this technique to provide real-time lifetime readout makes it extremely useful for a number of applications.

Fast bi-exponential fluorescence lifetime imaging analysis methods: publisher's note.

Table 1 of an earlier paper [Opt. Lett.40, 336 (2015)10.1364/OL.40.000336] contained an incorrect mathematical expression. The error is rectified here.

Artificial neural network approaches for fluorescence lifetime imaging techniques.

A novel high-speed fluorescence lifetime imaging (FLIM) analysis method based on artificial neural networks (ANN) has been proposed. In terms of image generation, the proposed ANN-FLIM method does not require iterative searching procedures or initial conditions, and it can generate lifetime images at least 180-fold faster than conventional least squares curve-fitting software tools. The advantages of ANN-FLIM were demonstrated on both synthesized and experimental data, showing that it has great potential to fuel current revolutions in rapid FLIM technologies.

Real-time fluorescence lifetime actuation for cell sorting using a CMOS SPAD silicon photomultiplier.

Time-correlated single photon counting (TCSPC) is a fundamental fluorescence lifetime measurement technique offering high signal to noise ratio (SNR). However, its requirement for complex software algorithms for histogram processing restricts throughput in flow cytometers and prevents on-the-fly sorting of cells. We present a single-point digital silicon photomultiplier (SiPM) detector accomplishing real-time fluorescence lifetime-activated actuation targeting cell sorting applications in flow cytometry. The sensor also achieves burst-integrated fluorescence lifetime (BIFL) detection by TCSPC. The SiPM is a single-chip complementary metal-oxide-semiconductor (CMOS) sensor employing a 32×32 single-photon avalanche diode (SPAD) array and eight pairs of time-interleaved time to digital converters (TI-TDCs) with a 50 ps minimum timing resolution. The sensor's pile-up resistant embedded center of mass method (CMM) processor accomplishes low-latency measurement and thresholding of fluorescence lifetime. A digital control signal is generated with a 16.6 µs latency for cell sorter actuation allowing a maximum cell throughput of 60,000 cells per second and an error rate of 0.6%.

Fast bi-exponential fluorescence lifetime imaging analysis methods.

A new hardware-friendly bi-exponential fluorescence lifetime imaging (FLIM) algorithm has been proposed. Compared to conventional FLIM software, the proposed algorithms are noniterative offering direct calculation of lifetimes and therefore suitable for real-time applications. They are applicable to single-channel or 2D multichannel time-correlated single-photon counting (TCSPC) systems. The proposed methods have been tested on both synthesized and realistic FLIM data, and we have compared their performances with other recently proposed nonfitting bi-exponential techniques showing promising applications in future massive solid-state TCSPC imagers.

Time-resolved multifocal multiphoton microscope for high speed FRET imaging in vivo.

Imaging the spatiotemporal interaction of proteins in vivo is essential to understanding the complexities of biological systems. The highest accuracy monitoring of protein-protein interactions is achieved using Förster resonance energy transfer (FRET) measured by fluorescence lifetime imaging, with measurements taking minutes to acquire a single frame, limiting their use in dynamic live cell systems. We present a diffraction limited, massively parallel, time-resolved multifocal multiphoton microscope capable of producing fluorescence lifetime images with 55 ps time-resolution, giving improvements in acquisition speed of a factor of 64. We present demonstrations with FRET imaging in a model cell system and demonstrate in vivo FLIM using a GTPase biosensor in the zebrafish embryo.

Quantification of blood flow index in diffuse correlation spectroscopy using a robust deep learning method.

Significance: Diffuse correlation spectroscopy (DCS) is a powerful, noninvasive optical technique for measuring blood flow. Traditionally the blood flow index (BFi) is derived through nonlinear least-square fitting the measured intensity autocorrelation function (ACF). However, the fitting process is computationally intensive, susceptible to measurement noise, and easily influenced by optical properties (absorption coefficient µa and reduced scattering coefficient µs') and scalp and skull thicknesses. Aim: We aim to develop a data-driven method that enables rapid and robust analysis of multiple-scattered light's temporal ACFs. Moreover, the proposed method can be applied to a range of source-detector distances instead of being limited to a specific source-detector distance. Approach: We present a deep learning architecture with one-dimensional convolution neural networks, called DCS neural network (DCS-NET), for BFi and coherent factor (ß) estimation. This DCS-NET was performed using simulated DCS data based on a three-layer brain model. We quantified the impact from physiologically relevant optical property variations, layer thicknesses, realistic noise levels, and multiple source-detector distances (5, 10, 15, 20, 25, and 30 mm) on BFi and ß estimations among DCS-NET, semi-infinite, and three-layer fitting models. Results: DCS-NET shows a much faster analysis speed, around 17,000-fold and 32-fold faster than the traditional three-layer and semi-infinite models, respectively. It offers higher intrinsic sensitivity to deep tissues compared with fitting methods. DCS-NET shows excellent anti-noise features and is less sensitive to variations of µa and µs' at a source-detector separation of 30 mm. Also, we have demonstrated that relative BFi (rBFi) can be extracted by DCS-NET with a much lower error of 8.35%. By contrast, the semi-infinite and three-layer fitting models result in significant errors in rBFi of 43.76% and 19.66%, respectively. Conclusions: DCS-NET can robustly quantify blood flow measurements at considerable source-detector distances, corresponding to much deeper biological tissues. It has excellent potential for hardware implementation, promising continuous real-time blood flow measurements.

Compact and robust deep learning architecture for fluorescence lifetime imaging and FPGA implementation.

This paper reports a bespoke adder-based deep learning network for time-domain fluorescence lifetime imaging (FLIM). By leveraging thel1-norm extraction method, we propose a 1D Fluorescence Lifetime AdderNet (FLAN) without multiplication-based convolutions to reduce the computational complexity. Further, we compressed fluorescence decays in temporal dimension using a log-scale merging technique to discard redundant temporal information derived as log-scaling FLAN (FLAN+LS). FLAN+LS achieves 0.11 and 0.23 compression ratios compared with FLAN and a conventional 1D convolutional neural network (1D CNN) while maintaining high accuracy in retrieving lifetimes. We extensively evaluated FLAN and FLAN+LS using synthetic and real data. A traditional fitting method and other non-fitting, high-accuracy algorithms were compared with our networks for synthetic data. Our networks attained a minor reconstruction error in different photon-count scenarios. For real data, we used fluorescent beads' data acquired by a confocal microscope to validate the effectiveness of real fluorophores, and our networks can differentiate beads with different lifetimes. Additionally, we implemented the network architecture on a field-programmable gate array (FPGA) with a post-quantization technique to shorten the bit-width, thereby improving computing efficiency. FLAN+LS on hardware achieves the highest computing efficiency compared to 1D CNN and FLAN. We also discussed the applicability of our network and hardware architecture for other time-resolved biomedical applications using photon-efficient, time-resolved sensors.

Time-domain fluorescence lifetime imaging techniques suitable for solid-state imaging sensor arrays.

We have successfully demonstrated video-rate CMOS single-photon avalanche diode (SPAD)-based cameras for fluorescence lifetime imaging microscopy (FLIM) by applying innovative FLIM algorithms. We also review and compare several time-domain techniques and solid-state FLIM systems, and adapt the proposed algorithms for massive CMOS SPAD-based arrays and hardware implementations. The theoretical error equations are derived and their performances are demonstrated on the data obtained from 0.13 µm CMOS SPAD arrays and the multiple-decay data obtained from scanning PMT systems. In vivo two photon fluorescence lifetime imaging data of FITC-albumin labeled vasculature of a P22 rat carcinosarcoma (BD9 rat window chamber) are used to test how different algorithms perform on bi-decay data. The proposed techniques are capable of producing lifetime images with enough contrast.