Fiber Bundle Image Reconstruction Using Convolutional Neural Networks and Bundle Rotation in Endomicroscopy.

Fiber-bundle endomicroscopy has several recognized drawbacks, the most prominent being the honeycomb effect. We developed a multi-frame super-resolution algorithm exploiting bundle rotation to extract features and reconstruct underlying tissue. Simulated data was used with rotated fiber-bundle masks to create multi-frame stacks to train the model. Super-resolved images are numerically analyzed, which demonstrates that the algorithm can restore images with high quality. The mean structural similarity index measurement (SSIM) improved by a factor of 1.97 compared with linear interpolation. The model was trained using images taken from a single prostate slide, 1343 images were used for training, 336 for validation, and 420 for testing. The model had no prior information about the test images, adding to the robustness of the system. Image reconstruction was completed in 0.03 s for 256 × 256 images indicating future real-time performance is within reach. The combination of fiber bundle rotation and multi-frame image enhancement through machine learning has not been utilized before in an experimental setting but could provide a much-needed improvement to image resolution in practice.

Molecular detection of Gram-positive bacteria in the human lung through an optical fiber-based endoscope.

PURPOSE: The relentless rise in antimicrobial resistance is a major societal challenge and requires, as part of its solution, a better understanding of bacterial colonization and infection. To facilitate this, we developed a highly efficient no-wash red optical molecular imaging agent that enables the rapid, selective, and specific visualization of Gram-positive bacteria through a bespoke optical fiber-based delivery/imaging endoscopic device. METHODS: We rationally designed a no-wash, red, Gram-positive-specific molecular imaging agent (Merocy-Van) based on vancomycin and an environmental merocyanine dye. We demonstrated the specificity and utility of the imaging agent in escalating in vitro and ex vivo whole human lung models (n = 3), utilizing a bespoke fiber-based delivery and imaging device, coupled to a wide-field, two-color endomicroscopy system. RESULTS: The imaging agent (Merocy-Van) was specific to Gram-positive bacteria and enabled no-wash imaging of S. aureus within the alveolar space of whole ex vivo human lungs within 60 s of delivery into the field-of-view, using the novel imaging/delivery endomicroscopy device. CONCLUSION: This platform enables the rapid and specific detection of Gram-positive bacteria in the human lung.

Multifocal multiphoton volumetric imaging approach for high-speed time-resolved Förster resonance energy transfer imaging in vivo.

In this Letter, we will discuss the development of a multifocal multiphoton fluorescent lifetime imaging system where four individual fluorescent intensity and lifetime planes are acquired simultaneously, allowing us to obtain volumetric data without the need for sequential scanning at different axial depths. Using a phase-only spatial light modulator (SLM) with an appropriate algorithm to generate a holographic pattern, we project a beamlet array within a sample volume of a size, which can be preprogrammed by the user. We demonstrate the capabilities of the system to image live-cell interactions. While only four planes are shown, this technique can be rescaled to a large number of focal planes, enabling full 3D acquisition and reconstruction.

Bimodal fluorogenic sensing of matrix proteolytic signatures in lung cancer.

Optical biosensing based on the activation of fluorescent reporters offers a powerful methodology for the real-time molecular interrogation of pathology. Here we report a first-in-class, bimodal fluorescent reporter strategy for the simultaneous and highly specific detection of two independent proteases (thrombin and matrix metalloproteases (MMPs)) pivotal in the fibroproliferative process surrounding lung cancer, based on a dual, multiplexing, peptide FRET system. This sophisticated synthetic smartprobe, with a molecular weight of 6 kDa, contains two independent fluorophores and quenchers that generate photonic signatures at two specific wavelengths upon activation by target enzymes within human lung cancer tissue.

New high-speed centre of mass method incorporating background subtraction for accurate determination of fluorescence lifetime.

We demonstrate an implementation of a centre-of-mass method (CMM) incorporating background subtraction for use in multifocal fluorescence lifetime imaging microscopy to accurately determine fluorescence lifetime in live cell imaging using the Megaframe camera. The inclusion of background subtraction solves one of the major issues associated with centre-of-mass approaches, namely the sensitivity of the algorithm to background signal. The algorithm, which is predominantly implemented in hardware, provides real-time lifetime output and allows the user to effectively condense large amounts of photon data. Instead of requiring the transfer of thousands of photon arrival times, the lifetime is simply represented by one value which allows the system to collect data up to limit of pulse pile-up without any limitations on data transfer rates. In order to evaluate the performance of this new CMM algorithm with existing techniques (i.e. rapid lifetime determination and Levenburg-Marquardt), we imaged live MCF-7 human breast carcinoma cells transiently transfected with FRET standards. We show that, it offers significant advantages in terms of lifetime accuracy and insensitivity to variability in dark count rate (DCR) between Megaframe camera pixels. Unlike other algorithms no prior knowledge of the expected lifetime is required to perform lifetime determination. The ability of this technique to provide real-time lifetime readout makes it extremely useful for a number of applications.

Real-time fluorescence lifetime actuation for cell sorting using a CMOS SPAD silicon photomultiplier.

Time-correlated single photon counting (TCSPC) is a fundamental fluorescence lifetime measurement technique offering high signal to noise ratio (SNR). However, its requirement for complex software algorithms for histogram processing restricts throughput in flow cytometers and prevents on-the-fly sorting of cells. We present a single-point digital silicon photomultiplier (SiPM) detector accomplishing real-time fluorescence lifetime-activated actuation targeting cell sorting applications in flow cytometry. The sensor also achieves burst-integrated fluorescence lifetime (BIFL) detection by TCSPC. The SiPM is a single-chip complementary metal-oxide-semiconductor (CMOS) sensor employing a 32×32 single-photon avalanche diode (SPAD) array and eight pairs of time-interleaved time to digital converters (TI-TDCs) with a 50 ps minimum timing resolution. The sensor's pile-up resistant embedded center of mass method (CMM) processor accomplishes low-latency measurement and thresholding of fluorescence lifetime. A digital control signal is generated with a 16.6 µs latency for cell sorter actuation allowing a maximum cell throughput of 60,000 cells per second and an error rate of 0.6%.

256 × 2 SPAD line sensor for time resolved fluorescence spectroscopy.

We present a CMOS chip 256 × 2 single photon avalanche diode (SPAD) line sensor, 23.78 µm pitch, 43.7% fill factor, custom designed for time resolved emission spectroscopy (TRES). Integrating time-to-digital converters (TDCs) implement on-chip mono-exponential fluorescence lifetime pre-calculation allowing timing of 65k photons/pixel at 200 Hz line rate at 40 ps resolution using centre-of-mass method (CMM). Per pixel time-correlated single-photon counting (TCSPC) histograms can also be generated with 320 ps bin resolution. We characterize performance in terms of dark count rate, instrument response function and lifetime uniformity for a set of fluorophores with lifetimes ranging from 4 ns to 6 ns. Lastly, we present fluorescence lifetime spectra of multicolor microspheres and skin autofluorescence acquired using a custom built spectrometer. In TCSPC mode, time-resolved spectra are acquired within 5 minutes whilst in CMM mode spectral lifetime signatures are acquired within 2 ms for fluorophore in cuvette and 200 ms for skin autofluorescence. We demonstrate CMOS line sensors to be a versatile tool for time-resolved fluorescence spectroscopy by providing parallelized and flexible spectral detection of fluorescence decay.

0.5 billion events per second time correlated single photon counting using CMOS SPAD arrays.

We present a digital architecture for fast acquisition of time correlated single photon counting (TCSPC) events from a 32×32 complementary metal oxide semiconductor (CMOS) single photon avalanche detector (SPAD) array (Megaframe) to the computer memory. Custom firmware was written to transmit event codes from 1024-TCSPC-enabled pixels for fast transfer of TCSPC events. Our 1024-channel TCSPC system is capable of acquiring up to 0.5×10(9) TCSPC events per second with 16 histogram bins spanning a 14 ns width. Other options include 320×10(6) TCSPC events per second with 256 histogram bins spanning either a 14 or 56 ns time window. We present a wide-field fluorescence microscopy setup demonstrating fast fluorescence lifetime data acquisition. To the best of our knowledge, this is the fastest direct TCSPC transfer from a single photon counting device to the computer to date.

Development of a doubly weighted Gerchberg-Saxton algorithm for use in multibeam imaging applications.

We report on the development of a doubly weighted Gerchberg-Saxton algorithm (DWGS) to enable generation of uniform beamlet arrays with a spatial light modulator (SLM) for use in multiphoton multifocal imaging applications. The algorithm incorporates the WGS algorithm as well as feedback of fluorescence signals from the sample measured with a single-photon avalanche diode (SPAD) detector array. This technique compensates for issues associated with nonuniform illumination onto the SLM, the effects due to aberrations and the variability in gain between detectors within the SPAD array to generate a uniformly illuminated multiphoton fluorescence image. We demonstrate the use of the DWGS with a number of beamlet array patterns to image muscle fibers of a 5-day-old fixed zebrafish larvae.

Time-resolved multifocal multiphoton microscope for high speed FRET imaging in vivo.

Imaging the spatiotemporal interaction of proteins in vivo is essential to understanding the complexities of biological systems. The highest accuracy monitoring of protein-protein interactions is achieved using Förster resonance energy transfer (FRET) measured by fluorescence lifetime imaging, with measurements taking minutes to acquire a single frame, limiting their use in dynamic live cell systems. We present a diffraction limited, massively parallel, time-resolved multifocal multiphoton microscope capable of producing fluorescence lifetime images with 55 ps time-resolution, giving improvements in acquisition speed of a factor of 64. We present demonstrations with FRET imaging in a model cell system and demonstrate in vivo FLIM using a GTPase biosensor in the zebrafish embryo.

In vivo laser speckle contrast imaging of microvascular blood perfusion using a chip-on-tip camera.

Laser speckle contrast imaging (LSCI) is an important non-invasive capability for real-time imaging for tissue-perfusion assessment. Yet, the size and weight of current clinical standard LSCI instrumentation restricts usage to mainly peripheral skin perfusion. Miniaturization of LSCI could enable hand-held instrumentation to image internal organ/tissue to produce accurate speckle-perfusion maps. We characterized a 1mm2 chip-on-tip camera for LSCI of blood perfusion in vivo and with a flow model. A dedicated optical setup was built to compare chip-on-tip camera to a high specification reference camera (GS3) for LSCI. We compared LSCI performance using a calibration standard and a flow phantom. Subsequently the camera assessed placenta perfusion in a small animal model. Lastly, a human study was conducted on the perfusion in fingertips of 13-volunteers. We demonstrate that the chip-on-tip camera can perform wide-field, in vivo, LSCI of tissue perfusion with the ability to measure physiological blood flow changes comparable with a standard reference camera.

Quantitative real-time imaging of intracellular FRET biosensor dynamics using rapid multi-beam confocal FLIM.

Fluorescence lifetime imaging (FLIM) is a quantitative, intensity-independent microscopical method for measurement of diverse biochemical and physical properties in cell biology. It is a highly effective method for measurements of Förster resonance energy transfer (FRET), and for quantification of protein-protein interactions in cells. Time-domain FLIM-FRET measurements of these dynamic interactions are particularly challenging, since the technique requires excellent photon statistics to derive experimental parameters from the complex decay kinetics often observed from fluorophores in living cells. Here we present a new time-domain multi-confocal FLIM instrument with an array of 64 visible beamlets to achieve parallelised excitation and detection with average excitation powers of ~ 1-2 µW per beamlet. We exemplify this instrument with up to 0.5 frames per second time-lapse FLIM measurements of cAMP levels using an Epac-based fluorescent biosensor in live HeLa cells with nanometer spatial and picosecond temporal resolution. We demonstrate the use of time-dependent phasor plots to determine parameterisation for multi-exponential decay fitting to monitor the fractional contribution of the activated conformation of the biosensor. Our parallelised confocal approach avoids having to compromise on speed, noise, accuracy in lifetime measurements and provides powerful means to quantify biochemical dynamics in living cells.

High-speed dual color fluorescence lifetime endomicroscopy for highly-multiplexed pulmonary diagnostic applications and detection of labeled bacteria.

We present a dual-color laser scanning endomicroscope capable of fluorescence lifetime endomicroscopy at one frame per second (FPS). The scanning system uses a coherent imaging fiber with 30,000 cores. High-speed lifetime imaging is achieved by distributing the signal over an array of 1024 parallel single-photon avalanche diode detectors (SPADs), minimizing detection dead-time maximizing the number of photons detected per excitation pulse without photon pile-up to achieve the high frame rate. This also enables dual color fluorescence imaging by temporally shifting the dual excitation lasers, with respect to each other, to separate the two spectrally distinct fluorescent decays in time. Combining the temporal encoding, to provide spectral separation, with lifetime measurements we show a one FPS, multi-channel endomicroscopy platform for clinical applications and diagnosis. We demonstrate the potential of the system by imaging SmartProbe labeled bacteria in ex vivo samples of human lung using lifetime to differentiate bacterial fluorescence from the strong background lung autofluorescence which was used to provide structural information.

Low-cost high sensitivity pulsed endomicroscopy to visualize tricolor optical signatures.

A highly sensitive, modular three-color fluorescence endomicroscopy imaging platform spanning the visible to near-infrared (NIR) range is demonstrated. Light-emitting diodes (LEDs) were sequentially pulsed along with the camera acquisition to provide up to 20 frames per second (fps) three-color imaging performance or 60 fps single color imaging. The system was characterized for bacterial and cellular molecular imaging in ex vivo human lung tissue and for bacterial and indocyanine green imaging in ex vivo perfused sheep lungs. A practical method to reduce background tissue autofluorescence is also proposed. The platform was clinically translated into six patients with pulmonary disease to delineate healthy, cancerous, and fibrotic tissue autofluorescent structures. The instrument is the most broadband clinical endomicroscopy system developed to date (covering visible to the NIR, 500 to 900 nm) and demonstrates significant potential for future clinical utility due to its low cost and modular capability to suit a wide variety of molecular imaging applications.

Characterization of a parallel-beam CCD optical-CT apparatus for 3D radiation dosimetry.

3D measurement of optical attenuation is of interest in a variety of fields of biomedical importance, including spectrophotometry, optical projection tomography (OPT) and analysis of 3D radiation dosimeters. Accurate, precise and economical 3D measurements of optical density (OD) are a crucial step in enabling 3D radiation dosimeters to enter wider use in clinics. Polymer gels and Fricke gels, as well as dosimeters not based around gels, have been characterized for 3D dosimetry over the last two decades. A separate problem is the verification of the best readout method. A number of different imaging modalities (magnetic resonance imaging (MRI), optical CT, x-ray CT and ultrasound) have been suggested for the readout of information from 3D dosimeters. To date only MRI and laser-based optical CT have been characterized in detail. This paper describes some initial steps we have taken in establishing charge coupled device (CCD)-based optical CT as a viable alternative to MRI for readout of 3D radiation dosimeters. The main advantage of CCD-based optical CT over traditional laser-based optical CT is a speed increase of at least an order of magnitude, while the simplicity of its architecture would lend itself to cheaper implementation than both MRI and laser-based optical CT if the camera itself were inexpensive enough. Specifically, we study the following aspects of optical metrology, using high quality test targets: (i) calibration and quality of absorbance measurements and the camera requirements for 3D dosimetry; (ii) the modulation transfer function (MTF) of individual projections; (iii) signal-to-noise ratio (SNR) in the projection and reconstruction domains; (iv) distortion in the projection domain, depth-of-field (DOF) and telecentricity. The principal results for our current apparatus are as follows: (i) SNR of optical absorbance in projections is better than 120:1 for uniform phantoms in absorbance range 0.3 to 1.6 (and better than 200:1 for absorbances 1.0 to 3.5 with the test target and a novel absorbance range extension method), (ii) the spatial resolution is shown to be at worst 0.5 mm (and often better than this) with an associated DOF of 8 cm, (iii) the SNR of uniform phantoms in reconstruction domain is above 80:1 (one standard deviation) over an absorbance dynamic range of 0.3 to 1.6, (iv) the apparatus is telecentric and without distortion. Finally, a sample scan and reconstruction of a scan of a PRESAGE dosimeter are shown, demonstrating the capabilities of the apparatus.

Fast laser scanning optical-CT apparatus for 3D radiation dosimetry.

Optical computed tomography (optical-CT) of 3D radiation dosimeters is a promising avenue for delivering an economic and reliable quality control of radiotherapy treatments such as intensity modulated radiotherapy, brachytherapy and stereotactic radiosurgery. The main problems in transferring 3D dosimeters to clinical setting have been in (1) the complexity of manufacture and behaviour of 3D dosimeters and (2) time-consuming readout and analysis of 3D dosimeters. This paper addresses the readout problem by showing that fast (20 min tomography scan), precise (projection absorbance signal-to-noise ratio is greater than 100:1 across the absorbance range 0.2 to 1.5) and accurate (good linearity in the calibration curve) measurements are possible using a novel method of optically scanning a laser beam across the 3D dosimeter.

Multiplexed single-mode wavelength-to-time mapping of multimode light.

When an optical pulse propagates along an optical fibre, different wavelengths travel at different group velocities. As a result, wavelength information is converted into arrival-time information, a process known as wavelength-to-time mapping. This phenomenon is most cleanly observed using a single-mode fibre transmission line, where spatial mode dispersion is not present, but the use of such fibres restricts possible applications. Here we demonstrate that photonic lanterns based on tapered single-mode multicore fibres provide an efficient way to couple multimode light to an array of single-photon avalanche detectors, each of which has its own time-to-digital converter for time-correlated single-photon counting. Exploiting this capability, we demonstrate the multiplexed single-mode wavelength-to-time mapping of multimode light using a multicore fibre photonic lantern with 121 single-mode cores, coupled to 121 detectors on a 32 × 32 detector array. This work paves the way to efficient multimode wavelength-to-time mapping systems with the spectral performance of single-mode systems.

Focusing optics of a parallel beam CCD optical tomography apparatus for 3D radiation gel dosimetry.

Optical tomography of gel dosimeters is a promising and cost-effective avenue for quality control of radiotherapy treatments such as intensity-modulated radiotherapy (IMRT). Systems based on a laser coupled to a photodiode have so far shown the best results within the context of optical scanning of radiosensitive gels, but are very slow ( approximately 9 min per slice) and poorly suited to measurements that require many slices. Here, we describe a fast, three-dimensional (3D) optical computed tomography (optical-CT) apparatus, based on a broad, collimated beam, obtained from a high power LED and detected by a charged coupled detector (CCD). The main advantages of such a system are (i) an acquisition speed approximately two orders of magnitude higher than a laser-based system when 3D data are required, and (ii) a greater simplicity of design. This paper advances our previous work by introducing a new design of focusing optics, which take information from a suitably positioned focal plane and project an image onto the CCD. An analysis of the ray optics is presented, which explains the roles of telecentricity, focusing, acceptance angle and depth-of-field (DOF) in the formation of projections. A discussion of the approximation involved in measuring the line integrals required for filtered backprojection reconstruction is given. Experimental results demonstrate (i) the effect on projections of changing the position of the focal plane of the apparatus, (ii) how to measure the acceptance angle of the optics, and (iii) the ability of the new scanner to image both absorbing and scattering gel phantoms. The quality of reconstructed images is very promising and suggests that the new apparatus may be useful in a clinical setting for fast and accurate 3D dosimetry.

Electrophysiological and Anatomical Correlates of Spinal Cord Optical Coherence Tomography.

Despite the continuous improvement in medical imaging technology, visualizing the spinal cord poses severe problems due to structural or incidental causes, such as small access space and motion artifacts. In addition, positional guidance on the spinal cord is not commonly available during surgery, with the exception of neuronavigation techniques based on static pre-surgical data and of radiation-based methods, such as fluoroscopy. A fast, bedside, intraoperative real-time imaging, particularly necessary during the positioning of endoscopic probes or tools, is an unsolved issue. The objective of our work, performed on experimental rats, is to demonstrate potential intraoperative spinal cord imaging and probe guidance by optical coherence tomography (OCT). Concurrently, we aimed to demonstrate that the electromagnetic OCT irradiation exerted no particular effect at the neuronal and synaptic levels. OCT is a user-friendly, low-cost and endoscopy-compatible photonics-based imaging technique. In particular, by using a Fourier-domain OCT imager, operating at 850 nm wavelength and scanning transversally with respect to the spinal cord, we have been able to: 1) accurately image tissue structures in an animal model (muscle, spine bone, cerebro-spinal fluid, dura mater and spinal cord), and 2) identify the position of a recording microelectrode approaching and inserting into the cord tissue 3) check that the infrared radiation has no actual effect on the electrophysiological activity of spinal neurons. The technique, potentially extendable to full three-dimensional image reconstruction, shows prospective further application not only in endoscopic intraoperative analyses and for probe insertion guidance, but also in emergency and adverse situations (e.g. after trauma) for damage recognition, diagnosis and fast image-guided intervention.

Two-color widefield fluorescence microendoscopy enables multiplexed molecular imaging in the alveolar space of human lung tissue.

We demonstrate a fast two-color widefield fluorescence microendoscopy system capable of simultaneously detecting several disease targets in intact human ex vivo lung tissue. We characterize the system for light throughput from the excitation light emitting diodes, fluorescence collection efficiency, and chromatic focal shifts. We demonstrate the effectiveness of the instrument by imaging bacteria (Pseudomonas aeruginosa) in ex vivo human lung tissue. We describe a mechanism of bacterial detection through the fiber bundle that uses blinking effects of bacteria as they move in front of the fiber core providing detection of objects smaller than the fiber core and cladding (∼3 µm ∼3 µm ). This effectively increases the measured spatial resolution of 4 µm 4 µm . We show simultaneous imaging of neutrophils, monocytes, and fungus (Aspergillus fumigatus) in ex vivo human lung tissue. The instrument has 10 nM and 50 nM sensitivity for fluorescein and Cy5 solutions, respectively. Lung tissue autofluorescence remains visible at up to 200 fps camera acquisition rate. The optical system lends itself to clinical translation due to high-fluorescence sensitivity, simplicity, and the ability to multiplex several pathological molecular imaging targets simultaneously.