Assessment of open-field fluorescence guided surgery systems: implementing a standardized method for characterization and comparison.

Significance: Fluorescence guided surgery (FGS) has demonstrated improvements in decision making and patient outcomes for a wide range of surgical procedures. Not only can FGS systems provide a higher level of structural perfusion accuracy in tissue reconstruction cases but they can also serve for real-time functional characterization. Multiple FGS devices have been Food and Drug administration (FDA) cleared for use in open and laparoscopic surgery. Despite the rapid growth of the field, there has been a lack standardization methods. Aim: This work overviews commonalities inherent to optical imaging methods that can be exploited to produce such a standardization procedure. Furthermore, a system evaluation pipeline is proposed and executed through the use of photo-stable indocyanine green fluorescence phantoms. Five different FDA-approved open-field FGS systems are used and evaluated with the proposed method. Approach: The proposed pipeline encompasses the following characterization: (1) imaging spatial resolution and sharpness, (2) sensitivity and linearity, (3) imaging depth into tissue, (4) imaging system DOF, (5) uniformity of illumination, (6) spatial distortion, (7) signal to background ratio, (8) excitation bands, and (9) illumination wavelength and power. Results: The results highlight how such a standardization approach can be successfully implemented for inter-system comparisons as well as how to better understand essential features within each FGS setup. Conclusions: Despite clinical use being the end goal, a robust yet simple standardization pipeline before clinical trials, such as the one presented herein, should benefit regulatory agencies, manufacturers, and end-users to better assess basic performance and improvements to be made in next generation FGS systems.

Mapping estimates of vascular permeability with a clinical indocyanine green fluorescence imaging system in experimental pancreatic adenocarcinoma tumors.

Significance: Pancreatic cancer tumors are known to be avascular, but their neovascular capillaries are still chaotic leaky vessels. Capillary permeability could have significant value for therapy assessment, and its quantification might be possible with macroscopic imaging of indocyanine green (ICG) kinetics in tissue. Aim: The capacity of using standard fluorescence surgical systems for ICG kinetic imaging as a probe for capillary leakage was evaluated using a clinical surgical fluorescence imaging system, as interpreted through vascular permeability modeling. Approach: Xenograft pancreatic adenocarcinoma models were imaged in mice during bolus injection of ICG to capture the kinetics of uptake. Image analysis included ratiometric data, normalization, and match to theoretical modeling. Kinetic data were converted into the extraction fraction of the capillary leakage. Results: Pancreatic tumors were usually less fluorescent than the surrounding healthy tissues, but still the rate of tumor perfusion could be assessed to quantify capillary extraction. Model simulations showed that flow kinetics stabilized after about 1 min beyond the initial bolus injection and that the relative extraction fraction model estimates matched the experimental data of normalized uptake within the tissue. The kinetics in the time period of 1 to 2 min post-injection provided optimal differential data between AsPC1 and BxPC3 tumors, although high individual variation exists between tumors. Conclusions: ICG kinetic imaging during the initial leakage phase was diagnostic for quantitative vascular permeability within pancreatic tumors. Methods for autogain correction and normalized model-based interpretation allowed for quantification of extraction fraction and difference identification between tumor types in early timepoints.

Deep learning-based fusion of widefield diffuse optical tomography and micro-CT structural priors for accurate 3D reconstructions.

Widefield illumination and detection strategies leveraging structured light have enabled fast and robust probing of tissue properties over large surface areas and volumes. However, when applied to diffuse optical tomography (DOT) applications, they still require a time-consuming and expert-centric solving of an ill-posed inverse problem. Deep learning (DL) models have been recently proposed to facilitate this challenging step. Herein, we expand on a previously reported deep neural network (DNN) -based architecture (modified AUTOMAP - ModAM) for accurate and fast reconstructions of the absorption coefficient in 3D DOT based on a structured light illumination and detection scheme. Furthermore, we evaluate the improved performances when incorporating a micro-CT structural prior in the DNN-based workflow, named Z-AUTOMAP. This Z-AUTOMAP significantly improves the widefield imaging process's spatial resolution, especially in the transverse direction. The reported DL-based strategies are validated both in silico and in experimental phantom studies using spectral micro-CT priors. Overall, this is the first successful demonstration of micro-CT and DOT fusion using deep learning, greatly enhancing the prospect of rapid data-integration strategies, often demanded in challenging pre-clinical scenarios.

Roadmap on Deep Learning for Microscopy.

Through digital imaging, microscopy has evolved from primarily being a means for visual observation of life at the micro- and nano-scale, to a quantitative tool with ever-increasing resolution and throughput. Artificial intelligence, deep neural networks, and machine learning are all niche terms describing computational methods that have gained a pivotal role in microscopy-based research over the past decade. This Roadmap is written collectively by prominent researchers and encompasses selected aspects of how machine learning is applied to microscopy image data, with the aim of gaining scientific knowledge by improved image quality, automated detection, segmentation, classification and tracking of objects, and efficient merging of information from multiple imaging modalities. We aim to give the reader an overview of the key developments and an understanding of possibilities and limitations of machine learning for microscopy. It will be of interest to a wide cross-disciplinary audience in the physical sciences and life sciences.

Computational macroscopic lifetime imaging and concentration unmixing of autofluorescence.

Single-pixel computational imaging can leverage highly sensitive detectors that concurrently acquire data across spectral and temporal domains. For molecular imaging, such methodology enables to collect rich intensity and lifetime multiplexed fluorescence datasets. Herein we report on the application of a single-pixel structured light-based platform for macroscopic imaging of tissue autofluorescence. The super-continuum visible excitation and hyperspectral single-pixel detection allow for parallel characterization of autofluorescence intensity and lifetime. Furthermore, we exploit a deep learning based data processing pipeline, to perform autofluorescence unmixing while yielding the autofluorophores' concentrations. The full scheme (setup and processing) is validated in silico and in vitro with clinically relevant autofluorophores flavin adenine dinucleotide, riboflavin, and protoporphyrin. The presented results demonstrate the potential of the methodology for macroscopically quantifying the intensity and lifetime of autofluorophores, with higher specificity for cases of mixed emissions, which are ubiquitous in autofluorescence and multiplexed in vivo imaging.

Monte Carlo-based data generation for efficient deep learning reconstruction of macroscopic diffuse optical tomography and topography applications.

SIGNIFICANCE: Deep learning (DL) models are being increasingly developed to map sensor data to the image domain directly. However, DL methodologies are data-driven and require large and diverse data sets to provide robust and accurate image formation performances. For research modalities such as 2D/3D diffuse optical imaging, the lack of large publicly available data sets and the wide variety of instrumentation designs, data types, and applications leads to unique challenges in obtaining well-controlled data sets for training and validation. Meanwhile, great efforts over the last four decades have focused on developing accurate and computationally efficient light propagation models that are flexible enough to simulate a wide variety of experimental conditions. AIM: Recent developments in Monte Carlo (MC)-based modeling offer the unique advantage of simulating accurately light propagation spatially, temporally, and over an extensive range of optical parameters, including minimally to highly scattering tissue within a computationally efficient platform. Herein, we demonstrate how such MC platforms, namely "Monte Carlo eXtreme" and "Mesh-based Monte Carlo," can be leveraged to generate large and representative data sets for training the DL model efficiently. APPROACH: We propose data generator pipeline strategies using these platforms and demonstrate their potential in fluorescence optical topography, fluorescence optical tomography, and single-pixel diffuse optical tomography. These applications represent a large variety in instrumentation design, sample properties, and contrast function. RESULTS: DL models trained using the MC-based in silico datasets, validated further with experimental data not used during training, show accurate and promising results. CONCLUSION: Overall, these MC-based data generation pipelines are expected to support the development of DL models for rapid, robust, and user-friendly image formation in a wide variety of applications.

3D k-space reflectance fluorescence tomography via deep learning.

We report on the potential to perform image reconstruction in 3D k-space reflectance fluorescence tomography (FT) using deep learning (DL). Herein, we adopt a modified AUTOMAP architecture and develop a training methodology that leverages an open-source Monte-Carlo-based simulator to generate a large dataset. Using an enhanced EMNIST (EEMNIST) dataset as an embedded contrast function allows us to train the network efficiently. The optical strategy utilizes k-space illumination in a reflectance configuration to probe tissue in the mesoscopic regime with high sensitivity and resolution. The proposed DL model training and validation is performed with both in silico data and a phantom experiment. Overall, our results indicate that the approach can correctly reconstruct both single and multiple fluorescent embedding(s) in a 3D volume. Furthermore, the presented technique is shown to outperform the traditional approaches [least-squares (LSQ) and total-variation minimization (TVAL)], especially at higher depths. We, therefore, expect the proposed computational technique to have future implications in preclinical studies.

Deep learning in macroscopic diffuse optical imaging.

SIGNIFICANCE: Biomedical optics system design, image formation, and image analysis have primarily been guided by classical physical modeling and signal processing methodologies. Recently, however, deep learning (DL) has become a major paradigm in computational modeling and has demonstrated utility in numerous scientific domains and various forms of data analysis. AIM: We aim to comprehensively review the use of DL applied to macroscopic diffuse optical imaging (DOI). APPROACH: First, we provide a layman introduction to DL. Then, the review summarizes current DL work in some of the most active areas of this field, including optical properties retrieval, fluorescence lifetime imaging, and diffuse optical tomography. RESULTS: The advantages of using DL for DOI versus conventional inverse solvers cited in the literature reviewed herein are numerous. These include, among others, a decrease in analysis time (often by many orders of magnitude), increased quantitative reconstruction quality, robustness to noise, and the unique capability to learn complex end-to-end relationships. CONCLUSIONS: The heavily validated capability of DL's use across a wide range of complex inverse solving methodologies has enormous potential to bring novel DOI modalities, otherwise deemed impractical for clinical translation, to the patient's bedside.

Macroscopic Fluorescence Lifetime Imaging for Monitoring of Drug-Target Engagement.

Precision medicine promises to improve therapeutic efficacy while reducing adverse effects, especially in oncology. However, despite great progresses in recent years, precision medicine for cancer treatment is not always part of routine care. Indeed, the ability to specifically tailor therapies to distinct patient profiles requires still significant improvements in targeted therapy development as well as decreases in drug treatment failures. In this regard, preclinical animal research is fundamental to advance our understanding of tumor biology, and diagnostic and therapeutic response. Most importantly, the ability to measure drug-target engagement accurately in live and intact animals is critical in guiding the development and optimization of targeted therapy. However, a major limitation of preclinical molecular imaging modalities is their lack of capability to directly and quantitatively discriminate between drug accumulation and drug-target engagement at the pathological site. Recently, we have developed Macroscopic Fluorescence Lifetime Imaging (MFLI) as a unique feature of optical imaging to quantitate in vivo drug-target engagement. MFLI quantitatively reports on nanoscale interactions via lifetime-sensing of Förster Resonance Energy Transfer (FRET) in live, intact animals. Hence, MFLI FRET acts as a direct reporter of receptor dimerization and target engagement via the measurement of the fraction of labeled-donor entity undergoing binding to its respective receptor. MFLI is expected to greatly impact preclinical imaging and also adjacent fields such as image-guided surgery and drug development.

Deep Learning in Biomedical Optics.

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

UNMIX-ME: spectral and lifetime fluorescence unmixing via deep learning.

Hyperspectral fluorescence lifetime imaging allows for the simultaneous acquisition of spectrally resolved temporal fluorescence emission decays. In turn, the acquired rich multidimensional data set enables simultaneous imaging of multiple fluorescent species for a comprehensive molecular assessment of biotissues. However, to enable quantitative imaging, inherent spectral overlap between the considered fluorescent probes and potential bleed-through must be considered. Such a task is performed via either spectral or lifetime unmixing, typically independently. Herein, we present "UNMIX-ME" (unmix multiple emissions), a deep learning-based fluorescence unmixing routine, capable of quantitative fluorophore unmixing by simultaneously using both spectral and temporal signatures. UNMIX-ME was trained and validated using an in silico framework replicating the data acquisition process of a compressive hyperspectral fluorescent lifetime imaging platform (HMFLI). It was benchmarked against a conventional LSQ method for tri and quadri-exponential simulated samples. Last, UNMIX-ME's potential was assessed for NIR FRET in vitro and in vivo preclinical applications.

Multiplexed non-invasive tumor imaging of glucose metabolism and receptor-ligand engagement using dark quencher FRET acceptor.

Rationale: Following an ever-increased focus on personalized medicine, there is a continuing need to develop preclinical molecular imaging modalities to guide the development and optimization of targeted therapies. Near-Infrared (NIR) Macroscopic Fluorescence Lifetime Förster Resonance Energy Transfer (MFLI-FRET) imaging offers a unique method to robustly quantify receptor-ligand engagement in live intact animals, which is critical to assess the delivery efficacy of therapeutics. However, to date, non-invasive imaging approaches that can simultaneously measure cellular drug delivery efficacy and metabolic response are lacking. A major challenge for the implementation of concurrent optical and MFLI-FRET in vivo whole-body preclinical imaging is the spectral crowding and cross-contamination between fluorescent probes. Methods: We report on a strategy that relies on a dark quencher enabling simultaneous assessment of receptor-ligand engagement and tumor metabolism in intact live mice. Several optical imaging approaches, such as in vitro NIR FLI microscopy (FLIM) and in vivo wide-field MFLI, were used to validate a novel donor-dark quencher FRET pair. IRDye 800CW 2-deoxyglucose (2-DG) imaging was multiplexed with MFLI-FRET of NIR-labeled transferrin FRET pair (Tf-AF700/Tf-QC-1) to monitor tumor metabolism and probe uptake in breast tumor xenografts in intact live nude mice. Immunohistochemistry was used to validate in vivo imaging results. Results: First, we establish that IRDye QC-1 (QC-1) is an effective NIR dark acceptor for the FRET-induced quenching of donor Alexa Fluor 700 (AF700). Second, we report on simultaneous in vivo imaging of the metabolic probe 2-DG and MFLI-FRET imaging of Tf-AF700/Tf-QC-1 uptake in tumors. Such multiplexed imaging revealed an inverse relationship between 2-DG uptake and Tf intracellular delivery, suggesting that 2-DG signal may predict the efficacy of intracellular targeted delivery. Conclusions: Overall, our methodology enables for the first time simultaneous non-invasive monitoring of intracellular drug delivery and metabolic response in preclinical studies.

Net-FLICS: fast quantitative wide-field fluorescence lifetime imaging with compressed sensing - a deep learning approach.

Macroscopic fluorescence lifetime imaging (MFLI) via compressed sensed (CS) measurements enables efficient and accurate quantification of molecular interactions in vivo over a large field of view (FOV). However, the current data-processing workflow is slow, complex and performs poorly under photon-starved conditions. In this paper, we propose Net-FLICS, a novel image reconstruction method based on a convolutional neural network (CNN), to directly reconstruct the intensity and lifetime images from raw time-resolved CS data. By carefully designing a large simulated dataset, Net-FLICS is successfully trained and achieves outstanding reconstruction performance on both in vitro and in vivo experimental data and even superior results at low photon count levels for lifetime quantification.

Review of structured light in diffuse optical imaging.

Diffuse optical imaging probes deep living tissue enabling structural, functional, metabolic, and molecular imaging. Recently, due to the availability of spatial light modulators, wide-field quantitative diffuse optical techniques have been implemented, which benefit greatly from structured light methodologies. Such implementations facilitate the quantification and characterization of depth-resolved optical and physiological properties of thick and deep tissue at fast acquisition speeds. We summarize the current state of work and applications in the three main techniques leveraging structured light: spatial frequency-domain imaging, optical tomography, and single-pixel imaging. The theory, measurement, and analysis of spatial frequency-domain imaging are described. Then, advanced theories, processing, and imaging systems are summarized. Preclinical and clinical applications on physiological measurements for guidance and diagnosis are summarized. General theory and method development of tomographic approaches as well as applications including fluorescence molecular tomography are introduced. Lastly, recent developments of single-pixel imaging methodologies and applications are reviewed.

Sensitivity of spatially offset Raman spectroscopy (SORS) to subcortical bone tissue.

The development of spatially offset Raman spectroscopy (SORS) has enabled deep, non-invasive chemical characterization of turbid media. Here, we use SORS to measure subcortical bone tissue and depth-resolved biochemical variability in intact, exposed murine bones. We also apply the technique to study a mouse model of the genetic bone disorder osteogenesis imperfecta. The results suggest that SORS is more sensitive to disease-related biochemical differences in subcortical trabecular bone and marrow than conventional Raman measurements.