Luminescence lifetime imaging of three-dimensional biological objects.

A major focus of current biological studies is to fill the knowledge gaps between cell, tissue and organism scales. To this end, a wide array of contemporary optical analytical tools enable multiparameter quantitative imaging of live and fixed cells, three-dimensional (3D) systems, tissues, organs and organisms in the context of their complex spatiotemporal biological and molecular features. In particular, the modalities of luminescence lifetime imaging, comprising fluorescence lifetime imaging (FLI) and phosphorescence lifetime imaging microscopy (PLIM), in synergy with Förster resonance energy transfer (FRET) assays, provide a wealth of information. On the application side, the luminescence lifetime of endogenous molecules inside cells and tissues, overexpressed fluorescent protein fusion biosensor constructs or probes delivered externally provide molecular insights at multiple scales into protein-protein interaction networks, cellular metabolism, dynamics of molecular oxygen and hypoxia, physiologically important ions, and other physical and physiological parameters. Luminescence lifetime imaging offers a unique window into the physiological and structural environment of cells and tissues, enabling a new level of functional and molecular analysis in addition to providing 3D spatially resolved and longitudinal measurements that can range from microscopic to macroscopic scale. We provide an overview of luminescence lifetime imaging and summarize key biological applications from cells and tissues to organisms.

Portable neuroimaging differentiates novices from those with experience for the Fundamentals of Laparoscopic Surgery (FLS) suturing with intracorporeal knot tying task.

BACKGROUND: The goal of this study was to compare the brain activation patterns of experienced and novice individuals when performing the Fundamentals of Laparoscopic Surgery (FLS) suture with intracorporeal knot tying task, which requires bimanual motor control. METHODS: Twelve experienced and fourteen novice participants completed this cross-sectional observational study. Participants performed three repetitions of the FLS suture with intracorporeal knot tying task in a standard box trainer. Functional near infrared spectroscopy (fNIRS) data was recorded using an optode montage that covered the prefrontal and sensorimotor brain areas throughout the task. Data processing was conducted using the HOMER3 and AtlasViewer toolboxes to determine the oxy-hemoglobin (HbO) and deoxyhemoglobin (HbR) concentrations. The hemodynamic response function based on HbO changes during the task relative to the resting state was averaged for each repetition and by participant. Group-level differences were evaluated using a general linear model of the HbO changes with significance set at p < 0.05. RESULTS: The average performance score for the experienced group was significantly higher than the novice group (p < 0.01). There were significant cortical activations (p < 0.05) in the prefrontal and sensorimotor areas for the experienced and novice groups. Areas of statistically significant differences in activation included the right dorsolateral prefrontal cortex (PFC), the right precentral gyrus, and the right postcentral gyrus. CONCLUSIONS: Portable neuroimaging allowed for the differentiation of brain regions activated by experienced and novice participants for a complex surgical motor task. This information can be used to support the objective evaluation of expertise during surgical skills training and assessment.

Performance and learning rate prediction models development in FLS and RAS surgical tasks using electroencephalogram and eye gaze data and machine learning.

OBJECTIVE: This study explored the use of electroencephalogram (EEG) and eye gaze features, experience-related features, and machine learning to evaluate performance and learning rates in fundamentals of laparoscopic surgery (FLS) and robotic-assisted surgery (RAS). METHODS: EEG and eye-tracking data were collected from 25 participants performing three FLS and 22 participants performing two RAS tasks. Generalized linear mixed models, using L1-penalized estimation, were developed to objectify performance evaluation using EEG and eye gaze features, and linear models were developed to objectify learning rate evaluation using these features and performance scores at the first attempt. Experience metrics were added to evaluate their role in learning robotic surgery. The differences in performance across experience levels were tested using analysis of variance. RESULTS: EEG and eye gaze features and experience-related features were important for evaluating performance in FLS and RAS tasks with reasonable results. Residents outperformed faculty in FLS peg transfer (p value = 0.04), while faculty and residents both excelled over pre-medical students in the FLS pattern cut (p value = 0.01 and p value < 0.001, respectively). Fellows outperformed pre-medical students in FLS suturing (p value = 0.01). In RAS tasks, both faculty and fellows surpassed pre-medical students (p values for the RAS pattern cut were 0.001 for faculty and 0.003 for fellows, while for RAS tissue dissection, the p value was less than 0.001 for both groups), with residents also showing superior skills in tissue dissection (p value = 0.03). CONCLUSION: Findings could be used to develop training interventions for improving surgical skills and have implications for understanding motor learning and designing interventions to enhance learning outcomes.

3D k-space reflectance fluorescence tomography via deep learning.

We report on the potential to perform image reconstruction in 3D k-space reflectance fluorescence tomography (FT) using deep learning (DL). Herein, we adopt a modified AUTOMAP architecture and develop a training methodology that leverages an open-source Monte-Carlo-based simulator to generate a large dataset. Using an enhanced EMNIST (EEMNIST) dataset as an embedded contrast function allows us to train the network efficiently. The optical strategy utilizes k-space illumination in a reflectance configuration to probe tissue in the mesoscopic regime with high sensitivity and resolution. The proposed DL model training and validation is performed with both in silico data and a phantom experiment. Overall, our results indicate that the approach can correctly reconstruct both single and multiple fluorescent embedding(s) in a 3D volume. Furthermore, the presented technique is shown to outperform the traditional approaches [least-squares (LSQ) and total-variation minimization (TVAL)], especially at higher depths. We, therefore, expect the proposed computational technique to have future implications in preclinical studies.

Fast fit-free analysis of fluorescence lifetime imaging via deep learning.

Fluorescence lifetime imaging (FLI) provides unique quantitative information in biomedical and molecular biology studies but relies on complex data-fitting techniques to derive the quantities of interest. Herein, we propose a fit-free approach in FLI image formation that is based on deep learning (DL) to quantify fluorescence decays simultaneously over a whole image and at fast speeds. We report on a deep neural network (DNN) architecture, named fluorescence lifetime imaging network (FLI-Net) that is designed and trained for different classes of experiments, including visible FLI and near-infrared (NIR) FLI microscopy (FLIM) and NIR gated macroscopy FLI (MFLI). FLI-Net outputs quantitatively the spatially resolved lifetime-based parameters that are typically employed in the field. We validate the utility of the FLI-Net framework by performing quantitative microscopic and preclinical lifetime-based studies across the visible and NIR spectra, as well as across the 2 main data acquisition technologies. These results demonstrate that FLI-Net is well suited to accurately quantify complex fluorescence lifetimes in cells and, in real time, in intact animals without any parameter settings. Hence, FLI-Net paves the way to reproducible and quantitative lifetime studies at unprecedented speeds, for improved dissemination and impact of FLI in many important biomedical applications ranging from fundamental discoveries in molecular and cellular biology to clinical translation.

Accelerating vasculature imaging in tumor using mesoscopic fluorescence molecular tomography via a hybrid reconstruction strategy.

Mesoscopic fluorescent molecular tomography (MFMT) enables to image fluorescent molecular probes beyond the typical depth limits of microscopic imaging and with enhanced resolution compared to macroscopic imaging. However, MFMT is a scattering-based inverse problem that is an ill-posed inverse problem and hence, requires relative complex iterative solvers coupled with regularization strategies. Inspired by the potential of deep learning in performing image formation tasks from raw measurements, this work proposes a hybrid approach to solve the MFMT inverse problem. This methodology combines a convolutional symmetric network and a conventional iterative algorithm to accelerate the reconstruction procedure. By the proposed deep neural network, the principal components of the sensitivity matrix are extracted and the accompanying noise in measurements is suppressed, which helps to accelerate the reconstruction and improve the accuracy of results. We apply the proposed method to reconstruct in silico and vascular tree models. The results demonstrate that reconstruction accuracy and speed are highly improved due to the reduction of redundant entries of the sensitivity matrix and noise suppression.

Deep Learning in Biomedical Optics.

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

Macroscopic fluorescence lifetime topography enhanced via spatial frequency domain imaging.

We report on a macroscopic fluorescence lifetime imaging (MFLI) topography computational framework based around machine learning with the main goal of retrieving the depth of fluorescent inclusions deeply seated in bio-tissues. This approach leverages the depth-resolved information inherent to time-resolved fluorescence data sets coupled with the retrieval of in situ optical properties as obtained via spatial frequency domain imaging (SFDI). Specifically, a Siamese network architecture is proposed with optical properties (OPs) and time-resolved fluorescence decays as input followed by simultaneous retrieval of lifetime maps and depth profiles. We validate our approach using comprehensive in silico data sets as well as with a phantom experiment. Overall, our results demonstrate that our approach can retrieve the depth of fluorescence inclusions, especially when coupled with optical properties estimation, with high accuracy. We expect the presented computational approach to find great utility in applications such as optical-guided surgery.

Accelerating Monte Carlo modeling of structured-light-based diffuse optical imaging via "photon sharing".

The increasing use of spatially modulated imaging and single-pixel detection techniques demands computationally efficient methods for light transport modeling. Herein, we report an easy-to-implement yet significantly more efficient Monte Carlo (MC) method for simultaneously simulating spatially modulated illumination and detection patterns accurately in 3D complex domains. We have implemented this accelerated algorithm, named "photon sharing," in our open-source MC simulators, reporting 13.6× and 5.5× speedups in mesh- and voxel-based MC benchmarks, respectively. In addition, the proposed algorithm is readily used to accelerate the solving of inverse problems in spatially modulated imaging systems by building Jacobians of all illumination-detection pattern pairs concurrently, resulting in a 12.4-fold speed improvement.

Quantification of Trastuzumab-HER2 Engagement In Vitro and In Vivo.

Human EGF Receptor 2 (HER2) is an important oncogene driving aggressive metastatic growth in up to 20% of breast cancer tumors. At the same time, it presents a target for passive immunotherapy such as trastuzumab (TZM). Although TZM has been widely used clinically since 1998, not all eligible patients benefit from this therapy due to primary and acquired drug resistance as well as potentially lack of drug exposure. Hence, it is critical to directly quantify TZM-HER2 binding dynamics, also known as cellular target engagement, in undisturbed tumor environments in live, intact tumor xenograft models. Herein, we report the direct measurement of TZM-HER2 binding in HER2-positive human breast cancer cells and tumor xenografts using fluorescence lifetime Forster Resonance Energy Transfer (FLI-FRET) via near-infrared (NIR) microscopy (FLIM-FRET) as well as macroscopy (MFLI-FRET) approaches. By sensing the reduction of fluorescence lifetime of donor-labeled TZM in the presence of acceptor-labeled TZM, we successfully quantified the fraction of HER2-bound and internalized TZM immunoconjugate both in cell culture and tumor xenografts in live animals. Ex vivo immunohistological analysis of tumors confirmed the binding and internalization of TZM-HER2 complex in breast cancer cells. Thus, FLI-FRET imaging presents a powerful analytical tool to monitor and quantify cellular target engagement and subsequent intracellular drug delivery in live HER2-positive tumor xenografts.

Objective assessment of surgical skill transfer using non-invasive brain imaging.

BACKGROUND: Physical and virtual surgical simulators are increasingly being used in training technical surgical skills. However, metrics such as completion time or subjective performance checklists often show poor correlation to transfer of skills into clinical settings. We hypothesize that non-invasive brain imaging can objectively differentiate and classify surgical skill transfer, with higher accuracy than established metrics, for subjects based on motor skill levels. STUDY DESIGN: 18 medical students at University at Buffalo were randomly assigned into control, physical surgical trainer, or virtual trainer groups. Training groups practiced a surgical technical task on respective simulators for 12 consecutive days. To measure skill transfer post-training, all subjects performed the technical task in an ex-vivo environment. Cortical activation was measured using functional near-infrared spectroscopy (fNIRS) in the prefrontal cortex, primary motor cortex, and supplementary motor area, due to their direct impact on motor skill learning. RESULTS: Classification between simulator trained and untrained subjects based on traditional metrics is poor, where misclassification errors range from 20 to 41%. Conversely, fNIRS metrics can successfully classify physical or virtual trained subjects from untrained subjects with misclassification errors of 2.2% and 8.9%, respectively. More importantly, untrained subjects are successfully classified from physical or virtual simulator trained subjects with misclassification errors of 2.7% and 9.1%, respectively. CONCLUSION: fNIRS metrics are significantly more accurate than current established metrics in classifying different levels of surgical motor skill transfer. Our approach brings robustness, objectivity, and accuracy in validating the effectiveness of future surgical trainers in translating surgical skills to clinically relevant environments.

Radiative transfer with delta-Eddington-type phase functions.

The radiative transfer equation (RTE) arises in a wide variety of applications, in particular, in biomedical imaging applications associated with the propagation of light through the biological tissue. However, highly forward-peaked scattering feature in a biological medium makes it very challenging to numerically solve the RTE problem accurately. One idea to overcome the difficulty associated with the highly forward-peaked scattering is through the use of a delta-Eddington phase function. This paper is devoted to an RTE framework with a family of delta-Eddington-type phase functions. Significance in biomedical imaging applications of the RTE with delta-Eddington-type phase functions are explained. Mathematical studies of the problems include solution existence, uniqueness, and continuous dependence on the problem data: the inflow boundary value, the source function, the absorption coefficient, and the scattering coefficient. Numerical results are presented to show that employing a delta-Eddington-type phase function with properly chosen parameters provides accurate simulation results for light propagation within highly forward-peaked scattering media.

Hyperspectral time-resolved wide-field fluorescence molecular tomography based on structured light and single-pixel detection.

We present a time-resolved fluorescence diffuse optical tomography platform that is based on wide-field structured illumination, single-pixel detection, and hyperspectral acquisition. Two spatial light modulators (digital micro-mirror devices) are employed to generate independently wide-field illumination and detection patterns, coupled with a 16-channel spectrophotometer detection module to capture hyperspectral time-resolved tomographic data sets. The main system characteristics are reported, and we demonstrate the feasibility of acquiring dense 4D tomographic data sets (space, time, spectra) for time domain 3D quantitative multiplexed fluorophore concentration mapping in turbid media.

L(p) regularization for early gate fluorescence molecular tomography.

Time domain fluorescence molecular tomography (TD-FMT) provides a unique dataset for enhanced quantification and spatial resolution. The time-gate dataset can be divided into two temporal groups around the maximum counts gate, which are early gates and late gates. It is well established that early gates allow for improved spatial resolution and late gates are essential for fluorophore unmixing and concentration quantification. However, the inverse problem of FMT is ill-posed and typically underdetermined, which makes image reconstruction highly susceptible to data noise. More specifically, photon counts are inherently very low at early gates due to high absorption and scattering of tissue, resulting in a low signal-to-noise ratio and unstable reconstructions. In this work, an L(p) regularization-based reconstruction algorithm was developed and tested with our wide-field mesh-based Monte Carlo simulation strategy. We compared the early time-gate reconstructions obtained with the different p (p∈{1/16,1/8,1/4,1/3,1/2,1,2}) from a synthetic murine model simulating the fluorophore uptake in the kidneys and preclinical data. The results from a 3D mouse atlas and a mouse experiment show that our L(1/4) regularization methods give the best performance for early time gates reconstructions.

Contrast-enhanced photon-counting micro-CT of tumor xenograft models.

Objective. Photon-counting micro-computed tomography (micro-CT) is a major advance in small animal preclinical imaging. Small molecule- and nanoparticle-based contrast agents have been widely used to enable the differentiation of liver tumors from surrounding tissues using photon-counting micro-CT. However, there is a notable gap in the application of these market-available agents to the imaging of breast and ovarian tumors using photon-counting micro-CT. Herein, we have used photon-counting micro-CT to determine the effectiveness of these contrast agents in differentiating ovarian and breast tumor xenografts in live, intact mice.Approach. Nude mice carrying different types of breast and ovarian tumor xenografts (AU565, MDA-MB-231 and SKOV-3 human cancer cells) were injected with ISOVUE-370 (a small molecule-based agent) or Exitron Nano 12000 (a nanoparticle-based agent) and subjected to photon-counting micro-CT. To improve tumor visualization using photon-counting micro-CT, we developed a novel color visualization method, which changes color tones to highlight contrast media distribution, offering a robust alternative to traditional material decomposition methods with less computational demand.Main results. Ourin vivoexperiments confirm the effectiveness of this color visualization approach, showing distinct enhancement characteristics for each contrast agent. Qualitative and quantitative analyses suggest that Exitron Nano 12000 provides superior vasculature enhancement and better quantitative consistency across scans, while ISOVUE-370 delivers a more comprehensive tumor enhancement but with significant variance between scans due to its short blood half-time. Further, a paired t-test on mean and standard deviation values within tumor volumes showed significant differences between the AU565 and SKOV-3 tumor models with the nanoparticle-based contrast agent (p-values < 0.02), attributable to their distinct vascularity, as confirmed by immunohistochemical analysis.Significance. These findings underscore the utility of photon-counting micro-CT in non-invasively assessing the morphology and anatomy of different tumor xenografts, which is crucial for tumor characterization and longitudinal monitoring of tumor progression and response to treatments.

Contrast-enhanced photon-counting micro-CT of tumor xenograft models.

Photon-counting micro computed tomography (micro-CT) offers new potential in preclinical imaging, particularly in distinguishing materials. It becomes especially helpful when combined with contrast agents, enabling the differentiation of tumors from surrounding tissues. There are mainly two types of contrast agents in the market for micro-CT: small molecule-based and nanoparticle-based. However, despite their widespread use in liver tumor studies, there is a notable gap in research on the application of these commercially available agents for photon-counting micro-CT in breast and ovarian tumors. Herein, we explored the effectiveness of these agents in differentiating tumor xenografts from various origins (AU565, MDA-MB-231, and SKOV-3) in nude mice, using photon-counting micro-CT. Specifically, ISOVUE-370 (a small molecule-based agent) and Exitrone Nano 12000 (a nanoparticle-based agent) were investigated in this context. To improve tumor visualization, we proposed a novel color visualization method for photon-counting micro-CT, which changes color tones to highlight contrast media distribution, offering a robust alternative to traditional material decomposition methods with less computational demand. Our in vivo experiments confirm its effectiveness, showing distinct enhancement characteristics for each contrast agent. Qualitative and quantitative analyses suggested that Exitrone Nano 12000 provides superior vasculature enhancement and better quantitative consistency across scans, while ISOVUE-370 gives more comprehensive tumor enhancement but with a significant variance between scans due to its short blood half-time. This variability leads to high sensitivity to timing and individual differences among mice. Further, a paired t-test on mean and standard deviation values within tumor volumes showed significant differences between the AU565 and SKOV-3 tumor models with the nanoparticle-based (p-values < 0.02), attributable to their distinct vascularity, as confirmed by immunohistochemistry. These findings underscore the utility of photon-counting micro-CT in non-invasively assessing the morphology and anatomy of different tumor xenografts, which is crucial for tumor characterization and longitudinal monitoring of tumor development and response to treatments.

One-shot skill assessment in high-stakes domains with limited data via meta learning.

Deep Learning (DL) has achieved robust competency assessment in various high-stakes fields. However, the applicability of DL models is often hampered by their substantial data requirements and confinement to specific training domains. This prevents them from transitioning to new tasks where data is scarce. Therefore, domain adaptation emerges as a critical element for the practical implementation of DL in real-world scenarios. Herein, we introduce A-VBANet, a novel meta-learning model capable of delivering domain-agnostic skill assessment via one-shot learning. Our methodology has been tested by assessing surgical skills on five laparoscopic and robotic simulators and real-life laparoscopic cholecystectomy. Our model successfully adapted with accuracies up to 99.5 % in one-shot and 99.9 % in few-shot settings for simulated tasks and 89.7 % for laparoscopic cholecystectomy. This study marks the first instance of a domain-agnostic methodology for skill assessment in critical fields setting a precedent for the broad application of DL across diverse real-life domains with limited data.

Fluorescence Lifetime Imaging for Quantification of Targeted Drug Delivery in Varying Tumor Microenvironments.

Rationale: Trastuzumab (TZM) is a monoclonal antibody that targets the human epidermal growth factor receptor (HER2) and is clinically used for the treatment of HER2-positive breast tumors. However, the tumor microenvironment can limit the access of TZM to the HER2 targets across the whole tumor and thereby compromise TZM's therapeutic efficacy. An imaging methodology that can non-invasively quantify the binding of TZM-HER2, which is required for therapeutic action, and distribution within tumors with varying tumor microenvironments is much needed. Methods: We performed near-infrared (NIR) fluorescence lifetime (FLI) Forster Resonance Energy Transfer (FRET) to measure TZM-HER2 binding, using in vitro microscopy and in vivo widefield macroscopy, in HER2 overexpressing breast and ovarian cancer cells and tumor xenografts, respectively. Immunohistochemistry was used to validate in vivo imaging results. Results: NIR FLI FRET in vitro microscopy data show variations in intracellular distribution of bound TZM in HER2-positive breast AU565 and AU565 tumor-passaged XTM cell lines in comparison to SKOV-3 ovarian cancer cells. Macroscopy FLI (MFLI) FRET in vivo imaging data show that SKOV-3 tumors display reduced TZM binding compared to AU565 and XTM tumors, as validated by ex vivo immunohistochemistry. Moreover, AU565/XTM and SKOV-3 tumor xenografts display different amounts and distributions of TME components, such as collagen and vascularity. Therefore, these results suggest that SKOV-3 tumors are refractory to TZM delivery due to their disrupted vasculature and increased collagen content. Conclusion: Our study demonstrates that FLI is a powerful analytical tool to monitor the delivery of antibody drug tumor both in cell cultures and in vivo live systems. Especially, MFLI FRET is a unique imaging modality that can directly quantify target engagement with potential to elucidate the role of the TME in drug delivery efficacy in intact live tumor xenografts.

Active wide-field illumination for high-throughput fluorescence lifetime imaging.

Wide-field fluorescence lifetime imaging allows for fast imaging of large sample areas at the cost of low sensitivity to weak fluorescence signals. To overcome this challenge, we developed an active wide-field illumination (AWFI) strategy to optimize the impinging spatial intensity for acquiring optimal fluorescence signals over the whole sample. We demonstrated the ability of AWFI to accurately estimate lifetimes from a multiwell plate sample with concentrations ranging over two orders of magnitude. We further reported its successful application to a quantitative Förster resonance energy transfer lifetime cell-based assay. Overall, this method allows for enhanced accuracy in lifetime-based imaging at high acquisition speed over samples with large fluorescence intensity distributions.

Video-based formative and summative assessment of surgical tasks using deep learning.

To ensure satisfactory clinical outcomes, surgical skill assessment must be objective, time-efficient, and preferentially automated-none of which is currently achievable. Video-based assessment (VBA) is being deployed in intraoperative and simulation settings to evaluate technical skill execution. However, VBA is manual, time-intensive, and prone to subjective interpretation and poor inter-rater reliability. Herein, we propose a deep learning (DL) model that can automatically and objectively provide a high-stakes summative assessment of surgical skill execution based on video feeds and low-stakes formative assessment to guide surgical skill acquisition. Formative assessment is generated using heatmaps of visual features that correlate with surgical performance. Hence, the DL model paves the way for the quantitative and reproducible evaluation of surgical tasks from videos with the potential for broad dissemination in surgical training, certification, and credentialing.