Robust deep learning optical autofocus system applied to automated multiwell plate single molecule localization microscopy.

We presenta robust, long-range optical autofocus system for microscopy utilizing machine learning. This can be useful for experiments with long image data acquisition times that may be impacted by defocusing resulting from drift of components, for example due to changes in temperature or mechanical drift. It is also useful for automated slide scanning or multiwell plate imaging where the sample(s) to be imaged may not be in the same horizontal plane throughout the image data acquisition. To address the impact of (thermal or mechanical) fluctuations over time in the optical autofocus system itself, we utilize a convolutional neural network (CNN) that is trained over multiple days to account for such fluctuations. To address the trade-off between axial precision and range of the autofocus, we implement orthogonal optical readouts with separate CNN training data, thereby achieving an accuracy well within the 600 nm depth of field of our 1.3 numerical aperture objective lens over a defocus range of up to approximately +/-100 µm. We characterize the performance of this autofocus system and demonstrate its application to automated multiwell plate single molecule localization microscopy.

Many microscopy experiments involve extended imaging of samples over timescales from minutes to days, during which the microscope can 'drift' out of focus. When imaging at high magnification, the depth of field is of the order of one micron and so the imaging system should keep the sample in the focal plane of the microscope objective lens to this precision. Unfortunately, temperature changes in the laboratory can cause thermal expansion of microscope components that can move the focal plane by more than a micron and such changes can occur on a timescale of minutes. This is a particular issue for super-resolved microscopy experiments using single molecule localization microscopy (SMLM) techniques, for which 1000s of images are acquired, and for automated imaging of multiple samples in multiwell plates. It is possible to maintain the sample in the focal plane focus position by either automatically moving the sample or adjusting the imaging system, for example by moving the objective lens. This is called 'autofocus' and is frequently achieved by reflecting a light beam from the microscope coverslip and measuring its position of beam profile as a function of defocus of the microscope. The correcting adjustment is then usually calculated analytically but there is recent interest in using machine learning techniques to determine the required focussing adjustment. Here, we present a system that uses a neural network to determine the required defocus correcting adjustment from camera images of a laser beam that is reflected from the coverslip. Unfortunately, this approach will only work when the microscope is in the same condition as it was when the neural network was trained - and this can be compromised by the same drift of the optical system that causes the defocus needing to be corrected. We show, however, that by training a neural network over an extended period, for example 10 days, this approach can 'learn' about the optical system drifts and provide the required autofocus function. We also show that an optical system utilizing a rectangular slit can make two measurements of the defocus simultaneously, with one measurement being optimized for high accuracy over a limited range (±10 µm) near focus and the other providing lower accuracy but over a much longer range (±100 µm). This robust autofocus system is suitable for automated super-resolved microscopy of arrays of samples in a multiwell plate using SMLM, for which an experiment routinely lasts more than 5 h.

Smad4 controls signaling robustness and morphogenesis by differentially contributing to the Nodal and BMP pathways.

The transcriptional effector SMAD4 is a core component of the TGF-ß family signaling pathways. However, its role in vertebrate embryo development remains unresolved. To address this, we deleted Smad4 in zebrafish and investigated the consequences of this on signaling by the TGF-ß family morphogens, BMPs and Nodal. We demonstrate that in the absence of Smad4, dorsal/ventral embryo patterning is disrupted due to the loss of BMP signaling. However, unexpectedly, Nodal signaling is maintained, but lacks robustness. This Smad4-independent Nodal signaling is sufficient for mesoderm specification, but not for optimal endoderm specification. Furthermore, using Optical Projection Tomography in combination with 3D embryo morphometry, we have generated a BMP morphospace and demonstrate that Smad4 mutants are morphologically indistinguishable from embryos in which BMP signaling has been genetically/pharmacologically perturbed. Smad4 is thus differentially required for signaling by different TGF-ß family ligands, which has implications for diseases where Smad4 is mutated or deleted.

Single-shot phase contrast microscopy using polarisation-resolved differential phase contrast.

We present a robust, low-cost single-shot implementation of differential phase microscopy utilising a polarisation-sensitive camera to simultaneously acquire four images from which phase contrast images can be calculated. This polarisation-resolved differential phase contrast (pDPC) microscopy technique can be easily integrated with fluorescence microscopy.

Application of direct stochastic optical reconstruction microscopy (dSTORM) to the histological analysis of human glomerular disease.

Electron microscopy (EM) following immunofluorescence (IF) imaging is a vital tool for the diagnosis of human glomerular diseases, but the implementation of EM is limited to specialised institutions and it is not available in many countries. Recent progress in fluorescence microscopy now enables conventional widefield fluorescence microscopes to be adapted at modest cost to provide resolution below 50 nm in biological specimens. We show that stochastically switched single-molecule localisation microscopy can be applied to clinical histological sections stained with standard IF techniques and that such super-resolved IF may provide an alternative means to resolve ultrastructure to aid the diagnosis of kidney disease where EM is not available. We have implemented the direct stochastic optical reconstruction microscopy technique with human kidney biopsy frozen sections stained with clinically approved immunofluorescent probes for the basal laminae and immunoglobulin G deposits. Using cases of membranous glomerulonephritis, thin basement membrane lesion, and lupus nephritis, we compare this approach to clinical EM images and demonstrate enhanced imaging compared to conventional IF microscopy. With minor modifications in established IF protocols of clinical frozen renal biopsies, we believe the cost-effective adaptation of conventional widefield microscopes can be widely implemented to provide super-resolved image information to aid diagnosis of human glomerular disease.

DDR1 autophosphorylation is a result of aggregation into dense clusters.

The collagen receptor DDR1 is a receptor tyrosine kinase that promotes progression of a wide range of human disorders. Little is known about how ligand binding triggers DDR1 kinase activity. We previously reported that collagen induces DDR1 activation through lateral dimer association and phosphorylation between dimers, a process that requires specific transmembrane association. Here we demonstrate ligand-induced DDR1 clustering by widefield and super-resolution imaging and provide evidence for a mechanism whereby DDR1 kinase activity is determined by its molecular density. Ligand binding resulted in initial DDR1 reorganisation into morphologically distinct clusters with unphosphorylated DDR1. Further compaction over time led to clusters with highly aggregated and phosphorylated DDR1. Ligand-induced DDR1 clustering was abolished by transmembrane mutations but did not require kinase activity. Our results significantly advance our understanding of the molecular events underpinning ligand-induced DDR1 kinase activity and provide an explanation for the unusually slow DDR1 activation kinetics.

Convolutional neural networks for reconstruction of undersampled optical projection tomography data applied to in vivo imaging of zebrafish.

Optical projection tomography (OPT) is a 3D mesoscopic imaging modality that can utilize absorption or fluorescence contrast. 3D images can be rapidly reconstructed from tomographic data sets sampled with sufficient numbers of projection angles using the Radon transform, as is typically implemented with optically cleared samples of the mm-to-cm scale. For in vivo imaging, considerations of phototoxicity and the need to maintain animals under anesthesia typically preclude the acquisition of OPT data at a sufficient number of angles to avoid artifacts in the reconstructed images. For sparse samples, this can be addressed with iterative algorithms to reconstruct 3D images from undersampled OPT data, but the data processing times present a significant challenge for studies imaging multiple animals. We show here that convolutional neural networks (CNN) can be used in place of iterative algorithms to remove artifacts-reducing processing time for an undersampled in vivo zebrafish dataset from 77 to 15 minutes. We also show that using CNN produces reconstructions of equivalent quality to compressed sensing with 40% fewer projections. We further show that diverse training data classes, for example, ex vivo mouse tissue data, can be used for CNN-based reconstructions of OPT data of other species including live zebrafish.

Automated Fluorescence Lifetime Imaging High-Content Analysis of Förster Resonance Energy Transfer between Endogenously Labeled Kinetochore Proteins in Live Budding Yeast Cells.

We describe an open-source automated multiwell plate fluorescence lifetime imaging (FLIM) methodology to read out Förster resonance energy transfer (FRET) between fluorescent proteins (FPs) labeling endogenous kinetochore proteins (KPs) in live budding yeast cells. The low copy number of many KPs and their small spatial extent present significant challenges for the quantification of donor fluorescence lifetime in the presence of significant cellular autofluorescence and photobleaching. Automated FLIM data acquisition was controlled by µManager and incorporated wide-field time-gated imaging with optical sectioning to reduce background fluorescence. For data analysis, we used custom MATLAB-based software tools to perform kinetochore foci segmentation and local cellular background subtraction and fitted the fluorescence lifetime data using the open-source FLIMfit software. We validated the methodology using endogenous KPs labeled with mTurquoise2 FP and/or yellow FP and measured the donor fluorescence lifetimes for foci comprising 32 kinetochores with KP copy numbers as low as ~2 per kinetochore under an average labeling efficiency of 50%. We observed changes of median donor lifetime ≥250 ps for KPs known to form dimers. Thus, this FLIM high-content analysis platform enables the screening of relatively low-copy-number endogenous protein-protein interactions at spatially confined macromolecular complexes.

Performance of a wearable acoustic system for fetal movement discrimination.

Fetal movements (FM) are a key factor in clinical management of high-risk pregnancies such as fetal growth restriction. While maternal perception of reduced FM can trigger self-referral to obstetric services, maternal sensation is highly subjective. Objective, reliable monitoring of fetal movement patterns outside clinical environs is not currently possible. A wearable and non-transmitting system capable of sensing fetal movements over extended periods of time would be extremely valuable, not only for monitoring individual fetal health, but also for establishing normal levels of movement in the population at large. Wearable monitors based on accelerometers have previously been proposed as a means of tracking FM, but such systems have difficulty separating maternal and fetal activity and have not matured to the level of clinical use. We introduce a new wearable system based on a novel combination of accelerometers and bespoke acoustic sensors as well as an advanced signal processing architecture to identify and discriminate between types of fetal movements. We validate the system with concurrent ultrasound tests on a cohort of 44 pregnant women and demonstrate that the garment is capable of both detecting and discriminating the vigorous, whole-body 'startle' movements of a fetus. These results demonstrate the promise of multimodal sensing for the development of a low-cost, non-transmitting wearable monitor for fetal movements.

Quantitative time domain analysis of lifetime-based Förster resonant energy transfer measurements with fluorescent proteins: Static random isotropic fluorophore orientation distributions.

Förster resonant energy transfer (FRET) measurements are widely used to obtain information about molecular interactions and conformations through the dependence of FRET efficiency on the proximity of donor and acceptor fluorophores. Fluorescence lifetime measurements can provide quantitative analysis of FRET efficiency and interacting population fraction. Many FRET experiments exploit the highly specific labelling of genetically expressed fluorescent proteins, applicable in live cells and organisms. Unfortunately, the typical assumption of fast randomization of fluorophore orientations in the analysis of fluorescence lifetime-based FRET readouts is not valid for fluorescent proteins due to their slow rotational mobility compared to their upper state lifetime. Here, previous analysis of effectively static isotropic distributions of fluorophore dipoles on FRET measurements is incorporated into new software for fitting donor emission decay profiles. Calculated FRET parameters, including molar population fractions, are compared for the analysis of simulated and experimental FRET data under the assumption of static and dynamic fluorophores and the intermediate regimes between fully dynamic and static fluorophores, and mixtures within FRET pairs, is explored. Finally, a method to correct the artefact resulting from fitting the emission from static FRET pairs with isotropic angular distributions to the (incorrect) typically assumed dynamic FRET decay model is presented.

In vivo multiphoton microscopy using a handheld scanner with lateral and axial motion compensation.

This paper reports a handheld multiphoton fluorescence microscope designed for clinical imaging that incorporates axial motion compensation and lateral image stabilization. Spectral domain optical coherence tomography is employed to track the axial position of the skin surface, and lateral motion compensation is realised by imaging the speckle pattern arising from the optical coherence tomography beam illuminating the sample. Our system is able to correct lateral sample velocities of up to approximately 65 µm s-1 . Combined with the use of negative curvature microstructured optical fibre to deliver tunable ultrafast radiation to the handheld multiphoton scanner without the need of a dispersion compensation unit, this instrument has potential for a range of clinical applications. The system is used to compensate for both lateral and axial motion of the sample when imaging human skin in vivo.

Open Source High Content Analysis Utilizing Automated Fluorescence Lifetime Imaging Microscopy.

We present an open source high content analysis instrument utilizing automated fluorescence lifetime imaging (FLIM) for assaying protein interactions using Förster resonance energy transfer (FRET) based readouts of fixed or live cells in multiwell plates. This provides a means to screen for cell signaling processes read out using intramolecular FRET biosensors or intermolecular FRET of protein interactions such as oligomerization or heterodimerization, which can be used to identify binding partners. We describe here the functionality of this automated multiwell plate FLIM instrumentation and present exemplar data from our studies of HIV Gag protein oligomerization and a time course of a FRET biosensor in live cells. A detailed description of the practical implementation is then provided with reference to a list of hardware components and a description of the open source data acquisition software written in µManager. The application of FLIMfit, an open source MATLAB-based client for the OMERO platform, to analyze arrays of multiwell plate FLIM data is also presented. The protocols for imaging fixed and live cells are outlined and a demonstration of an automated multiwell plate FLIM experiment using cells expressing fluorescent protein-based FRET constructs is presented. This is complemented by a walk-through of the data analysis for this specific FLIM FRET data set.

Conformational transition of FGFR kinase activation revealed by site-specific unnatural amino acid reporter and single molecule FRET.

Protein kinases share significant structural similarity; however, structural features alone are insufficient to explain their diverse functions. Thus, bridging the gap between static structure and function requires a more detailed understanding of their dynamic properties. For example, kinase activation may occur via a switch-like mechanism or by shifting a dynamic equilibrium between inactive and active states. Here, we utilize a combination of FRET and molecular dynamics (MD) simulations to probe the activation mechanism of the kinase domain of Fibroblast Growth Factor Receptor (FGFR). Using genetically-encoded, site-specific incorporation of unnatural amino acids in regions essential for activation, followed by specific labeling with fluorescent moieties, we generated a novel class of FRET-based reporter to monitor conformational differences corresponding to states sampled by non phosphorylated/inactive and phosphorylated/active forms of the kinase. Single molecule FRET analysis in vitro, combined with MD simulations, shows that for FGFR kinase, there are populations of inactive and active states separated by a high free energy barrier resulting in switch-like activation. Compared to recent studies, these findings support diversity in features of kinases that impact on their activation mechanisms. The properties of these FRET-based constructs will also allow further studies of kinase dynamics as well as applications in vivo.

Quantitative in vivo optical tomography of cancer progression & vasculature development in adult zebrafish.

We describe a novel approach to study tumour progression and vasculature development in vivo via global 3-D fluorescence imaging of live non-pigmented adult zebrafish utilising angularly multiplexed optical projection tomography with compressive sensing (CS-OPT). This "mesoscopic" imaging method bridges a gap between established ~µm resolution 3-D fluorescence microscopy techniques and ~mm-resolved whole body planar imaging and diffuse tomography. Implementing angular multiplexing with CS-OPT, we demonstrate the in vivo global imaging of an inducible fluorescently labelled genetic model of liver cancer in adult non-pigmented zebrafish that also present fluorescently labelled vasculature. In this disease model, addition of a chemical inducer (doxycycline) drives expression of eGFP tagged oncogenic K-RASV12 in the liver of immune competent animals. We show that our novel in vivo global imaging methodology enables non-invasive quantitative imaging of the development of tumour and vasculature throughout the progression of the disease, which we have validated against established methods of pathology including immunohistochemistry. We have also demonstrated its potential for longitudinal imaging through a study of vascular development in the same zebrafish from early embryo to adulthood. We believe that this instrument, together with its associated analysis and data management tools, constitute a new platform for in vivo cancer studies and drug discovery in zebrafish disease models.

Visualising apoptosis in live zebrafish using fluorescence lifetime imaging with optical projection tomography to map FRET biosensor activity in space and time.

Fluorescence lifetime imaging (FLIM) combined with optical projection tomography (OPT) has the potential to map Förster resonant energy transfer (FRET) readouts in space and time in intact transparent or near transparent live organisms such as zebrafish larvae, thereby providing a means to visualise cell signalling processes in their physiological context. Here the first application of FLIM OPT to read out biological function in live transgenic zebrafish larvae using a genetically expressed FRET biosensor is reported. Apoptosis, or programmed cell death, is mapped in 3-D by imaging the activity of a FRET biosensor that is cleaved by Caspase 3, which is a key effector of apoptosis. Although apoptosis is a naturally occurring process during development, it can also be triggered in a variety of ways, including through gamma irradiation. FLIM OPT is shown here to enable apoptosis to be monitored over time, in live zebrafish larvae via changes in Caspase 3 activation following gamma irradiation at 24 hours post fertilisation. Significant apoptosis was observed at 3.5 hours post irradiation, predominantly in the head region.

Mesoscopic in vivo 3-D tracking of sparse cell populations using angular multiplexed optical projection tomography.

We describe an angular multiplexed imaging technique for 3-D in vivo cell tracking of sparse cell distributions and optical projection tomography (OPT) with superior time-lapse resolution and a significantly reduced light dose compared to volumetric time-lapse techniques. We demonstrate that using dual axis OPT, where two images are acquired simultaneously at different projection angles, can enable localization and tracking of features in 3-D with a time resolution equal to the camera frame rate. This is achieved with a 200x reduction in light dose compared to an equivalent volumetric time-lapse single camera OPT acquisition with 200 projection angles. We demonstrate the application of this technique to mapping the 3-D neutrophil migration pattern observed over ~25.5 minutes in a live 2 day post-fertilisation transgenic LysC:GFP zebrafish embryo following a tail wound.

Rapid global fitting of large fluorescence lifetime imaging microscopy datasets.

Fluorescence lifetime imaging (FLIM) is widely applied to obtain quantitative information from fluorescence signals, particularly using Förster Resonant Energy Transfer (FRET) measurements to map, for example, protein-protein interactions. Extracting FRET efficiencies or population fractions typically entails fitting data to complex fluorescence decay models but such experiments are frequently photon constrained, particularly for live cell or in vivo imaging, and this leads to unacceptable errors when analysing data on a pixel-wise basis. Lifetimes and population fractions may, however, be more robustly extracted using global analysis to simultaneously fit the fluorescence decay data of all pixels in an image or dataset to a multi-exponential model under the assumption that the lifetime components are invariant across the image (dataset). This approach is often considered to be prohibitively slow and/or computationally expensive but we present here a computationally efficient global analysis algorithm for the analysis of time-correlated single photon counting (TCSPC) or time-gated FLIM data based on variable projection. It makes efficient use of both computer processor and memory resources, requiring less than a minute to analyse time series and multiwell plate datasets with hundreds of FLIM images on standard personal computers. This lifetime analysis takes account of repetitive excitation, including fluorescence photons excited by earlier pulses contributing to the fit, and is able to accommodate time-varying backgrounds and instrument response functions. We demonstrate that this global approach allows us to readily fit time-resolved fluorescence data to complex models including a four-exponential model of a FRET system, for which the FRET efficiencies of the two species of a bi-exponential donor are linked, and polarisation-resolved lifetime data, where a fluorescence intensity and bi-exponential anisotropy decay model is applied to the analysis of live cell homo-FRET data. A software package implementing this algorithm, FLIMfit, is available under an open source licence through the Open Microscopy Environment.

Automated fluorescence lifetime imaging plate reader and its application to Förster resonant energy transfer readout of Gag protein aggregation.

Fluorescence lifetime measurements can provide quantitative readouts of local fluorophore environment and can be applied to biomolecular interactions via Förster resonant energy transfer (FRET). Fluorescence lifetime imaging (FLIM) can therefore provide a high content analysis (HCA) modality to map protein-protein interactions (PPIs) with applications in drug discovery, systems biology and basic research. We present here an automated multiwell plate reader able to perform rapid unsupervised optically sectioned FLIM of fixed and live biological samples and illustrate its potential to assay PPIs through application to Gag protein aggregation during the HIV life cycle. We demonstrate both hetero-FRET and homo-FRET readouts of protein aggregation and report the first quantitative evaluation of a FLIM HCA assay by generating dose response curves through addition of an inhibitor of Gag myristoylation. Z' factors exceeding 0.6 are realised for this FLIM FRET assay.

Multiphoton multispectral fluorescence lifetime tomography for the evaluation of basal cell carcinomas.

We present the first detailed study using multispectral multiphoton fluorescence lifetime imaging to differentiate basal cell carcinoma cells (BCCs) from normal keratinocytes. Images were acquired from 19 freshly excised BCCs and 27 samples of normal skin (in & ex vivo). Features from fluorescence lifetime images were used to discriminate BCCs with a sensitivity/specificity of 79%/93% respectively. A mosaic of BCC fluorescence lifetime images covering >1 mm(2) is also presented, demonstrating the potential for tumour margin delineation. Using 10,462 manually segmented cells from the image data, we quantify the cellular morphology and spectroscopic differences between BCCs and normal skin for the first time. Statistically significant increases were found in the fluorescence lifetimes of cells from BCCs in all spectral channels, ranging from 19.9% (425-515 nm spectral emission) to 39.8% (620-655 nm emission). A discriminant analysis based diagnostic algorithm allowed the fraction of cells classified as malignant to be calculated for each patient. This yielded a receiver operator characteristic area under the curve for the detection of BCC of 0.83. We have used both morphological and spectroscopic parameters to discriminate BCC from normal skin, and provide a comprehensive base for how this technique could be used for BCC assessment in clinical practice.

Quantification of cellular autofluorescence of human skin using multiphoton tomography and fluorescence lifetime imaging in two spectral detection channels.

We explore the diagnostic potential of imaging endogenous fluorophores using two photon microscopy and fluorescence lifetime imaging (FLIM) in human skin with two spectral detection channels. Freshly excised benign dysplastic nevi (DN) and malignant nodular Basal Cell Carcinomas (nBCCs) were excited at 760 nm. The resulting fluorescence signal was binned manually on a cell by cell basis. This improved the reliability of fitting using a double exponential decay model and allowed the fluorescence signatures from different cell populations within the tissue to be identified and studied. We also performed a direct comparison between different diagnostic groups. A statistically significant difference between the median mean fluorescence lifetime of 2.79 ns versus 2.52 ns (blue channel, 300-500 nm) and 2.08 ns versus 1.33 ns (green channel, 500-640 nm) was found between nBCCs and DN respectively, using the Mann-Whitney U test (p < 0.01). Further differences in the distribution of fluorescence lifetime parameters and inter-patient variability are also discussed.

FLIM FRET technology for drug discovery: automated multiwell-plate high-content analysis, multiplexed readouts and application in situ.

A fluorescence lifetime imaging (FLIM) technology platform intended to read out changes in Förster resonance energy transfer (FRET) efficiency is presented for the study of protein interactions across the drug-discovery pipeline. FLIM provides a robust, inherently ratiometric imaging modality for drug discovery that could allow the same sensor constructs to be translated from automated cell-based assays through small transparent organisms such as zebrafish to mammals. To this end, an automated FLIM multiwell-plate reader is described for high content analysis of fixed and live cells, tomographic FLIM in zebrafish and FLIM FRET of live cells via confocal endomicroscopy. For cell-based assays, an exemplar application reading out protein aggregation using FLIM FRET is presented, and the potential for multiple simultaneous FLIM (FRET) readouts in microscopy is illustrated.