Robust deep learning optical autofocus system applied to automated multiwell plate single molecule localization microscopy.

We presenta robust, long-range optical autofocus system for microscopy utilizing machine learning. This can be useful for experiments with long image data acquisition times that may be impacted by defocusing resulting from drift of components, for example due to changes in temperature or mechanical drift. It is also useful for automated slide scanning or multiwell plate imaging where the sample(s) to be imaged may not be in the same horizontal plane throughout the image data acquisition. To address the impact of (thermal or mechanical) fluctuations over time in the optical autofocus system itself, we utilize a convolutional neural network (CNN) that is trained over multiple days to account for such fluctuations. To address the trade-off between axial precision and range of the autofocus, we implement orthogonal optical readouts with separate CNN training data, thereby achieving an accuracy well within the 600 nm depth of field of our 1.3 numerical aperture objective lens over a defocus range of up to approximately +/-100 µm. We characterize the performance of this autofocus system and demonstrate its application to automated multiwell plate single molecule localization microscopy.

Many microscopy experiments involve extended imaging of samples over timescales from minutes to days, during which the microscope can 'drift' out of focus. When imaging at high magnification, the depth of field is of the order of one micron and so the imaging system should keep the sample in the focal plane of the microscope objective lens to this precision. Unfortunately, temperature changes in the laboratory can cause thermal expansion of microscope components that can move the focal plane by more than a micron and such changes can occur on a timescale of minutes. This is a particular issue for super-resolved microscopy experiments using single molecule localization microscopy (SMLM) techniques, for which 1000s of images are acquired, and for automated imaging of multiple samples in multiwell plates. It is possible to maintain the sample in the focal plane focus position by either automatically moving the sample or adjusting the imaging system, for example by moving the objective lens. This is called 'autofocus' and is frequently achieved by reflecting a light beam from the microscope coverslip and measuring its position of beam profile as a function of defocus of the microscope. The correcting adjustment is then usually calculated analytically but there is recent interest in using machine learning techniques to determine the required focussing adjustment. Here, we present a system that uses a neural network to determine the required defocus correcting adjustment from camera images of a laser beam that is reflected from the coverslip. Unfortunately, this approach will only work when the microscope is in the same condition as it was when the neural network was trained - and this can be compromised by the same drift of the optical system that causes the defocus needing to be corrected. We show, however, that by training a neural network over an extended period, for example 10 days, this approach can 'learn' about the optical system drifts and provide the required autofocus function. We also show that an optical system utilizing a rectangular slit can make two measurements of the defocus simultaneously, with one measurement being optimized for high accuracy over a limited range (±10 µm) near focus and the other providing lower accuracy but over a much longer range (±100 µm). This robust autofocus system is suitable for automated super-resolved microscopy of arrays of samples in a multiwell plate using SMLM, for which an experiment routinely lasts more than 5 h.

GPU-accelerated real-time reconstruction in Python of three-dimensional datasets from structured illumination microscopy with hexagonal patterns.

We present a structured illumination microscopy system that projects a hexagonal pattern by the interference among three coherent beams, suitable for implementation in a light-sheet geometry. Seven images acquired as the illumination pattern is shifted laterally can be processed to produce a super-resolved image that surpasses the diffraction-limited resolution by a factor of over 2 in an exemplar light-sheet arrangement. Three methods of processing data are discussed depending on whether the raw images are available in groups of seven, individually in a stream or as a larger batch representing a three-dimensional stack. We show that imaging axially moving samples can introduce artefacts, visible as fine structures in the processed images. However, these artefacts are easily removed by a filtering operation carried out as part of the batch processing algorithm for three-dimensional stacks. The reconstruction algorithms implemented in Python include specific optimizations for calculation on a graphics processing unit and we demonstrate its operation on experimental data of static objects and on simulated data of moving objects. We show that the software can process over 239 input raw frames per second at 512 × 512 pixels, generating over 34 super-resolved frames per second at 1024 × 1024 pixels. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 1)'.

Improving axial resolution in Structured Illumination Microscopy using deep learning.

Structured Illumination Microscopy (SIM) is a widespread methodology to image live and fixed biological structures smaller than the diffraction limits of conventional optical microscopy. Using recent advances in image up-scaling through deep learning models, we demonstrate a method to reconstruct 3D SIM image stacks with twice the axial resolution attainable through conventional SIM reconstructions. We further demonstrate our method is robust to noise and evaluate it against two-point cases and axial gratings. Finally, we discuss potential adaptions of the method to further improve resolution. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 1)'.

All-optical crosstalk-free manipulation and readout of Chronos-expressing neurons.

All optical neurophysiology allows manipulation and readout of neural network activity with single-cell spatial resolution and millisecond temporal resolution. Neurons can be made to express proteins that actuate transmembrane currents upon light absorption, enabling optical control of membrane potential and action potential signalling. In addition, neurons can be genetically or synthetically labelled with fluorescent reporters of changes in intracellular calcium concentration or membrane potential. Thus, to optically manipulate and readout neural activity in parallel, two spectra are involved: the action spectrum of the actuator, and the absorption spectrum of the fluorescent reporter. Due to overlap in these spectra, previous all-optical neurophysiology paradigms have been hindered by spurious activation of neuronal activity caused by the readout light. Here, we pair the blue-green absorbing optogenetic actuator, Chronos, with a deep red-emitting fluorescent calcium reporter CaSiR-1. We show that cultured Chinese hamster ovary cells transfected with Chronos do not exhibit transmembrane currents when illuminated with wavelengths and intensities suitable for exciting one-photon CaSiR-1 fluorescence. We then demonstrate crosstalk-free, high signal-to-noise ratio CaSiR-1 red fluorescence imaging at 100 frames s-1 of Chronos-mediated calcium transients evoked in neurons with blue light pulses at rates up to 20 Hz. These results indicate that the spectral separation between red light excited fluorophores, excited efficiently at or above 640 nm, with blue-green absorbing opsins such as Chronos, is sufficient to avoid spurious opsin actuation by the imaging wavelengths and therefore enable crosstalk-free all-optical neuronal manipulation and readout.

Adaptive multiphoton endomicroscopy through a dynamically deformed multicore optical fiber using proximal detection.

This paper demonstrates multiphoton excited fluorescence imaging through a polarisation maintaining multicore fiber (PM-MCF) while the fiber is dynamically deformed using all-proximal detection. Single-shot proximal measurement of the relative optical path lengths of all the cores of the PM-MCF in double pass is achieved using a Mach-Zehnder interferometer read out by a scientific CMOS camera operating at 416 Hz. A non-linear least squares fitting procedure is then employed to determine the deformation-induced lateral shift of the excitation spot at the distal tip of the PM-MCF. An experimental validation of this approach is presented that compares the proximally measured deformation-induced lateral shift in focal spot position to an independent distally measured ground truth. The proximal measurement of deformation-induced shift in focal spot position is applied to correct for deformation-induced shifts in focal spot position during raster-scanning multiphoton excited fluorescence imaging.

Optically assembled droplet interface bilayer (OptiDIB) networks from cell-sized microdroplets.

We report a new platform technology to systematically assemble droplet interface bilayer (DIB) networks in user-defined 3D architectures from cell-sized droplets using optical tweezers. Our OptiDIB platform is the first demonstration of optical trapping to precisely construct 3D DIB networks, paving the way for the development of a new generation of modular bio-systems.

Sonic hedgehog multimerization: a self-organizing event driven by post-translational modifications?

Sonic hedgehog (Shh) is a morphogen active during vertebrate development and tissue homeostasis in adulthood. Dysregulation of the Shh signalling pathway is known to incite carcinogenesis. Due to the highly lipophilic nature of this protein imparted by two post-translational modifications, Shh's method of transit through the aqueous extracellular milieu has been a long-standing conundrum, prompting the proposition of numerous hypotheses to explain the manner of its displacement from the surface of the producing cell. Detection of high molecular-weight complexes of Shh in the intercellular environment has indicated that the protein achieves this by accumulating into multimeric structures prior to release from producing cells. The mechanism of assembly of the multimers, however, has hitherto remained mysterious and contentious. Here, with the aid of high-resolution optical imaging and post-translational modification mutants of Shh, we show that the C-terminal cholesterol and the N-terminal palmitate adducts contribute to the assembly of large multimers and regulate their shape. Moreover, we show that small Shh multimers are produced in the absence of any lipid modifications. Based on an assessment of the distribution of various dimensional characteristics of individual Shh clusters, in parallel with deductions about the kinetics of release of the protein from the producing cells, we conclude that multimerization is driven by self-assembly underpinned by the law of mass action. We speculate that the lipid modifications augment the size of the multimolecular complexes through prolonging their association with the exoplasmic membrane.

Correction Approach for Delta Function Convolution Model Fitting of Fluorescence Decay Data in the Case of a Monoexponential Reference Fluorophore.

A correction is proposed to the Delta function convolution method (DFCM) for fitting a multiexponential decay model to time-resolved fluorescence decay data using a monoexponential reference fluorophore. A theoretical analysis of the discretised DFCM multiexponential decay function shows the presence an extra exponential decay term with the same lifetime as the reference fluorophore that we denote as the residual reference component. This extra decay component arises as a result of the discretised convolution of one of the two terms in the modified model function required by the DFCM. The effect of the residual reference component becomes more pronounced when the fluorescence lifetime of the reference is longer than all of the individual components of the specimen under inspection and when the temporal sampling interval is not negligible compared to the quantity (τR (-1) - τ(-1))(-1), where τR and τ are the fluorescence lifetimes of the reference and the specimen respectively. It is shown that the unwanted residual reference component results in systematic errors when fitting simulated data and that these errors are not present when the proposed correction is applied. The correction is also verified using real data obtained from experiment.

Experimental proof of concept of nanoparticle-assisted STED.

We imaged core-shell nanoparticles, consisting of a dye-doped silica core covered with a layer of gold, with a stimulated emission depletion, fluorescence lifetime imaging (STED-FLIM) microscope. Because of the field enhancement provided by the localized surface plasmon resonance of the gold shell, we demonstrate a reduction of the STED depletion power required to obtain resolution improvement by a factor of 4. This validates the concept of nanoparticle-assisted STED (NP-STED), where hybrid dye-plasmonic nanoparticles are used as labels for STED in order to decrease the depletion powers required for subwavelength imaging.

Analysis of DNA binding and nucleotide flipping kinetics using two-color two-photon fluorescence lifetime imaging microscopy.

Uracil DNA glycosylase plays a key role in DNA maintenance via base excision repair. Its role is to bind to DNA, locate unwanted uracil, and remove it using a base flipping mechanism. To date, kinetic analysis of this complex process has been achieved using stopped-flow analysis but, due to limitations in instrumental dead-times, discrimination of the "binding" and "base flipping" steps is compromised. Herein we present a novel approach for analyzing base flipping using a microfluidic mixer and two-color two-photon (2c2p) fluorescence lifetime imaging microscopy (FLIM). We demonstrate that 2c2p FLIM can simultaneously monitor binding and base flipping kinetics within the continuous flow microfluidic mixer, with results showing good agreement with computational fluid dynamics simulations.

Remodelling of cortical actin where lytic granules dock at natural killer cell immune synapses revealed by super-resolution microscopy.

Natural Killer (NK) cells are innate immune cells that secrete lytic granules to directly kill virus-infected or transformed cells across an immune synapse. However, a major gap in understanding this process is in establishing how lytic granules pass through the mesh of cortical actin known to underlie the NK cell membrane. Research has been hampered by the resolution of conventional light microscopy, which is too low to resolve cortical actin during lytic granule secretion. Here we use two high-resolution imaging techniques to probe the synaptic organisation of NK cell receptors and filamentous (F)-actin. A combination of optical tweezers and live cell confocal microscopy reveals that microclusters of NKG2D assemble into a ring-shaped structure at the centre of intercellular synapses, where Vav1 and Grb2 also accumulate. Within this ring-shaped organisation of NK cell proteins, lytic granules accumulate for secretion. Using 3D-structured illumination microscopy (3D-SIM) to gain super-resolution of ~100 nm, cortical actin was detected in a central region of the NK cell synapse irrespective of whether activating or inhibitory signals dominate. Strikingly, the periodicity of the cortical actin mesh increased in specific domains at the synapse when the NK cell was activated. Two-colour super-resolution imaging revealed that lytic granules docked precisely in these domains which were also proximal to where the microtubule-organising centre (MTOC) polarised. Together, these data demonstrate that remodelling of the cortical actin mesh occurs at the central region of the cytolytic NK cell immune synapse. This is likely to occur for other types of cell secretion and also emphasises the importance of emerging super-resolution imaging technology for revealing new biology.

Absolute quantification of protein copy number using a single-molecule-sensitive microarray.

We report the use of a microfluidic microarray incorporating single molecule detection for the absolute quantification of protein copy number in solution. In this paper we demonstrate protocols which enable calibration free detection for two protein detection assays. An EGFP protein assay has a limit of detection of <30 EGFP proteins in a microfluidic analysis chamber (limited by non-specific background binding), with a measured limit of linearity of approximately 6 × 10(6) molecules of analyte in the analysis chamber and a dynamic range of >5 orders of magnitude in protein concentration. An antibody sandwich assay was used to detect unlabelled human tumour suppressor protein p53 with a limit of detection of approximately 21 p53 proteins and a dynamic range of >3 orders of magnitude. We show that these protocols can be used to calibrate data retrospectively to determine the absolute protein copy number at the single cell level in two human cancer cell lines.

The grab-and-drop protocol: a novel strategy for membrane protein isolation and reconstitution from single cells.

We present a rapid and robust technique for the sampling of membrane-associated proteins from the surface of a single, live cell and their subsequent deposition onto a solid-supported lipid bilayer. As a proof of principle, this method has been used to extract green fluorescent protein (EGFP) labelled K-ras proteins located at the inner leaflet of the plasma membrane of colon carcinoma cells and to transfer them to an S-layer supported lipid bilayer system. The technique is non-destructive, meaning that both the cell and proteins are intact after the sampling operation, offering the potential for repeated measurements of the same cell of interest. This system provides the ideal tool for the investigation of cellular heterogeneity, as well as a platform for the investigation of rare cell types such as circulating tumour cells.

Double pass, common path method for arbitrary polarization control using a ferroelectric liquid crystal spatial light modulator.

We present a method for arbitrary control of the polarization of a light beam. Our method uses two holograms on a binary ferroelectric liquid crystal spatial light modulator (FLCSLM), and so has the potential to allow polarization state switching at kilohertz rates. Unlike previous methods that achieve polarization control using FLCSLMs, our method is common path and requires only the simplest optical components. For this reason, the method is very easy to setup, align, and maintain. In addition, it has the ability to modulate unpolarized input light. We demonstrate the formation of radially, azimuthally, and circularly polarized beams by imaging their focal spots formed at low numerical aperture.

Application of ultrafast gold luminescence to measuring the instrument response function for multispectral multiphoton fluorescence lifetime imaging.

When performing multiphoton fluorescence lifetime imaging in multiple spectral emission channels, an instrument response function must be acquired in each channel if accurate measurements of complex fluorescence decays are to be performed. Although this can be achieved using the reference reconvolution technique, it is difficult to identify suitable fluorophores with a mono-exponential fluorescence decay across a broad emission spectrum. We present a solution to this problem by measuring the IRF using the ultrafast luminescence from gold nanorods. We show that ultrafast gold nanorod luminescence allows the IRF to be directly obtained in multiple spectral channels simultaneously across a wide spectral range. We validate this approach by presenting an analysis of multispectral autofluorescence FLIM data obtained from human skin ex vivo.

Interferometry using binary holograms without high order diffraction effects.

We describe a technique for a phase-stepping interferometer based on programmable binary phase holograms, particularly useful for optical testing of aspheric or free-form surfaces. It is well-known that binary holograms can be used to generate reference surfaces for interferometry, but a major problem is that cross talk from higher diffraction orders and aliasing can reduce the fidelity of the system. Here, we propose a new encoding technique which improves the accuracy of the technique and demonstrate its implementation using a binary liquid crystal spatial light modulator.

Adaptive phase compensation for ultracompact laser scanning endomicroscopy.

We present an approach to laser scanning endomicroscopy that requires no moving parts and can be implemented with no distal scanners or optics, permitting extremely compact endoscopic probes to be developed. Our approach utilizes a spatial light modulator to correct for phase variations across a fiber imaging bundle and to encode for arbitrary wavefronts at the distal end of the fiber bundle. Thus, it is possible to realize both focusing and beam scanning at the output of the fiber bundle with no distal components. We present proof of principle results to illustrate three-dimensional scanning of the focal spot and exemplar images of a United States Air Force resolution test chart.

Microclusters of inhibitory killer immunoglobulin-like receptor signaling at natural killer cell immunological synapses.

We report the supramolecular organization of killer Ig-like receptor (KIR) phosphorylation using a technique applicable to imaging phosphorylation of any green fluorescent protein-tagged receptor at an intercellular contact or immune synapse. Specifically, we use fluorescence lifetime imaging (FLIM) to report Förster resonance energy transfer (FRET) between GFP-tagged KIR2DL1 and a Cy3-tagged generic anti-phosphotyrosine monoclonal antibody. Visualization of KIR phosphorylation in natural killer (NK) cells contacting target cells expressing cognate major histocompatibility complex class I proteins revealed that inhibitory signaling is spatially restricted to the immune synapse. This explains how NK cells respond appropriately when simultaneously surveying susceptible and resistant target cells. More surprising, phosphorylated KIR was confined to microclusters within the aggregate of KIR, contrary to an expected homogeneous distribution of KIR signaling across the immune synapse. Also, yellow fluorescent protein-tagged Lck, a kinase important for KIR phosphorylation, accumulated in a multifocal distribution at inhibitory synapses. Spatial confinement of receptor phosphorylation within the immune synapse may be critical to how activating and inhibitory signals are integrated in NK cells.

FLIM FRET technology for drug discovery: automated multiwell-plate high-content analysis, multiplexed readouts and application in situ.

A fluorescence lifetime imaging (FLIM) technology platform intended to read out changes in Förster resonance energy transfer (FRET) efficiency is presented for the study of protein interactions across the drug-discovery pipeline. FLIM provides a robust, inherently ratiometric imaging modality for drug discovery that could allow the same sensor constructs to be translated from automated cell-based assays through small transparent organisms such as zebrafish to mammals. To this end, an automated FLIM multiwell-plate reader is described for high content analysis of fixed and live cells, tomographic FLIM in zebrafish and FLIM FRET of live cells via confocal endomicroscopy. For cell-based assays, an exemplar application reading out protein aggregation using FLIM FRET is presented, and the potential for multiple simultaneous FLIM (FRET) readouts in microscopy is illustrated.

High plasma membrane lipid order imaged at the immunological synapse periphery in live T cells.

Cholesterol- and glycosphingolipid-enriched membrane lipid microdomains, frequently called lipid rafts, are thought to play an important role in the spatial and temporal organization of immunological synapses. Higher ordering of lipid acyl chains was suggested for these entities and imaging of membrane order in living cells during activation can therefore help to understand the mechanisms responsible for the supramolecular organization of molecules involved in the activation of T cells. Here, we employ the phase-sensitive membrane dye di-4-ANEPPDHQ together with a variety of spectrally-resolved microscopy techniques, including 2-channel ratiometric TIRF microscopy and fluorescence lifetime imaging, to characterize membrane order at the T cell immunological synapse at high spatial and temporal resolution in live cells at physiological temperature. We find that higher membrane order resides at the immunological synapse periphery where proximal signalling through the immunoreceptors and accessory proteins in microclusters has previously been shown to take place. The observed spatial patterning of membrane order in the immunological synapse depends on active receptor signalling.