Robust deep learning optical autofocus system applied to automated multiwell plate single molecule localization microscopy.

We presenta robust, long-range optical autofocus system for microscopy utilizing machine learning. This can be useful for experiments with long image data acquisition times that may be impacted by defocusing resulting from drift of components, for example due to changes in temperature or mechanical drift. It is also useful for automated slide scanning or multiwell plate imaging where the sample(s) to be imaged may not be in the same horizontal plane throughout the image data acquisition. To address the impact of (thermal or mechanical) fluctuations over time in the optical autofocus system itself, we utilize a convolutional neural network (CNN) that is trained over multiple days to account for such fluctuations. To address the trade-off between axial precision and range of the autofocus, we implement orthogonal optical readouts with separate CNN training data, thereby achieving an accuracy well within the 600 nm depth of field of our 1.3 numerical aperture objective lens over a defocus range of up to approximately +/-100 µm. We characterize the performance of this autofocus system and demonstrate its application to automated multiwell plate single molecule localization microscopy.

Many microscopy experiments involve extended imaging of samples over timescales from minutes to days, during which the microscope can 'drift' out of focus. When imaging at high magnification, the depth of field is of the order of one micron and so the imaging system should keep the sample in the focal plane of the microscope objective lens to this precision. Unfortunately, temperature changes in the laboratory can cause thermal expansion of microscope components that can move the focal plane by more than a micron and such changes can occur on a timescale of minutes. This is a particular issue for super-resolved microscopy experiments using single molecule localization microscopy (SMLM) techniques, for which 1000s of images are acquired, and for automated imaging of multiple samples in multiwell plates. It is possible to maintain the sample in the focal plane focus position by either automatically moving the sample or adjusting the imaging system, for example by moving the objective lens. This is called 'autofocus' and is frequently achieved by reflecting a light beam from the microscope coverslip and measuring its position of beam profile as a function of defocus of the microscope. The correcting adjustment is then usually calculated analytically but there is recent interest in using machine learning techniques to determine the required focussing adjustment. Here, we present a system that uses a neural network to determine the required defocus correcting adjustment from camera images of a laser beam that is reflected from the coverslip. Unfortunately, this approach will only work when the microscope is in the same condition as it was when the neural network was trained - and this can be compromised by the same drift of the optical system that causes the defocus needing to be corrected. We show, however, that by training a neural network over an extended period, for example 10 days, this approach can 'learn' about the optical system drifts and provide the required autofocus function. We also show that an optical system utilizing a rectangular slit can make two measurements of the defocus simultaneously, with one measurement being optimized for high accuracy over a limited range (±10 µm) near focus and the other providing lower accuracy but over a much longer range (±100 µm). This robust autofocus system is suitable for automated super-resolved microscopy of arrays of samples in a multiwell plate using SMLM, for which an experiment routinely lasts more than 5 h.

GPU-accelerated real-time reconstruction in Python of three-dimensional datasets from structured illumination microscopy with hexagonal patterns.

We present a structured illumination microscopy system that projects a hexagonal pattern by the interference among three coherent beams, suitable for implementation in a light-sheet geometry. Seven images acquired as the illumination pattern is shifted laterally can be processed to produce a super-resolved image that surpasses the diffraction-limited resolution by a factor of over 2 in an exemplar light-sheet arrangement. Three methods of processing data are discussed depending on whether the raw images are available in groups of seven, individually in a stream or as a larger batch representing a three-dimensional stack. We show that imaging axially moving samples can introduce artefacts, visible as fine structures in the processed images. However, these artefacts are easily removed by a filtering operation carried out as part of the batch processing algorithm for three-dimensional stacks. The reconstruction algorithms implemented in Python include specific optimizations for calculation on a graphics processing unit and we demonstrate its operation on experimental data of static objects and on simulated data of moving objects. We show that the software can process over 239 input raw frames per second at 512 × 512 pixels, generating over 34 super-resolved frames per second at 1024 × 1024 pixels. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 1)'.

Improving axial resolution in Structured Illumination Microscopy using deep learning.

Structured Illumination Microscopy (SIM) is a widespread methodology to image live and fixed biological structures smaller than the diffraction limits of conventional optical microscopy. Using recent advances in image up-scaling through deep learning models, we demonstrate a method to reconstruct 3D SIM image stacks with twice the axial resolution attainable through conventional SIM reconstructions. We further demonstrate our method is robust to noise and evaluate it against two-point cases and axial gratings. Finally, we discuss potential adaptions of the method to further improve resolution. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 1)'.

Multidimensional luminescence microscope for imaging defect colour centres in diamond.

We report a multidimensional luminescence microscope providing hyperspectral imaging and time-resolved (luminescence lifetime) imaging for the study of luminescent diamond defects. The instrument includes crossed-polariser white light transmission microscopy to reveal any birefringence that would indicate strain in the diamond lattice. We demonstrate the application of this new instrument to detect defects in natural and synthetic diamonds including N3, nitrogen and silicon vacancies. Hyperspectral imaging provides contrast that is not apparent in conventional intensity images and the luminescence lifetime provides further contrast.

All-optical crosstalk-free manipulation and readout of Chronos-expressing neurons.

All optical neurophysiology allows manipulation and readout of neural network activity with single-cell spatial resolution and millisecond temporal resolution. Neurons can be made to express proteins that actuate transmembrane currents upon light absorption, enabling optical control of membrane potential and action potential signalling. In addition, neurons can be genetically or synthetically labelled with fluorescent reporters of changes in intracellular calcium concentration or membrane potential. Thus, to optically manipulate and readout neural activity in parallel, two spectra are involved: the action spectrum of the actuator, and the absorption spectrum of the fluorescent reporter. Due to overlap in these spectra, previous all-optical neurophysiology paradigms have been hindered by spurious activation of neuronal activity caused by the readout light. Here, we pair the blue-green absorbing optogenetic actuator, Chronos, with a deep red-emitting fluorescent calcium reporter CaSiR-1. We show that cultured Chinese hamster ovary cells transfected with Chronos do not exhibit transmembrane currents when illuminated with wavelengths and intensities suitable for exciting one-photon CaSiR-1 fluorescence. We then demonstrate crosstalk-free, high signal-to-noise ratio CaSiR-1 red fluorescence imaging at 100 frames s-1 of Chronos-mediated calcium transients evoked in neurons with blue light pulses at rates up to 20 Hz. These results indicate that the spectral separation between red light excited fluorophores, excited efficiently at or above 640 nm, with blue-green absorbing opsins such as Chronos, is sufficient to avoid spurious opsin actuation by the imaging wavelengths and therefore enable crosstalk-free all-optical neuronal manipulation and readout.

easySLM-STED: Stimulated emission depletion microscopy with aberration correction, extended field of view and multiple beam scanning.

We demonstrate a simplified set-up for STED microscopy with a straightforward alignment procedure that uses a single spatial light modulator (SLM) with collinear incident excitation and depletion beams to provide phase modulation of the beam profiles and correction of optical aberrations. We show that this approach can be used to extend the field of view for STED microscopy by correcting chromatic aberration that otherwise leads to walk-off between the focused excitation and depletion beams. We further show how this arrangement can be adapted to increase the imaging speed through multibeam excitation and depletion. Fine adjustments to the alignment can be accomplished using the SLM only, conferring the potential for automation.

In vivo multiphoton microscopy using a handheld scanner with lateral and axial motion compensation.

This paper reports a handheld multiphoton fluorescence microscope designed for clinical imaging that incorporates axial motion compensation and lateral image stabilization. Spectral domain optical coherence tomography is employed to track the axial position of the skin surface, and lateral motion compensation is realised by imaging the speckle pattern arising from the optical coherence tomography beam illuminating the sample. Our system is able to correct lateral sample velocities of up to approximately 65 µm s-1 . Combined with the use of negative curvature microstructured optical fibre to deliver tunable ultrafast radiation to the handheld multiphoton scanner without the need of a dispersion compensation unit, this instrument has potential for a range of clinical applications. The system is used to compensate for both lateral and axial motion of the sample when imaging human skin in vivo.

Open Source High Content Analysis Utilizing Automated Fluorescence Lifetime Imaging Microscopy.

We present an open source high content analysis instrument utilizing automated fluorescence lifetime imaging (FLIM) for assaying protein interactions using Förster resonance energy transfer (FRET) based readouts of fixed or live cells in multiwell plates. This provides a means to screen for cell signaling processes read out using intramolecular FRET biosensors or intermolecular FRET of protein interactions such as oligomerization or heterodimerization, which can be used to identify binding partners. We describe here the functionality of this automated multiwell plate FLIM instrumentation and present exemplar data from our studies of HIV Gag protein oligomerization and a time course of a FRET biosensor in live cells. A detailed description of the practical implementation is then provided with reference to a list of hardware components and a description of the open source data acquisition software written in µManager. The application of FLIMfit, an open source MATLAB-based client for the OMERO platform, to analyze arrays of multiwell plate FLIM data is also presented. The protocols for imaging fixed and live cells are outlined and a demonstration of an automated multiwell plate FLIM experiment using cells expressing fluorescent protein-based FRET constructs is presented. This is complemented by a walk-through of the data analysis for this specific FLIM FRET data set.

Plasmonic Nanoprobes for Stimulated Emission Depletion Nanoscopy.

Plasmonic nanoparticles influence the absorption and emission processes of nearby emitters due to local enhancements of the illuminating radiation and the photonic density of states. Here, we use the plasmon resonance of metal nanoparticles in order to enhance the stimulated depletion of excited molecules for super-resolved nanoscopy. We demonstrate stimulated emission depletion (STED) nanoscopy with gold nanorods with a long axis of only 26 nm and a width of 8 nm. These particles provide an enhancement of up to 50% of the resolution compared to fluorescent-only probes without plasmonic components irradiated with the same depletion power. The nanoparticle-assisted STED probes reported here represent a ∼2 × 103 reduction in probe volume compared to previously used nanoparticles. Finally, we demonstrate their application toward plasmon-assisted STED cellular imaging at low-depletion powers, and we also discuss their current limitations.

Optically assembled droplet interface bilayer (OptiDIB) networks from cell-sized microdroplets.

We report a new platform technology to systematically assemble droplet interface bilayer (DIB) networks in user-defined 3D architectures from cell-sized droplets using optical tweezers. Our OptiDIB platform is the first demonstration of optical trapping to precisely construct 3D DIB networks, paving the way for the development of a new generation of modular bio-systems.

Adaptive multiphoton endomicroscopy through a dynamically deformed multicore optical fiber using proximal detection.

This paper demonstrates multiphoton excited fluorescence imaging through a polarisation maintaining multicore fiber (PM-MCF) while the fiber is dynamically deformed using all-proximal detection. Single-shot proximal measurement of the relative optical path lengths of all the cores of the PM-MCF in double pass is achieved using a Mach-Zehnder interferometer read out by a scientific CMOS camera operating at 416 Hz. A non-linear least squares fitting procedure is then employed to determine the deformation-induced lateral shift of the excitation spot at the distal tip of the PM-MCF. An experimental validation of this approach is presented that compares the proximally measured deformation-induced lateral shift in focal spot position to an independent distally measured ground truth. The proximal measurement of deformation-induced shift in focal spot position is applied to correct for deformation-induced shifts in focal spot position during raster-scanning multiphoton excited fluorescence imaging.

easySTORM: a robust, lower-cost approach to localisation and TIRF microscopy.

TIRF and STORM microscopy are super-resolving fluorescence imaging modalities for which current implementations on standard microscopes can present significant complexity and cost. We present a straightforward and low-cost approach to implement STORM and TIRF taking advantage of multimode optical fibres and multimode diode lasers to provide the required excitation light. Combined with open source software and relatively simple protocols to prepare samples for STORM, including the use of Vectashield for non-TIRF imaging, this approach enables TIRF and STORM imaging of cells labelled with appropriate dyes or expressing suitable fluorescent proteins to become widely accessible at low cost.

Tunable fibre-coupled multiphoton microscopy with a negative curvature fibre.

Negative curvature fibre (NCF) guides light in its core by inhibiting the coupling of core and cladding modes. In this work, an NCF was designed and fabricated to transmit ultrashort optical pulses for multiphoton microscopy with low group velocity dispersion (GVD) at 800 nm. Its attenuation was measured to be <0.3 dB m(-1) over the range 600-850 nm and the GVD was -180 ± 70 fs(2)  m(-1) at 800 nm. Using an average fibre output power of ∼20 mW and pulse repetition rate of 80 MHz, the NCF enabled pulses with a duration of <200 fs to be transmitted through a length of 1.5 m of fibre over a tuning range of 180 nm without the need for dispersion compensation. In a 4 m fibre, temporal and spectral pulse widths were maintained to within 10% of low power values up to the maximum fibre output power achievable with the laser system used of 278 mW at 700 nm, 808 mW at 800 nm and 420 mW at 860 nm. When coupled to a multiphoton microscope, it enabled imaging of ex vivo tissue using excitation wavelengths from 740 nm to 860 nm without any need for adjustments to the set-up.

Sonic hedgehog multimerization: a self-organizing event driven by post-translational modifications?

Sonic hedgehog (Shh) is a morphogen active during vertebrate development and tissue homeostasis in adulthood. Dysregulation of the Shh signalling pathway is known to incite carcinogenesis. Due to the highly lipophilic nature of this protein imparted by two post-translational modifications, Shh's method of transit through the aqueous extracellular milieu has been a long-standing conundrum, prompting the proposition of numerous hypotheses to explain the manner of its displacement from the surface of the producing cell. Detection of high molecular-weight complexes of Shh in the intercellular environment has indicated that the protein achieves this by accumulating into multimeric structures prior to release from producing cells. The mechanism of assembly of the multimers, however, has hitherto remained mysterious and contentious. Here, with the aid of high-resolution optical imaging and post-translational modification mutants of Shh, we show that the C-terminal cholesterol and the N-terminal palmitate adducts contribute to the assembly of large multimers and regulate their shape. Moreover, we show that small Shh multimers are produced in the absence of any lipid modifications. Based on an assessment of the distribution of various dimensional characteristics of individual Shh clusters, in parallel with deductions about the kinetics of release of the protein from the producing cells, we conclude that multimerization is driven by self-assembly underpinned by the law of mass action. We speculate that the lipid modifications augment the size of the multimolecular complexes through prolonging their association with the exoplasmic membrane.

Correction Approach for Delta Function Convolution Model Fitting of Fluorescence Decay Data in the Case of a Monoexponential Reference Fluorophore.

A correction is proposed to the Delta function convolution method (DFCM) for fitting a multiexponential decay model to time-resolved fluorescence decay data using a monoexponential reference fluorophore. A theoretical analysis of the discretised DFCM multiexponential decay function shows the presence an extra exponential decay term with the same lifetime as the reference fluorophore that we denote as the residual reference component. This extra decay component arises as a result of the discretised convolution of one of the two terms in the modified model function required by the DFCM. The effect of the residual reference component becomes more pronounced when the fluorescence lifetime of the reference is longer than all of the individual components of the specimen under inspection and when the temporal sampling interval is not negligible compared to the quantity (τR (-1) - τ(-1))(-1), where τR and τ are the fluorescence lifetimes of the reference and the specimen respectively. It is shown that the unwanted residual reference component results in systematic errors when fitting simulated data and that these errors are not present when the proposed correction is applied. The correction is also verified using real data obtained from experiment.

Analysis of DNA binding and nucleotide flipping kinetics using two-color two-photon fluorescence lifetime imaging microscopy.

Uracil DNA glycosylase plays a key role in DNA maintenance via base excision repair. Its role is to bind to DNA, locate unwanted uracil, and remove it using a base flipping mechanism. To date, kinetic analysis of this complex process has been achieved using stopped-flow analysis but, due to limitations in instrumental dead-times, discrimination of the "binding" and "base flipping" steps is compromised. Herein we present a novel approach for analyzing base flipping using a microfluidic mixer and two-color two-photon (2c2p) fluorescence lifetime imaging microscopy (FLIM). We demonstrate that 2c2p FLIM can simultaneously monitor binding and base flipping kinetics within the continuous flow microfluidic mixer, with results showing good agreement with computational fluid dynamics simulations.

Experimental proof of concept of nanoparticle-assisted STED.

We imaged core-shell nanoparticles, consisting of a dye-doped silica core covered with a layer of gold, with a stimulated emission depletion, fluorescence lifetime imaging (STED-FLIM) microscope. Because of the field enhancement provided by the localized surface plasmon resonance of the gold shell, we demonstrate a reduction of the STED depletion power required to obtain resolution improvement by a factor of 4. This validates the concept of nanoparticle-assisted STED (NP-STED), where hybrid dye-plasmonic nanoparticles are used as labels for STED in order to decrease the depletion powers required for subwavelength imaging.

The grab-and-drop protocol: a novel strategy for membrane protein isolation and reconstitution from single cells.

We present a rapid and robust technique for the sampling of membrane-associated proteins from the surface of a single, live cell and their subsequent deposition onto a solid-supported lipid bilayer. As a proof of principle, this method has been used to extract green fluorescent protein (EGFP) labelled K-ras proteins located at the inner leaflet of the plasma membrane of colon carcinoma cells and to transfer them to an S-layer supported lipid bilayer system. The technique is non-destructive, meaning that both the cell and proteins are intact after the sampling operation, offering the potential for repeated measurements of the same cell of interest. This system provides the ideal tool for the investigation of cellular heterogeneity, as well as a platform for the investigation of rare cell types such as circulating tumour cells.

Absolute quantification of protein copy number using a single-molecule-sensitive microarray.

We report the use of a microfluidic microarray incorporating single molecule detection for the absolute quantification of protein copy number in solution. In this paper we demonstrate protocols which enable calibration free detection for two protein detection assays. An EGFP protein assay has a limit of detection of <30 EGFP proteins in a microfluidic analysis chamber (limited by non-specific background binding), with a measured limit of linearity of approximately 6 × 10(6) molecules of analyte in the analysis chamber and a dynamic range of >5 orders of magnitude in protein concentration. An antibody sandwich assay was used to detect unlabelled human tumour suppressor protein p53 with a limit of detection of approximately 21 p53 proteins and a dynamic range of >3 orders of magnitude. We show that these protocols can be used to calibrate data retrospectively to determine the absolute protein copy number at the single cell level in two human cancer cell lines.

3-D stimulated emission depletion microscopy with programmable aberration correction.

We present a stimulated emission depletion (STED) microscope that provides 3-D super resolution by simultaneous depletion using beams with both a helical phase profile for enhanced lateral resolution and an annular phase profile to enhance axial resolution. The 3-D depletion point spread function is realised using a single spatial light modulator that can also be programmed to compensate for aberrations in the microscope and the sample. We apply it to demonstrate the first 3-D super-resolved imaging of an immunological synapse between a Natural Killer cell and its target cell.