Structured illumination microscopy with noise-controlled image reconstructions.

Super-resolution structured illumination microscopy (SIM) has become a widely used method for biological imaging. Standard reconstruction algorithms, however, are prone to generate noise-specific artifacts that limit their applicability for lower signal-to-noise data. Here we present a physically realistic noise model that explains the structured noise artifact, which we then use to motivate new complementary reconstruction approaches. True-Wiener-filtered SIM optimizes contrast given the available signal-to-noise ratio, and flat-noise SIM fully overcomes the structured noise artifact while maintaining resolving power. Both methods eliminate ad hoc user-adjustable reconstruction parameters in favor of physical parameters, enhancing objectivity. The new reconstructions point to a trade-off between contrast and a natural noise appearance. This trade-off can be partly overcome by further notch filtering but at the expense of a decrease in signal-to-noise ratio. The benefits of the proposed approaches are demonstrated on focal adhesion and tubulin samples in two and three dimensions, and on nanofabricated fluorescent test patterns.

Single image Fourier ring correlation.

We address resolution assessment for (light super-resolution) microscopy imaging. In modalities where imaging is not diffraction limited, correlation between two noise independent images is the standard way to infer the resolution. Here we take away the need for two noise independent images by computationally splitting one image acquisition into two noise independent realizations. This procedure generates two Poisson noise distributed images if the input is Poissonian distributed. As most modern cameras are shot-noise limited this procedure is directly applicable. However, also in the presence of readout noise we can compute the resolution faithfully via a correction factor. We evaluate our method on simulations and experimental data of widefield microscopy, STED microscopy, rescan confocal microscopy, image scanning microscopy, conventional confocal microscopy, and transmission electron microscopy. In all situations we find that using one image instead of two results in the same computed image resolution.

Localization microscopy at doubled precision with patterned illumination.

MINFLUX offers a breakthrough in single molecule localization precision, but is limited in field of view. Here we combine centroid estimation and illumination pattern induced photon count variations in a conventional widefield imaging setup to extract position information over a typical micrometer-sized field of view. We show a near two-fold improvement in precision over standard localization with the same photon count on DNA-origami nanostructures and tubulin in cells, using DNA-PAINT and STORM imaging.

Joint registration of multiple point clouds for fast particle fusion in localization microscopy.

SUMMARY: We present a fast particle fusion method for particles imaged with single-molecule localization microscopy. The state-of-the-art approach based on all-to-all registration has proven to work well but its computational cost scales unfavorably with the number of particles N, namely as N2. Our method overcomes this problem and achieves a linear scaling of computational cost with N by making use of the Joint Registration of Multiple Point Clouds (JRMPC) method. Straightforward application of JRMPC fails as mostly locally optimal solutions are found. These usually contain several overlapping clusters that each consist of well-aligned particles, but that have different poses. We solve this issue by repeated runs of JRMPC for different initial conditions, followed by a classification step to identify the clusters, and a connection step to link the different clusters obtained for different initializations. In this way a single well-aligned structure is obtained containing the majority of the particles. RESULTS: We achieve reconstructions of experimental DNA-origami datasets consisting of close to 400 particles within only 10 min on a CPU, with an image resolution of 3.2 nm. In addition, we show artifact-free reconstructions of symmetric structures without making any use of the symmetry. We also demonstrate that the method works well for poor data with a low density of labeling and for 3D data. AVAILABILITY AND IMPLEMENTATION: The code is available for download from https://github.com/wexw/Joint-Registration-of-Multiple-Point-Clouds-for-Fast-Particle-Fusion-in-Localization-Microscopy. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Precision in iterative modulation enhanced single-molecule localization microscopy.

Modulation enhanced single-molecule localization microscopy (meSMLM) methods improve the localization precision by using patterned illumination to encode additional position information. Iterative meSMLM (imeSMLM) methods iteratively generate prior information on emitter positions, used to locally improve the localization precision during subsequent iterations. The Cramér-Rao lower bound cannot incorporate prior information to bound the best achievable localization precision because it requires estimators to be unbiased. By treating estimands as random variables with a known prior distribution, the Van Trees inequality (VTI) can be used to bound the best possible localization precision of imeSMLM methods. An imeSMLM method is considered, where the positions of in-plane standing-wave illumination patterns are controlled over the course of multiple iterations. Using the VTI, we analytically approximate a lower bound on the maximum localization precision of imeSMLM methods that make use of standing-wave illumination patterns. In addition, we evaluate the maximally achievable localization precision for different illumination pattern placement strategies using Monte Carlo simulations. We show that in the absence of background and under perfect modulation, the information content of signal photons increases exponentially as a function of the iteration count. However, the information increase is no longer exponential as a function of the iteration count under non-zero background, imperfect modulation, or limited mechanical resolution of the illumination positioning system. As a result, imeSMLM with two iterations reaches at most a fivefold improvement over SMLM at 8 expected background photons per pixel and 95% modulation contrast. Moreover, the information increase from imeSMLM is balanced by a reduced signal photon rate. Therefore, SMLM outperforms imeSMLM when considering an equal measurement time and illumination power per iteration. Finally, the VTI is an excellent tool for the assessment of the performance of illumination control and is therefore the method of choice for optimal design and control of imeSMLM methods.

Axial accuracy in localization microscopy with 3D point spread function engineering.

Single-molecule localization microscopy has developed into a widely used technique to overcome the diffraction limit and enables 3D localization of single-emitters with nanometer precision. A widely used method to enable 3D encoding is to use a cylindrical lens or a phase mask to engineer the point spread function (PSF). The performance of these PSFs is often assessed by comparing the precision they achieve, ignoring accuracy. Nonetheless, accurate localization is required in many applications, such as multi-plane imaging, measuring and modelling of physical processes based on volumetric data, and 3D particle averaging. However, there are PSF model mismatches in the localization schemes due to how reference PSFs are obtained, look-up-tables are created, or spots are fitted. Currently there is little insight in how these model mismatches give rise to systematic axial localization errors, how large these errors are, and how to mitigate them. In this theoretical and simulation work we use a vector PSF model, which incorporates super-critical angle fluorescence (SAF) and the appropriate aplanatic correction factor, to analyze the errors in z-localization. We introduce theory for defining the focal plane in SAF conditions and analyze the predicted axial errors for an astigmatic PSF, double-helix PSF, and saddle-point PSF. These simulations indicate that the absolute axial biases can be as large as 140 nm, 250 nm, and 120 nm for the astigmatic, saddle-point, and double-helix PSF respectively, with relative errors of more than 50%. Finally, we discuss potential experimental methods to verify these findings and propose a workflow to mitigate these effects.

Template-free 2D particle fusion in localization microscopy.

Methods that fuse multiple localization microscopy images of a single structure can improve signal-to-noise ratio and resolution, but they generally suffer from template bias or sensitivity to registration errors. We present a template-free particle-fusion approach based on an all-to-all registration that provides robustness against individual misregistrations and underlabeling. We achieved 3.3-nm Fourier ring correlation (FRC) image resolution by fusing 383 DNA origami nanostructures with 80% labeling density, and 5.0-nm resolution for structures with 30% labeling density.

Waveguide-based total internal reflection fluorescence microscope enabling cellular imaging under cryogenic conditions.

Total internal reflection fluorescence (TIRF) microscopy is an important imaging tool for the investigation of biological structures, especially the study on cellular events near the plasma membrane. Imaging at cryogenic temperatures not only enables observing structures in a near-native and fixed state but also suppresses irreversible photo-bleaching rates, resulting in increased photo-stability of fluorophores. Traditional TIRF microscopes produce an evanescent field based on high numerical aperture immersion objective lenses with high magnification, which results in a limited field of view and is incompatible with cryogenic conditions. Here, we present a waveguide-based TIRF microscope, which is able to generate a uniform evanescent field using high refractive index waveguides on photonic chips and to obtain cellular observation at cryogenic temperatures. Our method provides an inexpensive way to achieve total-internal-reflection fluorescence imaging under cryogenic conditions.

Image based aberration retrieval using helical point spread functions.

A practical method for determining wavefront aberrations in optical systems based on the acquisition of an extended, unknown object is presented. The approach utilizes a conventional phase diversity approach in combination with a pupil-engineered, helical point spread function (PSF) to discriminate the aberrated PSF from the object features. The analysis of the image's power cepstrum enables an efficient retrieval of the aberration coefficients by solving a simple linear system of equations. An extensive Monte Carlo simulation is performed to demonstrate that the approach makes it possible to measure low-order Zernike modes including defocus, primary astigmatism, coma, and trefoil. The presented approach is tested experimentally by retrieving the two-dimensional aberration distribution of a test setup by imaging an extended, unknown scene.

Computational imaging modalities for multi-focal whole-slide imaging systems.

Whole-slide imaging systems can generate full-color image data of tissue slides efficiently, which are needed for digital pathology applications. This paper focuses on a scanner architecture that is based on a multi-line image sensor that is tilted with respect to the optical axis, such that every line of the sensor scans the tissue slide at a different focus level. This scanner platform is designed for imaging with continuous autofocus and inherent color registration at a throughput of the order of 400 MPx/s. Here, single-scan multi-focal whole-slide imaging, enabled by this platform, is explored. In particular, two computational imaging modalities based on multi-focal image data are studied. First, 3D imaging of thick absorption stained slides (∼60µm) is demonstrated in combination with deconvolution to ameliorate the inherently weak contrast in thick-tissue imaging. Second, quantitative phase tomography is demonstrated on unstained tissue slides and on fluorescently stained slides, revealing morphological features complementary to features made visible with conventional absorption or fluorescence stains. For both computational approaches simplified algorithms are proposed, targeted for straightforward parallel processing implementation at ∼GPx/s throughputs.

High-order-helix point spread functions for monocular three-dimensional imaging with superior aberration robustness.

An approach for designing purely refractive optical elements that generate engineered, multi-order-helix point spread functions (PSFs) with large peak separation for passive, optical depth measurement is presented. The influence of aberrations on the PSF's rotation angle, which limits the depth retrieval accuracy, is studied numerically and analytically. It appears that only Zernike modes with an azimuthal index that is an integer multiple of the number of PSF peaks introduce PSF rotation, and hence depth estimation errors. This implies that high-order-helix designs have superior robustness with respect to aberrations. This is experimentally demonstrated by imaging an extended scene in the presence of severe system aberrations using novel, cost-efficient phase elements based on UV-replication on the wafer-scale.

Impact of partial coherence on the apparent optical transfer function derived from the response to amplitude edges.

We present an investigation of the impact of partial coherence on optical imaging systems with the focus on whole slide imaging (WSI) systems for digital pathology. The investigation is based on the analysis of the edge response of the optical system, which gives rise to an apparent optical transfer function (OTF) that can be linked to two elementary complex functions Q and U. The function Q is directly related to the transmission cross coefficient (TCC) and can be identified with the performance function first introduced by Kintner and Sillitto. The function U depends on the TCC in a more involved way. When there are no aberrations the Q-function corresponds to the real part of the apparent OTF and the U function to the imaginary part of the apparent OTF. Close to the incoherent limit the effect of the U function is a mere shift of the edge compared to the fully incoherent case. We propose a new expression for the dependence of the depth of focus (DOF) on spatial frequency and on the partial coherence factor σ, and validate it by simulation. Partial coherence effects are investigated experimentally on a WSI system with a compact LED-based Köhler illumination unit with variable condenser NA. This unit incorporates a top hat diffuser for providing a reasonably uniform illumination field, with variations below 10% across the imaged field of view. The measurements of the apparent through-focus OTF derived from edges on a custom resolution chart for different σ were substantially in agreement with the simulations. Finding an optimal value for σ is not straightforward as lateral resolution and the level of edge ringing improve with increasing σ, whereas edge contrast and DOF improve with decreasing σ. We assess that the trade-off for the particular application of WSI systems for digital pathology is optimized for a σ value in the range of 0.55-0.75.

Measuring image resolution in optical nanoscopy.

Resolution in optical nanoscopy (or super-resolution microscopy) depends on the localization uncertainty and density of single fluorescent labels and on the sample's spatial structure. Currently there is no integral, practical resolution measure that accounts for all factors. We introduce a measure based on Fourier ring correlation (FRC) that can be computed directly from an image. We demonstrate its validity and benefits on two-dimensional (2D) and 3D localization microscopy images of tubulin and actin filaments. Our FRC resolution method makes it possible to compare achieved resolutions in images taken with different nanoscopy methods, to optimize and rank different emitter localization and labeling strategies, to define a stopping criterion for data acquisition, to describe image anisotropy and heterogeneity, and even to estimate the average number of localizations per emitter. Our findings challenge the current focus on obtaining the best localization precision, showing instead how the best image resolution can be achieved as fast as possible.

Fast multicolor 3D imaging using aberration-corrected multifocus microscopy.

Conventional acquisition of three-dimensional (3D) microscopy data requires sequential z scanning and is often too slow to capture biological events. We report an aberration-corrected multifocus microscopy method capable of producing an instant focal stack of nine 2D images. Appended to an epifluorescence microscope, the multifocus system enables high-resolution 3D imaging in multiple colors with single-molecule sensitivity, at speeds limited by the camera readout time of a single image.

Single shot three-dimensional imaging using an engineered point spread function.

A system approach to acquire a three-dimensional object distribution is presented using a compact and cost efficient camera system with an engineered point spread function. The corresponding monocular setup incorporates a phase-only computer-generated hologram in combination with a conventional imaging objective in order to optically encode the axial information within a single two-dimensional image. The object's depth map is calculated using a novel approach based on the power cepstrum of the image. The in-plane RGB image information is restored with an extended depth of focus by applying an adapted Wiener filter. The presented approach is tested experimentally by estimating the three-dimensional distribution of an extended passively illuminated scene.

Simultaneous measurement of emission color and 3D position of single molecules.

We show that the position of single molecules in all three spatial dimensions can be estimated alongside its emission color by diffractive optics based design of the Point Spread Function (PSF). The phase in a plane conjugate to the aperture stop of the objective lens is modified by a diffractive structure that splits the spot on the camera into closely spaced diffraction orders. The distance between and the size of these sub-spots are a measure of the emission color. Estimation of the axial position is enabled by imprinting aberrations such as astigmatism and defocus onto the orders. The overall spot shape is fitted with a fully vectorial PSF model. Proof-of-principle experiments on quantum dots indicate that a spectral precision of 10 to 20 nm, an axial localization precision of 25 to 50 nm, and a lateral localization precision of 10 to 30 nm can be achieved over a 1 µm range of axial positions for on average 800 signal photons and 17 background photons/pixel. The method appears to be rather sensitive to PSF model errors such as aberrations, giving in particular rise to biases in the fitted wavelength of up to 15 nm.

Deconvolution methods for structured illumination microscopy.

We compare two recently developed multiple-frame deconvolution approaches for the reconstruction of structured illumination microscopy (SIM) data: the pattern-illuminated Fourier ptychography algorithm (piFP) and the joint Richardson-Lucy deconvolution (jRL). The quality of the images reconstructed by these methods is compared in terms of the achieved resolution improvement, noise enhancement, and inherent artifacts. Furthermore, we study the issue of object-dependent resolution improvement by considering the modulation transfer functions derived from different types of objects. The performance of the considered methods is tested in experiments and benchmarked with a commercial SIM microscope. We find that the piFP method resolves periodic and isolated structures equally well, whereas the jRL method provides significantly higher resolution for isolated objects compared to periodic ones. Images reconstructed by the piFP and jRL algorithms are comparable to the images reconstructed using the generalized Wiener filter applied in most commercial SIM microscopes. An advantage of the discussed algorithms is that they allow the reconstruction of SIM images acquired under different types of illumination, such as multi-spot or random illumination.

Single molecule image formation, reconstruction and processing: introduction.

The ability to image at the single molecule scale has revolutionized research in molecular biology. This feature issue presents a collection of articles that provides new insights into the fundamental limits of single molecule imaging and reports novel techniques for image formation and analysis.

Studying different illumination patterns for resolution improvement in fluorescence microscopy.

Various types of non-uniform illumination can be used for resolution improvement in fluorescence microscopy. Here we study the differences between several types of incoherent illumination patterns, such as multi-spot, line and pseudo-random patterns. This requires an imaging setup and an image reconstruction algorithm that are flexible enough to incorporate any type of illumination pattern. We employ fluorescence microscope with structured illumination generated by a Digital Micro-mirror Device (DMD) and the pattern-illuminated Fourier Ptychography reconstruction algorithm (piFP) to this end. The piFP method is modified and improved by identifying the algorithm as steepest descent optimization of a least squares function. We find that illumination patterns with regular structure are superior to those with irregular structure in terms of resolution enhancement and noise level in the reconstructed images.