Single image Fourier ring correlation.

We address resolution assessment for (light super-resolution) microscopy imaging. In modalities where imaging is not diffraction limited, correlation between two noise independent images is the standard way to infer the resolution. Here we take away the need for two noise independent images by computationally splitting one image acquisition into two noise independent realizations. This procedure generates two Poisson noise distributed images if the input is Poissonian distributed. As most modern cameras are shot-noise limited this procedure is directly applicable. However, also in the presence of readout noise we can compute the resolution faithfully via a correction factor. We evaluate our method on simulations and experimental data of widefield microscopy, STED microscopy, rescan confocal microscopy, image scanning microscopy, conventional confocal microscopy, and transmission electron microscopy. In all situations we find that using one image instead of two results in the same computed image resolution.

Detecting continuous structural heterogeneity in single-molecule localization microscopy data.

Fusion of multiple chemically identical complexes, so-called particles, in localization microscopy, can improve the signal-to-noise ratio and overcome under-labeling. To this end, structural homogeneity of the data must be assumed. Biological heterogeneity, however, could be present in the data originating from distinct conformational variations or (continuous) variations in particle shapes. We present a prior-knowledge-free method for detecting continuous structural variations with localization microscopy. Detecting this heterogeneity leads to more faithful fusions and reconstructions of the localization microscopy data as their heterogeneity is taken into account. In experimental datasets, we show the continuous variation of the height of DNA origami tetrahedrons imaged with 3D PAINT and of the radius of Nuclear Pore Complexes imaged in 2D with STORM. In simulation, we study the impact on the heterogeneity detection pipeline of Degree Of Labeling and of structural variations in the form of two independent modes.

Particle fusion of super-resolution data reveals the unit structure of Nup96 in Nuclear Pore Complex.

Single molecule localization microscopy offers resolution nearly down to the molecular level with specific molecular labelling, and is thereby a promising tool for structural biology. In practice, however, the actual value to this field is limited primarily by incomplete fluorescent labelling of the structure. This missing information can be completed by merging information from many structurally identical particles in a particle fusion approach similar to cryo-EM single-particle analysis. In this paper, we present a data analysis of particle fusion results of fluorescently labelled Nup96 nucleoporins in the Nuclear Pore Complex to show that Nup96 occurs in a spatial arrangement of two rings of 8 units with two Nup96 copies per unit giving a total of 32 Nup96 copies per pore. We use Artificial Intelligence assisted modeling in Alphafold to extend the existing cryo-EM model of Nup96 to accurately pinpoint the positions of the fluorescent labels and show the accuracy of the match between fluorescent and cryo-EM data to be better than 3 nm in-plane and 5 nm out-of-plane.

Electron-beam patterned calibration structures for structured illumination microscopy.

Super-resolution fluorescence microscopy can be achieved by image reconstruction after spatially patterned illumination or sequential photo-switching and read-out. Reconstruction algorithms and microscope performance are typically tested using simulated image data, due to a lack of strategies to pattern complex fluorescent patterns with nanoscale dimension control. Here, we report direct electron-beam patterning of fluorescence nanopatterns as calibration standards for super-resolution fluorescence. Patterned regions are identified with both electron microscopy and fluorescence labelling of choice, allowing precise correlation of predefined pattern dimensions, a posteriori obtained electron images, and reconstructed super-resolution images.

Axial accuracy in localization microscopy with 3D point spread function engineering.

Single-molecule localization microscopy has developed into a widely used technique to overcome the diffraction limit and enables 3D localization of single-emitters with nanometer precision. A widely used method to enable 3D encoding is to use a cylindrical lens or a phase mask to engineer the point spread function (PSF). The performance of these PSFs is often assessed by comparing the precision they achieve, ignoring accuracy. Nonetheless, accurate localization is required in many applications, such as multi-plane imaging, measuring and modelling of physical processes based on volumetric data, and 3D particle averaging. However, there are PSF model mismatches in the localization schemes due to how reference PSFs are obtained, look-up-tables are created, or spots are fitted. Currently there is little insight in how these model mismatches give rise to systematic axial localization errors, how large these errors are, and how to mitigate them. In this theoretical and simulation work we use a vector PSF model, which incorporates super-critical angle fluorescence (SAF) and the appropriate aplanatic correction factor, to analyze the errors in z-localization. We introduce theory for defining the focal plane in SAF conditions and analyze the predicted axial errors for an astigmatic PSF, double-helix PSF, and saddle-point PSF. These simulations indicate that the absolute axial biases can be as large as 140 nm, 250 nm, and 120 nm for the astigmatic, saddle-point, and double-helix PSF respectively, with relative errors of more than 50%. Finally, we discuss potential experimental methods to verify these findings and propose a workflow to mitigate these effects.

Photon efficient orientation estimation using polarization modulation in single-molecule localization microscopy.

Combining orientation estimation with localization microscopy opens up the possibility to analyze the underlying orientation of biomolecules on the nanometer scale. Inspired by the recent improvement of the localization precision by shifting excitation patterns (MINFLUX, SIMFLUX), we have adapted the idea towards the modulation of excitation polarization to enhance the orientation precision. For this modality two modes are analyzed: i) normally incident excitation with three polarization steps to retrieve the in-plane angle of emitters and ii) obliquely incident excitation with p-polarization with five different azimuthal angles of incidence to retrieve the full orientation. Firstly, we present a theoretical study of the lower precision limit with a Cramér-Rao bound for these modes. For the oblique incidence mode we find a favorable isotropic orientation precision for all molecular orientations if the polar angle of incidence is equal to arccos ⁡ 2 / 3 ≈ 35 degrees. Secondly, a simulation study is performed to assess the performance for low signal-to-background ratios and how inaccurate illumination polarization angles affect the outcome. We show that a precision, at the Cramér-Rao bound (CRB) limit, of just 2.4 and 1.6 degrees in the azimuthal and polar angles can be achieved with only 1000 detected signal photons and 10 background photons per pixel (about twice better than reported earlier). Lastly, the alignment and calibration of an optical microscope with polarization control is described in detail. With this microscope a proof-of-principle experiment is carried out, demonstrating an experimental in-plane precision close to the CRB limit for signal photon counts ranging from 400 to 10,000.

Joint registration of multiple point clouds for fast particle fusion in localization microscopy.

SUMMARY: We present a fast particle fusion method for particles imaged with single-molecule localization microscopy. The state-of-the-art approach based on all-to-all registration has proven to work well but its computational cost scales unfavorably with the number of particles N, namely as N2. Our method overcomes this problem and achieves a linear scaling of computational cost with N by making use of the Joint Registration of Multiple Point Clouds (JRMPC) method. Straightforward application of JRMPC fails as mostly locally optimal solutions are found. These usually contain several overlapping clusters that each consist of well-aligned particles, but that have different poses. We solve this issue by repeated runs of JRMPC for different initial conditions, followed by a classification step to identify the clusters, and a connection step to link the different clusters obtained for different initializations. In this way a single well-aligned structure is obtained containing the majority of the particles. RESULTS: We achieve reconstructions of experimental DNA-origami datasets consisting of close to 400 particles within only 10 min on a CPU, with an image resolution of 3.2 nm. In addition, we show artifact-free reconstructions of symmetric structures without making any use of the symmetry. We also demonstrate that the method works well for poor data with a low density of labeling and for 3D data. AVAILABILITY AND IMPLEMENTATION: The code is available for download from https://github.com/wexw/Joint-Registration-of-Multiple-Point-Clouds-for-Fast-Particle-Fusion-in-Localization-Microscopy. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Precision in iterative modulation enhanced single-molecule localization microscopy.

Modulation enhanced single-molecule localization microscopy (meSMLM) methods improve the localization precision by using patterned illumination to encode additional position information. Iterative meSMLM (imeSMLM) methods iteratively generate prior information on emitter positions, used to locally improve the localization precision during subsequent iterations. The Cramér-Rao lower bound cannot incorporate prior information to bound the best achievable localization precision because it requires estimators to be unbiased. By treating estimands as random variables with a known prior distribution, the Van Trees inequality (VTI) can be used to bound the best possible localization precision of imeSMLM methods. An imeSMLM method is considered, where the positions of in-plane standing-wave illumination patterns are controlled over the course of multiple iterations. Using the VTI, we analytically approximate a lower bound on the maximum localization precision of imeSMLM methods that make use of standing-wave illumination patterns. In addition, we evaluate the maximally achievable localization precision for different illumination pattern placement strategies using Monte Carlo simulations. We show that in the absence of background and under perfect modulation, the information content of signal photons increases exponentially as a function of the iteration count. However, the information increase is no longer exponential as a function of the iteration count under non-zero background, imperfect modulation, or limited mechanical resolution of the illumination positioning system. As a result, imeSMLM with two iterations reaches at most a fivefold improvement over SMLM at 8 expected background photons per pixel and 95% modulation contrast. Moreover, the information increase from imeSMLM is balanced by a reduced signal photon rate. Therefore, SMLM outperforms imeSMLM when considering an equal measurement time and illumination power per iteration. Finally, the VTI is an excellent tool for the assessment of the performance of illumination control and is therefore the method of choice for optimal design and control of imeSMLM methods.

Waveguide-based total internal reflection fluorescence microscope enabling cellular imaging under cryogenic conditions.

Total internal reflection fluorescence (TIRF) microscopy is an important imaging tool for the investigation of biological structures, especially the study on cellular events near the plasma membrane. Imaging at cryogenic temperatures not only enables observing structures in a near-native and fixed state but also suppresses irreversible photo-bleaching rates, resulting in increased photo-stability of fluorophores. Traditional TIRF microscopes produce an evanescent field based on high numerical aperture immersion objective lenses with high magnification, which results in a limited field of view and is incompatible with cryogenic conditions. Here, we present a waveguide-based TIRF microscope, which is able to generate a uniform evanescent field using high refractive index waveguides on photonic chips and to obtain cellular observation at cryogenic temperatures. Our method provides an inexpensive way to achieve total-internal-reflection fluorescence imaging under cryogenic conditions.

Simultaneous orientation and 3D localization microscopy with a Vortex point spread function.

Estimating the orientation and 3D position of rotationally constrained emitters with localization microscopy typically requires polarization splitting or a large engineered Point Spread Function (PSF). Here we utilize a compact modified PSF for single molecule emitter imaging to estimate simultaneously the 3D position, dipole orientation, and degree of rotational constraint from a single 2D image. We use an affordable and commonly available phase plate, normally used for STED microscopy in the excitation light path, to alter the PSF in the emission light path. This resulting Vortex PSF does not require polarization splitting and has a compact PSF size, making it easy to implement and combine with localization microscopy techniques. In addition to a vectorial PSF fitting routine we calibrate for field-dependent aberrations which enables orientation and position estimation within 30% of the Cramér-Rao bound limit over a 66 µm field of view. We demonstrate this technique on reorienting single molecules adhered to the cover slip, λ-DNA with DNA intercalators using binding-activated localization microscopy, and we reveal periodicity on intertwined structures on supercoiled DNA.

Structured illumination microscopy with noise-controlled image reconstructions.

Super-resolution structured illumination microscopy (SIM) has become a widely used method for biological imaging. Standard reconstruction algorithms, however, are prone to generate noise-specific artifacts that limit their applicability for lower signal-to-noise data. Here we present a physically realistic noise model that explains the structured noise artifact, which we then use to motivate new complementary reconstruction approaches. True-Wiener-filtered SIM optimizes contrast given the available signal-to-noise ratio, and flat-noise SIM fully overcomes the structured noise artifact while maintaining resolving power. Both methods eliminate ad hoc user-adjustable reconstruction parameters in favor of physical parameters, enhancing objectivity. The new reconstructions point to a trade-off between contrast and a natural noise appearance. This trade-off can be partly overcome by further notch filtering but at the expense of a decrease in signal-to-noise ratio. The benefits of the proposed approaches are demonstrated on focal adhesion and tubulin samples in two and three dimensions, and on nanofabricated fluorescent test patterns.

Detecting structural heterogeneity in single-molecule localization microscopy data.

Particle fusion for single molecule localization microscopy improves signal-to-noise ratio and overcomes underlabeling, but ignores structural heterogeneity or conformational variability. We present a-priori knowledge-free unsupervised classification of structurally different particles employing the Bhattacharya cost function as dissimilarity metric. We achieve 96% classification accuracy on mixtures of up to four different DNA-origami structures, detect rare classes of origami occuring at 2% rate, and capture variation in ellipticity of nuclear pore complexes.

3D particle averaging and detection of macromolecular symmetry in localization microscopy.

Single molecule localization microscopy offers in principle resolution down to the molecular level, but in practice this is limited primarily by incomplete fluorescent labeling of the structure. This missing information can be completed by merging information from many structurally identical particles. In this work, we present an approach for 3D single particle analysis in localization microscopy which hugely increases signal-to-noise ratio and resolution and enables determining the symmetry groups of macromolecular complexes. Our method does not require a structural template, and handles anisotropic localization uncertainties. We demonstrate 3D reconstructions of DNA-origami tetrahedrons, Nup96 and Nup107 subcomplexes of the nuclear pore complex acquired using multiple single molecule localization microscopy techniques, with their structural symmetry deducted from the data.

High-speed multicolor structured illumination microscopy using a hexagonal single mode fiber array.

Structured illumination microscopy (SIM) is a widely used imaging technique that doubles the effective resolution of widefield microscopes. Most current implementations rely on diffractive elements, either gratings or programmable devices, to generate structured light patterns in the sample. These can be limited by spectral efficiency, speed, or both. Here we introduce the concept of fiber SIM that allows for camera frame rate limited pattern generation and manipulation over a broad wavelength range. Illumination patterns are generated by coupling laser beams into radially opposite pairs of fibers in a hexagonal single mode fiber array where the exit beams are relayed to the microscope objective's back focal plane. The phase stepping and rotation of the illumination patterns are controlled by fast electro-optic devices. We achieved a rate of 111 SIM frames per second and imaged with excitation patterns generated by both 488 nm and 532 nm lasers.

Automated detection of cribriform growth patterns in prostate histology images.

Cribriform growth patterns in prostate carcinoma are associated with poor prognosis. We aimed to introduce a deep learning method to detect such patterns automatically. To do so, convolutional neural network was trained to detect cribriform growth patterns on 128 prostate needle biopsies. Ensemble learning taking into account other tumor growth patterns during training was used to cope with heterogeneous and limited tumor tissue occurrences. ROC and FROC analyses were applied to assess network performance regarding detection of biopsies harboring cribriform growth pattern. The ROC analysis yielded a mean area under the curve up to 0.81. FROC analysis demonstrated a sensitivity of 0.9 for regions larger than [Formula: see text] with on average 7.5 false positives. To benchmark method performance for intra-observer annotation variability, false positive and negative detections were re-evaluated by the pathologists. Pathologists considered 9% of the false positive regions as cribriform, and 11% as possibly cribriform; 44% of the false negative regions were not annotated as cribriform. As a final experiment, the network was also applied on a dataset of 60 biopsy regions annotated by 23 pathologists. With the cut-off reaching highest sensitivity, all images annotated as cribriform by at least 7/23 of the pathologists, were all detected as cribriform by the network and 9/60 of the images were detected as cribriform whereas no pathologist labelled them as such. In conclusion, the proposed deep learning method has high sensitivity for detecting cribriform growth patterns at the expense of a limited number of false positives. It can detect cribriform regions that are labelled as such by at least a minority of pathologists. Therefore, it could assist clinical decision making by suggesting suspicious regions.

Image based aberration retrieval using helical point spread functions.

A practical method for determining wavefront aberrations in optical systems based on the acquisition of an extended, unknown object is presented. The approach utilizes a conventional phase diversity approach in combination with a pupil-engineered, helical point spread function (PSF) to discriminate the aberrated PSF from the object features. The analysis of the image's power cepstrum enables an efficient retrieval of the aberration coefficients by solving a simple linear system of equations. An extensive Monte Carlo simulation is performed to demonstrate that the approach makes it possible to measure low-order Zernike modes including defocus, primary astigmatism, coma, and trefoil. The presented approach is tested experimentally by retrieving the two-dimensional aberration distribution of a test setup by imaging an extended, unknown scene.

Computational imaging modalities for multi-focal whole-slide imaging systems.

Whole-slide imaging systems can generate full-color image data of tissue slides efficiently, which are needed for digital pathology applications. This paper focuses on a scanner architecture that is based on a multi-line image sensor that is tilted with respect to the optical axis, such that every line of the sensor scans the tissue slide at a different focus level. This scanner platform is designed for imaging with continuous autofocus and inherent color registration at a throughput of the order of 400 MPx/s. Here, single-scan multi-focal whole-slide imaging, enabled by this platform, is explored. In particular, two computational imaging modalities based on multi-focal image data are studied. First, 3D imaging of thick absorption stained slides (∼60µm) is demonstrated in combination with deconvolution to ameliorate the inherently weak contrast in thick-tissue imaging. Second, quantitative phase tomography is demonstrated on unstained tissue slides and on fluorescently stained slides, revealing morphological features complementary to features made visible with conventional absorption or fluorescence stains. For both computational approaches simplified algorithms are proposed, targeted for straightforward parallel processing implementation at ∼GPx/s throughputs.

25th Anniversary of STED Microscopy and the 20th Anniversary of SIM: feature introduction.

This feature issue commemorating 25 years of STED microscopy and 20 years of SIM is intended to highlight the incredible progress and growth in the field of superresolution microscopy since Stefan Hell and Jan Wichmann published the article Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy in Optics Letters in 1994.