Precision in iterative modulation enhanced single-molecule localization microscopy.

Modulation enhanced single-molecule localization microscopy (meSMLM) methods improve the localization precision by using patterned illumination to encode additional position information. Iterative meSMLM (imeSMLM) methods iteratively generate prior information on emitter positions, used to locally improve the localization precision during subsequent iterations. The Cramér-Rao lower bound cannot incorporate prior information to bound the best achievable localization precision because it requires estimators to be unbiased. By treating estimands as random variables with a known prior distribution, the Van Trees inequality (VTI) can be used to bound the best possible localization precision of imeSMLM methods. An imeSMLM method is considered, where the positions of in-plane standing-wave illumination patterns are controlled over the course of multiple iterations. Using the VTI, we analytically approximate a lower bound on the maximum localization precision of imeSMLM methods that make use of standing-wave illumination patterns. In addition, we evaluate the maximally achievable localization precision for different illumination pattern placement strategies using Monte Carlo simulations. We show that in the absence of background and under perfect modulation, the information content of signal photons increases exponentially as a function of the iteration count. However, the information increase is no longer exponential as a function of the iteration count under non-zero background, imperfect modulation, or limited mechanical resolution of the illumination positioning system. As a result, imeSMLM with two iterations reaches at most a fivefold improvement over SMLM at 8 expected background photons per pixel and 95% modulation contrast. Moreover, the information increase from imeSMLM is balanced by a reduced signal photon rate. Therefore, SMLM outperforms imeSMLM when considering an equal measurement time and illumination power per iteration. Finally, the VTI is an excellent tool for the assessment of the performance of illumination control and is therefore the method of choice for optimal design and control of imeSMLM methods.

Phase retrieval of large-scale time-varying aberrations using a non-linear Kalman filtering framework.

This paper presents a computationally efficient framework in which a single focal-plane image is used to obtain a high-resolution reconstruction of dynamic aberrations. Assuming small-phase aberrations, a non-linear Kalman filter implementation is developed whose computational complexity scales close to linearly with the number of pixels of the focal-plane camera. The performance of the method is tested in a simulation of an adaptive optics system, where the small-phase assumption is enforced by considering a closed-loop system that uses a low-resolution wavefront sensor to control a deformable mirror. The results confirm the computational efficiency of the algorithm and show a large robustness against noise and model uncertainties.

Sparse data-driven wavefront prediction for large-scale adaptive optics.

This paper presents a computationally efficient wavefront aberration prediction framework for data-driven control in large-scale adaptive optics systems. Our novel prediction algorithm splits prediction into two stages: a high-resolution and a low-resolution stage. For the former, we exploit sparsity structures in the system matrices in a data-driven Kalman filtering algorithm and constrain the identified gain to be likewise sparse; for the latter, we identify a dense Kalman gain and perform corrections to the suboptimal predictions of the former on a smaller grid. This novel prediction framework is able to retain the robustness to measurement noise of the standard Kalman filter in a much more computationally efficient manner, in both its offline and online aspects, while minimally sacrificing performance; its data-driven nature further compensates for modeling errors. As an intermediate result, we present a sparsity-exploiting data-driven Kalman filtering algorithm able to quickly estimate an approximate Kalman gain without solving the Riccati equation.

Modeling and state-space identification of deformable mirrors.

To develop high-performance controllers for adaptive optics (AO) systems, it is essential to first derive sufficiently accurate state-space models of deformable mirrors (DMs). However, it is often challenging to develop realistic large-scale finite element (FE) state-space models that take into account the system damping, actuator dynamics, boundary conditions, and multi-physics phenomena affecting the system dynamics. Furthermore, it is challenging to establish a modeling framework capable of the automated and quick derivation of state-space models for different actuator configurations and system geometries. On the other hand, for accurate model-based control and system monitoring, it is often necessary to estimate state-space models from the experimental data. However, this is a challenging problem since the DM dynamics is inherently infinite-dimensional and it is characterized by a large number of eigenmodes and eigenfrequencies. In this paper, we provide modeling and estimation frameworks that address these challenges. We develop an FE state-space model of a faceplate DM that incorporates damping and actuator dynamics. We investigate the frequency and time domain responses for different model parameters. The state-space modeling process is completely automated using the LiveLink for MATLAB toolbox that is incorporated into the COMSOL Multiphysics software package. The developed state-space model is used to generate the estimation data. This data, together with a subspace identification algorithm, is used to estimate reduced-order DM models. We address the model-order selection and model validation problems. The results of this paper provide essential modeling and estimation tools to broad AO and mechatronics scientific communities. The developed Python, MATLAB, and COMSOL Multiphysics codes are available online.

Anisoplanatic adaptive optics in parallelized laser scanning microscopy.

Inhomogeneities in the refractive index of a biological microscopy sample can introduce phase aberrations, severely impairing the quality of images. Adaptive optics can be employed to correct for phase aberrations and improve image quality. However, conventional adaptive optics can only correct a single phase aberration for the whole field of view (isoplanatic correction) while, due to the highly heterogeneous nature of biological tissues, the sample induced aberrations in microscopy often vary throughout the field of view (anisoplanatic aberration), limiting significantly the effectiveness of adaptive optics. This paper reports on a new approach for aberration correction in laser scanning confocal microscopy, in which a spatial light modulator is used to generate multiple excitation points in the sample to simultaneously scan different portions of the field of view with completely independent correction, achieving anisoplanatic compensation of sample induced aberrations, in a significantly shorter time compared to sequential isoplanatic correction of multiple image subregions. The method was tested in whole Drosophila brains and in larval Zebrafish, each showing a dramatic improvement in resolution and sharpness when compared to conventional isoplanatic adaptive optics.

Phase retrieval based on the vectorial model of point spread function.

We present an efficient phase retrieval approach for imaging systems with high numerical aperture based on the vectorial model of the point spread function. The algorithm is in the class of alternating minimization methods and can be adjusted for applications with either known or unknown amplitude of the field in the pupil. The algorithm outperforms existing solutions for high-numerical-aperture phase retrieval: (1) the generalization of the method of Hanser et al., based on extension of the scalar diffraction theory by representing the out-of-focus diversity applied to the image by a spherical cap, and (2) the method of Braat et al., which assumes through the use of extended Nijboer-Zernike expansion the phase to be smooth. The former is limited in terms of accuracy due to model deviations, while the latter is of high computational complexity and excludes phase retrieval problems where the phase is discontinuous or sparse. Extensive numerical results demonstrate the efficiency, robustness, and practicability of the proposed algorithm in various practically relevant simulations.

Convex optimization-based blind deconvolution for images taken with coherent illumination.

A rank-constrained reformulation of the blind deconvolution problem on images taken with coherent illumination is proposed. Since in the reformulation the rank constraint is imposed on a matrix that is affine in the decision variables, we propose a novel convex heuristic for the blind deconvolution problem. The proposed heuristic allows for easy incorporation of prior information on the decision variables and the use of the phase diversity concept. The convex optimization problem can be iteratively re-parameterized to obtain better estimates. The proposed methods are demonstrated on numerically illustrative examples.

Predictive wavefront sensorless adaptive optics for time-varying aberrations.

A new wavefront sensorless adaptive optics method is presented that can accurately correct for time-varying aberrations using a single focal plane image at each sample instance. The linear relation between the mean square of the aberration gradient and the change in second moment of the image forms the basis of the presented method. The new algorithm results in significant improvements when an accurate model of the aberration's temporal dynamics is known, by applying a Kalman filter and optimal control. Moreover, where existing wavefront sensorless adaptive optics methods update all modes sequentially, the information of the Kalman filter is used to select and update the modes that are expected to give the greatest improvement in performance. The performance is analyzed in a simulation of an adaptive optics system for atmospheric turbulence. The results show that the new method is able to correct for the aberration more accurately for higher wind speeds and higher noise levels than existing algorithms.

Identification of the dynamics of time-varying phase aberrations from time histories of the point-spread function.

To optimally compensate for time-varying phase aberrations with adaptive optics, a model of the dynamics of the aberrations is required to predict the phase aberration at the next time step. We model the time-varying behavior of a phase aberration, expressed in Zernike modes, by assuming that the temporal dynamics of the Zernike coefficients can be described by a vector-valued autoregressive (VAR) model. We propose an iterative method based on a convex heuristic for a rank-constrained optimization problem, to jointly estimate the parameters of the VAR model and the Zernike coefficients from a time series of measurements of the point-spread function (PSF) of the optical system. By assuming the phase aberration is small, the relation between aberration and PSF measurements can be approximated by a quadratic function. As such, our method is a blind identification method for linear dynamics in a stochastic Wiener system with a quadratic nonlinearity at the output and a phase retrieval method that uses a time-evolution-model constraint and a single image at every time step.

Pupil mask diversity for image correction in microscopy.

Three-dimensional microscopy suffers from sample-induced aberrations that reduce the resolution and lead to misinterpretations of the object distribution. In this paper, the resolution of a three-dimensional fluorescent microscope is significantly improved by introducing an amplitude diversity in the form of a binary amplitude mask positioned in several different orientations within the pupil, followed by computer processing of the diversity images. The method has proved to be fast, easy to implement, and cost-effective in high-resolution imaging of casper fli:GFP zebrafish.

Extended-image-based correction of aberrations using a deformable mirror with hysteresis.

With a view to the next generation of large space telescopes, we investigate guide-star-free, image-based aberration correction using a unimorph deformable mirror in a plane conjugate to the primary mirror. We designed and built a high-resolution imaging testbed to evaluate control algorithms. In this paper we use an algorithm based on the heuristic hill climbing technique and compare the correction in three different domains, namely the voltage domain, the domain of the Zernike modes, and the domain of the singular modes of the deformable mirror. Through our systematic experimental study, we found that successive control in two domains effectively counteracts uncompensated hysteresis of the deformable mirror.

Tensor-based predictive control for extremely large-scale single conjugate adaptive optics.

In this paper we propose a data-driven predictive control algorithm for large-scale single conjugate adaptive optics systems. At each time sample, the Shack-Hartmann wavefront sensor signal sampled on a spatial grid of size N×N is reshuffled into a d-dimensional tensor. Its spatial-temporal dynamics are modeled with a d-dimensional autoregressive model of temporal order p, where each tensor storing past data undergoes a multilinear transformation by factor matrices of small sizes. Equivalently, the vector form of this autoregressive model features coefficient matrices parametrized with a sum of Kronecker products between d-factor matrices. We propose an Alternating Least Squares algorithm for identifying the factor matrices from open-loop sensor data. When modeling each coefficient matrix with a sum of r terms, the computational complexity for updating the sensor prediction online reduces from O(pN4) in the unstructured matrix case to O(prd N2(d+1)d). Most importantly, this model structure breaks away from assuming any prior spatial-temporal coupling as it is discovered from the data. The algorithm is validated on a laboratory testbed that demonstrates the ability to accurately decompose the coefficient matrices of large-scale autoregressive models with a tensor-based representation, hence achieving high data compression rates and reducing the temporal error especially for a large Greenwood per sample frequency ratio.

Solving large-scale general phase retrieval problems via a sequence of convex relaxations.

We present a convex relaxation-based algorithm for large-scale general phase retrieval problems. General phase retrieval problems include, e.g., the estimation of the phase of the optical field in the pupil plane based on intensity measurements of a point source recorded in the image (focal) plane. The non-convex problem of finding the complex field that generates the correct intensity is reformulated into a rank constraint problem. The nuclear norm is used to obtain the convex relaxation of the phase retrieval problem. A new iterative method referred to as convex optimization-based phase retrieval (COPR) is presented, with each iteration consisting of solving a convex problem. In the noise-free case and for a class of phase retrieval problems, the solutions of the minimization problems converge linearly or faster towards a correct solution. Since the solutions to nuclear norm minimization problems can be computed using semidefinite programming, and this tends to be an expensive optimization in terms of scalability, we provide a fast algorithm called alternating direction method of multipliers (ADMM) that exploits the problem structure. The performance of the COPR algorithm is demonstrated in a realistic numerical simulation study, demonstrating its improvements in reliability and speed with respect to state-of-the-art methods.

GPU implementation for spline-based wavefront reconstruction.

This paper presents an adaptation of the distributed-spline-based aberration reconstruction method for Shack-Hartmann (SH) slope measurements to extremely large-scale adaptive optics systems and the execution on graphics processing units (GPUs). The introduction of a hierarchical multi-level scheme for the elimination of piston offsets between the locally computed wavefront (WF) estimates solves the piston error propagation observed for a large number of partitions with the original version. To obtain a fully distributed method for WF correction, the projection of the phase estimates is locally approximated and applied in a distributed fashion, providing stable results for low and medium actuator coupling. An implementation of the method with the parallel computing platform CUDA exploits the inherently distributed nature of the algorithm. With a standard off-the-shelf GPU, the computation of the adaptive optics correction updates is accomplished in under 1 ms for the benchmark case of a 200×200 SH array.

Optical path difference microscopy with a Shack-Hartmann wavefront sensor.

In this Letter, we show that a Shack-Hartmann wavefront sensor can be used for the quantitative measurement of the specimen optical path difference (OPD) in an ordinary incoherent optical microscope, if the spatial coherence of the illumination light in the plane of the specimen is larger than the microscope resolution. To satisfy this condition, the illumination numerical aperture should be smaller than the numerical aperture of the imaging lens. This principle has been successfully applied to build a high-resolution reference-free instrument for the characterization of the OPD of micro-optical components and microscopic biological samples.

Aliasing, coherence, and resolution in a lensless holographic microscope.

We have shown that the maximum achievable resolution of an in-line lensless holographic microscope is limited by aliasing and, for collimated illumination, cannot exceed the camera pixel size. This limit can be achieved only when the optimal conditions on the spatial and temporal coherence state of the illumination are satisfied. The expressions defining the configuration, delivering maximum resolution with given spatial and temporal coherence of the illumination, are obtained. The validity of these conditions is confirmed experimentally.

Nonlinear spline wavefront reconstruction from Shack-Hartmann intensity measurements through small aberration approximations.

We propose an extension of the Spline based ABerration Reconstruction (SABRE) method to Shack-Hartmann (SH) intensity measurements, through small aberration approximations of the focal spot models. The original SABRE for SH slope measurements is restricted to the use of linear spline polynomials, due to the limited amount of data, and the resolution of its reconstruction is determined by the number of lenslets. In this work, a fast algorithm is presented that directly processes the pixel information of the focal spots, allowing the employment of nonlinear polynomials for high accuracy reconstruction. In order to guarantee the validity of the small aberration approximations, the method is applied in two correction steps, with a first compensation of large, low-order aberrations through the gradient-based linear SABRE followed by compensation of the remaining high-order aberrations with the intensity-based nonlinear SABRE.

Adaptive illumination based on direct wavefront sensing in a light-sheet fluorescence microscope.

A methodology for the adaptive control and correction of phase aberrations in the illumination arm of a light-sheet fluorescence microscope has been developed. The method uses direct wavefront sensing on epi-fluorescent light to detect the aberration present in the sample. Using this signal, the aberrations in the illumination arm are subsequently corrected with a spatial light modulator in a feedforward mode. Adaptive correction, resulting in significant improvement in the axial resolution, has been demonstrated by imaging Tg(fli:GFP) zebrafish embryos.

Holographic imaging with a Shack-Hartmann wavefront sensor.

A high-resolution Shack-Hartmann wavefront sensor has been used for coherent holographic imaging, by computer reconstruction and propagation of the complex field in a lensless imaging setup. The resolution of the images obtained with the experimental data is in a good agreement with the diffraction theory. Although a proper calibration with a reference beam improves the image quality, the method has a potential for reference-less holographic imaging with spatially coherent monochromatic and narrowband polychromatic sources in microscopy and imaging through turbulence.

Framework to trade optimality for local processing in large-scale wavefront reconstruction problems.

We show that the minimum variance wavefront estimation problems permit localized approximate solutions, in the sense that the wavefront value at a point (excluding unobservable modes, such as the piston mode) can be approximated by a linear combination of the wavefront slope measurements in the point's neighborhood. This enables us to efficiently compute a wavefront estimate by performing a single sparse matrix-vector multiplication. Moreover, our results open the possibility for the development of wavefront estimators that can be easily implemented in a decentralized/distributed manner, and in which the estimate optimality can be easily traded for computational efficiency. We numerically validate our approach on Hudgin wavefront sensor geometries, and the results can be easily generalized to Fried geometries.