Adaptive optics in single objective inclined light sheet microscopy enables three-dimensional localization microscopy in adult Drosophila brains.

Single-molecule localization microscopy (SMLM) enables the high-resolution visualization of organelle structures and the precise localization of individual proteins. However, the expected resolution is not achieved in tissue as the imaging conditions deteriorate. Sample-induced aberrations distort the point spread function (PSF), and high background fluorescence decreases the localization precision. Here, we synergistically combine sensorless adaptive optics (AO), in-situ 3D-PSF calibration, and a single-objective lens inclined light sheet microscope (SOLEIL), termed (AO-SOLEIL), to mitigate deep tissue-induced deteriorations. We apply AO-SOLEIL on several dSTORM samples including brains of adult Drosophila. We observed a 2x improvement in the estimated axial localization precision with respect to widefield without aberration correction while we used synergistic solution. AO-SOLEIL enhances the overall imaging resolution and further facilitates the visualization of sub-cellular structures in tissue.

Precision in iterative modulation enhanced single-molecule localization microscopy.

Modulation enhanced single-molecule localization microscopy (meSMLM) methods improve the localization precision by using patterned illumination to encode additional position information. Iterative meSMLM (imeSMLM) methods iteratively generate prior information on emitter positions, used to locally improve the localization precision during subsequent iterations. The Cramér-Rao lower bound cannot incorporate prior information to bound the best achievable localization precision because it requires estimators to be unbiased. By treating estimands as random variables with a known prior distribution, the Van Trees inequality (VTI) can be used to bound the best possible localization precision of imeSMLM methods. An imeSMLM method is considered, where the positions of in-plane standing-wave illumination patterns are controlled over the course of multiple iterations. Using the VTI, we analytically approximate a lower bound on the maximum localization precision of imeSMLM methods that make use of standing-wave illumination patterns. In addition, we evaluate the maximally achievable localization precision for different illumination pattern placement strategies using Monte Carlo simulations. We show that in the absence of background and under perfect modulation, the information content of signal photons increases exponentially as a function of the iteration count. However, the information increase is no longer exponential as a function of the iteration count under non-zero background, imperfect modulation, or limited mechanical resolution of the illumination positioning system. As a result, imeSMLM with two iterations reaches at most a fivefold improvement over SMLM at 8 expected background photons per pixel and 95% modulation contrast. Moreover, the information increase from imeSMLM is balanced by a reduced signal photon rate. Therefore, SMLM outperforms imeSMLM when considering an equal measurement time and illumination power per iteration. Finally, the VTI is an excellent tool for the assessment of the performance of illumination control and is therefore the method of choice for optimal design and control of imeSMLM methods.

Sparse data-driven wavefront prediction for large-scale adaptive optics.

This paper presents a computationally efficient wavefront aberration prediction framework for data-driven control in large-scale adaptive optics systems. Our novel prediction algorithm splits prediction into two stages: a high-resolution and a low-resolution stage. For the former, we exploit sparsity structures in the system matrices in a data-driven Kalman filtering algorithm and constrain the identified gain to be likewise sparse; for the latter, we identify a dense Kalman gain and perform corrections to the suboptimal predictions of the former on a smaller grid. This novel prediction framework is able to retain the robustness to measurement noise of the standard Kalman filter in a much more computationally efficient manner, in both its offline and online aspects, while minimally sacrificing performance; its data-driven nature further compensates for modeling errors. As an intermediate result, we present a sparsity-exploiting data-driven Kalman filtering algorithm able to quickly estimate an approximate Kalman gain without solving the Riccati equation.

Phase retrieval of large-scale time-varying aberrations using a non-linear Kalman filtering framework.

This paper presents a computationally efficient framework in which a single focal-plane image is used to obtain a high-resolution reconstruction of dynamic aberrations. Assuming small-phase aberrations, a non-linear Kalman filter implementation is developed whose computational complexity scales close to linearly with the number of pixels of the focal-plane camera. The performance of the method is tested in a simulation of an adaptive optics system, where the small-phase assumption is enforced by considering a closed-loop system that uses a low-resolution wavefront sensor to control a deformable mirror. The results confirm the computational efficiency of the algorithm and show a large robustness against noise and model uncertainties.

Anisoplanatic adaptive optics in parallelized laser scanning microscopy.

Inhomogeneities in the refractive index of a biological microscopy sample can introduce phase aberrations, severely impairing the quality of images. Adaptive optics can be employed to correct for phase aberrations and improve image quality. However, conventional adaptive optics can only correct a single phase aberration for the whole field of view (isoplanatic correction) while, due to the highly heterogeneous nature of biological tissues, the sample induced aberrations in microscopy often vary throughout the field of view (anisoplanatic aberration), limiting significantly the effectiveness of adaptive optics. This paper reports on a new approach for aberration correction in laser scanning confocal microscopy, in which a spatial light modulator is used to generate multiple excitation points in the sample to simultaneously scan different portions of the field of view with completely independent correction, achieving anisoplanatic compensation of sample induced aberrations, in a significantly shorter time compared to sequential isoplanatic correction of multiple image subregions. The method was tested in whole Drosophila brains and in larval Zebrafish, each showing a dramatic improvement in resolution and sharpness when compared to conventional isoplanatic adaptive optics.

Modeling and state-space identification of deformable mirrors.

To develop high-performance controllers for adaptive optics (AO) systems, it is essential to first derive sufficiently accurate state-space models of deformable mirrors (DMs). However, it is often challenging to develop realistic large-scale finite element (FE) state-space models that take into account the system damping, actuator dynamics, boundary conditions, and multi-physics phenomena affecting the system dynamics. Furthermore, it is challenging to establish a modeling framework capable of the automated and quick derivation of state-space models for different actuator configurations and system geometries. On the other hand, for accurate model-based control and system monitoring, it is often necessary to estimate state-space models from the experimental data. However, this is a challenging problem since the DM dynamics is inherently infinite-dimensional and it is characterized by a large number of eigenmodes and eigenfrequencies. In this paper, we provide modeling and estimation frameworks that address these challenges. We develop an FE state-space model of a faceplate DM that incorporates damping and actuator dynamics. We investigate the frequency and time domain responses for different model parameters. The state-space modeling process is completely automated using the LiveLink for MATLAB toolbox that is incorporated into the COMSOL Multiphysics software package. The developed state-space model is used to generate the estimation data. This data, together with a subspace identification algorithm, is used to estimate reduced-order DM models. We address the model-order selection and model validation problems. The results of this paper provide essential modeling and estimation tools to broad AO and mechatronics scientific communities. The developed Python, MATLAB, and COMSOL Multiphysics codes are available online.

Phase retrieval based on the vectorial model of point spread function.

We present an efficient phase retrieval approach for imaging systems with high numerical aperture based on the vectorial model of the point spread function. The algorithm is in the class of alternating minimization methods and can be adjusted for applications with either known or unknown amplitude of the field in the pupil. The algorithm outperforms existing solutions for high-numerical-aperture phase retrieval: (1) the generalization of the method of Hanser et al., based on extension of the scalar diffraction theory by representing the out-of-focus diversity applied to the image by a spherical cap, and (2) the method of Braat et al., which assumes through the use of extended Nijboer-Zernike expansion the phase to be smooth. The former is limited in terms of accuracy due to model deviations, while the latter is of high computational complexity and excludes phase retrieval problems where the phase is discontinuous or sparse. Extensive numerical results demonstrate the efficiency, robustness, and practicability of the proposed algorithm in various practically relevant simulations.

Predictive wavefront sensorless adaptive optics for time-varying aberrations.

A new wavefront sensorless adaptive optics method is presented that can accurately correct for time-varying aberrations using a single focal plane image at each sample instance. The linear relation between the mean square of the aberration gradient and the change in second moment of the image forms the basis of the presented method. The new algorithm results in significant improvements when an accurate model of the aberration's temporal dynamics is known, by applying a Kalman filter and optimal control. Moreover, where existing wavefront sensorless adaptive optics methods update all modes sequentially, the information of the Kalman filter is used to select and update the modes that are expected to give the greatest improvement in performance. The performance is analyzed in a simulation of an adaptive optics system for atmospheric turbulence. The results show that the new method is able to correct for the aberration more accurately for higher wind speeds and higher noise levels than existing algorithms.

Convex optimization-based blind deconvolution for images taken with coherent illumination.

A rank-constrained reformulation of the blind deconvolution problem on images taken with coherent illumination is proposed. Since in the reformulation the rank constraint is imposed on a matrix that is affine in the decision variables, we propose a novel convex heuristic for the blind deconvolution problem. The proposed heuristic allows for easy incorporation of prior information on the decision variables and the use of the phase diversity concept. The convex optimization problem can be iteratively re-parameterized to obtain better estimates. The proposed methods are demonstrated on numerically illustrative examples.

Identification of the dynamics of time-varying phase aberrations from time histories of the point-spread function.

To optimally compensate for time-varying phase aberrations with adaptive optics, a model of the dynamics of the aberrations is required to predict the phase aberration at the next time step. We model the time-varying behavior of a phase aberration, expressed in Zernike modes, by assuming that the temporal dynamics of the Zernike coefficients can be described by a vector-valued autoregressive (VAR) model. We propose an iterative method based on a convex heuristic for a rank-constrained optimization problem, to jointly estimate the parameters of the VAR model and the Zernike coefficients from a time series of measurements of the point-spread function (PSF) of the optical system. By assuming the phase aberration is small, the relation between aberration and PSF measurements can be approximated by a quadratic function. As such, our method is a blind identification method for linear dynamics in a stochastic Wiener system with a quadratic nonlinearity at the output and a phase retrieval method that uses a time-evolution-model constraint and a single image at every time step.

Extended-image-based correction of aberrations using a deformable mirror with hysteresis.

With a view to the next generation of large space telescopes, we investigate guide-star-free, image-based aberration correction using a unimorph deformable mirror in a plane conjugate to the primary mirror. We designed and built a high-resolution imaging testbed to evaluate control algorithms. In this paper we use an algorithm based on the heuristic hill climbing technique and compare the correction in three different domains, namely the voltage domain, the domain of the Zernike modes, and the domain of the singular modes of the deformable mirror. Through our systematic experimental study, we found that successive control in two domains effectively counteracts uncompensated hysteresis of the deformable mirror.

Imaging & identification of malaria parasites using cellphone microscope with a ball lens.

We have optimized the design and imaging procedures, to clearly resolve the malaria parasite in Giemsa-stained thin blood smears, using simple low-cost cellphone-based microscopy with oil immersion. The microscope uses a glass ball as the objective and the phone camera as the tube lens. Our optimization includes the optimal choice of the ball lens diameter, the size and the position of the aperture diaphragm, and proper application of immersion, to achieve diagnostic capacity in a wide field of view. The resulting system is potentially applicable to low-cost in-the-field optical diagnostics of malaria as it clearly resolves micron-sized features and allows for analysis of parasite morphology in the field of 50 × 50 µm, and parasite detection in the field of at least 150 × 150 µm.

Tensor-based predictive control for extremely large-scale single conjugate adaptive optics.

In this paper we propose a data-driven predictive control algorithm for large-scale single conjugate adaptive optics systems. At each time sample, the Shack-Hartmann wavefront sensor signal sampled on a spatial grid of size N×N is reshuffled into a d-dimensional tensor. Its spatial-temporal dynamics are modeled with a d-dimensional autoregressive model of temporal order p, where each tensor storing past data undergoes a multilinear transformation by factor matrices of small sizes. Equivalently, the vector form of this autoregressive model features coefficient matrices parametrized with a sum of Kronecker products between d-factor matrices. We propose an Alternating Least Squares algorithm for identifying the factor matrices from open-loop sensor data. When modeling each coefficient matrix with a sum of r terms, the computational complexity for updating the sensor prediction online reduces from O(pN4) in the unstructured matrix case to O(prd N2(d+1)d). Most importantly, this model structure breaks away from assuming any prior spatial-temporal coupling as it is discovered from the data. The algorithm is validated on a laboratory testbed that demonstrates the ability to accurately decompose the coefficient matrices of large-scale autoregressive models with a tensor-based representation, hence achieving high data compression rates and reducing the temporal error especially for a large Greenwood per sample frequency ratio.

Pupil mask diversity for image correction in microscopy.

Three-dimensional microscopy suffers from sample-induced aberrations that reduce the resolution and lead to misinterpretations of the object distribution. In this paper, the resolution of a three-dimensional fluorescent microscope is significantly improved by introducing an amplitude diversity in the form of a binary amplitude mask positioned in several different orientations within the pupil, followed by computer processing of the diversity images. The method has proved to be fast, easy to implement, and cost-effective in high-resolution imaging of casper fli:GFP zebrafish.

Solving large-scale general phase retrieval problems via a sequence of convex relaxations.

We present a convex relaxation-based algorithm for large-scale general phase retrieval problems. General phase retrieval problems include, e.g., the estimation of the phase of the optical field in the pupil plane based on intensity measurements of a point source recorded in the image (focal) plane. The non-convex problem of finding the complex field that generates the correct intensity is reformulated into a rank constraint problem. The nuclear norm is used to obtain the convex relaxation of the phase retrieval problem. A new iterative method referred to as convex optimization-based phase retrieval (COPR) is presented, with each iteration consisting of solving a convex problem. In the noise-free case and for a class of phase retrieval problems, the solutions of the minimization problems converge linearly or faster towards a correct solution. Since the solutions to nuclear norm minimization problems can be computed using semidefinite programming, and this tends to be an expensive optimization in terms of scalability, we provide a fast algorithm called alternating direction method of multipliers (ADMM) that exploits the problem structure. The performance of the COPR algorithm is demonstrated in a realistic numerical simulation study, demonstrating its improvements in reliability and speed with respect to state-of-the-art methods.

GPU implementation for spline-based wavefront reconstruction.

This paper presents an adaptation of the distributed-spline-based aberration reconstruction method for Shack-Hartmann (SH) slope measurements to extremely large-scale adaptive optics systems and the execution on graphics processing units (GPUs). The introduction of a hierarchical multi-level scheme for the elimination of piston offsets between the locally computed wavefront (WF) estimates solves the piston error propagation observed for a large number of partitions with the original version. To obtain a fully distributed method for WF correction, the projection of the phase estimates is locally approximated and applied in a distributed fashion, providing stable results for low and medium actuator coupling. An implementation of the method with the parallel computing platform CUDA exploits the inherently distributed nature of the algorithm. With a standard off-the-shelf GPU, the computation of the adaptive optics correction updates is accomplished in under 1 ms for the benchmark case of a 200×200 SH array.

Optimal model-based sensorless adaptive optics for epifluorescence microscopy.

We report on a universal sample-independent sensorless adaptive optics method, based on modal optimization of the second moment of the fluorescence emission from a point-like excitation. Our method employs a sample-independent precalibration, performed only once for the particular system, to establish the direct relation between the image quality and the aberration. The method is potentially applicable to any form of microscopy with epifluorescence detection, including the practically important case of incoherent fluorescence emission from a three dimensional object, through minor hardware modifications. We have applied the technique successfully to a widefield epifluorescence microscope and to a multiaperture confocal microscope.

Online Optimization With Costly and Noisy Measurements Using Random Fourier Expansions.

This paper analyzes data-based online nonlinear extremum-seeker (DONE), an online optimization algorithm that iteratively minimizes an unknown function based on costly and noisy measurements. The algorithm maintains a surrogate of the unknown function in the form of a random Fourier expansion. The surrogate is updated whenever a new measurement is available, and then used to determine the next measurement point. The algorithm is comparable with Bayesian optimization algorithms, but its computational complexity per iteration does not depend on the number of measurements. We derive several theoretical results that provide insight on how the hyperparameters of the algorithm should be chosen. The algorithm is compared with a Bayesian optimization algorithm for an analytic benchmark problem and three applications, namely, optical coherence tomography, optical beam-forming network tuning, and robot arm control. It is found that the DONE algorithm is significantly faster than Bayesian optimization in the discussed problems while achieving a similar or better performance.

Fast Calculation of Computer Generated Holograms for 3D Photostimulation through Compressive-Sensing Gerchberg-Saxton Algorithm.

The use of spatial light modulators to project computer generated holograms is a common strategy for optogenetic stimulation of multiple structures of interest within a three-dimensional volume. A common requirement when addressing multiple targets sparsely distributed in three dimensions is the generation of a points cloud, focusing excitation light in multiple diffraction-limited locations throughout the sample. Calculation of this type of holograms is most commonly performed with either the high-speed, low-performance random superposition algorithm, or the low-speed, high performance Gerchberg-Saxton algorithm. This paper presents a variation of the Gerchberg-Saxton algorithm that, by only performing iterations on a subset of the data, according to compressive sensing principles, is rendered significantly faster while maintaining high quality outputs. The algorithm is presented in high-efficiency and high-uniformity variants. All source code for the method implementation is available as Supplementary Materials and as open-source software. The method was tested computationally against existing algorithms, and the results were confirmed experimentally on a custom setup for in-vivo multiphoton optogenetics. The results clearly show that the proposed method can achieve computational speed performances close to the random superposition algorithm, while retaining the high performance of the Gerchberg-Saxton algorithm, with a minimal hologram quality loss.

Nonlinear spline wavefront reconstruction from Shack-Hartmann intensity measurements through small aberration approximations.

We propose an extension of the Spline based ABerration Reconstruction (SABRE) method to Shack-Hartmann (SH) intensity measurements, through small aberration approximations of the focal spot models. The original SABRE for SH slope measurements is restricted to the use of linear spline polynomials, due to the limited amount of data, and the resolution of its reconstruction is determined by the number of lenslets. In this work, a fast algorithm is presented that directly processes the pixel information of the focal spots, allowing the employment of nonlinear polynomials for high accuracy reconstruction. In order to guarantee the validity of the small aberration approximations, the method is applied in two correction steps, with a first compensation of large, low-order aberrations through the gradient-based linear SABRE followed by compensation of the remaining high-order aberrations with the intensity-based nonlinear SABRE.