Information optimization of laser scanning microscopes for real-time feedback-driven single particle tracking.

Real-time feedback-driven single particle tracking (RT-FD-SPT) is a class of microscopy techniques that uses measurements of finite excitation/detection volume in a feedback control loop to actuate that volume and track with high spatio-temporal resolution a single particle moving in three dimensions. A variety of methods have been developed, each defined by a set of user-defined choices. Selection of those values is typically done through ad hoc, off-line tuning for the best perceived performance. Here we present a mathematical framework, based on optimization of the Fisher information, to select those parameters such that the best information is acquired for estimating parameters of interest, such as the location of the particle, specifics of the excitation beam such as its dimensions or peak intensity, or the background noise. For concreteness, we focus on tracking of a fluorescently-labeled particle and apply this framework to determine the optimal parameters for three existing fluorescence-based RT-FD-SPT techniques with respect to particle localization.

Quantitative nanopatterning of fg-scale liquids with dip-pen nanolithography.

The ability to precisely pattern nanoscale amounts of liquids is essential for biotechnology and high-throughput chemistry, but controlling fluid flow on these scales is very challenging. Scanning probe lithography methods such as dip-pen nanolithography (DPN) provide a mechanism to write fluids at the nanoscale, but this is an open loop process as methods to provide feedback while patterning sub-pg features have yet to be reported. Here, we demonstrate a novel method for programmably nanopatterning liquid features at the fg-scale through a combination of ultrafast atomic force microscopy probes, the use of spherical tips, and inertial mass sensing. We begin by investigating the required probe properties that would provide sufficient mass responsivity to detect fg-scale mass changes and find ultrafast probes to be capable of this resolution. Further, we attach a spherical bead to the tip of an ultrafast probe as we hypothesize that the spherical tip could hold a drop at its apex which both facilitates interpretation of inertial sensing and maintains a consistent fluid environment for reliable patterning. We experimentally find that sphere-tipped ultrafast probes are capable of reliably patterning hundreds of features in a single experiment. Analyzing the changes in the vibrational resonance frequency during the patterning process, we find that drift in the resonance frequency complicates analysis, but that it can be removed through a systematic correction. Subsequently, we quantitatively study patterning using sphere-tipped ultrafast probes as a function of retraction speed and dwell time to find that the mass of fluid transferred can be modulated by greater than an order of magnitude and that liquid features as small as 6 fg can be patterned and resolved. Taken together, this work addresses a persistent concern in DPN by enabling quantitative feedback for nanopatterning of aL-scale features and lays the foundation for programmably nanopatterning fluids.

Real-Time Feedback-Driven Single-Particle Tracking: A Survey and Perspective.

Real-time feedback-driven single-particle tracking (RT-FD-SPT) is a class of techniques in the field of single-particle tracking that uses feedback control to keep a particle of interest in a detection volume. These methods provide high spatiotemporal resolution on particle dynamics and allow for concurrent spectroscopic measurements. This review article begins with a survey of existing techniques and of applications where RT-FD-SPT has played an important role. Each of the core components of RT-FD-SPT are systematically discussed in order to develop an understanding of the trade-offs that must be made in algorithm design and to create a clear picture of the important differences, advantages, and drawbacks of existing approaches. These components are feedback tracking and control, ranging from simple proportional-integral-derivative control to advanced nonlinear techniques, estimation to determine particle location from the measured data, including both online and offline algorithms, and techniques for calibrating and characterizing different RT-FD-SPT methods. Then a collection of metrics for RT-FD-SPT is introduced to help guide experimentalists in selecting a method for their particular application and to help reveal where there are gaps in the techniques that represent opportunities for further development. Finally, this review is concluded with a discussion on future perspectives in the field.

Synthetic Stochastic Motion Platform for Testing Single Particle Tracking Microscopes.

We describe the design and implementation of a control system for testing the performance of single particle tracking microscopes with the method of synthetic motion. Single particle tracking (SPT) has become a common and powerful tool in the study of biomolecular transport in cellular biology, providing the ability to track individual biological macromolecules in their native environment. Existing methods for testing SPT techniques rely on physical simulations and there is a clear need for experimental-based schemes for both comparing different approaches and for characterizing the accuracy and precision of techniques on particular experimental setups. Synthetic motion, that is, using an actuator such as a nanopositioning stage to drive a particle along a known ground-truth trajectory, is a means for achieving these ends. However, the resolution, accuracy, and flexibility of this method is limited by the actuator static and dynamic characteristics. In this work we apply system identification and model inverse feedforward control to increase actuator bandwidth and address some common actuator nonlinearities, develop a set of dimensionless numbers that describe system limitations, and provide a set of guidelines for the practical use of synthetic motion in SPT.

An Estimation Algorithm for General Linear Single Particle Tracking Models with Time-Varying Parameters.

Single Particle Tracking (SPT) is a powerful class of methods for studying the dynamics of biomolecules inside living cells. The techniques reveal the trajectories of individual particles, with a resolution well below the diffraction limit of light, and from them the parameters defining the motion model, such as diffusion coefficients and confinement lengths. Most existing algorithms assume these parameters are constant throughout an experiment. However, it has been demonstrated that they often vary with time as the tracked particles move through different regions in the cell or as conditions inside the cell change in response to stimuli. In this work, we propose an estimation algorithm to determine time-varying parameters of systems that discretely switch between different linear models of motion with Gaussian noise statistics, covering dynamics such as diffusion, directed motion, and Ornstein-Uhlenbeck dynamics. Our algorithm consists of three stages. In the first stage, we use a sliding window approach, combined with Expectation Maximization (EM) to determine maximum likelihood estimates of the parameters as a function of time. These results are only used to roughly estimate the number of model switches that occur in the data to guide the selection of algorithm parameters in the second stage. In the second stage, we use Change Detection (CD) techniques to identify where the models switch, taking advantage of the off-line nature of the analysis of SPT data to create non-causal algorithms with better precision than a purely causal approach. Finally, we apply EM to each set of data between the change points to determine final parameter estimates. We demonstrate our approach using experimental data generated in the lab under controlled conditions.

Feature Tracking for High Speed AFM Imaging of Biopolymers.

The scanning speed of atomic force microscopes continues to advance with some current commercial microscopes achieving on the order of one frame per second and at least one reaching 10 frames per second. Despite the success of these instruments, even higher frame rates are needed with scan ranges larger than are currently achievable. Moreover, there is a significant installed base of slower instruments that would benefit from algorithmic approaches to increasing their frame rate without requiring significant hardware modifications. In this paper, we present an experimental demonstration of high speed scanning on an existing, non-high speed instrument, through the use of a feedback-based, feature-tracking algorithm that reduces imaging time by focusing on features of interest to reduce the total imaging area. Experiments on both circular and square gratings, as well as silicon steps and DNA strands show a reduction in imaging time by a factor of 3-12 over raster scanning, depending on the parameters chosen.

A comparison of reconstruction methods for undersampled atomic force microscopy images.

Non-raster scanning and undersampling of atomic force microscopy (AFM) images is a technique for improving imaging rate and reducing the amount of tip-sample interaction needed to produce an image. Generation of the final image can be done using a variety of image processing techniques based on interpolation or optimization. The choice of reconstruction method has a large impact on the quality of the recovered image and the proper choice depends on the sample under study. In this work we compare interpolation through the use of inpainting algorithms with reconstruction based on optimization through the use of the basis pursuit algorithm commonly used for signal recovery in compressive sensing. Using four different sampling patterns found in non-raster AFM, namely row subsampling, spiral scanning, Lissajous scanning, and random scanning, we subsample data from existing images and compare reconstruction performance against the original image. The results illustrate that inpainting generally produces superior results when the image contains primarily low frequency content while basis pursuit is better when the images have mixed, but sparse, frequency content. Using support vector machines, we then classify images based on their frequency content and sparsity and, from this classification, develop a fast decision strategy to select a reconstruction algorithm to be used on subsampled data. The performance of the classification and decision test are demonstrated on test AFM images.

EM-based algorithms for single particle tracking of Ornstein-Uhlenbeck motion from sCMOS camera data.

Single particle tracking plays an important role in studying physical and kinetic properties of biomolecules. In this work, we introduce the application of Expectation Maximization (EM) based algorithms for solving localization and parameter estimation problems in SPT using data captured from scientific complementary metal-oxide semiconductor (sCMOS) camera sensors. Two representative methods are considered for generating the filtered and smoothed distributions needed by EM: Sequential Monte Carlo - EM, and Unscented - EM. The SMC method uses particle filtering and particle smoothing to handle general distributions, while the U scheme reduces the computational burden through the use of an unscented Kalman Filter and an unscented Rauch-Tung Striebel Smoother. We also investigate the influence of the number of images in the dataset on the final estimates through intensive simulations as well as the computational efficiency of the two methods.

Computationally efficient application of Sequential Monte Carlo expectation maximization to confined single particle tracking.

Single Particle Tracking (SPT) plays a crucial role in biophysics through its ability to reveal dynamic mechanisms and physical properties of biological macromolecules moving inside living cells. Such molecules are often subject to confinement and important information can be revealed by understanding the mobility of the molecules and the size of the domain they are restricted to. In previous work, we introduced a method known as Sequential Monte Carlo-Expectation Maximization (SMC-EM) to simultaneously estimate particle trajectories and model parameters. In this paper, we describe three modifications to SMC-EM aimed at improving its computationally efficiency and demonstrate it through analysis of simulated SPT data of a particle in a three dimensional confined environment. The first two modifications use approximation methods to reduce the complexity of the original motion and measurement models without significant loss of accuracy. The third modification replaces the previous SMC methods with a Gaussian particle filter combined with a backward simulation particle smoother, trading off some level of generality for improved computational performance. In addition, we take advantage of the improved efficiency to investigate the effect of data length on performance in localization and parameter estimation.

Expectation maximization based framework for joint localization and parameter estimation in single particle tracking from segmented images.

Single Particle Tracking (SPT) is a well known class of tools for studying the dynamics of biological macromolecules moving inside living cells. In this paper, we focus on the problem of localization and parameter estimation given a sequence of segmented images. In the standard paradigm, the location of the emitter inside each frame of a sequence of camera images is estimated using, for example, Gaussian fitting (GF), and these locations are linked to provide an estimate of the trajectory. Trajectories are then analyzed by using Mean Square Displacement (MSD) or Maximum Likelihood Estimation (MLE) techniques to determine motion parameters such as diffusion coefficients. However, the problems of localization and parameter estimation are clearly coupled. Motivated by this, we have created an Expectation Maximization (EM) based framework for simultaneous localization and parameter estimation. We demonstrate this framework through two representative methods, namely, Sequential Monte Carlo combined with Expectation Maximization (SMC-EM) and Unscented Kalman Filter combined with Expectation Maximization (U-EM). Using diffusion in two-dimensions as a prototypical example, we conduct quantitative investigations on localization and parameter estimation performance across a wide range of signal to background ratios and diffusion coefficients and compare our methods to the standard techniques based on GF-MSD/MLE. To demonstrate the flexibility of the EM based framework, we do comparisons using two different camera models, an ideal camera with Poisson distributed shot noise but no readout noise, and a camera with both shot noise and the pixel-dependent readout noise that is common to scientific complementary metal-oxide semiconductor (sCMOS) camera. Our results indicate our EM based methods outperform the standard techniques, especially at low signal levels. While U-EM and SMC-EM have similar accuracy, U-EM is significantly more computationally efficient, though the use of the Unscented Kalman Filter limits U-EM to lower diffusion rates.

Analysis of an Extremum Seeking Controller Under Bounded Disturbance.

One of the applications of Extremum Seeking (ES) is to localize the source of a scalar field by using a mobile agent that can measure this field at its current location. While the scientific literature has presented many approaches to this problem, a formal analysis of the behavior of ES controllers for source seeking in the presence of disturbances is still lacking. This paper aims to fill this gap by analyzing a specific version of an ES control algorithm in the presence of source movement and measurement disturbances. We define an approximate version of this controller that captures the main features but allows for a simplified analysis and then formally characterize the convergence properties of this approximation. Through simulations and physical experiments, we compare the theoretically-predicted regions of attraction of the simplified system with the behavior of the full system and show that the simplified version is a good predictor of the behavior of the initial ES controller.

Three-dimensional localization refinement and motion model parameter estimation for confined single particle tracking under low-light conditions.

Confined diffusion is an important model for describing the motion of biological macromolecules moving in the crowded, three-dimensional environment of the cell. In this work we build upon the technique known as sequential Monte Carlo - expectation maximization (SMC-EM) to simultaneously localize the particle and estimate the motion model parameters from single particle tracking data. We extend SMC-EM to handle the double-helix point spread function (DH-PSF) for encoding the three-dimensional position of the particle in the two-dimensional image plane of the camera. SMC-EM can handle a wide range of camera models and here we assume the data was acquired using a scientific CMOS (sCMOS) camera. The sensitivity and speed of these cameras make them well suited for SPT, though the pixel-dependent nature of the camera noise presents a challenge for analysis. We focus on the low signal setting and compare our method through simulation to more standard approaches that use the paradigm of localize-then-estimate. To localize the particle under the standard paradigm, we use both a Gaussian fit and a maximum likelihood estimator (MLE) that accounts for both the DH-PSF and the pixel-dependent noise of the camera. Model estimation is then carried out either by fitting the model to the mean squared displacement (MSD) curve, or through an optimal estimation approach. Our results indicate that in the low signal regime, the SMC-EM approach outperforms the other methods while at higher signal-to-background levels, SMC-EM and the MLE-based methods perform equally well and both are significantly better than fitting to the MSD. In addition our results indicate that at smaller confinement lengths where the nonlinearities dominate the motion model, the SMC-EM approach is superior to the alternative approaches.

Joint Estimation of Trajectory and Model Parameters for Single Particle Tracking of 3D Confined Diffusion Using the Double-Helix Point Spread Function.

Single particle tracking plays a significant role in biophysics through its ability to reveal dynamic mechanisms and physical properties of biological macromolecules inside living cells. The motion of these molecules can often be modeled as a confined diffusion. The standard paradigm in the biophysics community is to first estimate the trajectory of a particle and then use a technique such as the Mean Square Displacement or the Maximum Likelihood Estimation (MLE) to determine model parameters. These approaches, however, ignore the fact that localization and parameter estimation problems are coupled. We have previously introduced a framework based on optimal estimation theory to simultaneously do localization and parameter estimation. Here we build upon that work by expanding it to include a recent advance in imaging three dimensional motion, namely the Double-Helix (DH) engineered Point Spread Function (PSF). The DH-PSF encodes the axial position of the particle directly into the 2D image acquired by the camera mounted to the microscope. Our approach uses Expectation Maximization (EM) and Sequential Monte Carlo (SMC) to handle the nonlinearities in the observation and motion models. In this paper, we also improve upon the computational complexity of this scheme, using a Gaussian Particle Filter and Backward Simulation Particle Smoother in the SMC elements of the algorithm. We compare our scheme through simulation to state of the art methods based on localization using Gaussian fitting followed by MLE of the model parameters. These results show that our method outperforms GF-MLE at the low signal intensity levels common to biophysical experiments.

Information Optimal Control for Single Particle Tracking Microscopy.

We consider the problem of designing a control policy for a laser scanning microscope (LSM) which will minimize the estimation uncertainty when identifying the state and motion model of a fluorescent biological particle. Using the information optimal design framework we pose an optimization problem which seeks to maximize the Fisher information of the particle's state. We then apply optimal control methods to determine the laser trajectory that maximizes a criterion based on the Fisher information. The resulting optimal control policy is a Bang-Singular control which moves the laser to the set of measurement locations that maximize the rate of information accumulation. Simulations demonstrate the ability of the resulting control system to position the laser to measure the particles location with a minimum uncertainty.

Model Segmentation in Single Particle Tracking.

In this paper, we implement and compare two different change detection techniques applied to determining the time points in Single Particle Tracking (SPT) data where the particle changes the dynamic model of motion. The goal is to use this change detection to segment the data in order to estimate the relevant parameters of such models. We consider two well-known statistics commonly used for change detection: the likelihood ratio test (LRT) and the Kullback-Leibler divergence (KLD). We assume that our time-varying system is subject to step-like changes in the parameters that drive the process. The techniques are then applied to experimental data acquired on a microscope under controlled settings to validate our results.

Tracking Multiple Diffusing Particles Using Information Optimal Control.

We study the problem of tracking multiple diffusing particles using a laser scanning fluorescence microscope. The goal is to design trajectories for the laser to maximize the information contained in the measured intensity signal about the particles' trajectories. Our approach consists of a two level scheme: in the lower level we use an extremum seeking controller to track a single particle by first seeking it then orbiting around it. In the higher level controller, we decide which particle should be observed at each instant, with the goal of efficiently estimating each particle position while not losing track of any of them. Using simulations, we show that this technique is able to collect photons efficiently and to track multiple particles with low position estimation error.

Closed-Loop Nanopatterning of Liquids with Dip-Pen Nanolithography.

The ability to reliably manipulate small quantities of liquids is the backbone of high-throughput chemistry, but the continual drive for miniaturization necessitates creativity in how nanoscale samples of liquids are handled. Here, we describe a closed-loop method for patterning liquid samples on pL to sub-fL scales using scanning probe lithography. Specifically, we employ tipless scanning probes and identify liquid properties that enable probe-sample transport that is readily tuned using probe withdrawal speed. Subsequently, we introduce a novel two-harmonic inertial sensing scheme for tracking the mass of liquid on the probe. Finally, this is combined with a fluid mechanics-based iterative control scheme that selects printing conditions to meet a target feature mass to enable closed-loop patterning with better than 1% accuracy and ∼4% precision in terms of mass. Taken together, these advances address a pervasive issue in scanning probe lithography, namely, real-time closed-loop control over patterning, and position scanning probe lithography of liquids as a candidate for the robust nanoscale manipulation of liquids for advanced high-throughput chemistry.

Image reconstruction for sub-sampled atomic force microscopy images using deep neural networks.

Undersampling is a simple but efficient way to increase the imaging rate of atomic force microscopy (AFM). One major challenge in this approach is that of accurate image reconstruction from a limited number of measurements. In this work, we present a deep neural network (DNN) approach to reconstruct µ-path sub-sampled AFM images. Our network consists of two sub-networks, namely a RED-net and a U-net, in series, and is trained end-to-end from random images masked according to µ-path sub-sampling patterns. Using both simulation and experiments, the DNN is shown to yield better image quality than three existing optimization-based methods for reconstruction: basis pursuit, a variant of total variation minimization, and inpainting.

Feedforward Control for Single Particle Tracking Synthetic Motion.

Single particle tracking (SPT) is a method to study the transport of biomolecules with nanometer resolution. Unfortunately, recent reports show that systematic errors in position localization and uncertainty in model parameter estimates limits the utility of these techniques in studying biological processes. There is a need for an experimental method with a known ground-truth that tests the total SPT system (sample, microscope, algorithm) on both localization and estimation of model parameters. Synthetic motion is a known ground-truth method that moves a particle along a trajectory. This trajectory is a realization of a Markovian stochastic process that represents models of biomolecular transport. Here we describe a platform for creating synthetic motion using common equipment and well-known, simple methods that can be easily adopted by the biophysics community. In this paper we describe the synthetic motion system and calibration to achieve nanometer accuracy and precision. Steady state input-output characteristics are analyzed with both line scans and grid scans. The resulting relationship is described by an affine transformation, which is inverted and used as a prefilter. Model inverse feed forward control is used to increase the system bandwidth. The system model was identified from frequency response function measurements using an integrated stepped-sine with coherent demodulation built into the FPGA controller. Zero magnitude error tracking controller method was used to invert non-minimum phase zeros to achieve a stable discrete time feed forward filter.