3D flow field estimation and assessment for live cell fluorescence microscopy.

MOTIVATION: The revolution in light sheet microscopy enables the concurrent observation of thousands of dynamic processes, from single molecules to cellular organelles, with high spatiotemporal resolution. However, challenges in the interpretation of multidimensional data requires the fully automatic measurement of those motions to link local processes to cellular functions. This includes the design and the implementation of image processing pipelines able to deal with diverse motion types, and 3D visualization tools adapted to the human visual system. RESULTS: Here, we describe a new method for 3D motion estimation that addresses the aforementioned issues. We integrate 3D matching and variational approach to handle a diverse range of motion without any prior on the shape of moving objects. We compare different similarity measures to cope with intensity ambiguities and demonstrate the effectiveness of the Census signature for both stages. Additionally, we present two intuitive visualization approaches to adapt complex 3D measures into an interpretable 2D view, and a novel way to assess the quality of flow estimates in absence of ground truth. AVAILABILITY AND IMPLEMENTATION: https://team.inria.fr/serpico/data/3d-optical-flow-data/. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

An extended model of vesicle fusion at the plasma membrane to estimate protein lateral diffusion from TIRF microscopy images.

BACKGROUND: Characterizing membrane dynamics is a key issue to understand cell exchanges with the extra-cellular medium. Total internal reflection fluorescence microscopy (TIRFM) is well suited to focus on the late steps of exocytosis at the plasma membrane. However, it is still a challenging task to quantify (lateral) diffusion and estimate local dynamics of proteins. RESULTS: A new model was introduced to represent the behavior of cargo transmembrane proteins during the vesicle fusion to the plasma membrane at the end of the exocytosis process. Two biophysical parameters, the diffusion coefficient and the release rate parameter, are automatically estimated from TIRFM image sequences, to account for both the lateral diffusion of molecules at the membrane and the continuous release of the proteins from the vesicle to the plasma membrane. Quantitative evaluation on 300 realistic computer-generated image sequences demonstrated the efficiency and accuracy of the method. The application of our method on 16 real TIRFM image sequences additionally revealed differences in the dynamic behavior of Transferrin Receptor (TfR) and Langerin proteins. CONCLUSION: An automated method has been designed to simultaneously estimate the diffusion coefficient and the release rate for each individual vesicle fusion event at the plasma membrane in TIRFM image sequences. It can be exploited for further deciphering cell membrane dynamics.

EM-Driven Unsupervised Learning for Efficient Motion Segmentation.

In this paper, we present a CNN-based fully unsupervised method for motion segmentation from optical flow. We assume that the input optical flow can be represented as a piecewise set of parametric motion models, typically, affine or quadratic motion models. The core idea of our work is to leverage the Expectation-Maximization (EM) framework in order to design in a well-founded manner a loss function and a training procedure of our motion segmentation neural network that does not require either ground-truth or manual annotation. However, in contrast to the classical iterative EM, once the network is trained, we can provide a segmentation for any unseen optical flow field in a single inference step and without estimating any motion models. We investigate different loss functions including robust ones and propose a novel efficient data augmentation technique on the optical flow field, applicable to any network taking optical flow as input. In addition, our method is able by design to segment multiple motions. Our motion segmentation network was tested on four benchmarks, DAVIS2016, SegTrackV2, FBMS59, and MoCA, and performed very well, while being fast at test time.

ROAM: A Rich Object Appearance Model with Application to Rotoscoping.

Rotoscoping, the detailed delineation of scene elements through a video shot, is a painstaking task of tremendous importance in professional post-production pipelines. While pixel-wise segmentation techniques can help for this task, professional rotoscoping tools rely on parametric curves that offer the artists a much better interactive control on the definition, editing and manipulation of the segments of interest. Sticking to this prevalent rotoscoping paradigm, we propose a novel framework to capture and track the visual aspect of an arbitrary object in a scene, given an initial closed outline of this object. This model combines a collection of local foreground/background appearance models spread along the outline, a global appearance model of the enclosed object and a set of distinctive foreground landmarks. The structure of this rich appearance model allows simple initialization, efficient iterative optimization with exact minimization at each step, and on-line adaptation in videos. We further extend this model by so-called trimaps which serve as an input to alpha-matting algorithms to allow truly seamless compositing. To this end, we leverage local classifiers attached to the roto-curves to define a confidence measure that is well-suited to define trimaps with adaptive band-widths. The resulting trimaps are parametric, temporally consistent and remain fully editable by the artist. We demonstrate qualitatively and quantitatively the merit of this framework through comparisons with tools based on either dynamic segmentation with a closed curve or pixel-wise binary labelling.

Space-time adaptation for patch-based image sequence restoration.

We present a novel space-time patch-based method for image sequence restoration. We propose an adaptive statistical estimation framework based on the local analysis of the bias-variance trade-off. At each pixel, the space-time neighborhood is adapted to improve the performance of the proposed patch-based estimator. The proposed method is unsupervised and requires no motion estimation. Nevertheless, it can also be combined with motion estimation to cope with very large displacements due to camera motion. Experiments show that this method is able to drastically improve the quality of highly corrupted image sequences. Quantitative evaluations on standard artificially noise-corrupted image sequences demonstrate that our method outperforms other recent competitive methods. We also report convincing results on real noisy image sequences.

Recognition of dynamic video contents with global probabilistic models of visual motion.

The exploitation of video data requires methods able to extract high-level information from the images. Video summarization, video retrieval, or video surveillance are examples of applications. In this paper, we tackle the challenging problem of recognizing dynamic video contents from low-level motion features. We adopt a statistical approach involving modeling, (supervised) learning, and classification issues. Because of the diversity of video content (even for a given class of events), we have to design appropriate models of visual motion and learn them from videos. We have defined original parsimonious global probabilistic motion models, both for the dominant image motion (assumed to be due to the camera motion) and the residual image motion (related to scene motion). Motion measurements include affine motion models to capture the camera motion and low-level local motion features to account for scene motion. Motion learning and recognition are solved using maximum likelihood criteria. To validate the interest of the proposed motion modeling and recognition framework, we report dynamic content recognition results on sports videos.

Adaptive Spot Detection With Optimal Scale Selection in Fluorescence Microscopy Images.

Accurately detecting subcellular particles in fluorescence microscopy is of primary interest for further quantitative analysis such as counting, tracking, or classification. Our primary goal is to segment vesicles likely to share nearly the same size in fluorescence microscopy images. Our method termed adaptive thresholding of Laplacian of Gaussian (LoG) images with autoselected scale (ATLAS) automatically selects the optimal scale corresponding to the most frequent spot size in the image. Four criteria are proposed and compared to determine the optimal scale in a scale-space framework. Then, the segmentation stage amounts to thresholding the LoG of the intensity image. In contrast to other methods, the threshold is locally adapted given a probability of false alarm (PFA) specified by the user for the whole set of images to be processed. The local threshold is automatically derived from the PFA value and local image statistics estimated in a window whose size is not a critical parameter. We also propose a new data set for benchmarking, consisting of six collections of one hundred images each, which exploits backgrounds extracted from real microscopy images. We have carried out an extensive comparative evaluation on several data sets with ground-truth, which demonstrates that ATLAS outperforms existing methods. ATLAS does not need any fine parameter tuning and requires very low computation time. Convincing results are also reported on real total internal reflection fluorescence microscopy images.

Background fluorescence estimation and vesicle segmentation in live cell imaging with conditional random fields.

Image analysis applied to fluorescence live cell microscopy has become a key tool in molecular biology since it enables to characterize biological processes in space and time at the subcellular level. In fluorescence microscopy imaging, the moving tagged structures of interest, such as vesicles, appear as bright spots over a static or nonstatic background. In this paper, we consider the problem of vesicle segmentation and time-varying background estimation at the cellular scale. The main idea is to formulate the joint segmentation-estimation problem in the general conditional random field framework. Furthermore, segmentation of vesicles and background estimation are alternatively performed by energy minimization using a min cut-max flow algorithm. The proposed approach relies on a detection measure computed from intensity contrasts between neighboring blocks in fluorescence microscopy images. This approach permits analysis of either 2D + time or 3D + time data. We demonstrate the performance of the so-called C-CRAFT through an experimental comparison with the state-of-the-art methods in fluorescence video-microscopy. We also use this method to characterize the spatial and temporal distribution of Rab6 transport carriers at the cell periphery for two different specific adhesion geometries.

Nonparametric motion characterization using causal probabilistic models for video indexing and retrieval.

This paper describes an original approach for content-based video indexing and retrieval. We aim at providing a global interpretation of the dynamic content of video shots without any prior motion segmentation and without any use of dense optic flow fields. To this end, we exploit the spatio-temporal distribution, within a shot, of appropriate local motion-related measurements derived from the spatio-temporal derivatives of the intensity function. These distributions are then represented by causal Gibbs models. To be independent of camera movement, the motion-related measurements are computed in the image sequence generated by compensating the estimated dominant image motion in the original sequence. The statistical modeling framework considered makes the exact computation of the conditional likelihood of a video shot belonging to a given motion or more generally to an activity class feasible. This property allows us to develop a general statistical framework for video indexing and retrieval with query-by-example. We build a hierarchical structure of the processed video database according to motion content similarity. This results in a binary tree where each node is associated to an estimated causal Gibbs model. We consider a similarity measure inspired from Kullback-Leibler divergence. Then, retrieval with query-by-example is performed through this binary tree using the maximum a posteriori (MAP) criterion. We have obtained promising results on a set of various real image sequences.

Patch-based nonlocal functional for denoising fluorescence microscopy image sequences.

We present a nonparametric regression method for denoising 3-D image sequences acquired via fluorescence microscopy. The proposed method exploits the redundancy of the 3-D+time information to improve the signal-to-noise ratio of images corrupted by Poisson-Gaussian noise. A variance stabilization transform is first applied to the image-data to remove the dependence between the mean and variance of intensity values. This preprocessing requires the knowledge of parameters related to the acquisition system, also estimated in our approach. In a second step, we propose an original statistical patch-based framework for noise reduction and preservation of space-time discontinuities. In our study, discontinuities are related to small moving spots with high velocity observed in fluorescence video-microscopy. The idea is to minimize an objective nonlocal energy functional involving spatio-temporal image patches. The minimizer has a simple form and is defined as the weighted average of input data taken in spatially-varying neighborhoods. The size of each neighborhood is optimized to improve the performance of the pointwise estimator. The performance of the algorithm (which requires no motion estimation) is then evaluated on both synthetic and real image sequences using qualitative and quantitative criteria.

A simulation and estimation framework for intracellular dynamics and trafficking in video-microscopy and fluorescence imagery.

Image sequence analysis in video-microscopy has now gained importance since molecular biology is presently having a profound impact on the way research is being conducted in medicine. However, image processing techniques that are currently used for modeling intracellular dynamics, are still relatively crude and yield imprecise results. Indeed, complex interactions between a large number of small moving particles in a complex scene cannot be easily modeled, limiting the performance of object detection and tracking algorithms. This motivates our present research effort which is to develop a general estimation/simulation framework able to produce image sequences showing small moving spots in interaction, with variable velocities, and corresponding to intracellular dynamics and trafficking in biology. It is now well established that spot/object trajectories can play a role in the analysis of living cell dynamics and simulating realistic image sequences is then of major importance. We demonstrate the potential of the proposed simulation/estimation framework in experiments, and show that this approach can also be used to evaluate the performance of object detection/tracking algorithms in video-microscopy and fluorescence imagery.

Adaptive spatio-temporal restoration for 4D fluorescence microscopic imaging.

We present a spatio-temporal filtering method for significantly increasing the signal-to-noise ratio (SNR) in noisy fluorescence microscopic image sequences where small particles have to be tracked from frame to frame. Image sequence restoration is achieved using a statistical approach involving an appropriate on-line window geometry specification. We have applied this method to noisy synthetic and real microscopic image sequences where a large number of small fluorescently labeled vesicles are moving in regions close to the Golgi apparatus. The SNR is shown to be drastically improved and the enhanced vesicles can be segmented. This novel approach can be further exploited for biological studies where the dynamics of small objects of interest have to be analyzed in molecular and sub-cellular bio-imaging.

Multiresolution parametric estimation of transparent motions and denoising of fluoroscopic images.

We describe a novel multiresolution parametric framework to estimate transparent motions typically present in X-Ray exams. Assuming the presence if two transparent layers, it computes two affine velocity fields by minimizing an appropriate objective function with an incremental Gauss-Newton technique. We have designed a realistic simulation scheme of fluoroscopic image sequences to validate our method on data with ground truth and different levels of noise. An experiment on real clinical images is also reported. We then exploit this transparent-motion estimation method to denoise two layers image sequences using a motion-compensated estimation method. In accordance with theory, we show that we reach a denoising factor of 2/3 in a few iterations without bringing any local artifacts in the image sequence.