Dense Semantic 3D Reconstruction.

Both image segmentation and dense 3D modeling from images represent an intrinsically ill-posed problem. Strong regularizers are therefore required to constrain the solutions from being 'too noisy'. These priors generally yield overly smooth reconstructions and/or segmentations in certain regions while they fail to constrain the solution sufficiently in other areas. In this paper, we argue that image segmentation and dense 3D reconstruction contribute valuable information to each other's task. As a consequence, we propose a mathematical framework to formulate and solve a joint segmentation and dense reconstruction problem. On the one hand knowing about the semantic class of the geometry provides information about the likelihood of the surface direction. On the other hand the surface direction provides information about the likelihood of the semantic class. Experimental results on several data sets highlight the advantages of our joint formulation. We show how weakly observed surfaces are reconstructed more faithfully compared to a geometry only reconstruction. Thanks to the volumetric nature of our formulation we also infer surfaces which cannot be directly observed for example the surface between the ground and a building. Finally, our method returns a semantic segmentation which is consistent across the whole dataset.

Automatic atlas-based three-label cartilage segmentation from MR knee images.

Osteoarthritis (OA) is the most common form of joint disease and often characterized by cartilage changes. Accurate quantitative methods are needed to rapidly screen large image databases to assess changes in cartilage morphology. We therefore propose a new automatic atlas-based cartilage segmentation method for future automatic OA studies. Atlas-based segmentation methods have been demonstrated to be robust and accurate in brain imaging and therefore also hold high promise to allow for reliable and high-quality segmentations of cartilage. Nevertheless, atlas-based methods have not been well explored for cartilage segmentation. A particular challenge is the thinness of cartilage, its relatively small volume in comparison to surrounding tissue and the difficulty to locate cartilage interfaces - for example the interface between femoral and tibial cartilage. This paper focuses on the segmentation of femoral and tibial cartilage, proposing a multi-atlas segmentation strategy with non-local patch-based label fusion which can robustly identify candidate regions of cartilage. This method is combined with a novel three-label segmentation method which guarantees the spatial separation of femoral and tibial cartilage, and ensures spatial regularity while preserving the thin cartilage shape through anisotropic regularization. Our segmentation energy is convex and therefore guarantees globally optimal solutions. We perform an extensive validation of the proposed method on 706 images of the Pfizer Longitudinal Study. Our validation includes comparisons of different atlas segmentation strategies, different local classifiers, and different types of regularizers. To compare to other cartilage segmentation approaches we validate based on the 50 images of the SKI10 dataset.

Multi-modal registration for correlative microscopy using image analogies.

Correlative microscopy is a methodology combining the functionality of light microscopy with the high resolution of electron microscopy and other microscopy technologies for the same biological specimen. In this paper, we propose an image registration method for correlative microscopy, which is challenging due to the distinct appearance of biological structures when imaged with different modalities. Our method is based on image analogies and allows to transform images of a given modality into the appearance-space of another modality. Hence, the registration between two different types of microscopy images can be transformed to a mono-modality image registration. We use a sparse representation model to obtain image analogies. The method makes use of corresponding image training patches of two different imaging modalities to learn a dictionary capturing appearance relations. We test our approach on backscattered electron (BSE) scanning electron microscopy (SEM)/confocal and transmission electron microscopy (TEM)/confocal images. We perform rigid, affine, and deformable registration via B-splines and show improvements over direct registration using both mutual information and sum of squared differences similarity measures to account for differences in image appearance.

What is optimized in convex relaxations for multilabel problems: connecting discrete and continuously inspired MAP inference.

In this work, we present a unified view on Markov random fields (MRFs) and recently proposed continuous tight convex relaxations for multilabel assignment in the image plane. These relaxations are far less biased toward the grid geometry than Markov random fields on grids. It turns out that the continuous methods are nonlinear extensions of the well-established local polytope MRF relaxation. In view of this result, a better understanding of these tight convex relaxations in the discrete setting is obtained. Further, a wider range of optimization methods is now applicable to find a minimizer of the tight formulation. We propose two methods to improve the efficiency of minimization. One uses a weaker, but more efficient continuously inspired approach as initialization and gradually refines the energy where it is necessary. The other one reformulates the dual energy enabling smooth approximations to be used for efficient optimization. We demonstrate the utility of our proposed minimization schemes in numerical experiments. Finally, we generalize the underlying energy formulation from isotropic metric smoothness costs to arbitrary nonmetric and orientation dependent smoothness terms.

Segmentation with area constraints.

Image segmentation approaches typically incorporate weak regularity conditions such as boundary length or curvature terms, or use shape information. High-level information such as a desired area or volume, or a particular topology are only implicitly specified. In this paper we develop a segmentation method with explicit bounds on the segmented area. Area constraints allow for the soft selection of meaningful solutions, and can counteract the shrinking bias of length-based regularization. We analyze the intrinsic problems of convex relaxations proposed in the literature for segmentation with size constraints. Hence, we formulate the area-constrained segmentation task as a mixed integer program, propose a branch and bound method for exact minimization, and use convex relaxations to obtain the required lower energy bounds on candidate solutions. We also provide a numerical scheme to solve the convex subproblems. We demonstrate the method for segmentations of vesicles from electron tomography images.

Globally Optimal Finsler Active Contours.

We present a continuous and convex formulation for Finsler active contours using seed regions or utilizing a regional bias term. The utilization of general Finsler metrics instead of Riemannian metrics allows the segmentation boundary to favor appropriate locations (e.g. with strong image discontinuities) and suitable directions (e.g. aligned with dark to bright image gradients). Strong edges are not required everywhere along the desired segmentation boundary due to incorporation of a regional bias. The resulting optimization procedure is simple and efficient, and leads to binary segmentation results regardless of the underlying continuous formulation. We demonstrate the proposed method in several examples.

Continuous Maximal Flows and Wulff Shapes: Application to MRFs.

Convex and continuous energy formulations for low level vision problems enable efficient search procedures for the corresponding globally optimal solutions. In this work we extend the well-established continuous, isotropic capacity-based maximal flow framework to the anisotropic setting. By using powerful results from convex analysis, a very simple and efficient minimization procedure is derived. Further, we show that many important properties carry over to the new anisotropic framework, e.g. globally optimal binary results can be achieved simply by thresholding the continuous solution. In addition, we unify the anisotropic continuous maximal flow approach with a recently proposed convex and continuous formulation for Markov random fields, thereby allowing more general smoothness priors to be incorporated. Dense stereo results are included to illustrate the capabilities of the proposed approach.

Near-tubular fiber bundle segmentation for diffusion weighted imaging: segmentation through frame reorientation.

This paper proposes a methodology to segment near-tubular fiber bundles from diffusion weighted magnetic resonance images (DW-MRI). Segmentation is simplified by locally reorienting diffusion information based on large-scale fiber bundle geometry. Segmentation is achieved through simple global statistical modeling of diffusion orientation. Utilizing a modification of a recent segmentation approach by Bresson et al. allows for a convex optimization formulation of the segmentation problem, combining orientation statistics and spatial regularization. The approach compares favorably with segmentation by full-brain streamline tractography.

A duality based algorithm for TV-L1-optical-flow image registration.

Nonlinear image registration is a challenging task in the field of medical image analysis. In many applications discontinuities may be present in the displacement field, and intensity variations may occur. In this work we therefore utilize an energy functional which is based on Total Variation regularization and a robust data term. We propose a novel, fast and stable numerical scheme to find the minimizer of this energy. Our approach combines a fixed-point procedure derived from duality principles combined with a fast thresholding step. We show experimental results on synthetic and clinical CT lung data sets at different breathing states as well as registration results on inter-subject brain MRIs.

Automatic point landmark matching for regularizing nonlinear intensity registration: application to thoracic CT images.

Nonlinear image registration is a prerequisite for a variety of medical image analysis tasks. A frequently used registration method is based on manually or automatically derived point landmarks leading to a sparse displacement field which is densified in a thin-plate spline (TPS) framework. A large problem of TPS interpolation/approximation is the requirement for evenly distributed landmark correspondences over the data set which can rarely be guaranteed by landmark matching algorithms. We propose to overcome this problem by combining the sparse correspondences with intensity-based registration in a generic nonlinear registration scheme based on the calculus of variations. Missing landmark information is compensated by a stronger intensity term, thus combining the strengths of both approaches. An explicit formulation of the generic framework is derived that constrains an intra-modality intensity data term with a regularization term from the corresponding landmarks and an anisotropic image-driven displacement regularization term. An evaluation of this algorithm is performed comparing it to an intensity- and a landmark-based method. Results on four synthetically deformed and four clinical thorax CT data sets at different breathing states are shown.