Unsupervised activity perception in crowded and complicated scenes using hierarchical bayesian models.

We propose a novel unsupervised learning framework to model activities and interactions in crowded and complicated scenes. Hierarchical Bayesian models are used to connect three elements in visual surveillance: low-level visual features, simple "atomic" activities, and interactions. Atomic activities are modeled as distributions over low-level visual features, and multi-agent interactions are modeled as distributions over atomic activities. These models are learnt in an unsupervised way. Given a long video sequence, moving pixels are clustered into different atomic activities and short video clips are clustered into different interactions. In this paper, we propose three hierarchical Bayesian models, Latent Dirichlet Allocation (LDA) mixture model, Hierarchical Dirichlet Process (HDP) mixture model, and Dual Hierarchical Dirichlet Processes (Dual-HDP) model. They advance existing language models, such as LDA [1] and HDP [2]. Our data sets are challenging video sequences from crowded traffic scenes and train station scenes with many kinds of activities co-occurring. Without tracking and human labeling effort, our framework completes many challenging visual surveillance tasks of board interest such as: (1) discovering typical atomic activities and interactions; (2) segmenting long video sequences into different interactions; (3) segmenting motions into different activities; (4) detecting abnormality; and (5) supporting high-level queries on activities and interactions.

A method for clustering white matter fiber tracts.

BACKGROUND/PURPOSE: Despite its potential for visualizing white matter fiber tracts in vivo, diffusion tensor tractography has found only limited applications in clinical research in which specific anatomic connections between distant regions need to be evaluated. We introduce a robust method for fiber clustering that guides the separation of anatomically distinct fiber tracts and enables further estimation of anatomic connectivity between distant brain regions. METHODS: Line scanning diffusion tensor images (LSDTI) were acquired on a 1.5T magnet. Regions of interest for several anatomically distinct fiber tracts were manually drawn; then, white matter tractography was performed by using the Runge-Kutta method to interpolate paths (fiber traces) following the major directions of diffusion, in which traces were seeded only within the defined regions of interest. Next, a fully automatic procedure was applied to fiber traces, grouping them according to a pairwise similarity function that takes into account the shapes of the fibers and their spatial locations. RESULTS: We demonstrated the ability of the clustering algorithm to separate several fiber tracts which are otherwise difficult to define (left and right fornix, uncinate fasciculus and inferior occipitofrontal fasciculus, and corpus callosum fibers). CONCLUSION: This method successfully delineates fiber tracts that can be further analyzed for clinical research purposes. Hypotheses regarding specific fiber connections and their abnormalities in various neuropsychiatric disorders can now be tested.

Visual hemifield mapping using transcranial magnetic stimulation coregistered with cortical surfaces derived from magnetic resonance images.

The perception of a visual stimulus can be inhibited by occipital transcranial magnetic stimulation. This visual suppression effect has been attributed to disruption in the cortical gray matter of primary visual cortex or in the fiber tracts leading to V1 from the thalamus. However, others have suggested that the visual suppression effect is caused by disruption in secondary visual cortex. Here the authors used a figure-eight coil, which produces a focal magnetic field, and a Quadropulse stimulator to produce visual suppression contralateral to the stimulated hemisphere in five normal volunteer subjects. The authors coregistered the stimulation sites with magnetic resonance images in these same subjects using optical digitization. The stimulation sites were mapped onto the surface of the occipital lobes in three-dimensional reconstructions of the cortical surface to show the distribution of the visual suppression effect. The results were consistent with disruption of secondary visual cortical areas.

Segmentation of brain tissue from magnetic resonance images.

Segmentation of medical imagery is a challenging problem due to the complexity of the images, as well as to the absence of models of the anatomy that fully capture the possible deformations in each structure. The brain is a particularly complex structure, and its segmentation is an important step for many problems, including studies in temporal change detection of morphology, and 3-D visualizations for surgical planning. We present a method for segmentation of brain tissue from magnetic resonance images that is a combination of three existing techniques from the computer vision literature: expectation/maximization segmentation, binary mathematical morphology, and active contour models. Each of these techniques has been customized for the problem of brain tissue segmentation such that the resultant method is more robust than its components. Finally, we present the results of a parallel implementation of this method on IBM's supercomputer Power Visualization System for a database of 20 brain scans each with 256 x 256 x 124 voxels and validate those results against segmentations generated by neuroanatomy experts.

CURVES: curve evolution for vessel segmentation.

The vasculature is of utmost importance in neurosurgery. Direct visualization of images acquired with current imaging modalities, however, cannot provide a spatial representation of small vessels. These vessels, and their branches which show considerable variations, are most important in planning and performing neurosurgical procedures. In planning they provide information on where the lesion draws its blood supply and where it drains. During surgery the vessels serve as landmarks and guidelines to the lesion. The more minute the information is, the more precise the navigation and localization of computer guided procedures. Beyond neurosurgery and neurological study, vascular information is also crucial in cardiovascular surgery, diagnosis, and research. This paper addresses the problem of automatic segmentation of complicated curvilinear structures in three-dimensional imagery, with the primary application of segmenting vasculature in magnetic resonance angiography (MRA) images. The method presented is based on recent curve and surface evolution work in the computer vision community which models the object boundary as a manifold that evolves iteratively to minimize an energy criterion. This energy criterion is based both on intensity values in the image and on local smoothness properties of the object boundary, which is the vessel wall in this application. In particular, the method handles curves evolving in 3D, in contrast with previous work that has dealt with curves in 2D and surfaces in 3D. Results are presented on cerebral and aortic MRA data as well as lung computed tomography (CT) data.

Experimentation with a transcranial magnetic stimulation system for functional brain mapping.

We describe functional brain mapping experiments using a transcranial magnetic stimulation (TMS) device. This device, when placed on a subject's scalp, stimulates the underlying neurons by generating focused magnetic field pulses. A brain mapping is then generated by measuring responses of different motor and sensory functions to this stimulation. The key process in generating this mapping is the association of the 3-D positions and orientations of the TMS probe on the scalp to a 3-D brain reconstruction such as is feasible with a magnetic resonance image (MRI). We have developed a registration system which not only generates functional brain maps using such a device, but also provides real-time feedback to guide the technician in placing the probe at appropriate points on the head to achieve the desired map resolution. Functional areas we have mapped are the motor and visual cortex. Validation experiments focus on repeatability tests for mapping the same subjects several times. Applications of the technique include neuroanatomy research, surgical planning and guidance, treatment and disease monitoring, and therapeutic procedures.

Computational experiments with a feature based stereo algorithm.

Computational models of the human stereo system can provide insight into general information processing constraints that apply to any stereo system, either artificial or biological. In 1977 Marr and Poggio proposed one such computational model, which was characterized as matching certain feature points in difference-of-Gaussian filtered images and using the information obtained by matching coarser resolution representations to restrict the search space for matching finer resolution representations. An implementation of the algorithm and its testing on a range of images was reported in 1980. Since then a number of psychophysical experiments have suggested possible refinements to the model and modifications to the algorithm. As well, recent computational experiments applying the algorithm to a variety of natural images, especially aerial photographs, have led to a number of modifications. In this paper, we present a version of the Marr-Poggio-Grimson algorithm that embodies these modifications, and we illustrate its performance on a series of natural images.

Localizing overlapping parts by searching the interpretation tree.

This paper discusses how local measurements of positions and surface normals may be used to identify and locate overlapping objects. The objects are modeled as polyhedra (or polygons) having up to six degrees of positional freedom relative to the sensors. The approach operates by examining all hypotheses about pairings between sensed data and object surfaces and efficiently discarding inconsistent ones by using local constraints on: distances between faces, angles between face normals, and angles (relative to the surface normals) of vectors between sensed points. The method described here is an extension of a method for recognition and localization of nonoverlapping parts previously described in [18] and [15].

Comments on ``digital step edges from zero crossings of second directional derivatives''.

In a recent paper,1 Haralick published an edge detection scheme that was supported, in part, by an evaluation against the Prewitt and the Marr-Hildreth (¿2G) operators. This evaluation led to the conclusion that Haralick's method performed the best, and the ¿2G operator performed the worst. The implementation of the ¿2G operator, on which this evaluation was based, differed significantly from that used by Marr and Hildreth. Evaluation of the performance of the Marr-Hildreth implementation of the ¿2G operator on similar images shows that this edge detection method in fact performs comparably to the Prewitt and Haralick operators.

AnatomyBrowser: A novel approach to visualization and integration of medical information.

In this article, we present a novel technique for visualization of three-dimensional (3D) surface models, as well as its implementation in a system called AnatomyBrowser. Using our approach, visualization of 3D surface models is performed in two separate steps: a pre-rendering step, in which the models are rendered and saved in a special format, and an actual display step, in which the final result of rendering is generated using information from the prerendering step. Whereas prerendering requires high-end graphics hardware, the final image generation and display can be implemented efficiently in software. Moreover, our current implementation of AnatomyBrowser interface uses the Java programming language and can therefore be readily run on a wide range of systems, including low-end computers with no special graphics hardware. In addition to visualization of 3D models and 2D slices, AnatomyBrowser provides a rich set of annotation and cross-referencing capabilities. We demonstrate several possible applications for AnatomyBrowser, including interactive anatomy atlases, surgery planning, and assistance in segmentation.

A computer implementation of a theory of human stereo vision.

Recently, Marr & Poggio (1979) presented a theory of human stereo vision. An implementation of that theory is presented, and consists of five steps. (i) The left and right images are each filtered with masks of four sizes that increase with eccentricity; the shape of these masks is given by delta 2G, the Laplacian of a Gaussian function. (ii) Zero crossings in the filtered images are found along horizontal scan lines. (iii) For each mask size, matching takes place between zero crossings of the same sign and roughly the same orientation in the two images, for a range of disparities up to about the width of the mask's central region. Within this disparity range, it can be shown that false targets pose only a simple problem. (iv) The output of the wide masks can control vergence movements, thus causing small masks to come into correspondence. In this way, the matching process gradually moves from dealing with large disparities at a low resolution to dealing with small disparities at a high resolution. (v) When a correspondence is achieved, it is stored in a dynamic buffer, called the 2 1/2-dimensional sketch. To support the adequacy of the Marr-Poggio model of human stereo vision, the implementation was tested on a wide range of stereograms from the human stereopsis literature. The performance of the implementation is illustrated and compared with human perception. Also statistical assumptions made by Marr & Poggio are supported by comparison with statistics found in practice. Finally, the process of implementing the theory has led to the clarification and refinement of a number of details within the theory; these are discussed in detail.

The intelligent camera: images of computer vision.

The field of computer vision is concerned with the problem of deducing properties of object surfaces and the position and orientation of objects in scenes. I illustrate the fundamental difficulty of the problem, describe common approaches taken in the field, and provide representative examples of current solutions.

A computational theory of visual surface interpolation.

Computational theories of structure-from-motion and stereo vision only specify the computation of three-dimensional surface information at special points in the image. Yet the visual perception is clearly of complete surfaces. To account for this a computational theory of the interpolation of surfaces from visual information is presented. The problem is constrained by the fact that the surface must agree with the information from stereo or motion correspondence, and not vary radically between these points. Using the image irradiance equation, an explicit form of this surface consistency constraint can be derived. To determine which of two possible surfaces is more consistent with the surface consistency constraint, one must be able to compare the two surfaces. To do this, a functional from the space of possible functions to the real numbers is required. In this way, the surface most consistent with the visual information will be that which minimizes the functional. To ensure that the functional has a unique minimal surface, conditions on the form of the functional are derived. In particular, if the functional is a complete semi-norm that satisfies the parallelogram law, or the space of functions is a semi-Hilbert space and the functional is a semi-inner product, then there is a unique (to within possibly an element of the null space of the functional) surface that is most consistent with the visual information. It can be shown, based on the above conditions plus a condition of rotational symmetry, that there is a vector space of possible functionals that measure surface consistency, this vector space being spanned by the functional of quadratic variation and the functional of square Laplacian. Arguments based on the null spaces of the respective functionals are used to justify the choice of the quadratic variation as the optimal functional. Possible refinements to the theory, concerning the role of discontinuities in depth and the effects of applying the interpolation process to scenes containing more than one object, are discussed.

An integrated visualization system for surgical planning and guidance using image fusion and an open MR.

A surgical guidance and visualization system is presented, which uniquely integrates capabilities for data analysis and on-line interventional guidance into the setting of interventional MRI. Various pre-operative scans (T1- and T2-weighted MRI, MR angiography, and functional MRI (fMRI)) are fused and automatically aligned with the operating field of the interventional MR system. Both pre-surgical and intra-operative data may be segmented to generate three-dimensional surface models of key anatomical and functional structures. Models are combined in a three-dimensional scene along with reformatted slices that are driven by a tracked surgical device. Thus, pre-operative data augments interventional imaging to expedite tissue characterization and precise localization and targeting. As the surgery progresses, and anatomical changes subsequently reduce the relevance of pre-operative data, interventional data is refreshed for software navigation in true real time. The system has been applied in 45 neurosurgical cases and found to have beneficial utility for planning and guidance. J. Magn. Reson. Imaging 2001;13:967-975.