Tracking and Classification of Head Movement for Augmentative and Alternative Communication Systems.

The use of assistive technologies can mitigate or reduce the challenges faced by individuals with motor disabilities to use computer systems. However, those who feature severe involuntary movements often have fewer options at hand. This work describes an application that can recognize the user's head using a conventional webcam, track its motion, model the desired functional movement, and recognize it to enable the use of a virtual keyboard. The proposed classifier features a flexible structure and may be personalized for different user need. Experimental results obtained with participants with no neurological disorders have shown that classifiers based on Hidden Markov Models provided similar or better performance than a classifier based on position threshold. However, motion segmentation and interpretation modules were sensitive to involuntary movements featured by participants with cerebral palsy that took part in the study.

Retinal slit lamp video mosaicking.

PURPOSE: To this day, the slit lamp remains the first tool used by an ophthalmologist to examine patient eyes. Imaging of the retina poses, however, a variety of problems, namely a shallow depth of focus, reflections from the optical system, a small field of view and non-uniform illumination. For ophthalmologists, the use of slit lamp images for documentation and analysis purposes, however, remains extremely challenging due to large image artifacts. For this reason, we propose an automatic retinal slit lamp video mosaicking, which enlarges the field of view and reduces amount of noise and reflections, thus enhancing image quality. METHODS: Our method is composed of three parts: (i) viable content segmentation, (ii) global registration and (iii) image blending. Frame content is segmented using gradient boosting with custom pixel-wise features. Speeded-up robust features are used for finding pair-wise translations between frames with robust random sample consensus estimation and graph-based simultaneous localization and mapping for global bundle adjustment. Foreground-aware blending based on feathering merges video frames into comprehensive mosaics. RESULTS: Foreground is segmented successfully with an area under the curve of the receiver operating characteristic curve of 0.9557. Mosaicking results and state-of-the-art methods were compared and rated by ophthalmologists showing a strong preference for a large field of view provided by our method. CONCLUSIONS: The proposed method for global registration of retinal slit lamp images of the retina into comprehensive mosaics improves over state-of-the-art methods and is preferred qualitatively.

Fundus image mosaicking for information augmentation in computer-assisted slit-lamp imaging.

Laser photocoagulation is currently the standard treatment for sight-threatening diseases worldwide, namely diabetic retinopathy and retinal vein occlusions. The slit lamp biomicroscope is the most commonly used device for this procedure, specially for the treatment of the eye periphery. However, only a small portion of the retina can be visualized through the biomicroscope, complicating the task of localizing and identifying surgical targets, increasing treatment duration and patient discomfort. In order to assist surgeons, we propose a method for creating intraoperative retina maps for view expansion using a slit-lamp device. Based on the mosaicking method described by Richa et al, 2012, the proposed method is a combination of direct and feature-based methods, suitable for the textured nature of the human retina. In this paper, we describe three major enhancements to the original formulation. The first is a visual tracking method using local illumination compensation to cope with the challenging visualization conditions. The second is an efficient pixel selection scheme for increased computational efficiency. The third is an entropy-based mosaic update method to dynamically improve the retina map during exploration. To evaluate the performance of the proposed method, we conducted several experiments on human subjects with a computer-assisted slit-lamp prototype. We also demonstrate the practical value of the system for photo documentation, diagnosis and intraoperative navigation.

Unified detection and tracking of instruments during retinal microsurgery.

Methods for tracking an object have generally fallen into two groups: tracking by detection and tracking through local optimization. The advantage of detection-based tracking is its ability to deal with target appearance and disappearance, but it does not naturally take advantage of target motion continuity during detection. The advantage of local optimization is efficiency and accuracy, but it requires additional algorithms to initialize tracking when the target is lost. To bridge these two approaches, we propose a framework for unified detection and tracking as a time-series Bayesian estimation problem. The basis of our approach is to treat both detection and tracking as a sequential entropy minimization problem, where the goal is to determine the parameters describing a target in each frame. To do this we integrate the Active Testing (AT) paradigm with Bayesian filtering, and this results in a framework capable of both detecting and tracking robustly in situations where the target object enters and leaves the field of view regularly. We demonstrate our approach on a retinal tool tracking problem and show through extensive experiments that our method provides an efficient and robust tracking solution.

Hybrid tracking and mosaicking for information augmentation in retinal surgery.

Current technical limitations in retinal surgery hinder the ability of surgeons to identify and localize surgical targets, increasing operating times and risks of surgical error. In this paper we present a hybrid tracking and mosaicking method for augmented reality in retinal surgery. The system is a combination of direct and feature-based tracking methods. A novel extension for direct visual tracking using a robust image similarity measure in color images is also proposed. Several experiments conducted on phantom, in vivo rabbit and human images attest the ability of the method to cope with the challenging retinal surgery scenario. Applications of the proposed method for tele-mentoring and intra-operative guidance are demonstrated.

Data-driven visual tracking in retinal microsurgery.

In the context of retinal microsurgery, visual tracking of instruments is a key component of robotics assistance. The difficulty of the task and major reason why most existing strategies fail on in-vivo image sequences lies in the fact that complex and severe changes in instrument appearance are challenging to model. This paper introduces a novel approach, that is both data-driven and complementary to existing tracking techniques. In particular, we show how to learn and integrate an accurate detector with a simple gradient-based tracker within a robust pipeline which runs at framerate. In addition, we present a fully annotated dataset of retinal instruments in in-vivo surgeries, which we use to quantitatively validate our approach. We also demonstrate an application of our method in a laparascopy image sequence.

Unified detection and tracking in retinal microsurgery.

Traditionally, tool tracking involves two subtasks: (i) detecting the tool in the initial image in which it appears, and (ii) predicting and refining the configuration of the detected tool in subsequent images. With retinal microsurgery in mind, we propose a unified tool detection and tracking framework, removing the need for two separate systems. The basis of our approach is to treat both detection and tracking as a sequential entropy minimization problem, where the goal is to determine the parameters describing a surgical tool in each frame. The resulting framework is capable of both detecting and tracking in situations where the tool enters and leaves the field of view regularly. We demonstrate the benefits of this method in the context of retinal tool tracking. Through extensive experimentation on a phantom eye, we show that this method provides efficient and robust tool tracking and detection.

Towards robust 3D visual tracking for motion compensation in beating heart surgery.

In the context of minimally invasive cardiac surgery, active vision-based motion compensation schemes have been proposed for mitigating problems related to physiological motion. However, robust and accurate visual tracking remains a difficult task. The purpose of this paper is to present a robust visual tracking method that estimates the 3D temporal and spatial deformation of the heart surface using stereo endoscopic images. The novelty is the combination of a visual tracking method based on a Thin-Plate Spline (TPS) model for representing the heart surface deformations with a temporal heart motion model based on a time-varying dual Fourier series for overcoming tracking disturbances or failures. The considerable improvements in tracking robustness facing specular reflections and occlusions are demonstrated through experiments using images of in vivo porcine and human beating hearts.

Robust 3D tracking for robotic-assisted beating heart surgery.

The past decades have witnessed the notable development of minimally invasive surgery (MIS). The benefits of this modality of surgery for patients are numerous, shortening convalescence, reducing trauma and surgery costs. In this context, robotic assistance aims to make the surgical act more intuitive and safer. In the domain of cardiac MIS, heartbeat and respiration represent two important sources of disturbances. Even though miniaturized versions of heart stabilizers have been conceived for the MIS scenario, residual motion is still considerable and has to be manually canceled by the surgeon. Our work focuses on computer vision techniques for estimating the 3D motion of the heart relying solely on natural structures on the heart surface for active compensation of physiological motions. We have developed in [2] a visual tracking method for estimating the 3D deformation of a region of interest on the heart surface based on the visual feedback of a stereo endoscope. The method is robust to illumination variations and large tissue deformations (Fig. 1).

Robust 3D visual tracking for robotic-assisted cardiac interventions.

In the context of minimally invasive cardiac surgery, active vision-based motion compensation schemes have been proposed for mitigating problems related to physiological motion. However, robust and accurate visual tracking is a difficult task. The purpose of this paper is to present a hybrid tracker that estimates the heart surface deformation using the outputs of multiple visual tracking techniques. In the proposed method, the failure of an individual technique can be circumvented by the success of others, enabling the robust estimation of the heart surface deformation with increased spatial resolution. In addition, for coping with the absence of visual information due to motion blur or occlusions, a temporal heart motion model is incorporated as an additional support for the visual tracking task. The superior performance of the proposed technique compared to existing techniques individually is demonstrated through experiments conducted on recorded images of an in vivo minimally invasive CABG using the DaVinci robotic platform.

Efficient 3D tracking for motion compensation in beating heart surgery.

The design of physiological motion compensation systems for robotic-assisted cardiac Minimally Invasive Surgery (MIS) is a challenging research topic. In this domain, vision-based techniques have proven to be a practical way to retrieve the motion of the beating heart. However due to the complexity of the heart motion and its surface characteristics, efficient tracking is still a complicated task. In this paper, we propose an algorithm for tracking the 3D motion of the beating heart, based on a Thin-Plate Splines (TPS) parametric model. The novelty of our approach lies in that no explicit matching between the stereo camera images is required and consequently no intermediate steps such as rectification are needed. Experiments conducted on ex-vivo and in-vivo tissue show the effectiveness of the proposed algorithm for tracking surfaces undergoing complex deformations.

Motion prediction for tracking the beating heart.

In the past few years, several research groups have worked on the design of efficient motion compensation systems for cardiac robotic-assisted Minimally Invasive Surgery (MIS). By providing surgeons with a stabilized work environment, significant improvements of the precision and repeatability of their gestures can be achieved. The design of a motion compensation system requires the accurate measurement of the heart motion, which can be achieved using computer vision techniques for tracking cardiac structures on the heart surface. However, most works in the literature focus on the representation and localization of cardiac structures while few explore their motion dynamics. In this paper we study and implement different adaptive methods for predicting the future heart motion using Kalman filtering. By exploiting the quasi-periodic nature of the heart motion, we are able to increase tracking robustness and computational efficiency. The experimental results indicate the significant increase in tracking performance when heart motion prediction is employed.