Survey and Synthesis of State of the Art in Driver Monitoring.

Road vehicle accidents are mostly due to human errors, and many such accidents could be avoided by continuously monitoring the driver. Driver monitoring (DM) is a topic of growing interest in the automotive industry, and it will remain relevant for all vehicles that are not fully autonomous, and thus for decades for the average vehicle owner. The present paper focuses on the first step of DM, which consists of characterizing the state of the driver. Since DM will be increasingly linked to driving automation (DA), this paper presents a clear view of the role of DM at each of the six SAE levels of DA. This paper surveys the state of the art of DM, and then synthesizes it, providing a unique, structured, polychotomous view of the many characterization techniques of DM. Informed by the survey, the paper characterizes the driver state along the five main dimensions-called here "(sub)states"-of drowsiness, mental workload, distraction, emotions, and under the influence. The polychotomous view of DM is presented through a pair of interlocked tables that relate these states to their indicators (e.g., the eye-blink rate) and the sensors that can access each of these indicators (e.g., a camera). The tables factor in not only the effects linked directly to the driver, but also those linked to the (driven) vehicle and the (driving) environment. They show, at a glance, to concerned researchers, equipment providers, and vehicle manufacturers (1) most of the options they have to implement various forms of advanced DM systems, and (2) fruitful areas for further research and innovation.

Multi-Timescale Drowsiness Characterization Based on a Video of a Driver's Face.

Drowsiness is a major cause of fatal accidents, in particular in transportation. It is therefore crucial to develop automatic, real-time drowsiness characterization systems designed to issue accurate and timely warnings of drowsiness to the driver. In practice, the least intrusive, physiology-based approach is to remotely monitor, via cameras, facial expressions indicative of drowsiness such as slow and long eye closures. Since the system's decisions are based upon facial expressions in a given time window, there exists a trade-off between accuracy (best achieved with long windows, i.e., at long timescales) and responsiveness (best achieved with short windows, i.e., at short timescales). To deal with this trade-off, we develop a multi-timescale drowsiness characterization system composed of four binary drowsiness classifiers operating at four distinct timescales (5 s, 15 s, 30 s, and 60 s) and trained jointly. We introduce a multi-timescale ground truth of drowsiness, based on the reaction times (RTs) performed during standard Psychomotor Vigilance Tasks (PVTs), that strategically enables our system to characterize drowsiness with diverse trade-offs between accuracy and responsiveness. We evaluated our system on 29 subjects via leave-one-subject-out cross-validation and obtained strong results, i.e., global accuracies of 70%, 85%, 89%, and 94% for the four classifiers operating at increasing timescales, respectively.

Probabilistic Framework for the Characterization of Surfaces and Edges in Range Images, with Application to Edge Detection.

We develop a powerful probabilistic framework for the local characterization of surfaces and edges in range images. We use the geometrical nature of the data to derive an analytic expression for the joint probability density function (pdf) for the random variables used to model the ranges of a set of pixels in a local neighborhood of an image. We decompose this joint pdf by considering independently the cases where two real world points corresponding to two neighboring pixels are locally on the same real world surface or not. In particular, we show that this joint pdf is linked to the Voigt pdf and not to the Gaussian pdf as it is assumed in some applications. We apply our framework to edge detection and develop a locally adaptive algorithm that is based on a probabilistic decision rule. We show in an objective evaluation that this new edge detector performs better than prior art edge detectors. This proves the benefits of the probabilistic characterization of the local neighborhood as a tool to improve applications that involve range images.

Objective assessment of visual pursuit in patients with disorders of consciousness: an exploratory study.

Visual pursuit is a key marker of residual consciousness in patients with disorders of consciousness (DOC). Currently, its assessment relies on subjective clinical decisions. In this study, we explore the variability of such clinical assessments, and present an easy-to-use device composed of cameras and video processing algorithms that could help the clinician to improve the detection of visual pursuit in a clinical context. Visual pursuit was assessed by an experienced research neuropsychologist on 31 patients with DOC and on 23 healthy subjects, while the device was used to simultaneously record videos of both one eye and the mirror. These videos were then scored by three researchers: the experienced research neuropsychologist who did the clinical assessment, another experienced research neuropsychologist, and a neurologist. For each video, a consensus was decided between the three persons, and used as the gold standard of the presence or absence of visual pursuit. Almost 10% of the patients were misclassified at the bedside according to their consensus. An automatic classifier analyzed eye and mirror trajectories, and was able to identify patients and healthy subjects with visual pursuit, in total agreement with the consensus on video. In conclusion, our device can be used easily in patients with DOC while respecting the current guidelines of visual pursuit assessment. Our results suggest that our material and our classification method can identify patients with visual pursuit, as well as the three researchers based on video recordings can.

Tests of a New Drowsiness Characterization and Monitoring System Based on Ocular Parameters.

Drowsiness is the intermediate state between wakefulness and sleep. It is characterized by impairments of performance, which can be very dangerous in many activities and can lead to catastrophic accidents in transportation or in industry. There is thus an obvious need for systems that are able to continuously, objectively, and automatically estimate the level of drowsiness of a person busy at a task. We have developed such a system, which is based on the physiological state of a person, and, more specifically, on the values of ocular parameters extracted from images of the eye (photooculography), and which produces a numerical level of drowsiness. In order to test our system, we compared the level of drowsiness determined by our system to two references: (1) the level of drowsiness obtained by analyzing polysomnographic signals; and (2) the performance of individuals in the accomplishment of a task. We carried out an experiment in which 24 participants were asked to perform several Psychomotor Vigilance Tests in different sleep conditions. The results show that the output of our system is well correlated with both references. We determined also the best drowsiness level threshold in order to warn individuals before they reach dangerous situations. Our system thus has significant potential for reliably quantifying the level of drowsiness of individuals accomplishing a task and, ultimately, for preventing drowsiness-related accidents.

Development and validation of an accelerometer-based method for quantifying gait events.

An original signal processing algorithm is presented to automatically extract, on a stride-by-stride basis, four consecutive fundamental events of walking, heel strike (HS), toe strike (TS), heel-off (HO), and toe-off (TO), from wireless accelerometers applied to the right and left foot. First, the signals recorded from heel and toe three-axis accelerometers are segmented providing heel and toe flat phases. Then, the four gait events are defined from these flat phases. The accelerometer-based event identification was validated in seven healthy volunteers and a total of 247 trials against reference data provided by a force plate, a kinematic 3D analysis system, and video camera. HS, TS, HO, and TO were detected with a temporal accuracy ± precision of 1.3 ms ± 7.2 ms, -4.2 ms ± 10.9 ms, -3.7 ms ± 14.5 ms, and -1.8 ms ± 11.8 ms, respectively, with the associated 95% confidence intervals ranging from -6.3 ms to 2.2 ms. It is concluded that the developed accelerometer-based method can accurately and precisely detect HS, TS, HO, and TO, and could thus be used for the ambulatory monitoring of gait features computed from these events when measured concurrently in both feet.

Evaluation of the performance of an experimental somnolence quantification system in terms of reaction times and lapses.

Somnolence is known to be a major cause of various types of accidents, and ocular parameters are recognized to be reliable physiological indicators of somnolence. We have thus developed an experimental somnolence quantification system that uses images of the eye and that produces a level of somnolence on a continuous numerical scale. The aim of this paper is to show that the level of somnolence produced by our system is well related to the level of performance of subjects accomplishing three reaction-time tests in different sleep conditions. Twenty seven subjects participated in the study and images of their right eye were continuously recorded during the tests. Levels of somnolence, reaction times (RTs), and percentages of lapses were computed for each minute of test. Results show that the values of these three parameters increase significantly with sleep deprivation. We determined the best threshold on our scale of somnolence to predict lapses, and we also shown that correlations exist with some of the ocular parameters. Our somnolence quantification system has thus significant potential to predict performance decrements of subjects accomplishing a task.

Serial FEM/XFEM-Based Update of Preoperative Brain Images Using Intraoperative MRI.

Current neuronavigation systems cannot adapt to changing intraoperative conditions over time. To overcome this limitation, we present an experimental end-to-end system capable of updating 3D preoperative images in the presence of brain shift and successive resections. The heart of our system is a nonrigid registration technique using a biomechanical model, driven by the deformations of key surfaces tracked in successive intraoperative images. The biomechanical model is deformed using FEM or XFEM, depending on the type of deformation under consideration, namely, brain shift or resection. We describe the operation of our system on two patient cases, each comprising five intraoperative MR images, and we demonstrate that our approach significantly improves the alignment of nonrigidly registered images.

3D XFEM-based modeling of retraction for preoperative image update.

Outcomes for neurosurgery patients can be improved by enhancing intraoperative navigation and guidance. Current navigation systems do not accurately account for intraoperative brain deformation. So far, most studies of brain deformation have focused on brain shift, whereas this paper focuses on the brain deformation due to retraction. The heart of our system is a 3D nonrigid registration technique using a biomechanical model driven by the deformations of key surfaces tracked between two intraoperative images. The key surfaces, e.g., the whole-brain region boundary and the lips of the retraction cut, thus deform due to the combination of gravity and retractor deployment. The tissue discontinuity due to retraction is handled via the eXtended Finite Element Method (XFEM), which has the appealing feature of being able to handle arbitrarily shaped discontinuity without any remeshing. Our approach is shown to significantly improve the alignment of intraoperative MRI.

2D XFEM-based modeling of retraction and successive resections for preoperative image update.

This paper considers an approach to improving outcomes for neurosurgery patients by enhancing intraoperative navigation and guidance. Currently, intraoperative navigation systems do not accurately account for brain shift or tissue resection. We describe how preoperative images can be incrementally updated to take into account any type of brain tissue deformation that may occur during surgery, and thus to improve the accuracy of image-guided navigation systems. For this purpose, we have developed a non-rigid image registration technique using a biomechanical model, which deforms based on the Finite Element Method (FEM). While the FEM has been used successfully for dealing with deformations such as brain shift, it has difficulty with tissue discontinuities. Here, we describe a novel application of the eXtended Finite Element Method (XFEM) in the field of image-guided surgery in order to model brain deformations that imply tissue discontinuities. In particular, this paper presents a detailed account of the use of XFEM for dealing with retraction and successive resections, and demonstrates the feasibility of the approach by considering 2D examples based on intraoperative MR images. To evaluate our results, we compute the modified Hausdorff distance between Canny edges extracted from images before and after registration. We show that this distance decreases after registration, and thus demonstrate that our approach improves alignment of intraoperative images.