Using content-based image retrieval of dermoscopic images for interpretation and education: A pilot study.

BACKGROUND: Dermoscopic content-based image retrieval (CBIR) systems provide a set of visually similar dermoscopic (magnified and illuminated) skin images with a pathology-confirmed diagnosis for a given dermoscopic query image of a skin lesion. Although recent advances in machine learning have spurred novel CBIR algorithms, we have few insights into how end users interact with CBIRs and to what extent CBIRs can be useful for education and image interpretation. MATERIALS AND METHODS: We developed an interactive user interface for a CBIR system with dermoscopic images as a decision support tool and investigated users' interactions and decisions with the system. We performed a pilot experiment with 14 non-medically trained users for a given set of annotated dermoscopic images. RESULTS: Our pilot showed that the number of correct classifications and users' confidence levels significantly increased with the CBIR interface compared with a non-CBIR interface, although the timing also increased significantly. The users found the CBIR interface of high educational value, engaging and easy to use. CONCLUSION: Overall, users became more accurate, found the CBIR approach provided a useful decision aid, and had educational value for learning about skin conditions.

Detection of Changes in Surgical Difficulty: Evidence From Pupil Responses.

BACKGROUND: Assessing the workload of surgeons requires technology to continuously monitor surgeons' behaviors without interfering with their performance. We investigated the feasibility of using eye-tracking to reveal surgeons' response to increasing task difficulty. METHODS: A controlled study was conducted in a simulated operating room, where 14 subjects were required to perform a laparoscopic procedure that includes 9 subtasks. The subtasks could be divided into 3 types with different levels of task difficulty, calculated by the index of task difficulty (ID) proposed by Fitts in 1954. Pupillary responses of subjects in performing the procedure were recorded using Tobii eye-tracking equipment. Peak pupil dilation and movement time were compared between subtasks with different IDs as well as between fast moving and slow aiming phases within each subtask. RESULTS: When the task difficulty was increased, task completion time increased. Meanwhile, the subjects' peak pupil size also increased. As the entire procedure was performed continuously, we found that pupil responses were not only affected by the ID in the current subtask but also influenced by subtasks before and after. DISCUSSION: Decomposing a surgical procedure into meaningful subtasks and examining the surgeon's pupil response to each subtask enables us to identify the challenging steps within a continuous surgical procedure. Psychomotor evidence on surgeon's performance may lead to an innovation for designing a task-specific training curriculum.

Video processing to locate the tooltip position in surgical eye-hand coordination tasks.

INTRODUCTION: Trajectories of surgical instruments in laparoscopic surgery contain rich information about surgeons' performance. In a simulation environment, instrument trajectories can be taken by motion sensors attached to the instruments. This method is not accepted by surgeons working in the operating room due to safety concerns. In this study, a novel approach of acquiring instrument trajectories from surgical videos is reported. METHODS: A total of 12 surgical videos were obtained for this study. The videos were captured during simulated laparoscopic procedures where subjects were required to pick up and transport an object over 3 different targets using a laparoscopic grasper. An algorithm was developed to allow the computer to identify the tip of the grasper on each frame of video, and then compute the trajectories of grasper movement. RESULTS: The newly developed algorithm successfully identified tool trajectories from all 12 surgical videos. To validate the accuracy of this algorithm, the location of the tooltip in these videos were also manually labeled. The rate of accurate matching between these 2 methods was 98.4% of all video frames. DISCUSSION: Identifying tool movement from surgical videos creates an effective way to track instrument trajectories. This builds up the foundation for assessing psychomotor performance of surgeons in the operating room without jeopardizing patient safety.

Pupil response to precision in surgical task execution.

Task-evoked pupil response (TEPR) has been extensively studied and well proven to be sensitive to mental workload changes. We aimed to explore how TEPR reflects mental workload changes in a surgical environment. We conducted a simulated surgical task that has 3 different subtasks with different levels of motor precision and different mental workload requirements. We found a significant effect among these different subtask groups by measuring pupil diameter change rate. This finding may improve patient safety in a real operating room by non-intrusively monitoring the surgeon's mental workload while performing a surgery using an eye-tracking system.

Identifying eye gaze mismatch during laparoscopic surgery.

During a laparoscopic operation, the surgical team should have a common understanding of the action plan which can be aided by focusing on the same surgical site. We show how measuring the overlap between two spatially and temporally aligned gaze recordings can be used to identify periods during which the primary operator and assistant were focused on different areas of the surgical display.

Detection and analysis of irregular streaks in dermoscopic images of skin lesions.

Irregular streaks are important clues for Melanoma (a potentially fatal form of skin cancer) diagnosis using dermoscopy images. This paper extends our previous algorithm to identify the absence or presence of streaks in a skin lesions, by further analyzing the appearance of detected streak lines, and performing a three-way classification for streaks, Absent, Regular, and Irregular, in a pigmented skin lesion. In addition, the directional pattern of detected lines is analyzed to extract their orientation features in order to detect the underlying pattern. The method uses a graphical representation to model the geometric pattern of valid streaks and the distribution and coverage of the structure. Using these proposed features of the valid streaks along with the color and texture features of the entire lesion, an accuracy of 76.1% and weighted average area under ROC curve (AUC) of 85% is achieved for classifying dermoscopy images into streaks Absent, Regular, or Irregular on 945 images compiled from atlases and the internet without any exclusion criteria. This challenging dataset is the largest validation dataset for streaks detection and classification published to date. The data set has also been applied to the two-class sub-problems of Absent/Present classification (accuracy of 78.3% with AUC of 83.2%) and to Regular/Irregular classification (accuracy 83.6% with AUC of 88.9%). When the method was tested on a cleaned subset of 300 images randomly selected from the 945 images, the AUC increased to 91.8%, 93.2% and 90.9% for the Absent/Regular/Irregular, Absent/Present, and Regular/Irregular problems, respectively.

What do surgeons see: capturing and synchronizing eye gaze for surgery applications.

Recording eye motions in surgical environments is challenging. This study describes the authors' experiences with performing eye-tracking for improving surgery training, both in the laboratory and in the operating room (OR). Three different eye-trackers were used, each with different capabilities and requirements. For monitoring eye gaze shifts over the room scene in a simulated OR, a head-mounted system was used. The number of surgeons' eye glances on the monitor displaying patient vital signs was successfully captured by this system. The resolution of the head-mounted eye-tracker was not sufficient to obtain the gaze coordinates in detail on the surgical display monitor. The authors then selected a high-resolution eye-tracker built in to a 17-inch computer monitor that is capable of recording gaze differences with resolution of 1° of visual angle. This system enables one to investigate surgeons' eye-hand coordination on the surgical monitor in the laboratory environment. However, the limited effective tracking distance restricts the use of this system in the dynamic environment in the real OR. Another eye-tracker system was found with equally high level of resolution but with more flexibility on the tracking distance, as the eye-tracker camera was detached from the monitor. With this system, the surgeon's gaze during 11 laparoscopic procedures in the OR was recorded successfully. There were many logistical challenges with unobtrusively integrating the eye-tracking equipment into the regular OR workflow and data processing issues in the form of image compatibility and data validation. The experiences and solutions to these challenges are discussed.

Automatic detection of blue-white veil by discrete colour matching in dermoscopy images.

Skin lesions are often comprised of various colours. The presence of multiple colours with an irregular distribution can signal malignancy. Among common colours under dermoscopy, blue-grey (blue-white veil) is a strong indicator of malignant melanoma. Since it is not always easy to visually identify and recognize this feature, a computerised automatic colour analysis method can provide the clinician with an objective second opinion. In this paper, we put forward an innovative method, through colour analysis and computer vision techniques, to automatically detect and segment blue-white veil areas in dermoscopy images. The proposed method is an attempt to mimic the human perception of lesion colours, and improves and outperforms the state-of-the-art as shown in our experiments.

Capturing and evaluating blinks from video-based eyetrackers.

Blinks are related to several emotional states, and the present report describes a simple, reliable way to measure blinks from the video stream of an eye obtained during eyetracking, where the source of the eye video is a video camera attached to a head-mounted eyetracker. Computer vision techniques are employed to determine the moments that a blink starts and ends, for the purpose of calculating blink frequency and duration. The video is first processed to show blocks of eyelid and pupil movements, and is then analyzed for blink starts and ends. The moment of a blink start is reported when the eyelid starts to move quickly, exceeding a predetermined threshold. The end of a blink arises when the pupil size increases by less than a separate threshold. We observed several different blink patterns from different subjects, and our algorithm was designed to work for all of these patterns. We evaluated our algorithm by manually measuring the true blinks of five different subjects while they were eyetracked. To test the sensitivity and specificity of the algorithm, we employed a series of threshold values to plot the receiver operating characteristic curves. Using the best thresholds, we achieved excellent sensitivity (>90 %) and specificity (>99 %) over the five subjects. Potential applications of this research include real-time, nonintrusive, continuous and automated measurements of mental workload and other emotional states related to blink rates and durations.

Symmetric positive semi-definite Cartesian Tensor fiber orientation distributions (CT-FOD).

A novel method for estimating a field of fiber orientation distribution (FOD) based on signal de-convolution from a given set of diffusion weighted magnetic resonance (DW-MR) images is presented. We model the FOD by higher order Cartesian tensor basis using a parametrization that explicitly enforces the positive semi-definite property to the computed FOD. The computed Cartesian tensors, dubbed Cartesian Tensor-FOD (CT-FOD), are symmetric positive semi-definite tensors whose coefficients can be efficiently estimated by solving a linear system with non-negative constraints. Next, we show how to use our method for converting higher-order diffusion tensors to CT-FODs, which is an essential task since the maxima of higher-order tensors do not correspond to the underlying fiber orientations. Finally, we propose a diffusion anisotropy index computed directly from CT-FODs using higher order tensor distance measures thus consolidating the whole analysis pipeline of diffusion imaging solely using CT-FODs. We evaluate our method qualitatively and quantitatively using simulated DW-MR images, phantom images, and human brain real dataset. The results conclusively demonstrate the superiority of the proposed technique over several existing multi-fiber reconstruction methods.

Analysis of eye gaze: do novice surgeons look at the same location as expert surgeons during a laparoscopic operation?

INTRODUCTION: Eye-gaze technology can be used to track the gaze of surgeons on the surgical monitor. We examine the gaze of surgeons performing a task in the operating room and later watching the operative video in a lab. We also examined gaze of video watching by surgical residents. METHODS: Data collection required two phases. Phase 1 involved recording the real-time eye gaze of expert surgeons while they were performing laparoscopic procedures in the operating room. The videos were used for phase 2. Phase 2 involved showing the recorded videos to the same expert surgeons, and while they were watching the videos (self-watching), their eye gaze was recorded. Junior residents (PGY 1-3) also were asked to watch the videos (other-watching) and their eye gaze was recorded. Dual eye-gaze similarity in self-watching was computed by the level of gaze overlay and compared with other-watching. RESULTS: Sixteen cases of laparoscopic cholecystectomy were recorded in the operating room. When experts watched the videos, there was a 55% overlap of eye gaze; yet when novices watched, only a 43.8% overlap (p < 0.001) was shown. CONCLUSIONS: These findings show that there is a significant difference in gaze patterns between novice and expert surgeons while watching surgical videos. Expert gaze recording from the operating room can be used to make teaching videos for gaze training to expedite learning curves of novice surgeons.

Workload assessment of surgeons: correlation between NASA TLX and blinks.

BACKGROUND: Blinks are known as an indicator of visual attention and mental stress. In this study, surgeons' mental workload was evaluated utilizing a paper assessment instrument (National Aeronautics and Space Administration Task Load Index, NASA TLX) and by examining their eye blinks. Correlation between these two assessments was reported. METHODS: Surgeons' eye motions were video-recorded using a head-mounted eye-tracker while the surgeons performed a laparoscopic procedure on a virtual reality trainer. Blink frequency and duration were computed using computer vision technology. The level of workload experienced during the procedure was reported by surgeons using the NASA TLX. RESULTS: A total of 42 valid videos were recorded from 23 surgeons. After blinks were computed, videos were divided into two groups based on the blink frequency: infrequent group (≤ 6 blinks/min) and frequent group (more than 6 blinks/min). Surgical performance (measured by task time and trajectories of tool tips) was not significantly different between these two groups, but NASA TLX scores were significantly different. Surgeons who blinked infrequently reported a higher level of frustration (46 vs. 34, P = 0.047) and higher overall level of workload (57 vs. 47, P = 0.045) than those who blinked more frequently. The correlation coefficients (Pearson test) between NASA TLX and the blink frequency and duration were -0.17 and 0.446. CONCLUSION: Reduction of blink frequency and shorter blink duration matched the increasing level of mental workload reported by surgeons. The value of using eye-tracking technology for assessment of surgeon mental workload was shown.

Oriented pattern analysis for streak detection in dermoscopy images.

There is an increasing demand for automated detection and analysis of dermoscopy structures and malignancy clues such as streaks in dermoscopy images, for computer-aided early diagnosis of deadly melanoma. This paper presents a novel approach for streak detection and visualization on dermoscopic images. We tackle the detection of streaks by means of ridge and valley estimation. Orientation estimation and correction is applied to detect low contrast and fuzzy streaks lines, and candidate streaks are used to classify dermoscopy images into streaks Absent or Present with the AUC of 90.5% on 300 dermoscopy images. Our approach can also detect starburst pattern of regular streaks using detected linear structures with accuracy of 81.5% and AUC of 87.7%.

Intrinsic melanin and hemoglobin colour components for skin lesion malignancy detection.

In this paper we propose a new log-chromaticity 2-D colour space, an extension of previous approaches, which succeeds in removing confounding factors from dermoscopic images: (i) the effects of the particular camera characteristics for the camera system used in forming RGB images; (ii) the colour of the light used in the dermoscope; (iii) shading induced by imaging non-flat skin surfaces; (iv) and light intensity, removing the effect of light-intensity falloff toward the edges of the dermoscopic image. In the context of a blind source separation of the underlying colour, we arrive at intrinsic melanin and hemoglobin images, whose properties are then used in supervised learning to achieve excellent malignant vs. benign skin lesion classification. In addition, we propose using the geometric-mean of colour for skin lesion segmentation based on simple grey-level thresholding, with results outperforming the state of the art.

Conditional random fields and supervised learning in automated skin lesion diagnosis.

Many subproblems in automated skin lesion diagnosis (ASLD) can be unified under a single generalization of assigning a label, from an predefined set, to each pixel in an image. We first formalize this generalization and then present two probabilistic models capable of solving it. The first model is based on independent pixel labeling using maximum a-posteriori (MAP) estimation. The second model is based on conditional random fields (CRFs), where dependencies between pixels are defined using a graph structure. Furthermore, we demonstrate how supervised learning and an appropriate training set can be used to automatically determine all model parameters. We evaluate both models' ability to segment a challenging dataset consisting of 116 images and compare our results to 5 previously published methods.

Generalizing common tasks in automated skin lesion diagnosis.

We present a general model using supervised learning and MAP estimation that is capable of performing many common tasks in automated skin lesion diagnosis. We apply our model to segment skin lesions, detect occluding hair, and identify the dermoscopic structure pigment network. Quantitative results are presented for segmentation and hair detection and are competitive when compared to other specialized methods. Additionally, we leverage the probabilistic nature of the model to produce receiver operating characteristic curves, show compelling visualizations of pigment networks, and provide confidence intervals on segmentations.

The bias/variance trade-off when estimating the MR signal magnitude from the complex average of repeated measurements.

The signal-dependent bias of MR images has been considered a hindrance to visual interpretation almost since the beginning of clinical MRI. Over time, a variety of procedures have been suggested to produce less-biased images from the complex average of repeated measurements. In this work, we re-evaluate these approaches using first a survey of previous estimators in the MRI literature, then a survey of the methods statisticians employ for our specific problem. Our conclusions are substantially different from much of the previous work: first, removing bias completely is impossible if we demand the estimator have bounded variance; second, reducing bias may not be beneficial to image quality.

Quantifying surgeons' vigilance during laparoscopic operations using eyegaze tracking.

The vigilance of surgeons while operating is an important consideration for patient safety. Using a lightweight mobile eyegaze tracker, we can objectively observe and quantify a surgeon's vigilance measured as the frequency and duration of time spent gazing at an anaesthesia monitor displaying various patient vital signs. Expert surgeons and training surgical residents had their eyegaze recorded while performing a mock partial cholecystectomy on a computer simulator. Results show that experts glanced at the patient vital signs more than the residents, indicating a higher level of surgical vigilance.

Chromatic aberration correction: an enhancement to the calibration of low-cost digital dermoscopes.

BACKGROUND/PURPOSE: We present a method for calibrating low-cost digital dermoscopes that corrects for color and inconsistent lighting and also corrects for chromatic aberration. Chromatic aberration is a form of radial distortion that often occurs in inexpensive digital dermoscopes and creates red and blue halo-like effects on edges. Being radial in nature, distortions due to chromatic aberration are not constant across the image, but rather vary in both magnitude and direction. As a result, distortions are not only visually distracting but could also mislead automated characterization techniques. METHODS: Two low-cost dermoscopes, based on different consumer-grade cameras, were tested. Color is corrected by imaging a reference and applying singular value decomposition to determine the transformation required to ensure accurate color reproduction. Lighting is corrected by imaging a uniform surface and creating lighting correction maps. Chromatic aberration is corrected using a second-order radial distortion model. RESULTS: Our results for color and lighting calibration are consistent with previously published results, while distortions due to chromatic aberration can be reduced by 42-47% in the two systems considered. CONCLUSION: The disadvantages of inexpensive dermoscopy can be quickly substantially mitigated with a suitable calibration procedure.

A novel method for detection of pigment network in dermoscopic images using graphs.

We describe a novel approach to detect and visualize pigment network structures in dermoscopic images, based on the fact that the edges of pigment network structures form cyclic graphs which can be automatically detected and analyzed. First we perform a pre-processing step of image enhancement and edge detection. The resulting binary edge image is converted to a graph and the defined feature patterns are extracted by finding cyclic subgraphs corresponding to skin texture structures. We filtered these cyclic subgraphs to remove other round structures such as globules, dots, and oil bubbles, based on their size and color. Another high-level graph is created from each correctly extracted subgraph, with a node corresponding to a hole in the pigment network. Nodes are connected by edges according to their distances. Finally the image is classified according to the density ratio of the graph. Our results over a set of 500 images from a well known atlas of dermoscopy show an accuracy of 94.3% on classification of the images as pigment network Present or Absent.