Generative Adversarial Networks and Conditional Random Fields for Hyperspectral Image Classification.

In this paper, we address the hyperspectral image (HSI) classification task with a generative adversarial network and conditional random field (GAN-CRF)-based framework, which integrates a semisupervised deep learning and a probabilistic graphical model, and make three contributions. First, we design four types of convolutional and transposed convolutional layers that consider the characteristics of HSIs to help with extracting discriminative features from limited numbers of labeled HSI samples. Second, we construct semisupervised generative adversarial networks (GANs) to alleviate the shortage of training samples by adding labels to them and implicitly reconstructing real HSI data distribution through adversarial training. Third, we build dense conditional random fields (CRFs) on top of the random variables that are initialized to the softmax predictions of the trained GANs and are conditioned on HSIs to refine classification maps. This semisupervised framework leverages the merits of discriminative and generative models through a game-theoretical approach. Moreover, even though we used very small numbers of labeled training HSI samples from the two most challenging and extensively studied datasets, the experimental results demonstrated that spectral-spatial GAN-CRF (SS-GAN-CRF) models achieved top-ranking accuracy for semisupervised HSI classification.

Non-contact hemodynamic imaging reveals the jugular venous pulse waveform.

Cardiovascular monitoring is important to prevent diseases from progressing. The jugular venous pulse (JVP) waveform offers important clinical information about cardiac health, but is not routinely examined due to its invasive catheterisation procedure. Here, we demonstrate for the first time that the JVP can be consistently observed in a non-contact manner using a photoplethysmographic imaging system. The observed jugular waveform was strongly negatively correlated to the arterial waveform (r = -0.73 ± 0.17), consistent with ultrasound findings. Pulsatile venous flow was observed over a spatially cohesive region of the neck. Critical inflection points (c, x, v, y waves) of the JVP were observed across all participants. The anatomical locations of the strongest pulsatile venous flow were consistent with major venous pathways identified through ultrasound.

Spectral-spatial fusion model for robust blood pulse waveform extraction in photoplethysmographic imaging.

Photoplethysmographic imaging is an optical solution for non-contact cardiovascular monitoring from a distance. This camera-based technology enables physiological monitoring in situations where contact-based devices may be problematic or infeasible, such as ambulatory, sleep, and multi-individual monitoring. However, automatically extracting the blood pulse waveform signal is challenging due to the unknown mixture of relevant (pulsatile) and irrelevant pixels in the scene. Here, we propose a signal fusion framework, FusionPPG, for extracting a blood pulse waveform signal with strong temporal fidelity from a scene without requiring anatomical priors. The extraction problem is posed as a Bayesian least squares fusion problem, and solved using a novel probabilistic pulsatility model that incorporates both physiologically derived spectral and spatial waveform priors to identify pulsatility characteristics in the scene. Evaluation was performed on a 24-participant sample with various ages (9-60 years) and body compositions (fat% 30.0 ± 7.9, muscle% 40.4 ± 5.3, BMI 25.5 ± 5.2 kg·m-2). Experimental results show stronger matching to the ground-truth blood pulse waveform signal compared to the FaceMeanPPG (p < 0.001) and DistancePPG (p < 0.001) methods. Heart rates predicted using FusionPPG correlated strongly with ground truth measurements (r2 = 0.9952). A cardiac arrhythmia was visually identified in FusionPPG's waveform via temporal analysis.

Spatial probabilistic pulsatility model for enhancing photoplethysmographic imaging systems.

Photoplethysmographic imaging (PPGI) is a widefield noncontact biophotonic technology able to remotely monitor cardiovascular function over anatomical areas. Although spatial context can provide insight into physiologically relevant sampling locations, existing PPGI systems rely on coarse spatial averaging with no anatomical priors for assessing arterial pulsatility. Here, we developed a continuous probabilistic pulsatility model for importance-weighted blood pulse waveform extraction. Using a data-driven approach, the model was constructed using a 23 participant sample with a large demographic variability (11/12 female/male, age 11 to 60 years, BMI 16.4 to 35.1??kg·m?2). Using time-synchronized ground-truth blood pulse waveforms, spatial correlation priors were computed and projected into a coaligned importance-weighted Cartesian space. A modified Parzen­Rosenblatt kernel density estimation method was used to compute the continuous resolution-agnostic probabilistic pulsatility model. The model identified locations that consistently exhibited pulsatility across the sample. Blood pulse waveform signals extracted with the model exhibited significantly stronger temporal correlation (W=35,p<0.01) and spectral SNR (W=31,p<0.01) compared to uniform spatial averaging. Heart rate estimation was in strong agreement with true heart rate [r2=0.9619, error (?,?)=(0.52,1.69) bpm].

Numerical Demultiplexing of Color Image Sensor Measurements via Non-linear Random Forest Modeling.

The simultaneous capture of imaging data at multiple wavelengths across the electromagnetic spectrum is highly challenging, requiring complex and costly multispectral image devices. In this study, we investigate the feasibility of simultaneous multispectral imaging using conventional image sensors with color filter arrays via a novel comprehensive framework for numerical demultiplexing of the color image sensor measurements. A numerical forward model characterizing the formation of sensor measurements from light spectra hitting the sensor is constructed based on a comprehensive spectral characterization of the sensor. A numerical demultiplexer is then learned via non-linear random forest modeling based on the forward model. Given the learned numerical demultiplexer, one can then demultiplex simultaneously-acquired measurements made by the color image sensor into reflectance intensities at discrete selectable wavelengths, resulting in a higher resolution reflectance spectrum. Experimental results demonstrate the feasibility of such a method for the purpose of simultaneous multispectral imaging.

Sorted self-similarity for multi-modal image registration.

In medical image analysis, registration of multimodal images has been challenging due to the complex intensity relationship between images. Classical multi-modal registration approaches evaluate the degree of the alignment by measuring the statistical dependency of the intensity values between images to be aligned. Employing statistical similarity measures, such as mutual information, is not promising in those cases with complex and spatially dependent intensity relations. A new similarity measure is proposed based on the assessing the similarity of pixels within an image, based on the idea that similar structures in an image are more probable to undergo similar intensity transformations. The most significant pixel similarity values are considered to transmit the most significant self-similarity information. The proposed method is employed in a framework to register different modalities of real brain scans and the performance of the method is compared to the conventional multi-modal registration approach. Quantitative evaluation of the method demonstrates the better registration accuracy in both rigid and non-rigid deformations.

Feasibility of long-distance heart rate monitoring using transmittance photoplethysmographic imaging (PPGI).

Photoplethysmography (PPG) devices are widely used for monitoring cardiovascular function. However, these devices require skin contact, which restricts their use to at-rest short-term monitoring. Photoplethysmographic imaging (PPGI) has been recently proposed as a non-contact monitoring alternative by measuring blood pulse signals across a spatial region of interest. Existing systems operate in reflectance mode, many of which are limited to short-distance monitoring and are prone to temporal changes in ambient illumination. This paper is the first study to investigate the feasibility of long-distance non-contact cardiovascular monitoring at the supermeter level using transmittance PPGI. For this purpose, a novel PPGI system was designed at the hardware and software level. Temporally coded illumination (TCI) is proposed for ambient correction, and a signal processing pipeline is proposed for PPGI signal extraction. Experimental results show that the processing steps yielded a substantially more pulsatile PPGI signal than the raw acquired signal, resulting in statistically significant increases in correlation to ground-truth PPG in both short- and long-distance monitoring. The results support the hypothesis that long-distance heart rate monitoring is feasible using transmittance PPGI, allowing for new possibilities of monitoring cardiovascular function in a non-contact manner.

Structure-guided statistical textural distinctiveness for salient region detection in natural images.

We propose a simple yet effective structure-guided statistical textural distinctiveness approach to salient region detection. Our method uses a multilayer approach to analyze the structural and textural characteristics of natural images as important features for salient region detection from a scale point of view. To represent the structural characteristics, we abstract the image using structured image elements and extract rotational-invariant neighborhood-based textural representations to characterize each element by an individual texture pattern. We then learn a set of representative texture atoms for sparse texture modeling and construct a statistical textural distinctiveness matrix to determine the distinctiveness between all representative texture atom pairs in each layer. Finally, we determine saliency maps for each layer based on the occurrence probability of the texture atoms and their respective statistical textural distinctiveness and fuse them to compute a final saliency map. Experimental results using four public data sets and a variety of performance evaluation metrics show that our approach provides promising results when compared with existing salient region detection approaches.

High-level intuitive features (HLIFs) for intuitive skin lesion description.

A set of high-level intuitive features (HLIFs) is proposed to quantitatively describe melanoma in standard camera images. Melanoma is the deadliest form of skin cancer. With rising incidence rates and subjectivity in current clinical detection methods, there is a need for melanoma decision support systems. Feature extraction is a critical step in melanoma decision support systems. Existing feature sets for analyzing standard camera images are comprised of low-level features, which exist in high-dimensional feature spaces and limit the system's ability to convey intuitive diagnostic rationale. The proposed HLIFs were designed to model the ABCD criteria commonly used by dermatologists such that each HLIF represents a human-observable characteristic. As such, intuitive diagnostic rationale can be conveyed to the user. Experimental results show that concatenating the proposed HLIFs with a full low-level feature set increased classification accuracy, and that HLIFs were able to separate the data better than low-level features with statistical significance. An example of a graphical interface for providing intuitive rationale is given.

Segmentation of skin lesions from digital images using joint statistical texture distinctiveness.

Melanoma is the deadliest form of skin cancer. Incidence rates of melanoma have been increasing, especially among non-Hispanic white males and females, but survival rates are high if detected early. Due to the costs for dermatologists to screen every patient, there is a need for an automated system to assess a patient's risk of melanoma using images of their skin lesions captured using a standard digital camera. One challenge in implementing such a system is locating the skin lesion in the digital image. A novel texture-based skin lesion segmentation algorithm is proposed. A set of representative texture distributions are learned from an illumination-corrected photograph and a texture distinctiveness metric is calculated for each distribution. Next, regions in the image are classified as normal skin or lesion based on the occurrence of representative texture distributions. The proposed segmentation framework is tested by comparing lesion segmentation results and melanoma classification results to results using other state-of-art algorithms. The proposed framework has higher segmentation accuracy compared to all other tested algorithms.

Enhanced Decoupled Active Contour Using Structural and Textural Variation Energy Functionals.

Active contours are a popular approach for object segmentation that uses an energy minimizing spline to extract an object's boundary. Nonparametric approaches can be computationally complex, whereas parametric approaches can be impacted by parameter sensitivity. A decoupled active contour (DAC) overcomes these problems by decoupling the external and internal energies and optimizing them separately. However a drawback of this approach is its reliance on the edge gradient as the external energy. This can lead to poor convergence toward the object boundary in the presence of weak object and strong background edges. To overcome these issues with convergence, a novel approach is proposed that takes advantage of a sparse texture model, which explicitly considers texture for boundary detection. The approach then defines the external energy as a weighted combination of textural and structural variation maps and feeds it into a multifunctional hidden Markov model for more robust object boundary detection. The enhanced DAC (EDAC) is qualitatively and visually analyzed on two natural image data sets as well as Brodatz images. The results demonstrate that EDAC effectively combines texture and structural information to extract the object boundary without impact on computation time and a reliance on color.

Physiological characterization of skin lesion using non-linear random forest regression model.

The current diagnostic technique for melanoma solely relies on the surface level of skin and under-skin information is neglected. Since physiological features of skin such as melanin are closely related to development of melanoma, the non-linear physiological feature extraction model based on random forest regression is proposed. The proposed model characterizes the concentration of eumelanin and pheomelanin from standard camera images or dermoscopic images, which are conventionally used for diagnosis of melanoma. For the validation, the phantom study and the separability test using clinical images were conducted and compared against the state-of-the art non-linear and linear feature extraction models. The results showed that the proposed model outperformed other comparing models in phantom and clinical experiments. Promising results show that the quantitative characterization of skin features, which is provided by the proposed method, can allow dermatologists and clinicians to make a more accurate and improved diagnosis of melanoma.

Multi-modal image registration using structural features.

Multi-modal image registration has been a challenging task in medical images because of the complex intensity relationship between images to be aligned. Registration methods often rely on the statistical intensity relationship between the images which suffers from problems such as statistical insufficiency. The proposed registration method works based on extracting structural features by utilizing the complex phase and gradient-based information. By employing structural relationships between different modalities instead of complex similarity measures, the multi-modal registration problem is converted into a mono-modal one. Therefore, conventional mono-modal similarity measures can be utilized to evaluate the registration results. This new registration paradigm has been tested on magnetic resonance (MR) brain images of different modes. The method has been evaluated based on target registration error (TRE) to determine alignment accuracy. Quantitative results demonstrate that the proposed method is capable of achieving comparable registration accuracy compared to the conventional mutual information.

Enhanced classification of malignant melanoma lesions via the integration of physiological features from dermatological photographs.

Traditional methods for early detection of melanoma rely upon a dermatologist to visually assess a skin lesion using the ABCDE (Asymmetry, Border irregularity, Color variegation, Diameter, Evolution) criteria before confirmation can be done through biopsy by a pathologist. However, this visual assessment strategy taken by dermatologists is hampered by clinician subjectivity and suffers from low sensitivity. Computer-aided diagnostic methods based on dermatological photographs are being developed to aid in the melanoma diagnosis process, but most of these methods rely only on superficial, topographic features that can be limiting in characterizing melanoma. In this work, a hybrid feature model is introduced for characterizing skin lesions that combines low-level and high-level features, and augments them with a set of physiological features extracted from dermatological photographs using a nearest-neighbor nonlinear model to improve classification performance. The physiological features extracted from the lesion for the proposed hybrid feature model include those based on: i) eumelanin concentrations, ii) pheomelanin concentrations, and iii) blood oxygen saturation. The proposed hybrid feature model was evaluated on 206 dermatological photographs of skin lesions (119 confirmed melanoma cases, 87 confirmed non-melanoma cases) using a cross validation scheme. The experimental results show that the proposed hybrid feature model, with integrated physiological features, provided improved sensitivity, specificity, precision and accuracy for the purpose of melanoma classification.

A decoupled approach to illumination-robust optical flow estimation.

Despite continuous improvements in optical flow in the last three decades, the ability for optical flow algorithms to handle illumination variation is still an unsolved challenge. To improve the ability to interpret apparent object motion in video containing illumination variation, an illumination-robust optical flow method is designed. This method decouples brightness into reflectance and illumination components using a stochastic technique; reflectance is given higher weight to ensure robustness against illumination, which is suppressed. Illumination experiments using the Middlebury and University of Oulu databases demonstrate the decoupled method's improvement when compared with state-of-the-art. In addition, a novel technique is implemented to visualize optical flow output, which is especially useful to compare different optical flow methods in the absence of the ground truth.

Sparse reconstruction of breast MRI using homotopic L0 minimization in a regional sparsified domain.

The use of MRI for early breast examination and screening of asymptomatic women has become increasing popular, given its ability to provide detailed tissue characteristics that cannot be obtained using other imaging modalities such as mammography and ultrasound. Recent application-oriented developments in compressed sensing theory have shown that certain types of magnetic resonance images are inherently sparse in particular transform domains, and as such can be reconstructed with a high level of accuracy from highly undersampled k-space data below Nyquist sampling rates using homotopic L0 minimization schemes, which holds great potential for significantly reducing acquisition time. An important consideration in the use of such homotopic L0 minimization schemes is the choice of sparsifying transform. In this paper, a regional differential sparsifying transform is investigated for use within a homotopic L0 minimization framework for reconstructing breast MRI. By taking local regional characteristics into account, the regional differential sparsifying transform can better account for signal variations and fine details that are characteristic of breast MRI than the popular finite differential transform, while still maintaining strong structure fidelity. Experimental results show that good breast MRI reconstruction accuracy can be achieved compared to existing methods.

Energy-guided learning approach to compressive FD-OCT.

High quality, large size volumetric imaging of biological tissue with optical coherence tomography (OCT) requires large number and high density of scans, which results in large data acquisition volume. This may lead to corruption of the data with motion artifacts related to natural motion of biological tissue, and could potentially cause conflicts with the maximum permissible exposure of biological tissue to optical radiation. Therefore, OCT can benefit greatly from different approaches to sparse or compressive sampling of the data where the signal is recovered from its sub-Nyquist measurements. In this paper, a new energy-guided compressive sensing approach is proposed for improving the quality of images acquired with Fourier domain OCT (FD-OCT) and reconstructed from sparse data sets. The proposed algorithm learns an optimized sampling probability density function based on the energy distribution of the training data set, which is then used for sparse sampling instead of the commonly used uniformly random sampling. It was demonstrated that the proposed energy-guided learning approach to compressive FD-OCT of retina images requires 45% fewer samples in comparison with the conventional uniform compressive sensing (CS) approach while achieving similar reconstruction performance. This novel approach to sparse sampling has the potential to significantly reduce data acquisition while maintaining image quality.

MSIM: multistage illumination modeling of dermatological photographs for illumination-corrected skin lesion analysis.

Melanoma is the most deadly form of skin cancer and it is costly for dermatologists to screen every patient for melanoma. There is a need for a system to assess the risk of melanoma based on dermatological photographs of a skin lesion. However, the presence of illumination variation in the photographs can have a negative impact on lesion segmentation and classification performance. A novel multistage illumination modeling algorithm is proposed to correct the underlying illumination variation in skin lesion photographs. The first stage is to compute an initial estimate of the illumination map of the photograph using a Monte Carlo nonparametric modeling strategy. The second stage is to obtain a final estimate of the illumination map via a parametric modeling strategy, where the initial nonparametric estimate is used as a prior. Finally, the corrected photograph is obtained using the final illumination map estimate. The proposed algorithm shows better visual, segmentation, and classification results when compared to three other illumination correction algorithms, one of which is designed specifically for lesion analysis.

Robust spectral clustering using statistical sub-graph affinity model.

Spectral clustering methods have been shown to be effective for image segmentation. Unfortunately, the presence of image noise as well as textural characteristics can have a significant negative effect on the segmentation performance. To accommodate for image noise and textural characteristics, this study introduces the concept of sub-graph affinity, where each node in the primary graph is modeled as a sub-graph characterizing the neighborhood surrounding the node. The statistical sub-graph affinity matrix is then constructed based on the statistical relationships between sub-graphs of connected nodes in the primary graph, thus counteracting the uncertainty associated with the image noise and textural characteristics by utilizing more information than traditional spectral clustering methods. Experiments using both synthetic and natural images under various levels of noise contamination demonstrate that the proposed approach can achieve improved segmentation performance when compared to existing spectral clustering methods.

Compressive fluorescence microscopy using saliency-guided sparse reconstruction ensemble fusion.

Compressive fluorescence microscopy has been proposed as a promising approach for fast acquisitions at sub-Nyquist sampling rates. Given that signal-to-noise ratio (SNR) is very important in the design of fluorescence microscopy systems, a new saliency-guided sparse reconstruction ensemble fusion system has been proposed for improving SNR in compressive fluorescence microscopy. This system produces an ensemble of sparse reconstructions using adaptively optimized probability density functions derived based on underlying saliency rather than the common uniform random sampling approach. The ensemble of sparse reconstructions are then fused together via ensemble expectation merging. Experimental results using real fluorescence microscopy data sets show that significantly improved SNR can be achieved when compared to existing compressive fluorescence microscopy approaches, with SNR increases of 16-9 dB within the noise range of 1.5%-10% standard deviation at the same compression rate.