Learning-Based Non-Intrusive Electric Load Monitoring for Smart Energy Management.

State-of-the-art smart cities have been calling for economic but efficient energy management over a large-scale network, especially for the electric power system. It is a critical issue to monitor, analyze, and control electric loads of all users in the system. In this study, a non-intrusive load monitoring method was designed for smart power management using computer vision techniques popular in artificial intelligence. First of all, one-dimensional current signals are mapped onto two-dimensional color feature images using signal transforms (including the wavelet transform and discrete Fourier transform) and Gramian Angular Field (GAF) methods. Second, a deep neural network with multi-scale feature extraction and attention mechanism is proposed to recognize all electrical loads from the color feature images. Third, a cloud-based approach was designed for the non-intrusive monitoring of all users, thereby saving energy costs during power system control. Experimental results on both public and private datasets demonstrate that the method achieves superior performances compared to its peers, and thus supports efficient energy management over a large-scale Internet of Things network.

Advances in Deep-Learning-Based Sensing, Imaging, and Video Processing.

Deep learning techniques have shown their capabilities to discover knowledge from massive unstructured data, providing data-driven solutions for representation and decision making [...].

UIF: An Objective Quality Assessment for Underwater Image Enhancement.

Due to complex and volatile lighting environment, underwater imaging can be readily impaired by light scattering, warping, and noises. To improve the visual quality, Underwater Image Enhancement (UIE) techniques have been widely studied. Recent efforts have also been contributed to evaluate and compare the UIE performances with subjective and objective methods. However, the subjective evaluation is time-consuming and uneconomic for all images, while existing objective methods have limited capabilities for the newly-developed UIE approaches based on deep learning. To fill this gap, we propose an Underwater Image Fidelity (UIF) metric for objective evaluation of enhanced underwater images. By exploiting the statistical features of these images in CIELab space, we present the naturalness, sharpness, and structure indexes. Among them, the naturalness and sharpness indexes represent the visual improvements of enhanced images; the structure index indicates the structural similarity between the underwater images before and after UIE. We combine all indexes with a saliency-based spatial pooling and thus obtain the final UIF metric. To evaluate the proposed metric, we also establish a first-of-its-kind large-scale UIE database with subjective scores, namely Underwater Image Enhancement Database (UIED). Experimental results confirm that the proposed UIF metric outperforms a variety of underwater and general-purpose image quality metrics. The database and source code are available at https://github.com/z21110008/UIF.

A Timestamp-Independent Haptic-Visual Synchronization Method for Haptic-Based Interaction System.

The booming haptic data significantly improve the users' immersion during multimedia interaction. As a result, the study of a Haptic-based Interaction System has attracted the attention of the multimedia community. To construct such a system, a challenging task is the synchronization of multiple sensorial signals that is critical to the user experience. Despite audio-visual synchronization efforts, there is still a lack of a haptic-aware multimedia synchronization model. In this work, we propose a timestamp-independent synchronization for haptic-visual signal transmission. First, we exploit the sequential correlations during delivery and playback of a haptic-visual communication system. Second, we develop a key sample extraction of haptic signals based on the force feedback characteristics and a key frame extraction of visual signals based on deep-object detection. Third, we combine the key samples and frames to synchronize the corresponding haptic-visual signals. Without timestamps in the signal flow, the proposed method is still effective and more robust in complicated network conditions. Subjective evaluation also shows a significant improvement of user experience with the proposed method.

An algorithm-assisted automated identification and enumeration system for sensitive hydrogen sulfide sensing under dark field microscopy.

Hydrogen sulfide (H2S) is an active physiological molecule, and its intracellular level has great significance to life functions. In this study, an effective and sensitive method was developed for H2S sensing with dark field microscopy (DFM). The proposed method employed AuNPs as the signal source, DFM as the readout system, and an intelligence algorithm as the image processing and output systems, respectively. The AuNP surface was modified with azido and alkynyl in advance, and then added into a tube cap. As the H2S evaporated from the solution and selectively reduced azido to amino, the click chemistry reaction was inhibited, which resulted in the AuNPs being well dispersed in the solution; otherwise, AuNP aggregation occurred. The scattering colour of single AuNPs could be easily distinguished from that of AuNP aggregations with DFM, and the number or ratio of single AuNPs could also be easily obtained by the custom algorithm. The results showed that the H2S content could be linearly analyzed in a range from 2-80 µM. Furthermore, the proposed sensing strategy has been applied for H2S detection in cell lysate. Compared with the traditional colorimetric method, the results showed no significant difference, indicating the good prospects of the algorithm and proposed H2S sensing method.

Embedding Regularizer Learning for Multi-View Semi-Supervised Classification.

Classification remains challenging when confronted with the existence of multi-view data with limited labels. In this paper, we propose an embedding regularizer learning scheme for multi-view semi-supervised classification (ERL-MVSC). The proposed framework integrates diversity, sparsity and consensus to dexterously manipulate multi-view data with limited labels. To encourage diversity, ERL-MVSC recasts a linear regression model to derive view-specific embedding regularizers and automatically determines their weights. This is able to tactfully incorporate complementary information of different views. To ensure sparsity, ERL-MVSC imposes l2,1 -norm on a fused embedding regularizer to exploit the sparse local structure of samples, thereby conveying valuable classification information and enhancing the robustness against noise/outliers. To enhance consensus, ERL-MVSC learns a shared predicted label matrix, which serves as the comment target of multi-view classification. With these techniques, we formulate ERL-MVSC as a joint optimization problem of an embedding regularizer and a predicted label matrix, which can be solved by a coordinate descent method. Extensive experimental results on real-world datasets demonstrate the effectiveness and superiority of the proposed algorithm.

Joint Learning of Latent Similarity and Local Embedding for Multi-View Clustering.

Spectral clustering has been an attractive topic in the field of computer vision due to the extensive growth of applications, such as image segmentation, clustering and representation. In this problem, the construction of the similarity matrix is a vital element affecting clustering performance. In this paper, we propose a multi-view joint learning (MVJL) framework to achieve both a reliable similarity matrix and a latent low-dimensional embedding. Specifically, the similarity matrix to be learned is represented as a convex hull of similarity matrices from different views, where the nuclear norm is imposed to capture the principal information of multiple views and improve robustness against noise/outliers. Moreover, an effective low-dimensional representation is obtained by applying local embedding on the similarity matrix, which preserves the local intrinsic structure of data through dimensionality reduction. With these techniques, we formulate the MVJL as a joint optimization problem and derive its mathematical solution with the alternating direction method of multipliers strategy and the proximal gradient descent method. The solution, which consists of a similarity matrix and a low-dimensional representation, is ultimately integrated with spectral clustering or K-means for multi-view clustering. Extensive experimental results on real-world datasets demonstrate that MVJL achieves superior clustering performance over other state-of-the-art methods.

Multi-View Data Fusion Oriented Clustering via Nuclear Norm Minimization.

Image clustering remains challenging when handling image data from heterogeneous sources. Fusing the independent and complementary information existing in heterogeneous sources together facilitates to improve the image clustering performance. To this end, we propose a joint learning framework of multi-view image data fusion and clustering based on nuclear norm minimization. Specifically, we first formulate the problem as matrix factorization to a shared clustering indicator matrix and a representative coefficient matrix. The former is constrained with orthogonality and nonnegativity, which ensures the validation of clustering assignments. The latter is imposed with nuclear norm minimization to achieve compression of principal components for performance improvement. Then, an alternating minimization strategy is employed to efficiently decompose the multi-variable optimization problem into several small solvable sub-problems with closed-form solutions. Extensive experimental results on real-world image and video datasets demonstrate the superiority of proposed method over other state-of-the-art methods.

Adaptive Quantization Parameter Cascading in HEVC Hierarchical Coding.

The state-of-the-art High Efficiency Video Coding (HEVC) standard adopts a hierarchical coding structure to improve its coding efficiency. This allows for the quantization parameter cascading (QPC) scheme that assigns quantization parameters (Qps) to different hierarchical layers in order to further improve the rate-distortion (RD) performance. However, only static QPC schemes have been suggested in HEVC test model, which are unable to fully explore the potentials of QPC. In this paper, we propose an adaptive QPC scheme for an HEVC hierarchical structure to code natural video sequences characterized by diversified textures, motions, and encoder configurations. We formulate the adaptive QPC scheme as a non-linear programming problem and solve it in a scientifically sound way with a manageable low computational overhead. The proposed model addresses a generic Qp assignment problem of video coding. Therefore, it also applies to group-of-picture-level, frame-level and coding unit-level Qp assignments. Comprehensive experiments have demonstrated that the proposed QPC scheme is able to adapt quickly to different video contents and coding configurations while achieving noticeable RD performance enhancement over all static and adaptive QPC schemes under comparison as well as HEVC default frame-level rate control. We have also made valuable observations on the distributions of adaptive QPC sets in the videos of different types of contents, which provide useful insights on how to further improve static QPC schemes.

Objective Quality Assessment for Color-to-Gray Image Conversion.

Color-to-gray (C2G) image conversion is the process of transforming a color image into a grayscale one. Despite its wide usage in real-world applications, little work has been dedicated to compare the performance of C2G conversion algorithms. Subjective evaluation is reliable but is also inconvenient and time consuming. Here, we make one of the first attempts to develop an objective quality model that automatically predicts the perceived quality of C2G converted images. Inspired by the philosophy of the structural similarity index, we propose a C2G structural similarity (C2G-SSIM) index, which evaluates the luminance, contrast, and structure similarities between the reference color image and the C2G converted image. The three components are then combined depending on image type to yield an overall quality measure. Experimental results show that the proposed C2G-SSIM index has close agreement with subjective rankings and significantly outperforms existing objective quality metrics for C2G conversion. To explore the potentials of C2G-SSIM, we further demonstrate its use in two applications: 1) automatic parameter tuning for C2G conversion algorithms and 2) adaptive fusion of C2G converted images.

Multiview coding mode decision with hybrid optimal stopping model.

In a generic decision process, optimal stopping theory aims to achieve a good tradeoff between decision performance and time consumed, with the advantages of theoretical decision-making and predictable decision performance. In this paper, optimal stopping theory is employed to develop an effective hybrid model for the mode decision problem, which aims to theoretically achieve a good tradeoff between the two interrelated measurements in mode decision, as computational complexity reduction and rate-distortion degradation. The proposed hybrid model is implemented and examined with a multiview encoder. To support the model and further promote coding performance, the multiview coding mode characteristics, including predicted mode probability and estimated coding time, are jointly investigated with inter-view correlations. Exhaustive experimental results with a wide range of video resolutions reveal the efficiency and robustness of our method, with high decision accuracy, negligible computational overhead, and almost intact rate-distortion performance compared to the original encoder.

H.264/SVC mode decision based on optimal stopping theory.

Fast mode decision algorithms have been widely used in the video encoder implementation to reduce encoding complexity yet without much sacrifice in the coding performance. Optimal stopping theory, which addresses early termination for a generic class of decision problems, is adopted in this paper to achieve fast mode decision for the H.264/Scalable Video Coding standard. A constrained model is developed with optimal stopping, and the solutions to this model are employed to initialize the candidate mode list and predict the early termination. Comprehensive simulation results are conducted to demonstrate that the proposed method strikes a good balance between low encoding complexity and high coding efficiency.