Improving biomedical named entity recognition by dynamic caching inter-sentence information.

MOTIVATION: Biomedical Named Entity Recognition (BioNER) aims to identify biomedical domain-specific entities (e.g. gene, chemical and disease) from unstructured texts. Despite deep learning-based methods for BioNER achieving satisfactory results, there is still much room for improvement. Firstly, most existing methods use independent sentences as training units and ignore inter-sentence context, which usually leads to the labeling inconsistency problem. Secondly, previous document-level BioNER works have approved that the inter-sentence information is essential, but what information should be regarded as context remains ambiguous. Moreover, there are still few pre-training-based BioNER models that have introduced inter-sentence information. Hence, we propose a cache-based inter-sentence model called BioNER-Cache to alleviate the aforementioned problems. RESULTS: We propose a simple but effective dynamic caching module to capture inter-sentence information for BioNER. Specifically, the cache stores recent hidden representations constrained by predefined caching rules. And the model uses a query-and-read mechanism to retrieve similar historical records from the cache as the local context. Then, an attention-based gated network is adopted to generate context-related features with BioBERT. To dynamically update the cache, we design a scoring function and implement a multi-task approach to jointly train our model. We build a comprehensive benchmark on four biomedical datasets to evaluate the model performance fairly. Finally, extensive experiments clearly validate the superiority of our proposed BioNER-Cache compared with various state-of-the-art intra-sentence and inter-sentence baselines. AVAILABILITYAND IMPLEMENTATION: Code will be available at https://github.com/zgzjdx/BioNER-Cache. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

A survey on causal inference for recommendation.

Causal inference has recently garnered significant interest among recommender system (RS) researchers due to its ability to dissect cause-and-effect relationships and its broad applicability across multiple fields. It offers a framework to model the causality in RSs such as confounding effects and deal with counterfactual problems such as offline policy evaluation and data augmentation. Although there are already some valuable surveys on causal recommendations, they typically classify approaches based on the practical issues faced in RS, a classification that may disperse and fragment the unified causal theories. Considering RS researchers' unfamiliarity with causality, it is necessary yet challenging to comprehensively review relevant studies from a coherent causal theoretical perspective, thereby facilitating a deeper integration of causal inference in RS. This survey provides a systematic review of up-to-date papers in this area from a causal theory standpoint and traces the evolutionary development of RS methods within the same causal strategy. First, we introduce the fundamental concepts of causal inference as the basis of the following review. Subsequently, we propose a novel theory-driven taxonomy, categorizing existing methods based on the causal theory employed, namely those based on the potential outcome framework, the structural causal model, and general counterfactuals. The review then delves into the technical details of how existing methods apply causal inference to address particular recommender issues. Finally, we highlight some promising directions for future research in this field. Representative papers and open-source resources will be progressively available at https://github.com/Chrissie-Law/Causal-Inference-for-Recommendation.

A syntactic evidence network model for fact verification.

In natural language processing, fact verification is a very challenging task, which requires retrieving multiple evidence sentences from a reliable corpus to verify the authenticity of a claim. Although most of the current deep learning methods use the attention mechanism for fact verification, they have not considered imposing attentional constraints on important related words in the claim and evidence sentences, resulting in inaccurate attention for some irrelevant words. In this paper, we propose a syntactic evidence network (SENet) model which incorporates entity keywords, syntactic information and sentence attention for fact verification. The SENet model extracts entity keywords from claim and evidence sentences, and uses a pre-trained syntactic dependency parser to extract the corresponding syntactic sentence structures and incorporates the extracted syntactic information into the attention mechanism for language-driven word representation. In addition, the sentence attention mechanism is applied to obtain a richer semantic representation. We have conducted experiments on the FEVER and UKP Snopes datasets for performance evaluation. Our SENet model has achieved 78.69% in Label Accuracy and 75.63% in FEVER Score on the FEVER dataset. In addition, our SENet model also has achieved 65.0% in precision and 61.2% in macro F1 on the UKP Snopes dataset. The experimental results have shown that our proposed SENet model has outperformed the baseline models and achieved the state-of-the-art performance for fact verification.

Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition.

The teacher-free online Knowledge Distillation (KD) aims to train an ensemble of multiple student models collaboratively and distill knowledge from each other. Although existing online KD methods achieve desirable performance, they often focus on class probabilities as the core knowledge type, ignoring the valuable feature representational information. We present a Mutual Contrastive Learning (MCL) framework for online KD. The core idea of MCL is to perform mutual interaction and transfer of contrastive distributions among a cohort of networks in an online manner. Our MCL can aggregate cross-network embedding information and maximize the lower bound to the mutual information between two networks. This enables each network to learn extra contrastive knowledge from others, leading to better feature representations, thus improving the performance of visual recognition tasks. Beyond the final layer, we extend MCL to intermediate layers and perform an adaptive layer-matching mechanism trained by meta-optimization. Experiments on image classification and transfer learning to visual recognition tasks show that layer-wise MCL can lead to consistent performance gains against state-of-the-art online KD approaches. The superiority demonstrates that layer-wise MCL can guide the network to generate better feature representations. Our code is publicly avaliable at https://github.com/winycg/L-MCL.

HarMI: Human Activity Recognition Via Multi-Modality Incremental Learning.

Nowadays, with the development of various kinds of sensors in smartphones or wearable devices, human activity recognition (HAR) has been widely researched and has numerous applications in healthcare, smart city, etc. Many techniques based on hand-crafted feature engineering or deep neural network have been proposed for sensor based HAR. However, these existing methods usually recognize activities offline, which means the whole data should be collected before training, occupying large-capacity storage space. Moreover, once the offline model training finished, the trained model can't recognize new activities unless retraining from the start, thus with a high cost of time and space. In this paper, we propose a multi-modality incremental learning model, called HarMI, with continuous learning ability. The proposed HarMI model can start training quickly with little storage space and easily learn new activities without storing previous training data. In detail, we first adopt attention mechanism to align heterogeneous sensor data with different frequencies. In addition, to overcome catastrophic forgetting in incremental learning, HarMI utilizes the elastic weight consolidation and canonical correlation analysis from a multi-modality perspective. Extensive experiments based on two public datasets demonstrate that HarMI can achieve a superior performance compared with several state-of-the-arts.

Real-time measurement of total nitrogen for agricultural runoff based on multiparameter sensors and intelligent algorithms.

Real-time monitoring of non-point source (NPS) pollution is challenging owing to the minute-scale change in runoff flow and concentration under rainfall condition. In this study, we proposed a real-time measurement method for total nitrogen (TN) by combining the timeliness of sensor detection and the accuracy of intelligent algorithms, based on the physical and chemical relationships between TN and sensor-measured indexes. Extra tree regression was selected as the TN inversion algorithm, which has high precision, high computational efficiency, and better ability in over-fitting control. The results show that: (1) the real-time inversion algorithm of TN can achieve the monitoring frequency at the minute scale (<5 min); (2) the method performs well (R2>0.9) when the training and testing datasets are from similar environmental backgrounds (fields or ditches); (3) in the case of partial variable missing, this method can still realize TN inversion, and the prediction accuracy is acceptable (R2>0.7) under the number of missing variables (n) ≤ 2, which makes up for the flaws of missing or abnormal data caused by sensor malfunctions. Overall, the proposed real-time measurement method of TN has stable data acquisition, high precision, and high monitoring frequency. In addition, the method is not limited by cloudy, rainy, or nighttime conditions. Compared with methods such as laboratory test, remote sensing inversion, and water quality automatic monitoring station, our method has obvious advantages in runoff monitoring of NPS pollution, which mainly occurs in small and micro water bodies. The new real-time measurement of TN for runoff may provide important technological support for pre-warning and emergency control of NPS pollution.

Market-oriented job skill valuation with cooperative composition neural network.

The value assessment of job skills is important for companies to select and retain the right talent. However, there are few quantitative ways available for this assessment. Therefore, we propose a data-driven solution to assess skill value from a market-oriented perspective. Specifically, we formulate the task of job skill value assessment as a Salary-Skill Value Composition Problem, where each job position is regarded as the composition of a set of required skills attached with the contextual information of jobs, and the job salary is assumed to be jointly influenced by the context-aware value of these skills. Then, we propose an enhanced neural network with cooperative structure, namely Salary-Skill Composition Network (SSCN), to separate the job skills and measure their value based on the massive job postings. Experiments show that SSCN can not only assign meaningful value to job skills, but also outperforms benchmark models for job salary prediction.

Coarse Alignment of Topic and Sentiment: A Unified Model for Cross-Lingual Sentiment Classification.

Cross-lingual sentiment classification (CLSC) aims to leverage rich-labeled resources in the source language to improve prediction models of a resource-scarce domain in the target language. Existing feature representation learning-based approaches try to minimize the difference of latent features between different domains by exact alignment, which is achieved by either one-to-one topic alignment or matrix projection. Exact alignment, however, restricts the representation flexibility and further degrades the model performances on CLSC tasks if the distribution difference between two language domains is large. On the other hand, most previous studies proposed document-level models or ignored sentiment polarities of topics that might lead to insufficient learning of latent features. To solve the abovementioned problems, we propose a coarse alignment mechanism to enhance the model's representation by a group-to-group topic alignment into an aspect-level fine-grained model. First, we propose an unsupervised aspect, opinion, and sentiment unification model (AOS), which trimodels aspects, opinions, and sentiments of reviews from different domains and helps capture more accurate latent feature representation by a coarse alignment mechanism. To further boost AOS, we propose ps-AOS, a partial supervised AOS model, in which labeled source language data help minimize the difference of feature representations between two language domains with the help of logistics regression. Finally, an expectation-maximization framework with Gibbs sampling is then proposed to optimize our model. Extensive experiments on various multilingual product review data sets show that ps-AOS significantly outperforms various kinds of state-of-the-art baselines.

Knowledge graph embedding with shared latent semantic units.

Knowledge graph embedding (KGE) aims to project both entities and relations into a continuous low-dimensional space. However, for a given knowledge graph (KG), only a small number of entities and relations occur many times, while the vast majority of entities and relations occur less frequently. This data sparsity problem has largely been ignored by most of the existing KGE models. To this end, in this paper, we propose a general technique to enable knowledge transfer among semantically similar entities or relations. Specifically, we define latent semantic units (LSUs), which are the sub-components of entity and relation embeddings. Semantically similar entities or relations are supposed to share the same LSUs, and thus knowledge can be transferred among entities or relations. Finally, extensive experiments show that the proposed technique is able to enhance existing KGE models and can provide better representations of KGs.

Deep Subdomain Adaptation Network for Image Classification.

For a target task where the labeled data are unavailable, domain adaptation can transfer a learner from a different source domain. Previous deep domain adaptation methods mainly learn a global domain shift, i.e., align the global source and target distributions without considering the relationships between two subdomains within the same category of different domains, leading to unsatisfying transfer learning performance without capturing the fine-grained information. Recently, more and more researchers pay attention to subdomain adaptation that focuses on accurately aligning the distributions of the relevant subdomains. However, most of them are adversarial methods that contain several loss functions and converge slowly. Based on this, we present a deep subdomain adaptation network (DSAN) that learns a transfer network by aligning the relevant subdomain distributions of domain-specific layer activations across different domains based on a local maximum mean discrepancy (LMMD). Our DSAN is very simple but effective, which does not need adversarial training and converges fast. The adaptation can be achieved easily with most feedforward network models by extending them with LMMD loss, which can be trained efficiently via backpropagation. Experiments demonstrate that DSAN can achieve remarkable results on both object recognition tasks and digit classification tasks. Our code will be available at https://github.com/easezyc/deep-transfer-learning.

Leveraging maximum entropy and correlation on latent factors for learning representations.

Many tasks involve learning representations from matrices, and Non-negative Matrix Factorization (NMF) has been widely used due to its excellent interpretability. Through factorization, sample vectors are reconstructed as additive combinations of latent factors, which are represented as non-negative distributions over the raw input features. NMF models are significantly affected by latent factors' distribution characteristics and the correlations among them. And NMF models are faced with the challenge of learning robust latent factor. To this end, we propose to learn representations with an awareness of the semantic quality evaluated from the aspects of intra- and inter-factors. On the one hand, a Maximum Entropy-based function is devised for the intra-factor semantic quality. On the other hand, the semantic uniqueness is evaluated via inter-factor correlation, which reinforces the aim of semantic compactness. Moreover, we present a novel non-linear NMF framework. The learning algorithm is presented and the convergence is theoretically analyzed and proved. Extensive experimental results on multiple datasets demonstrate that our method can be successfully applied to representative NMF models and boost performances over state-of-the-art models.

Rapid Learning of Earthquake Felt Area and Intensity Distribution with Real-time Search Engine Queries.

Immediately after a destructive earthquake, the real-time seismological community has a major focus on rapidly estimating the felt area and the extent of ground shaking. This estimate provides critical guidance for government emergency response teams to conduct orderly rescue and recovery operations in the damaged areas. While considerable efforts have been made in this direction, it still remains a realistic challenge for gathering macro-seismic data in a timely, accurate and cost-effective manner. To this end, we introduce a new direction to improve the information acquisition through monitoring the real-time information-seeking behaviors in the search engine queries, which are submitted by tens of millions of users after earthquakes. Specifically, we provide a very efficient, robust and machine-learning-assisted method for mapping the user-reported ground shaking distribution through the large-scale analysis of real-time search queries from a dominant search engine in China. In our approach, each query is regarded as a "crowd sensor" with a certain weight of confidence to proactively report the shaking location and extent. By fitting the epicenters of earthquakes occurred in mainland China from 2014 to 2018 with well-designed machine learning models, we can efficiently learn the realistic weight of confidence for each search query and sketch the felt areas and intensity distributions for most of the earthquakes. Indeed, this approach paves the way for using real-time search engine queries to efficiently map earthquake felt area in the regions with a relatively large population of search engine users.

Exploiting bi-directional global transition patterns and personal preferences for missing POI category identification.

Recent years have witnessed the increasing popularity of Location-based Social Network (LBSN) services, which provides unparalleled opportunities to build personalized Point-of-Interest (POI) recommender systems. Existing POI recommendation and location prediction tasks utilize past information for future recommendation or prediction from a single direction perspective, while the missing POI category identification task needs to utilize the check-in information both before and after the missing category. Therefore, a long-standing challenge is how to effectively identify the missing POI categories at any time in the real-world check-in data of mobile users. To this end, in this paper, we propose a novel neural network approach to identify the missing POI categories by integrating both bi-directional global non-personal transition patterns and personal preferences of users. Specifically, we delicately design an attention matching cell to model how well the check-in category information matches their non-personal transition patterns and personal preferences. Finally, we evaluate our model on two real-world datasets, which clearly validate its effectiveness compared with the state-of-the-art baselines. Furthermore, our model can be naturally extended to address next POI category recommendation and prediction tasks with competitive performance.

Softly Associative Transfer Learning for Cross-Domain Classification.

The main challenge of cross-domain text classification is to train a classifier in a source domain while applying it to a different target domain. Many transfer learning-based algorithms, for example, dual transfer learning, triplex transfer learning, etc., have been proposed for cross-domain classification, by detecting a shared low-dimensional feature representation for both source and target domains. These methods, however, often assume that the word clusters matrix or the clusters association matrix as knowledge transferring bridges are exactly the same across different domains, which is actually unrealistic in real-world applications and, therefore, could degrade classification performance. In light of this, in this paper, we propose a softly associative transfer learning algorithm for cross-domain text classification. Specifically, we integrate two non-negative matrix tri-factorizations into a joint optimization framework, with approximate constraints on both word clusters matrices and clusters association matrices so as to allow proper diversity in knowledge transfer, and with another approximate constraint on class labels in source domains in order to handle noisy labels. An iterative algorithm is then proposed to solve the above problem, with its convergence verified theoretically and empirically. Extensive experimental results on various text datasets demonstrate the effectiveness of our algorithm, even with the presence of abundant state-of-the-art competitors.

Adaptive Deep Modeling of Users and Items Using Side Information for Recommendation.

In the existing recommender systems, matrix factorization (MF) is widely applied to model user preferences and item features by mapping the user-item ratings into a low-dimension latent vector space. However, MF has ignored the individual diversity where the user's preference for different unrated items is usually different. A fixed representation of user preference factor extracted by MF cannot model the individual diversity well, which leads to a repeated and inaccurate recommendation. To this end, we propose a novel latent factor model called adaptive deep latent factor model (ADLFM), which learns the preference factor of users adaptively in accordance with the specific items under consideration. We propose a novel user representation method that is derived from their rated item descriptions instead of original user-item ratings. Based on this, we further propose a deep neural networks framework with an attention factor to learn the adaptive representations of users. Extensive experiments on Amazon data sets demonstrate that ADLFM outperforms the state-of-the-art baselines greatly. Also, further experiments show that the attention factor indeed makes a great contribution to our method.

Multi-representation adaptation network for cross-domain image classification.

In image classification, it is often expensive and time-consuming to acquire sufficient labels. To solve this problem, domain adaptation often provides an attractive option given a large amount of labeled data from a similar nature but different domains. Existing approaches mainly align the distributions of representations extracted by a single structure and the representations may only contain partial information, e.g., only contain part of the saturation, brightness, and hue information. Along this line, we propose Multi-Representation Adaptation which can dramatically improve the classification accuracy for cross-domain image classification and specially aims to align the distributions of multiple representations extracted by a hybrid structure named Inception Adaptation Module (IAM). Based on this, we present Multi-Representation Adaptation Network (MRAN) to accomplish the cross-domain image classification task via multi-representation alignment which can capture the information from different aspects. In addition, we extend Maximum Mean Discrepancy (MMD) to compute the adaptation loss. Our approach can be easily implemented by extending most feed-forward models with IAM, and the network can be trained efficiently via back-propagation. Experiments conducted on three benchmark image datasets demonstrate the effectiveness of MRAN.

NeuO: Exploiting the sentimental bias between ratings and reviews with neural networks.

Traditional recommender systems rely on user profiling based on either user ratings or reviews through bi-sentimental analysis. However, in real-world scenarios, there are two common phenomena: (1) users only provide ratings for items but without detailed review comments. As a result, the historical transaction data available for recommender systems are usually unbalanced and sparse; (2) in many cases, users' opinions can be better grasped in their reviews than ratings. For the reason that there is always a bias between ratings and reviews, it is really important that users' ratings and reviews should be mutually reinforced to grasp the users' true opinions. To this end, in this paper, we develop an opinion mining model based on convolutional neural networks for enhancing recommendation. Specifically, we exploit two-step training neural networks, which utilize both reviews and ratings to grasp users' true opinions in unbalanced data. Moreover, we propose a Sentiment Classification scoring (SC) method, which employs dual attention vectors to predict the users' sentiment scores of their reviews rather than using bi-sentiment analysis. Next, a combination function is designed to use the results of SC and user-item rating matrix to catch the opinion bias. It can filter the reviews and users, and build an enhanced user-item matrix. Finally, a Multilayer perceptron based Matrix Factorization (MMF) method is proposed to make recommendations with the enhanced user-item matrix. Extensive experiments on several real-world datasets (Yelp, Amazon, Taobao and Jingdong) demonstrate that (1) our approach can achieve a superior performance over state-of-the-art baselines; (2) our approach is able to tackle unbalanced data and achieve stable performances.

Semantic Feature Learning for Heterogeneous Multitask Classification via Non-Negative Matrix Factorization.

Multitask learning (MTL) aims to learn multiple related tasks simultaneously instead of separately to improve the generalization performance of each task. Most existing MTL methods assumed that the multiple tasks to be learned have the same feature representation. However, this assumption may not hold for many real-world applications. In this paper, we study the problem of MTL with heterogeneous features for each task. To address this problem, we first construct an integrated graph of a set of bipartite graphs to build a connection among different tasks. We then propose a non-negative matrix factorization-based multitask method (MTNMF) to learn a common semantic feature space underlying different heterogeneous feature spaces of each task. Moreover, an improved version of MTNMF (IMTNMF) is proposed, in which we do not need to construct the correlation matrix between input features and class labels, avoiding the information loss. Finally, based on the common semantic features and original heterogeneous features, we model the heterogenous MTL problem as a multitask multiview learning (MTMVL) problem. In this way, a number of existing MTMVL methods can be applied to solve the problem effectively. Extensive experiments on three real-world problems demonstrate the effectiveness of our proposed methods, and the improved version IMTNMF can gain about 2% average accuracy improvement compared with MTNMF.

Transfer collaborative filtering from multiple sources via consensus regularization.

Collaborative filtering is one of the most successful approaches to build recommendation systems. Recently, transfer learning has been applied to recommendation systems for incorporating information from external sources. However, most existing transfer collaborative filtering algorithms tend to transfer knowledge from one single source domain. Rich information is available in many source domains, which can better complement the data in the target domain than that from a single source. However, it is common to get inconsistent information from different sources. To this end, we proposed a TRA nsfer collaborative filtering framework from multiple sources via C onsE nsus R egularization, called TRACER for short. The TRACER framework handles the information inconsistency with a consensus regularization, which enforces the outputs from multiple sources to converge. In addition, our algorithm is to learn and transfer knowledge at the same time while most of the traditional transfer learning algorithms are to learn knowledge first and then transfer it. Experiments conducted on two real-world data sets validate the effectiveness of the proposed algorithm.

Representation learning via Dual-Autoencoder for recommendation.

Recommendation has provoked vast amount of attention and research in recent decades. Most previous works employ matrix factorization techniques to learn the latent factors of users and items. And many subsequent works consider external information, e.g., social relationships of users and items' attributions, to improve the recommendation performance under the matrix factorization framework. However, matrix factorization methods may not make full use of the limited information from rating or check-in matrices, and achieve unsatisfying results. Recently, deep learning has proven able to learn good representation in natural language processing, image classification, and so on. Along this line, we propose a new representation learning framework called Recommendation via Dual-Autoencoder (ReDa). In this framework, we simultaneously learn the new hidden representations of users and items using autoencoders, and minimize the deviations of training data by the learnt representations of users and items. Based on this framework, we develop a gradient descent method to learn hidden representations. Extensive experiments conducted on several real-world data sets demonstrate the effectiveness of our proposed method compared with state-of-the-art matrix factorization based methods.