A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection.

Time series are the primary data type used to record dynamic system measurements and generated in great volume by both physical sensors and online processes (virtual sensors). Time series analytics is therefore crucial to unlocking the wealth of information implicit in available data. With the recent advancements in graph neural networks (GNNs), there has been a surge in GNN-based approaches for time series analysis. These approaches can explicitly model inter-temporal and inter-variable relationships, which traditional and other deep neural network-based methods struggle to do. In this survey, we provide a comprehensive review of graph neural networks for time series analysis (GNN4TS), encompassing four fundamental dimensions: forecasting, classification, anomaly detection, and imputation. Our aim is to guide designers and practitioners to understand, build applications, and advance research of GNN4TS. At first, we provide a comprehensive task-oriented taxonomy of GNN4TS. Then, we present and discuss representative research works and introduce mainstream applications of GNN4TS. A comprehensive discussion of potential future research directions completes the survey. This survey, for the first time, brings together a vast array of knowledge on GNN-based time series research, highlighting foundations, practical applications, and opportunities of graph neural networks for time series analysis.

Advancing mRNA subcellular localization prediction with graph neural network and RNA structure.

MOTIVATION: The asymmetrical distribution of expressed mRNAs tightly controls the precise synthesis of proteins within human cells. This non-uniform distribution, a cornerstone of developmental biology, plays a pivotal role in numerous cellular processes. To advance our comprehension of gene regulatory networks, it is essential to develop computational tools for accurately identifying the subcellular localizations of mRNAs. However, considering multi-localization phenomena remains limited in existing approaches, with none considering the influence of RNA's secondary structure. RESULTS: In this study, we propose Allocator, a multi-view parallel deep learning framework that seamlessly integrates the RNA sequence-level and structure-level information, enhancing the prediction of mRNA multi-localization. The Allocator models equip four efficient feature extractors, each designed to handle different inputs. Two are tailored for sequence-based inputs, incorporating multilayer perceptron and multi-head self-attention mechanisms. The other two are specialized in processing structure-based inputs, employing graph neural networks. Benchmarking results underscore Allocator's superiority over state-of-the-art methods, showcasing its strength in revealing intricate localization associations. AVAILABILITY AND IMPLEMENTATION: The webserver of Allocator is available at http://Allocator.unimelb-biotools.cloud.edu.au; the source code and datasets are available on GitHub (https://github.com/lifuyi774/Allocator) and Zenodo (https://doi.org/10.5281/zenodo.13235798).

Deep transformer-based heterogeneous spatiotemporal graph learning for geographical traffic forecasting.

Accurate geographical traffic forecasting plays a critical role in urban transportation planning, traffic management, and geospatial artificial intelligence (GeoAI). Although deep learning models have made significant progress in geographical traffic forecasting, they still face challenges in effectively capturing long-term temporal dependencies and modeling heterogeneous dynamic spatial dependencies. To address these issues, we propose a novel deep transformer-based heterogeneous spatiotemporal graph learning model for geographical traffic forecasting. Our model incorporates a temporal transformer that captures long-term temporal patterns in traffic data without simple data fusion. Furthermore, we introduce adaptive normalized graph structures within different graph layers, enabling the model to capture dynamic spatial dependencies and adapt to diverse traffic scenarios, especially for the heterogeneous relationship. We conduct comprehensive experiments and visualization on four primary public datasets and demonstrate that our model achieves state-of-the-art results in comparison to existing methods.

Flow2GNN: Flexible Two-Way Flow Message Passing for Enhancing GNNs Beyond Homophily.

Message passing (MP) is crucial for effective graph neural networks (GNNs). Most local message-passing schemes have been shown to underperform on heterophily graphs due to the perturbation of updated representations caused by local redundant heterophily information. However, our experiment findings indicate that the distribution of heterophily information during MP can be disrupted by disentangling local neighborhoods. This finding can be applied to other GNNs, improving their performance on heterophily graphs in a more flexible manner compared to most heterophily GNNs with complex designs. This article proposes a new type of simple message-passing neural network called Flow2GNN. It uses a two-way flow message-passing scheme to enhance the ability of GNNs by disentangling and redistributing heterophily information in the topology space and the attribute space. Our proposed message-passing scheme consists of two steps in topology space and attribute space. First, we introduce a new disentangled operator with binary elements that disentangle topology information in-flow and out-flow between connected nodes. Second, we use an adaptive aggregation model that adjusts the flow amount between homophily and heterophily attribute information. Furthermore, we rigorously prove that disentangling in message-passing can reduce the generalization gap, offering a deeper understanding of how our model enhances other GNNs. The extensive experiment results show that the proposed model, Flow2GNN, not only outperforms state-of-the-art GNNs, but also helps improve the performance of other commonly used GNNs on heterophily graphs, including GCN, GAT, GCNII, and H 2 GCN, specifically for GCN, with up to a 25.88% improvement on the Wisconsin dataset.

Graph Convolutional Multi-Label Hashing for Cross-Modal Retrieval.

Cross-modal hashing encodes different modalities of multimodal data into low-dimensional Hamming space for fast cross-modal retrieval. In multi-label cross-modal retrieval, multimodal data are often annotated with multiple labels, and some labels, e.g.", ocean" and "cloud", often co-occur. However, existing cross-modal hashing methods overlook label dependency that is crucial for improving performance. To fulfill this gap, this article proposes graph convolutional multi-label hashing (GCMLH) for effective multi-label cross-modal retrieval. Specifically, GCMLH first generates word embedding of each label and develops label encoder to learn highly correlated label embedding via graph convolutional network (GCN). In addition, GCMLH develops feature encoder for each modality, and feature fusion module to generate highly semantic feature via GCN. GCMLH uses teacher-student learning scheme to transfer knowledge from the teacher modules, i.e., label encoder and feature fusion module, to the student module, i.e., feature encoder, such that learned hash code can well exploit multi-label dependency and multimodal semantic structure. Extensive empirical results on several benchmarks demonstrate the superiority of the proposed method over existing state-of-the-arts.

Characterizing Secretion System Effector Proteins With Structure-Aware Graph Neural Networks and Pre-Trained Language Models.

The Type III Secretion Systems (T3SSs) play a pivotal role in host-pathogen interactions by mediating the secretion of type III secretion system effectors (T3SEs) into host cells. These T3SEs mimic host cell protein functions, influencing interactions between Gram-negative bacterial pathogens and their hosts. Identifying T3SEs is essential in biomedical research for comprehending bacterial pathogenesis and its implications on human cells. This study presents EDIFIER, a novel multi-channel model designed for accurate T3SE prediction. It incorporates a graph structural channel, utilizing graph convolutional networks (GCN) to capture protein 3D structural features and a sequence channel based on the ProteinBERT pre-trained model to extract the sequence context features of T3SEs. Rigorous benchmarking tests, including ablation studies and comparative analysis, validate that EDIFIER outperforms current state-of-the-art tools in T3SE prediction. To enhance EDIFIER's accessibility to the broader scientific community, we developed a webserver that is publicly accessible at http://edifier.unimelb-biotools.cloud.edu.au/. We anticipate EDIFIER will contribute to the field by providing reliable T3SE predictions, thereby advancing our understanding of host-pathogen dynamics.

Towards complex dynamic physics system simulation with graph neural ordinary equations.

The great learning ability of deep learning facilitates us to comprehend the real physical world, making learning to simulate complicated particle systems a promising endeavour both in academia and industry. However, the complex laws of the physical world pose significant challenges to the learning based simulations, such as the varying spatial dependencies between interacting particles and varying temporal dependencies between particle system states in different time stamps, which dominate particles' interacting behavior and the physical systems' evolution patterns. Existing learning based methods fail to fully account for the complexities, making them unable to yield satisfactory simulations. To better comprehend the complex physical laws, we propose a novel model - Graph Networks with Spatial-Temporal neural Ordinary Differential Equations (GNSTODE) - that characterizes the varying spatial and temporal dependencies in particle systems using a united end-to-end framework. Through training with real-world particle-particle interaction observations, GNSTODE can simulate any possible particle systems with high precisions. We empirically evaluate GNSTODE's simulation performance on two real-world particle systems, Gravity and Coulomb, with varying levels of spatial and temporal dependencies. The results show that GNSTODE yields better simulations than state-of-the-art methods, showing that GNSTODE can serve as an effective tool for particle simulation in real-world applications. Our code is made available at https://github.com/Guangsi-Shi/AI-for-physics-GNSTODE.

Knowledge Graphs and Pretrained Language Models Enhanced Representation Learning for Conversational Recommender Systems.

Conversational recommender systems (CRSs) utilize natural language interactions and dialog history to infer user preferences and provide accurate recommendations. Due to the limited conversation context and background knowledge, existing CRSs rely on external sources such as knowledge graphs (KGs) to enrich the context and model entities based on their interrelations. However, these methods ignore the rich intrinsic information within entities. To address this, we introduce the knowledge-enhanced entity representation learning (KERL) framework, which leverages both the KG and a pretrained language model (PLM) to improve the semantic understanding of entities for CRS. In our KERL framework, entity textual descriptions are encoded via a PLM, while a KG helps reinforce the representation of these entities. We also employ positional encoding to effectively capture the temporal information of entities in a conversation. The enhanced entity representation is then used to develop a recommender component that fuses both entity and contextual representations for more informed recommendations, as well as a dialog component that generates informative entity-related information in the response text. A high-quality KG with aligned entity descriptions is constructed to facilitate this study, namely, the Wiki Movie Knowledge Graph (WikiMKG). The experimental results show that KERL achieves state-of-the-art results in both recommendation and response generation tasks. Our code is publicly available at the link: https://github.com/icedpanda/KERL.

MADE: Multicurvature Adaptive Embedding for Temporal Knowledge Graph Completion.

Temporal knowledge graphs (TKGs) are receiving increased attention due to their time-dependent properties and the evolving nature of knowledge over time. TKGs typically contain complex geometric structures, such as hierarchical, ring, and chain structures, which can often be mixed together. However, embedding TKGs into Euclidean space, as is typically done with TKG completion (TKGC) models, presents a challenge when dealing with high-dimensional nonlinear data and complex geometric structures. To address this issue, we propose a novel TKGC model called multicurvature adaptive embedding (MADE). MADE models TKGs in multicurvature spaces, including flat Euclidean space (zero curvature), hyperbolic space (negative curvature), and hyperspherical space (positive curvature), to handle multiple geometric structures. We assign different weights to different curvature spaces in a data-driven manner to strengthen the ideal curvature spaces for modeling and weaken the inappropriate ones. Additionally, we introduce the quadruplet distributor (QD) to assist the information interaction in each geometric space. Ultimately, we develop an innovative temporal regularization to enhance the smoothness of timestamp embeddings by strengthening the correlation of neighboring timestamps. Experimental results show that MADE outperforms the existing state-of-the-art TKGC models.

Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects.

Self-supervised learning (SSL) has recently achieved impressive performance on various time series tasks. The most prominent advantage of SSL is that it reduces the dependence on labeled data. Based on the pre-training and fine-tuning strategy, even a small amount of labeled data can achieve high performance. Compared with many published self-supervised surveys on computer vision and natural language processing, a comprehensive survey for time series SSL is still missing. To fill this gap, we review current state-of-the-art SSL methods for time series data in this article. To this end, we first comprehensively review existing surveys related to SSL and time series, and then provide a new taxonomy of existing time series SSL methods by summarizing them from three perspectives: generative-based, contrastive-based, and adversarial-based. These methods are further divided into ten subcategories with detailed reviews and discussions about their key intuitions, main frameworks, advantages and disadvantages. To facilitate the experiments and validation of time series SSL methods, we also summarize datasets commonly used in time series forecasting, classification, anomaly detection, and clustering tasks. Finally, we present the future directions of SSL for time series analysis.

An Inductive Reasoning Model based on Interpretable Logical Rules over temporal knowledge graph.

Extrapolating future events based on historical information in temporal knowledge graphs (TKGs) holds significant research value and practical applications. In this field, the methods currently utilized can be classified as either embedding-based or logical rule-based. Embedding-based methods depend on learned entity and relation embeddings for prediction, but they suffer from the lack of interpretability due to the opaque reasoning process. On the other hand, logical rule-based methods face scalability challenges as they heavily rely on predefined logical rules. To overcome these limitations, we propose a hybrid model that combines embedding-based and logical rule-based methods to capture deep causal logic. Our model, called the Inductive Reasoning Model based on Interpretable Logical Rule (ILR-IR), aims to provide interpretable insights while effectively predicting future events in TKGs. ILR-IR delves into historical information, extracting valuable insights from logical rules embedded within relations and interaction preferences between entities. By considering both logical rules and interaction preferences, ILR-IR offers a comprehensive perspective for predicting future events. In addition, we propose the incorporation of a one-class augmented matching loss during optimization, which serves to enhance performance of the model during training. We evaluate ILR-IR on multiple datasets, including ICEWS14, ICEWS0515, and ICEWS18. Experimental results demonstrate that ILR-IR outperforms state-of-the-art baselines, showcasing its superior performance in TKG extrapolation reasoning. Moreover, ILR-IR demonstrates remarkable generalization capabilities, even when applied to related datasets that share a common relation vocabulary. This suggests that our proposed model exhibits robust zero-shot reasoning abilities. For interested parties, we have made our code publicly available at https://github.com/mxadorable/ILR-IR.

ConTIG: Continuous representation learning on temporal interaction graphs.

Representation learning on temporal interaction graphs (TIG) aims to model complex networks with the dynamic evolution of interactions on a wide range of web and social graph applications. However, most existing works on TIG either (a) rely on discretely updated node embeddings merely when an interaction occurs that fail to capture the continuous evolution of embedding trajectories of nodes, or (b) overlook the rich temporal patterns hidden in the ever-changing graph data that presumably lead to sub-optimal models. In this paper, we propose a two-module framework named ConTIG, a novel representation learning method on TIG that captures the continuous dynamic evolution of node embedding trajectories. With two essential modules, our model exploits three-fold factors in dynamic networks including latest interaction, neighbor features, and inherent characteristics. In the first update module, we employ a continuous inference block to learn the nodes' state trajectories from time-adjacent interaction patterns using ordinary differential equations. In the second transform module, we introduce a self-attention mechanism to predict future node embeddings by aggregating historical temporal interaction information. Experiment results demonstrate the superiority of ConTIG on temporal link prediction, temporal node recommendation, and dynamic node classification tasks of four datasets compared with a range of state-of-the-art baselines, especially for long-interval interaction prediction.

PLANNER: a multi-scale deep language model for the origins of replication site prediction.

Origins of replication sites (ORIs) are crucial genomic regions where DNA replication initiation takes place, playing pivotal roles in fundamental biological processes like cell division, gene expression regulation, and DNA integrity. Accurate identification of ORIs is essential for comprehending cell replication, gene expression, and mutation-related diseases. However, experimental approaches for ORI identification are often expensive and time-consuming, leading to the growing popularity of computational methods. In this study, we present PLANNER (DeeP LeArNiNg prEdictor for ORI), a novel approach for species-specific and cell-specific prediction of eukaryotic ORIs. PLANNER uses the multi-scale ktuple sequences as input and employs the DNABERT pre-training model with transfer learning and ensemble learning strategies to train accurate predictive models. Extensive empirical test results demonstrate that PLANNER achieved superior predictive performance compared to state-of-the-art approaches, including iOri-Euk, Stack-ORI, and ORI-Deep, within specific cell types and across different cell types. Furthermore, by incorporating an interpretable analysis mechanism, we provide insights into the learned patterns, facilitating the mapping from discovering important sequential determinants to comprehensively analysing their biological functions. To facilitate the widespread utilisation of PLANNER, we developed an online webserver and local stand-alone software, available at http://planner.unimelb-biotools.cloud.edu.au/ and https://github.com/CongWang3/PLANNER, respectively.

SPAG6 regulates cell proliferation and apoptosis via TGF-ß/Smad signal pathway in adult B-cell acute lymphoblastic leukemia.

Adult B-cell acute lymphoblastic leukemia (B-ALL) prognosis remains unsatisfactory, and searching for new therapeutic targets is crucial for improving patient prognosis. Sperm-associated antigen 6 (SPAG6), a member of the cancer-testis antigen family, plays an important role in tumors, especially hematologic tumors; however, it is unknown whether SPAG6 plays a role in adult B-ALL. In this study, we demonstrated for the first time that SPAG6 expression was up-regulated in the bone marrow of adult B-ALL patients compared to healthy donors, and expression was significantly reduced in patients who achieved complete remission (CR) after treatment. In addition, patients with high SPAG6 expression were older (≥ 35 years; P = 0.015), had elevated white blood cell counts (WBC > 30 × 109/L; P = 0.021), and a low rate of CR (P = 0.036). We explored the SPAG6 effect on cell function by lentiviral transfection of adult B-ALL cell lines BALL-1 and NALM-6, and discovered that knocking down SPAG6 significantly inhibited cell proliferation and promoted apoptosis. We identified that SPAG6 knockdown might regulate cell proliferation and apoptosis via the transforming growth factor-ß (TGF-ß)/Smad signaling pathway.

Correlation-Aware Spatial-Temporal Graph Learning for Multivariate Time-Series Anomaly Detection.

Multivariate time-series anomaly detection is critically important in many applications, including retail, transportation, power grid, and water treatment plants. Existing approaches for this problem mostly employ either statistical models which cannot capture the nonlinear relations well or conventional deep learning (DL) models e.g., convolutional neural network (CNN) and long short-term memory (LSTM) that do not explicitly learn the pairwise correlations among variables. To overcome these limitations, we propose a novel method, correlation-aware spatial-temporal graph learning (termed ), for time-series anomaly detection. explicitly captures the pairwise correlations via a correlation learning (MTCL) module based on which a spatial-temporal graph neural network (STGNN) can be developed. Then, by employing a graph convolution network (GCN) that exploits one-and multihop neighbor information, our STGNN component can encode rich spatial information from complex pairwise dependencies between variables. With a temporal module that consists of dilated convolutional functions, the STGNN can further capture long-range dependence over time. A novel anomaly scoring component is further integrated into to estimate the degree of an anomaly in a purely unsupervised manner. Experimental results demonstrate that can detect and diagnose anomalies effectively in general settings as well as enable early detection across different time delays. Our code is available at https://github.com/huankoh/CST-GL.

Upregulated SPAG6 correlates with increased STAT1 and is associated with reduced sensitivity of interferon-α response in BCR::ABL1 negative myeloproliferative neoplasms.

Sperm-associated antigen 6 (SPAG6) has been identified as an oncogene or tumor suppressor in various types of human cancer. However, the role of SPAG6 in BCR::ABL1 negative myeloproliferative neoplasms (MPNs) remains unclear. Herein, we found that SPAG6 was upregulated at the mRNA level in primary MPN cells and MPN-derived leukemia cell lines. The SPAG6 protein was primarily located in the cytoplasm around the nucleus and positively correlated with ß-tubulin expression. In vitro, forced expression of SPAG6 increased cell clone formation and promoted G1 to S cell cycle progression. Downregulation of SPAG6 promoted apoptosis, reduced G1 to S phase transition, and impaired cell proliferation and cytokine release accompanied by downregulated signal transducer and activator of transcription 1 (STAT1) expression. Furthermore, the inhibitory effect of interferon-α (INF-α) on the primary MPN cells with high SPAG6 expression was decreased. Downregulation of SPAG6 enhanced STAT1 induction, thus enhancing the proapoptotic and cell cycle arrest effects of INF-α both in vitro and in vivo. Finally, a decrease in SPAG6 protein expression was noted when the STAT1 signaling was blocked. Chromatin immunoprecipitation assays indicated that STAT1 protein could bind to the SPAG6 promoter, while the dual-luciferase reporter assay indicated that STAT1 could promote the expression of SPAG6. Our results substantiate the relationship between upregulated SPAG6, increased STAT1, and reduced sensitivity to INF-α response in MPN.

Domain-Adaptive Graph Attention-Supervised Network for Cross-Network Edge Classification.

Graph neural networks (GNNs) have shown great ability in modeling graphs; however, their performance would significantly degrade when there are noisy edges connecting nodes from different classes. To alleviate negative effect of noisy edges on neighborhood aggregation, some recent GNNs propose to predict the label agreement between node pairs within a single network. However, predicting the label agreement of edges across different networks has not been investigated yet. Our work makes the pioneering attempt to study a novel problem of cross-network homophilous and heterophilous edge classification (CNHHEC) and proposes a novel domain-adaptive graph attention-supervised network (DGASN) to effectively tackle the CNHHEC problem. First, DGASN adopts multihead graph attention network (GAT) as the GNN encoder, which jointly trains node embeddings and edge embeddings via the node classification and edge classification losses. As a result, label-discriminative embeddings can be obtained to distinguish homophilous edges from heterophilous edges. In addition, DGASN applies direct supervision on graph attention learning based on the observed edge labels from the source network, thus lowering the negative effects of heterophilous edges while enlarging the positive effects of homophilous edges during neighborhood aggregation. To facilitate knowledge transfer across networks, DGASN employs adversarial domain adaptation to mitigate domain divergence. Extensive experiments on real-world benchmark datasets demonstrate that the proposed DGASN achieves the state-of-the-art performance in CNHHEC.

Boosting Graph Contrastive Learning via Adaptive Sampling.

Contrastive learning (CL) is a prominent technique for self-supervised representation learning, which aims to contrast semantically similar (i.e., positive) and dissimilar (i.e., negative) pairs of examples under different augmented views. Recently, CL has provided unprecedented potential for learning expressive graph representations without external supervision. In graph CL, the negative nodes are typically uniformly sampled from augmented views to formulate the contrastive objective. However, this uniform negative sampling strategy limits the expressive power of contrastive models. To be specific, not all the negative nodes can provide sufficiently meaningful knowledge for effective contrastive representation learning. In addition, the negative nodes that are semantically similar to the anchor are undesirably repelled from it, leading to degraded model performance. To address these limitations, in this article, we devise an adaptive sampling strategy termed "AdaS." The proposed AdaS framework can be trained to adaptively encode the importance of different negative nodes, so as to encourage learning from the most informative graph nodes. Meanwhile, an auxiliary polarization regularizer is proposed to suppress the adverse impacts of the false negatives and enhance the discrimination ability of AdaS. The experimental results on a variety of real-world datasets firmly verify the effectiveness of our AdaS in improving the performance of graph CL.

A survey on neural-symbolic learning systems.

In recent years, neural systems have demonstrated highly effective learning ability and superior perception intelligence. However, they have been found to lack effective reasoning and cognitive ability. On the other hand, symbolic systems exhibit exceptional cognitive intelligence but suffer from poor learning capabilities when compared to neural systems. Recognizing the advantages and disadvantages of both methodologies, an ideal solution emerges: combining neural systems and symbolic systems to create neural-symbolic learning systems that possess powerful perception and cognition. The purpose of this paper is to survey the advancements in neural-symbolic learning systems from four distinct perspectives: challenges, methods, applications, and future directions. By doing so, this research aims to propel this emerging field forward, offering researchers a comprehensive and holistic overview. This overview will not only highlight the current state-of-the-art but also identify promising avenues for future research.