Tackling higher-order relations and heterogeneity: Dynamic heterogeneous hypergraph network for spatiotemporal activity prediction.

Spatiotemporal activity prediction aims to predict user activities at a particular time and location, which is applicable in city planning, activity recommendations, and other domains. The fundamental endeavor in spatiotemporal activity prediction is to model the intricate interaction patterns among users, locations, time, and activities, which is characterized by higher-order relations and heterogeneity. Recently, graph-based methods have gained popularity due to the advancements in graph neural networks. However, these methods encounter two significant challenges. Firstly, higher-order relations and heterogeneity are not adequately modeled. Secondly, the majority of established methods are designed around the static graph structures that rely solely on co-occurrence relations, which can be imprecise. To overcome these challenges, we propose DyH2N, a dynamic heterogeneous hypergraph network for spatiotemporal activity prediction. Specifically, to enhance the capacity for modeling higher-order relations, hypergraphs are employed in lieu of graphs. Then we propose a set representation learning-inspired heterogeneous hyperedge learning module, which models higher-order relations and heterogeneity in spatiotemporal activity prediction using a non-decomposable manner. To improve the encoding of heterogeneous spatiotemporal activity hyperedges, a knowledge representation-regularized loss is introduced. Moreover, we present a hypergraph structure learning module to update the hypergraph structures dynamically. Our proposed DyH2N model has been extensively tested on four real-world datasets, proving to outperform previous state-of-the-art methods by 5.98% to 27.13%. The effectiveness of all framework components is demonstrated through ablation experiments.

Multi-Modal Remote Sensing Image Segmentation with Intuition-Inspired Hypergraph Modeling.

Multi-modal remote sensing (RS) image segmentation aims to comprehensively utilize multiple RS modalities to assign pixel-level semantics to the studied scenes, which can provide a new perspective for global city understanding. Multi-modal segmentation inevitably encounters the challenge of modeling intra- and inter-modal relationships, i.e., object diversity and modal gaps. However, the previous methods are usually designed for a single RS modality, limited by the noisy collection environment and poor discrimination information. Neuropsychology and neuroanatomy confirm that the human brain performs the guiding perception and integrative cognition of multi-modal semantics through intuitive reasoning. Therefore, establishing a semantic understanding framework inspired by intuition to realize multi-modal RS segmentation becomes the main motivation of this work. Drived by the superiority of hypergraphs in modeling high-order relationships, we propose an intuition-inspired hypergraph network (I2HN) for multi-modal RS segmentation. Specifically, we present a hypergraph parser to imitate guiding perception to learn intra-modal object-wise relationships. It parses the input modality into irregular hypergraphs to mine semantic clues and generate robust mono-modal representations. In addition, we also design a hypergraph matcher to dynamically update the hypergraph structure from the explicit correspondence of visual concepts, similar to integrative cognition, to improve cross-modal compatibility when fusing multi-modal features. Extensive experiments on two multi-modal RS datasets show that the proposed I2HN outperforms the state-of-the-art models, achieving F1/mIoU accuracy 91.4%/82.9% on the ISPRS Vaihingen dataset, and 92.1%/84.2% on the MSAW dataset. The complete algorithm and benchmark results will be available online.

M 2 DCapsN: Multimodal, Multichannel, and Dual-Step Capsule Network for Natural Language Moment Localization.

Natural language moment localization aims to localize the target moment that matches a given natural language query in an untrimmed video. The key to this challenging task is to capture fine-grained video-language correlations to establish the alignment between the query and target moment. Most existing works establish a single-pass interaction schema to capture correlations between queries and moments. Considering the complex feature space of lengthy video and diverse information between frames, the weight distribution of information interaction flow is prone to dispersion or misalignment, which leads to redundant information flow affecting the final prediction. We address this issue by proposing a capsule-based approach to model the query-video interactions, termed the Multimodal, Multichannel, and Dual-step Capsule Network (M 2 DCapsN), which is derived from the intuition that "multiple people viewing multiple times is better than one person viewing one time." First, we introduce a multimodal capsule network, replacing the single-pass interaction schema of "one person viewing one time" with the iterative interaction schema of "one person viewing multiple times", which cyclically updates cross-modal interactions and modifies potential redundant interactions via its routing-by-agreement. Then, considering that the conventional routing mechanism only learns a single iterative interaction schema, we further propose a multichannel dynamic routing mechanism to learn multiple iterative interaction schemas, where each channel performs independent routing iteration to collectively capture cross-modal correlations from multiple subspaces, that is", multiple people viewing." Moreover, we design a dual-step capsule network structure based on the multimodal, multichannel capsule network, bringing together the query and query-guided key moments to jointly enhance the original video, so as to select the target moments according to the enhanced part. Experimental results on three public datasets demonstrate the superiority of our approach in comparison with state-of-the-art methods, and comprehensive ablation and visualization analysis validate the effectiveness of each component of the proposed model.

Modeling High-Order Relationships: Brain-Inspired Hypergraph-Induced Multimodal-Multitask Framework for Semantic Comprehension.

Semantic comprehension aims to reasonably reproduce people's real intentions or thoughts, e.g., sentiment, humor, sarcasm, motivation, and offensiveness, from multiple modalities. It can be instantiated as a multimodal-oriented multitask classification issue and applied to scenarios, such as online public opinion supervision and political stance analysis. Previous methods generally employ multimodal learning alone to deal with varied modalities or solely exploit multitask learning to solve various tasks, a few to unify both into an integrated framework. Moreover, multimodal-multitask cooperative learning could inevitably encounter the challenges of modeling high-order relationships, i.e., intramodal, intermodal, and intertask relationships. Related research of brain sciences proves that the human brain possesses multimodal perception and multitask cognition for semantic comprehension via decomposing, associating, and synthesizing processes. Thus, establishing a brain-inspired semantic comprehension framework to bridge the gap between multimodal and multitask learning becomes the primary motivation of this work. Motivated by the superiority of the hypergraph in modeling high-order relations, in this article, we propose a hypergraph-induced multimodal-multitask (HIMM) network for semantic comprehension. HIMM incorporates monomodal, multimodal, and multitask hypergraph networks to, respectively, mimic the decomposing, associating, and synthesizing processes to tackle the intramodal, intermodal, and intertask relationships accordingly. Furthermore, temporal and spatial hypergraph constructions are designed to model the relationships in the modality with sequential and spatial structures, respectively. Also, we elaborate a hypergraph alternative updating algorithm to ensure that vertices aggregate to update hyperedges and hyperedges converge to update their connected vertices. Experiments on the dataset with two modalities and five tasks verify the effectiveness of HIMM on semantic comprehension.

Revealing influencing factors on global waste distribution via deep-learning based dumpsite detection from satellite imagery.

With the advancement of global civilisation, monitoring and managing dumpsites have become essential parts of environmental governance in various countries. Dumpsite locations are difficult to obtain in a timely manner by local government agencies and environmental groups. The World Bank shows that governments need to spend massive labour and economic costs to collect illegal dumpsites to implement management. Here we show that applying novel deep convolutional networks to high-resolution satellite images can provide an effective, efficient, and low-cost method to detect dumpsites. In sampled areas of 28 cities around the world, our model detects nearly 1000 dumpsites that appeared around 2021. This approach reduces the investigation time by more than 96.8% compared with the manual method. With this novel and powerful methodology, it is now capable of analysing the relationship between dumpsites and various social attributes on a global scale, temporally and spatially.

Mimicking the Brain's Cognition of Sarcasm From Multidisciplines for Twitter Sarcasm Detection.

Sarcasm is a sophisticated construct to express contempt or ridicule. It is well-studied in multiple disciplines (e.g., neuroanatomy and neuropsychology) but is still in its infancy in computational science (e.g., Twitter sarcasm detection). In contrast to previous methods that are usually geared toward a single discipline, we focus on the multidisciplinary cross-innovation, i.e., improving embryonic sarcasm detection in computational science by leveraging the advanced knowledge of sarcasm cognition in neuroanatomy and neuropsychology. In this work, we are oriented toward sarcasm detection in social media and correspondingly propose a multimodal, multi-interactive, and multihierarchical neural network ( M3N2 ). We select Twitter, image, text in image, and image caption as the input of M3N2 since the brain's perception of sarcasm requires multiple modalities. To reasonably address the multimodalities, we introduce singlewise, pairwise, triplewise, and tetradwise modality interactions incorporating gate mechanism and guide attention (GA) to simulate the interactions and collaborations of involved regions in the brain while perceiving multiple modes. Specifically, we exploit a multihop process for each modality interaction to extract modal information multiple times using GA for obtaining multiperspective information. Also, we adopt a two-hierarchical structure leveraging self-attention accompanied by attention pooling to integrate multimodal semantic information from different levels mimicking the brain's first- and second-order comprehensions of sarcasm. Experimental results show that M3N2 achieves competitive performance in sarcasm detection and displays powerful generalization ability in multimodal sentiment analysis and emotion recognition.

Niacin analogue, 6-Aminonicotinamide, a novel inhibitor of hepatitis B virus replication and HBsAg production.

BACKGROUND: Hepatitis B surface antigen (HBsAg) is one of the important clinical indexes for hepatitis B virus (HBV) infection diagnosis and sustained seroconversion of HBsAg is an indicator for functional cure. However, the level of HBsAg could not be reduced by interferons and nucleoside analogs effectively. Therefore, identification of a new drug targeting HBsAg is urgently needed. METHODS: In this study, 6-AN was screened out from 1500 compounds due to its low cytotoxicity and high antiviral activity. The effect of 6-AN on HBV was examined in HepAD38, HepG2-NTCP and PHHs cells. In addition, the antivirus effect of 6-AN was also identified in mouse model. FINDINGS: 6-AN treatment resulted in a significant decrease of HBsAg and other viral markers both in vitro and in vivo. Furthermore, we found that 6-AN inhibited the activities of HBV SpI, SpII and core promoter by decreasing transcription factor PPARα, subsequently reduced HBV RNAs transcription and HBsAg production. INTERPRETATION: We have identified a novel small molecule to inhibit HBV core DNA, HBV RNAs, HBsAg production, as well as cccDNA to a minor degree both in vitro and in vivo. This study may shed light on the development of a novel class of anti-HBV agent.