Dynamic range (DR) is a pivotal characteristic of imaging systems. Current frame-based cameras struggle to achieve high dynamic range imaging due to the conflict between globally uniform exposure and spatially variant scene illumination. In this paper, we propose AsynHDR, a pixel-asynchronous HDR imaging system, based on key insights into the challenges in HDR imaging and the unique event-generating mechanism of dynamic vision sensors (DVS). Our proposed AsynHDR system integrates the DVS with a set of LCD panels. The LCD panels modulate the irradiance incident upon the DVS by altering their transparency, thereby triggering the pixel-independent event streams. The HDR image is subsequently decoded from the event streams through our temporal-weighted algorithm. Experiments under the standard test platform and several challenging scenes have verified the feasibility of the system in HDR imaging tasks.
Small-cell lung cancer (SCLC) accounts for nearly 15% of all lung cancers. Although patients respond to first-line therapy readily, rapid relapse is inevitable, with few treatment options in the second-line setting. Here, we describe SCLC cell lines harboring amplification of MYC and MYCN, but not MYCL1 nor non-amplified MYC cell lines, exhibit superior sensitivity to treatment with the pan-BET bromodomain protein inhibitor Mivebresib (ABBV-075). Silencing MYC and MYCN partially rescued SCLC cell lines harboring these respective amplifications from the anti-proliferative effects of mivebresib. Further characterization of genome-wide binding of MYC, MYCN, and MYCL1 uncovered unique enhancer and epigenetic preferences. Implications: Our study suggests that chromatin landscapes could establish cell states with unique gene expression programs, conveying sensitivity to epigenetic inhibitors such as mivebresib.
Change captioning aims to describe the semantic change between two similar images. In this process, as the most typical distractor, viewpoint change leads to the pseudo changes about appearance and position of objects, thereby overwhelming the real change. Besides, since the visual signal of change appears in a local region with weak feature, it is difficult for the model to directly translate the learned change features into the sentence. In this paper, we propose a syntax-calibrated multi-aspect relation transformer to learn effective change features under different scenes, and build reliable cross-modal alignment between the change features and linguistic words during caption generation. Specifically, a multi-aspect relation learning network is designed to 1) explore the fine-grained changes under irrelevant distractors (e.g., viewpoint change) by embedding the relations of semantics and relative position into the features of each image; 2) learn two view-invariant image representations by strengthening their global contrastive alignment relation, so as to help capture a stable difference representation; 3) provide the model with the prior knowledge about whether and where the semantic change happened by measuring the relation between the representations of captured difference and the image pair. Through the above manner, the model can learn effective change features for caption generation. Further, we introduce the syntax knowledge of Part-of-Speech (POS) and devise a POS-based visual switch to calibrate the transformer decoder. The POS-based visual switch dynamically utilizes visual information during different word generation based on the POS of words. This enables the decoder to build reliable cross-modal alignment, so as to generate a high-level linguistic sentence about change. Extensive experiments show that the proposed method achieves the state-of-the-art performance on the three public datasets.
How to model the effect of reflection is crucial for single image reflection removal (SIRR) task. Modern SIRR methods usually simplify the reflection formulation with the assumption of linear combination of a transmission layer and a reflection layer. However, the large variations in image content and the real-world picture-taking conditions often result in far more complex reflection. In this paper, we introduce a new screen-blur combination based on two important factors, namely the intensity and the blurriness of reflection, to better characterize the reflection formulation in SIRR. Specifically, we present Screen-blur Reflection Networks (SRNet), which executes the screen-blur formulation in its network design and adapts to the complex reflection on real scenes. Technically, SRNet consists of three components: a blended image generator, a reflection estimator and a reflection removal module. The image generator exploits the screen-blur combination to synthesize the training blended images. The reflection estimator learns the reflection layer and a blur degree that measures the level of blurriness for reflection. The reflection removal module further uses the blended image, blur degree and reflection layer to filter out the transmission layer in a cascaded manner. Superior results on three different SIRR methods are reported when generating the training data on the principle of the screen-blur combination. Moreover, extensive experiments on six datasets quantitatively and qualitatively demonstrate the efficacy of SRNet over the state-of-the-art methods.
Generalized Zero-Shot Learning (GZSL) aims at recognizing images from both seen and unseen classes by constructing correspondences between visual images and semantic embedding. However, existing methods suffer from a strong bias problem, where unseen images in the target domain tend to be recognized as seen classes in the source domain. To address this issue, we propose a Prototype-augmented Self-supervised Generative Network by integrating self-supervised learning and prototype learning into a feature generating model for GZSL. The proposed model enjoys several advantages. First, we propose a Self-supervised Learning Module to exploit inter-domain relationships, where we introduce anchors as a bridge between seen and unseen categories. In the shared space, we pull the distribution of the target domain away from the source domain and obtain domain-aware features. To our best knowledge, this is the first work to introduce self-supervised learning into GZSL as learning guidance. Second, a Prototype Enhancing Module is proposed to utilize class prototypes to model reliable target domain distribution in finer granularity. In this module, a Prototype Alignment mechanism and a Prototype Dispersion mechanism are combined to guide the generation of better target class features with intra-class compactness and inter-class separability. Extensive experimental results on five standard benchmarks demonstrate that our model performs favorably against state-of-the-art GZSL methods.
With high temporal resolution, high dynamic range, and low latency, event cameras have made great progress in numerous low-level vision tasks. To help restore low-quality (LQ) video sequences, most existing event-based methods usually employ convolutional neural networks (CNNs) to extract sparse event features without considering the spatial sparse distribution or the temporal relation in neighboring events. It brings about insufficient use of spatial and temporal information from events. To address this problem, we propose a new spiking-convolutional network (SC-Net) architecture to facilitate event-driven video restoration. Specifically, to properly extract the rich temporal information contained in the event data, we utilize a spiking neural network (SNN) to suit the sparse characteristics of events and capture temporal correlation in neighboring regions; to make full use of spatial consistency between events and frames, we adopt CNNs to transform sparse events as an extra brightness prior to being aware of detailed textures in video sequences. In this way, both the temporal correlation in neighboring events and the mutual spatial information between the two types of features are fully explored and exploited to accurately restore detailed textures and sharp edges. The effectiveness of the proposed network is validated in three representative video restoration tasks: deblurring, super-resolution, and deraining. Extensive experiments on synthetic and real-world benchmarks have illuminated that our method performs better than existing competing methods.
Audiovisual event localization aims to localize the event that is both visible and audible in a video. Previous works focus on segment-level audio and visual feature sequence encoding and neglect the event proposals and boundaries, which are crucial for this task. The event proposal features provide event internal consistency between several consecutive segments constructing one proposal, while the event boundary features offer event boundary consistency to make segments located at boundaries be aware of the event occurrence. In this article, we explore the proposal-level feature encoding and propose a novel context-aware proposal-boundary (CAPB) network to address audiovisual event localization. In particular, we design a local-global context encoder (LGCE) to aggregate local-global temporal context information for visual sequence, audio sequence, event proposals, and event boundaries, respectively. The local context from temporally adjacent segments or proposals contributes to event discrimination, while the global context from the entire video provides semantic guidance of temporal relationship. Furthermore, we enhance the structural consistency between segments by exploiting the above-encoded proposal and boundary representations. CAPB leverages the context information and structural consistency to obtain context-aware event-consistent cross-modal representation for accurate event localization. Extensive experiments conducted on the audiovisual event (AVE) dataset show that our approach outperforms the state-of-the-art methods by clear margins in both supervised event localization and cross-modality localization.
Reflection from glasses is ubiquitous in daily life, but it is usually undesirable in photographs. To remove these unwanted noises, existing methods utilize either correlative auxiliary information or handcrafted priors to constrain this ill-posed problem. However, due to their limited capability to describe the properties of reflections, these methods are unable to handle strong and complex reflection scenes. In this article, we propose a hue guidance network (HGNet) with two branches for single image reflection removal (SIRR) by integrating image information and corresponding hue information. The complementarity between image information and hue information has not been noticed. The key to this idea is that we found that hue information can describe reflections well and thus can be used as a superior constraint for the specific SIRR task. Accordingly, the first branch extracts the salient reflection features by directly estimating the hue map. The second branch leverages these effective features, which can help locate salient reflection regions to obtain a high-quality restored image. Furthermore, we design a new cyclic hue loss to provide a more accurate optimization direction for the network training. Experiments substantiate the superiority of our network, especially its excellent generalization ability to various reflection scenes, as compared with state-of-the-arts both qualitatively and quantitatively. Source codes are available at https://github.com/zhuyr97/HGRR.
We study the problem of localizing audio-visual events that are both audible and visible in a video. Existing works focus on encoding and aligning audio and visual features at the segment level while neglecting informative correlation between segments of the two modalities and between multi-scale event proposals. We propose a novel Semantic and Relation Modulation Network (SRMN) to learn the above correlation and leverage it to modulate the related auditory, visual, and fused features. In particular, for semantic modulation, we propose intra-modal normalization and cross-modal normalization. The former modulates features of a single modality with the event-relevant semantic guidance of the same modality. The latter modulates features of two modalities by establishing and exploiting the cross-modal relationship. For relation modulation, we propose a multi-scale proposal modulating module and a multi-alignment segment modulating module to introduce multi-scale event proposals and enable dense matching between cross-modal segments, which strengthen correlations between successive segments within one proposal and between all segments. With the features modulated by the correlation information regarding audio-visual events, SRMN performs accurate event localization. Extensive experiments conducted on the public AVE dataset demonstrate that our method outperforms the state-of-the-art methods in both supervised event localization and cross-modality localization tasks.
Image deraining is a challenging task since rain streaks have the characteristics of a spatially long structure and have a complex diversity. Existing deep learning-based methods mainly construct the deraining networks by stacking vanilla convolutional layers with local relations, and can only handle a single dataset due to catastrophic forgetting, resulting in a limited performance and insufficient adaptability. To address these issues, we propose a new image deraining framework to effectively explore nonlocal similarity, and to continuously learn on multiple datasets. Specifically, we first design a patchwise hypergraph convolutional module, which aims to better extract the nonlocal properties with higher-order constraints on the data, to construct a new backbone and to improve the deraining performance. Then, to achieve better generalizability and adaptability in real-world scenarios, we propose a biological brain-inspired continual learning algorithm. By imitating the plasticity mechanism of brain synapses during the learning and memory process, our continual learning process allows the network to achieve a subtle stability-plasticity tradeoff. This it can effectively alleviate catastrophic forgetting and enables a single network to handle multiple datasets. Compared with the competitors, our new deraining network with unified parameters attains a state-of-the-art performance on seen synthetic datasets and has a significantly improved generalizability on unseen real rainy images.
Algorithms , Brain , Memory
Existing deep learning based de-raining approaches have resorted to the convolutional architectures. However, the intrinsic limitations of convolution, including local receptive fields and independence of input content, hinder the model's ability to capture long-range and complicated rainy artifacts. To overcome these limitations, we propose an effective and efficient transformer-based architecture for the image de-raining. First, we introduce general priors of vision tasks, i.e., locality and hierarchy, into the network architecture so that our model can achieve excellent de-raining performance without costly pre-training. Second, since the geometric appearance of rainy artifacts is complicated and of significant variance in space, it is essential for de-raining models to extract both local and non-local features. Therefore, we design the complementary window-based transformer and spatial transformer to enhance locality while capturing long-range dependencies. Besides, to compensate for the positional blindness of self-attention, we establish a separate representative space for modeling positional relationship, and design a new relative position enhanced multi-head self-attention. In this way, our model enjoys powerful abilities to capture dependencies from both content and position, so as to achieve better image content recovery while removing rainy artifacts. Experiments substantiate that our approach attains more appealing results than state-of-the-art methods quantitatively and qualitatively.
Weakly supervised Referring Expression Grounding (REG) aims to ground a particular target in an image described by a language expression while lacking the correspondence between target and expression. Two main problems exist in weakly supervised REG. First, the lack of region-level annotations introduces ambiguities between proposals and queries. Second, most previous weakly supervised REG methods ignore the discriminative location and context of the referent, causing difficulties in distinguishing the target from other same-category objects. To address the above challenges, we design an entity-enhanced adaptive reconstruction network (EARN). Specifically, EARN includes three modules: entity enhancement, adaptive grounding, and collaborative reconstruction. In entity enhancement, we calculate semantic similarity as supervision to select the candidate proposals. Adaptive grounding calculates the ranking score of candidate proposals upon subject, location and context with hierarchical attention. Collaborative reconstruction measures the ranking result from three perspectives: adaptive reconstruction, language reconstruction and attribute classification. The adaptive mechanism helps to alleviate the variance of different referring expressions. Experiments on five datasets show EARN outperforms existing state-of-the-art methods. Qualitative results demonstrate that the proposed EARN can better handle the situation where multiple objects of a particular category are situated together.
Hederacoside C (HSC) has attracted much attention as a novel modulator of inflammation, but its anti-inflammatory mechanism remains elusive. In the present study, we investigated how HSC attenuated intestinal inflammation in vivo and in vitro. HSC injection significantly alleviated TNBS-induced colitis by inhibiting pro-inflammatory cytokine production and colonic epithelial cell apoptosis, and partially restored colonic epithelial cell proliferation. The therapeutic effect of HSC injection was comparable to that of oral administration of mesalazine (200 mg·kg-1·d-1, i.g.). In LPS-stimulated human intestinal epithelial Caco-2 cells, pretreatment with HSC (0.1, 1, 10 µM) significantly inhibited activation of MAPK/NF-κB and its downstream signaling pathways. Pretreatment with HSC prevented LPS-induced TLR4 dimerization and MyD88 recruitment in vitro. Quantitative proteomic analysis revealed that HSC injection regulated 18 proteins in the colon samples, mainly clustered in neutrophil degranulation. Among them, S100A9 involved in the degranulation of neutrophils was one of the most significantly down-regulated proteins. HSC suppressed the expression of S100A9 and its downstream genes including TLR4, MAPK, and NF-κB axes in colon. In Caco-2 cells, recombinant S100A9 protein activated the MAPK/NF-κB signaling pathway and induced inflammation, which were ameliorated by pretreatment with HSC. Notably, HSC attenuated neutrophil recruitment and degranulation as well as S100A9 release in vitro and in vivo. In addition, HSC promoted the expression of tight junction proteins and repaired the epithelial barrier via inhibiting S100A9. Our results verify that HSC ameliorates colitis via restoring impaired intestinal barrier through moderating S100A9/MAPK and neutrophil recruitment inactivation, suggesting that HSC is a promising therapeutic candidate for colitis.
Colitis , NF-kappa B , Humans , NF-kappa B/metabolism , Caco-2 Cells , Calgranulin B/adverse effects , Neutrophil Infiltration , Toll-Like Receptor 4/metabolism , Lipopolysaccharides/pharmacology , Proteomics , Cytokines/metabolism , Colitis/chemically induced , Colitis/drug therapy , Colitis/metabolism , Inflammation
Background/Aims. Multiple sclerosis (MS) is an autoimmune disorder that affects the central nervous system (CNS) primarily hallmarked by neuroinflammation and demyelination. The activation of astrocytes exerts double-edged sword effects, which perform an integral function in demyelination and remyelination. In this research, we examined the therapeutic effects of the Bu Shen Yi Sui capsule (BSYS), a traditional Chinese medicine prescription, in a cuprizone- (CPZ-) triggered demyelination model of MS (CPZ mice). This research intended to evaluate if BSYS might promote remyelination by shifting A1 astrocytes to A2 astrocytes. Methods. The effects of BSYS on astrocyte polarization and the potential mechanisms were explored in vitro and in vivo utilizing real-time quantitative reverse transcription PCR, immunofluorescence, and Western blotting. Histopathology, expression of inflammatory cytokines (IL-10, IL-1ß, and IL-6), growth factors (TGF-ß, BDNF), and motor coordination were assessed to verify the effects of BSYS (3.02 g/kg/d) on CPZ mice. In vitro, A1 astrocytes were induced by TNF-α (30 ng/mL), IL-1α (3 ng/mL), and C1q (400 ng/mL), following which the effect of BSYS-containing serum (concentration of 15%) on the transformation of A1/A2 reactive astrocytes was also evaluated. Results and Conclusions. BSYS treatment improved motor function in CPZ mice as assessed by rotarod tests. Intragastric administration of BSYS considerably lowered the proportion of A1 astrocytes, but the number of A2 astrocytes, MOG+, PLP+, CNPase+, and MBP+ cells was upregulated. Meanwhile, dysregulation of glutathione peroxidase, malondialdehyde, and superoxide dismutase was reversed in CPZ mice after treatment with BSYS. In addition, the lesion area and expression of proinflammatory cytokines were decreased and neuronal protection factors and anti-inflammatory cytokines were increased. In vitro, BSYS-containing serum suppressed the A1 astrocytic markers' expression and elevated the expression levels of A2 markers in primary astrocytes triggered by C1q, TNF-α, and IL-1α. Importantly, the miR-155/SOCS1 signaling pathway was involved in the modulation of the A1/A2 phenotype shift. Overall, this study demonstrated that BSYS has neuroprotective effects in myelin repair by modulating astrocyte polarization via the miR-155/SOCS1 pathway.
MicroRNAs , Multiple Sclerosis , Animals , Astrocytes/metabolism , Central Nervous System , Complement C1q/metabolism , Complement C1q/pharmacology , Mice , Mice, Inbred C57BL , MicroRNAs/metabolism , Multiple Sclerosis/drug therapy , Multiple Sclerosis/metabolism , Myelin Sheath , Tumor Necrosis Factor-alpha/metabolism
The analysis of neuronal morphological data is essential to investigate the neuronal properties and brain mechanisms. The complex morphologies, absence of annotations, and sheer volume of these data pose significant challenges in neuronal morphological analysis, such as identifying neuron types and large-scale neuron retrieval, all of which require accurate measuring and efficient matching algorithms. Recently, many studies have been conducted to describe neuronal morphologies quantitatively using predefined measurements. However, hand-crafted features are usually inadequate for distinguishing fine-grained differences among massive neurons. In this article, we propose a novel morphology-aware contrastive graph neural network (MACGNN) for unsupervised neuronal morphological representation learning. To improve the retrieval efficiency in large-scale neuronal morphological datasets, we further propose Hash-MACGNN by introducing an improved deep hash algorithm to train the network end-to-end to learn binary hash representations of neurons. We conduct extensive experiments on the largest dataset, NeuroMorpho, which contains more than 100 000 neurons. The experimental results demonstrate the effectiveness and superiority of our MACGNN and Hash-MACGNN for large-scale neuronal morphological analysis.
Multiple sclerosis (MS) is an autoimmune-mediated degenerative disease of the central nervous system (CNS) characterized by immune cell infiltration, demyelination and axonal injury. Oxidative stress-induced inflammatory response, especially the destructive effect of immune cell-derived free radicals on neurons and oligodendrocytes, is crucial in the onset and progression of MS. Therefore, targeting oxidative stress-related processes may be a promising preventive and therapeutic strategy for MS. Animal models, especially rodent models, can be used to explore the in vivo molecular mechanisms of MS considering their similarity to the pathological processes and clinical signs of MS in humans and the significant oxidative damage observed within their CNS. Consequently, these models have been used widely in pre-clinical studies of oxidative stress in MS. To date, many natural products have been shown to exert antioxidant effects to attenuate the CNS damage in animal models of MS. This review summarized several common rodent models of MS and their association with oxidative stress. In addition, this review provides a comprehensive and concise overview of previously reported natural antioxidant products in inhibiting the progression of MS.
Methods: The potential active ingredients and corresponding potential targets of BSYS Capsule were obtained from the TCMSP, BATMAN-TCM, Swiss Target Prediction platform, and literature research. Disease targets of CNSD were explored through the GeneCards and the DisGeNET databases. The matching targets of BSYS in CNSD were identified from a Venn diagram. The protein-protein interaction (PPI) network was constructed using bioinformatics methods. Gene Ontology (GO) function and Kyoto Encyclopedia of Genes and Genomes (KEGG) pathway enrichment analyses were performed to predict the mechanisms of BSYS. Furthermore, the neuroprotective effects of BSYS were evaluated using a cell model of hydrogen peroxide- (H2O2-) induced cell death in OLN-93 cells. Results: A total of 59 potential bioactive components of BSYS Capsule and 227 intersection targets were obtained. Topological analysis showed that AKT had the highest connectivity degrees in the PPI network. Enrichment analysis revealed that the targets of BSYS in the treatment of CNSD were the PI3K-Akt and MAPK signaling pathway, among other pathways. GO analysis results showed that the targets were associated with various biological processes, including apoptosis, reactive oxygen species metabolic process, and response to oxidative stress, among others. The experimental results demonstrated that BSYS drug-containing serum alleviated the H2O2-induced increase in LDH, MDA, and ROS levels and reversed the decrease in SOD and mitochondrial membrane potential induced by H2O2. BSYS treatment also decreased the number of TUNEL (+) cells, downregulated Bcl-2 expression, and upregulated Bax and c-caspase-3 expression by promoting Akt phosphorylation. Conclusion: BSYS Capsule alleviated H2O2-induced OLN-93 cell injury by increasing Akt phosphorylation to suppress oxidative stress and cell apoptosis. Therefore, BSYS can be potentially used for CNSD treatment. However, the results of this study are only derived from in vitro experiments, lacking the validation of in vivo animal models, which is a limitation of our study. We will further verify the underlying mechanisms of BSYS in animal experiments in the future.
Drugs, Chinese Herbal , Medicine, Chinese Traditional , Animals , Central Nervous System , Drugs, Chinese Herbal/therapeutic use , Hydrogen Peroxide/pharmacology , Medicine, Chinese Traditional/methods , Network Pharmacology , Phosphatidylinositol 3-Kinases , Proto-Oncogene Proteins c-akt
Diabetes mellitus (DM) is a chronic disorder associated with severe metabolic derangement and comorbidities. The constant increase in the global population of diabetic patients coupled with some prevailing side effects associated with synthetic antidiabetic drugs has necessitated the urgent need for the search for alternative antidiabetic regimens. This study investigated the antidiabetic, antioxidant, and pancreatic protective effects of the Acacia pennata extract (APE) against nicotinamide/streptozotocin induced DM in rats. The antidiabetic activity of APE was evaluated and investigated at doses of 100 and 400 mg/kg body weight, while metformin (150 mg/kg bw) was used as a standard drug. APE markedly decreased blood glucose level, homeostatic model assessment for insulin resistance, serum total cholesterol, triglycerides, low-density lipoprotein, blood urea nitrogen, creatinine, alanine transaminase, aspartate transaminase, and alanine phosphatase levels. Additionally, treatment with APE increased the body weight, serum insulin concentration, and high-density lipoprotein. Moreover, activities of pancreatic superoxide dismutase, catalase, and glutathione peroxidase were increased, while the altered pancreatic architecture in the histopathological examination was notably restored in the treated rats. Ultra-high performance liquid chromatography combined with electrospray ionization quadrupole time-of-flight mass spectrometry (UHPLC-ESI-QTOF-MS) analysis of APE showcases the prevailing presence of polyphenolic compounds. Conclusively, this study showed the beneficial effects of the Acacia pennata in controlling metabolic derangement, pancreatic and hepatorenal dysfunction in diabetic rats.
Remyelination is a refractory feature of demyelinating diseases such as multiple sclerosis (MS). Studies have shown that promoting oligodendrocyte precursor cell (OPC) differentiation, which cannot be achieved by currently available therapeutic agents, is the key to enhancing remyelination. Bu Shen Yi Sui capsule (BSYSC) is a traditional Chinese herbal medicine over many years of clinical practice. We have found that BSYSC can effectively treat MS. In this study, the effects of BSYSC in promoting OPCs differentiation and remyelination were assessed using an experimental autoimmune encephalomyelitis (EAE) model in vivo and cultured OPCs in vitro. The results showed that BSYSC reduced clinical function scores and increased neuroprotection. The expression of platelet-derived growth factor receptor α (PDGFR-α) was decreased and the level of 2',3'-cyclic nucleotide 3'-phosphodiesterase (CNPase) was increased in the brains and spinal cords of mice as well as in OPCs after treatment with BSYSC. We further found that BSYSC elevated the expression of miR-219 or miR-338 in the serum exosomes of mice with EAE, thereby suppressing the expression of Sox6, Lingo1, and Hes5, which negatively regulate OPCs differentiation. Therefore, serum exosomes of BSYSC-treated mice (exos-BSYSC) were extracted and administered to OPCs in which miR-219 or miR-338 expression was knocked down by adenovirus, and the results showed that Sox6, Lingo1, and Hes5 expression was downregulated, MBP expression was upregulated, OPCs differentiation was increased, and the ability of OPCs to wrap around neuronal axons was improved. In conclusion, BSYSC may exert clinically relevant effects by regulating microRNA (miR) levels in exosomes and thus promoting the differentiation and maturation of OPCs.
Video captioning aims to generate a natural language sentence to describe the main content of a video. Since there are multiple objects in videos, taking full exploration of the spatial and temporal relationships among them is crucial for this task. The previous methods wrap the detected objects as input sequences, and leverage vanilla self-attention or graph neural network to reason about visual relations. This cannot make full use of the spatial and temporal nature of a video, and suffers from the problems of redundant connections, over-smoothing, and relation ambiguity. In order to address the above problems, in this paper we construct a long short-term graph (LSTG) that simultaneously captures short-term spatial semantic relations and long-term transformation dependencies. Further, to perform relational reasoning over the LSTG, we design a global gated graph reasoning module (G3RM), which introduces a global gating based on global context to control information propagation between objects and alleviate relation ambiguity. Finally, by introducing G3RM into Transformer instead of self-attention, we propose the long short-term relation transformer (LSRT) to fully mine objects' relations for caption generation. Experiments on MSVD and MSR-VTT datasets show that the LSRT achieves superior performance compared with state-of-the-art methods. The visualization results indicate that our method alleviates problem of over-smoothing and strengthens the ability of relational reasoning.
