Enriching contextualized language model from knowledge graph for biomedical information extraction.

Biomedical information extraction (BioIE) is an important task. The aim is to analyze biomedical texts and extract structured information such as named entities and semantic relations between them. In recent years, pre-trained language models have largely improved the performance of BioIE. However, they neglect to incorporate external structural knowledge, which can provide rich factual information to support the underlying understanding and reasoning for biomedical information extraction. In this paper, we first evaluate current extraction methods, including vanilla neural networks, general language models and pre-trained contextualized language models on biomedical information extraction tasks, including named entity recognition, relation extraction and event extraction. We then propose to enrich a contextualized language model by integrating a large scale of biomedical knowledge graphs (namely, BioKGLM). In order to effectively encode knowledge, we explore a three-stage training procedure and introduce different fusion strategies to facilitate knowledge injection. Experimental results on multiple tasks show that BioKGLM consistently outperforms state-of-the-art extraction models. A further analysis proves that BioKGLM can capture the underlying relations between biomedical knowledge concepts, which are crucial for BioIE.

Joint polarization detection and degradation mechanisms for underwater image enhancement.

Light absorption and scattering exist in the underwater environment, which can lead to blurring, reduced brightness, and color distortion in underwater images. Polarized images have the advantages of eliminating underwater scattering interference, enhancing contrast, and detecting material information of the object in underwater detection. In this paper, from the perspective of polarization imaging, different concentrations (0.15 g/ml, 0.30 g/ml, and 0.50 g/ml), different wave bands (red, green, and blue), different materials (copper, wood, high-density PVC, aluminum, cloth, foam, cloth sheet, low-density PVC, rubber, and porcelain tile), and different depths (10 cm, 20 cm, 30 cm, and 40 cm) are set up in a chamber for the experimental environment. By combining the degradation mechanism of underwater images and the analysis of polarization detection results, it is proved that the degree of polarization images have greater advantages than degree of linear polarization images, degree of circular polarization images, S1, S2, and S3 images, and visible images underwater. Finally, a fusion algorithm of underwater visible images and polarization images based on compressed sensing is proposed to enhance underwater degraded images. To improve the quality of fused images, we introduce orthogonal matching pursuit (OMP) in the high-frequency part to improve image sparsity and consistency detection in the low-frequency part to improve the image mutation phenomenon. The fusion results show that the peak SNR values of the fusion result maps using OMP in this paper are improved by 32.19% and 22.14% on average over those using backpropagation and subspace pursuit methods. With different materials and concentrations, the underwater image enhancement algorithm proposed in this paper improves information entropy, average gradient, and standard deviation by 7.76%, 18.12%, and 40.8%, respectively, on average over previous algorithms. The image NIQE value shows that the image quality obtained by this paper's algorithm is improved by about 69.26% over the original S0 image.

Hydrophobicity-tuned anion responsiveness underlies endosomolytic cargo delivery mediated by amphipathic vehicle peptides.

Peptide conformation can change subject to environment cues. This concept also applies to many cationic amphipathic peptides (CAPs) known to have cell membrane lytic or penetrative activities. Well-conditioned CAPs can match the properties of the target membrane to support their intended biological functions, e.g., intracellular cargo delivery; however, the intricacy in such conditioning surpasses our current understanding. Here we focused on hydrophobicity, a key biophysical property that dictates the membrane activity of CAPs, and applied a structure-function strategy to evolve a template peptide for endosomolytic cargo delivery. The template was subjected to iterative adjustment to balance hydrophobicity between its N-terminal linear and C-terminal helical domains. We demonstrate that the obtained peptide, LP6, could dramatically promote cargo cell entry and facilitate cytosolic delivery of biomacromolecules such as FITC-dextran, saporin, and human IgG. Among the evolved peptide series, LP6 has low cytotoxicity and moderate hydrophobicity, exhibits maximum change in helical conformation in response to negatively charged phospholipids, and also shows an apparent aggregational behavior in response to sialic acid enrichment. These attributes of LP6 collectively indicate that its anion-responsive conformational change is a critical underlining of its endosomolytic cargo delivery capability. Our results also suggest that modulation of hydrophobicity serves as a key to the precise tuning of CAP's membrane activity for future biomedical applications.

A span-graph neural model for overlapping entity relation extraction in biomedical texts.

MOTIVATION: Entity relation extraction is one of the fundamental tasks in biomedical text mining, which is usually solved by the models from natural language processing. Compared with traditional pipeline methods, joint methods can avoid the error propagation from entity to relation, giving better performances. However, the existing joint models are built upon sequential scheme, and fail to detect overlapping entity and relation, which are ubiquitous in biomedical texts. The main reason is that sequential models have relatively weaker power in capturing long-range dependencies, which results in lower performance in encoding longer sentences. In this article, we propose a novel span-graph neural model for jointly extracting overlapping entity relation in biomedical texts. Our model treats the task as relation triplets prediction, and builds the entity-graph by enumerating possible candidate entity spans. The proposed model captures the relationship between the correlated entities via a span scorer and a relation scorer, respectively, and finally outputs all valid relational triplets. RESULTS: Experimental results on two biomedical entity relation extraction tasks, including drug-drug interaction detection and protein-protein interaction detection, show that the proposed method outperforms previous models by a substantial margin, demonstrating the effectiveness of span-graph-based method for overlapping relation extraction in biomedical texts. Further in-depth analysis proves that our model is more effective in capturing the long-range dependencies for relation extraction compared with the sequential models. AVAILABILITY AND IMPLEMENTATION: Related codes are made publicly available at http://github.com/Baxelyne/SpanBioER.

Silver Ion-Histidine Interplay Switches Peptide Hydrogel from Antiparallel to Parallel ß-Assembly and Enables Controlled Antibacterial Activity.

Understanding the chemical absorption process of silver ions helps the rational design of functional materials for effective release to minimize unwanted toxicity. To this end, a histidine-containing aliphatic peptide (IH6) was designed to immobilize the silver ion (Ag+) through coordinate interaction. Using circular dichroism spectroscopy, Ag+ was found to dose-dependently induce parallel ß-sheet conformation of IH6 to a saturation molar ratio of 1:2. A conformational switch of IH6 from antiparallel to parallel ß-sheet assembly upon Ag + coordination was further revealed by Fourier transform infrared spectroscopy. The resultant Ag-IH6 hydrogel displayed substantially enhanced mechanical strength as well as controlled release of Ag+. Ag-IH6 hydrogel thus exhibited strong dose-dependent bactericidal activities that can be tuned selectively, sparing the cocultured human keratinocytes in normal. Overall, the study demonstrates an unusual silver ion-induced peptide conformational switch between ß-structure subtypes and the bilateral effects on hydrogel-based chemical control of silver ion absorption and release, thus, revealing the potential in antibacterial applications.

A hybrid neural network model for predicting kidney disease in hypertension patients based on electronic health records.

BACKGROUND: Disease prediction based on Electronic Health Records (EHR) has become one hot research topic in biomedical community. Existing work mainly focuses on the prediction of one target disease, and little work is proposed for multiple associated diseases prediction. Meanwhile, a piece of EHR usually contains two main information: the textual description and physical indicators. However, existing work largely adopts statistical models with discrete features from numerical physical indicators in EHR, and fails to make full use of textual description information. METHODS: In this paper, we study the problem of kidney disease prediction in hypertension patients by using neural network model. Specifically, we first model the prediction problem as a binary classification task. Then we propose a hybrid neural network which incorporates Bidirectional Long Short-Term Memory (BiLSTM) and Autoencoder networks to fully capture the information in EHR. RESULTS: We construct a dataset based on a large number of raw EHR data. The dataset consists of totally 35,332 records from hypertension patients. Experimental results show that the proposed neural model achieves 89.7% accuracy for the task. CONCLUSIONS: A hybrid neural network model was presented. Based on the constructed dataset, the comparison results of different models demonstrated the effectiveness of the proposed neural model. The proposed model outperformed traditional statistical models with discrete features and neural baseline systems.

Activation of AMP-activated protein kinase by compound 991 protects osteoblasts from dexamethasone.

Dexamethasone (Dex) induces direct cytotoxicity to cultured osteoblasts. The benzimidazole derivative compound 991 ("C991") is a novel and highly-efficient AMP-activated protein kinase (AMPK) activator. Here, in both MC3T3-E1 osteoblastic cells and primary murine osteoblasts, treatment with C991 activated AMPK signaling, and significantly attenuated Dex-induced apoptotic and non-apoptotic cell death. AMPKα1 knockdown (by shRNA), complete knockout (by CRISPR/Cas9 method) or dominant negative mutation (T172A) not only blocked C991-mediated AMPK activation, but also abolished its pro-survival effect against Dex in osteoblasts. Further studies showed that C991 boosted nicotinamide adenine dinucleotide phosphate (NADPH) activity and induced mRNA expression of NF-E2-related factor 2 (Nrf2)-regulated genes (heme oxygenase-1 and NADPH quinone oxidoreductase 1). Additionally, C991 alleviated Dex-induced reactive oxygen species (ROS) production in osteoblasts. Notably, genetic AMPK inhibition reversed the anti-oxidant actions by C991 in Dex-treated osteoblasts. Together, we conclude that C991 activates AMPK signaling to protect osteoblasts from Dex.

The redistribution of Drosophila vesicular monoamine transporter mutants from synaptic vesicles to large dense-core vesicles impairs amine-dependent behaviors.

Monoamine neurotransmitters are stored in both synaptic vesicles (SVs), which are required for release at the synapse, and large dense-core vesicles (LDCVs), which mediate extrasynaptic release. The contributions of each type of vesicular release to specific behaviors are not known. To address this issue, we generated mutations in the C-terminal trafficking domain of the Drosophila vesicular monoamine transporter (DVMAT), which is required for the vesicular storage of monoamines in both SVs and LDCVs. Deletion of the terminal 23 aa (DVMAT-Δ3) reduced the rate of endocytosis and localization of DVMAT to SVs, but supported localization to LDCVs. An alanine substitution mutation in a tyrosine-based motif (DVMAT-Y600A) also reduced sorting to SVs and showed an endocytic deficit specific to aminergic nerve terminals. Redistribution of DVMAT-Y600A from SV to LDCV fractions was also enhanced in aminergic neurons. To determine how these changes might affect behavior, we expressed DVMAT-Δ3 and DVMAT-Y600A in a dVMAT null genetic background that lacks endogenous dVMAT activity. When expressed ubiquitously, DVMAT-Δ3 showed a specific deficit in female fertility, whereas DVMAT-Y600A rescued behavior similarly to DVMAT-wt. In contrast, when expressed more specifically in octopaminergic neurons, both DVMAT-Δ3 and DVMAT-Y600A failed to rescue female fertility, and DVMAT-Y600A showed deficits in larval locomotion. DVMAT-Y600A also showed more severe dominant effects than either DVMAT-wt or DVMAT-Δ3. We propose that these behavioral deficits result from the redistribution of DVMAT from SVs to LDCVs. By extension, our data suggest that the balance of amine release from SVs versus that from LDCVs is critical for the function of some aminergic circuits.

Histidine-iridium(III) coordination-based peptide luminogenic cyclization and cyclo-RGD peptides for cancer-cell targeting.

In the field of peptide drug discovery, structural constraining and fluorescent labeling are two sought-after techniques important for both basic research and pharmaceutical development. In this work, we describe an easy-to-use approach for simultaneous peptide cyclization and luminescent labeling based on iridium(III)-histidine coordination (Ir-HH cyclization). Using a series of model peptides with histidine flanking each terminus, the binding activity and reaction kinetics of Ir-HH cyclization of different ring sizes were characterized. In the series, Ir-HAnH (n = 2, 3) with moderate ring sizes provides appropriate flexibility and proper distance between histidines for cyclic formation, which leads to the best binding affinity and structural stability in physiological conditions, as compared to other Ir-HH-cyclized peptides with smaller (n = 0, 1) or larger (n = 4, 5) ring sizes. Ir-HRGDH, an Ir-HH-cyclized peptide containing integrin targeting motif Arg-Gly-Asp (RGD), showed better targeting affinity than its linear form and enhanced membrane permeability in comparison with fluorescein-labeled cyclic RGDyK peptide. Cell death inducing peptide KLA-linked Ir-HRGDH (Ir-HRGDH-KLA) showed dramatically enhanced cytotoxicity and high selectivity for cancer cells versus noncancer cells. These data demonstrate that the method conveniently combines structural constraining of peptides with luminescent imaging capabilities, which facilitates functional and intracellular characterization of potential peptide-based drug leads, thus introducing a new tool to meet emerging needs in medicinal research.

Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment.

While pre-training large-scale video-language models (VLMs) has shown remarkable potential for various downstream video-language tasks, existing VLMs can still suffer from certain commonly seen limitations, e.g., coarse-grained cross-modal aligning, under-modeling of temporal dynamics, detached video-language view. In this work, we target enhancing VLMs with a fine-grained structural spatio-temporal alignment learning method (namely Finsta). First of all, we represent the input texts and videos with fine-grained scene graph (SG) structures, both of which are further unified into a holistic SG (HSG) for bridging two modalities. Then, an SG-based framework is built, where the textual SG (TSG) is encoded with a graph Transformer, while the video dynamic SG (DSG) and the HSG are modeled with a novel recurrent graph Transformer for spatial and temporal feature propagation. A spatial-temporal Gaussian differential graph Transformer is further devised to strengthen the sense of the changes in objects across spatial and temporal dimensions. Next, based on the fine-grained structural features of TSG and DSG, we perform object-centered spatial alignment and predicate-centered temporal alignment respectively, enhancing the video-language grounding in both the spatiality and temporality. We design our method as a plug&play system, which can be integrated into existing well-trained VLMs for further representation augmentation, without training from scratch or relying on SG annotations in downstream applications. On 6 representative VL modeling tasks over 12 datasets in both standard and long-form video scenarios, Finsta consistently improves the existing 13 strong-performing VLMs persistently, and refreshes the current state-of-the-art end task performance significantly in both the fine-tuning and zero-shot settings.

A density correction method for radioactive waste drum based on SRGS technology.

The segmented ringed gamma scanning (SRGS) technique represents an advancement in segmented gamma scanning (SGS) technology used for detecting the density of radioactive waste drums, offering enhanced measurement accuracy. However, significant occur errors in the reconstruction of matrix densities due to the non-uniform distribution of density in radioactive waste and the conical beam emitted from the transmission source collimator. This paper proposes a density correction method based on dichotomy to address this issue. The efficacy of this method was verified through both simulations and experiments on a sample containing five different materials, utilizing 137Cs and 60Co for transmission and emission measurements, respectively. The experimental results demonstrate that the errors in the corrected matrix densities are reduced, falling within a margin of 16.8%. Additionally, the corrected reconstruction error of the activity is approximately 25% of the uncorrected results.

[Combination of bone marrow concentrate and PGA scaffolds enhancing bone marrow stimulation in the repair of rabbit articular cartilage].

OBJECTIVE: To observe the effects of bone marrow concentrate (BMC)-PGA scaffolds for bone marrow stimulation enhancement and repairing rabbit articular cartilage. METHODS: A rabbit model of articular cartilage defect was established for BMC-PGA stent implantation. After 8 weeks, the experimental animals were sacrificed. And the methods of hematoxylin and eosin stain, toluidine blue stain and immunohistochemistry were used to evaluate the effects of bone marrow stimulation enhancement and rabbit cartilage defect repairing. RESULTS: Visible new cartilage formation was evident after implantation. As compared with other groups, the repairing effect was better. CONCLUSION: The implantation of BMC-PGA scaffolds is both simple and effective in the repair of articular cartilage.

[Selection of suitable procedure in the treatment of degenerative lumbar scoliosis based upon imaging studies].

OBJECTIVE: To compare the clinical efficacies of two different procedures in the treatment of degenerative lumbar scoliosis. METHODS: From August 2008 to August 2011, 28 patients of lumbar degenerative scoliosis were divided into one group (n = 14) undergoing modified transforaminal lumbar interbody fusion (TLIF) instrumented surgery and another group (n = 14) undergoing posterolateral fusion (PLF) instrumented surgery. There were 12 males and 16 females with a mean age of 66.2 years (range: 54-79). The operative durations and bleeding volumes of two groups were recorded. The post-operative efficacy was evaluated with VAS (visual analogue scale) for low back pain, ODI (Oswestry disability index), Cobb' angle and lumbar lordosis angle on plain film. RESULTS: The mean follow-up period was 25.9 months. The operative duration was 192.0 ± 44.7 min in modified TLIF group versus 163.0 ± 39.0 min in PLF group. The bleeding volume was 718.0 ± 197.2 ml in modified TLIF group versus 546.0 ± 226.6 ml in PLF group. All operated lumbar intervertebral achieved bony fusion in modified TLIF group by the last follow-up. Two cases had no bony fusion and there was one case of pseudarthrosis in PLF group. Significant differences existed between two groups in pre-operative and post-operative values of VAS, ODI, Cobb' s angle and lumbar lordosis angle (P < 0.05). There were significant differences between two groups in the values of pre-operative and post-operative VAS and lumbar lordosis angle (P < 0.05) but not in the values of pre-operative and post-operative ODI and Cobb' s angle (P > 0.05). CONCLUSION: As an alternative, safe and effective procedure, modified TLIF instrumented is superior to PLF instrumented in the treatment of lumbar degenerative scoliosis.

Nonautoregressive Encoder-Decoder Neural Framework for End-to-End Aspect-Based Sentiment Triplet Extraction.

Aspect-based sentiment triplet extraction (ASTE) aims at recognizing the joint triplets from texts, i.e., aspect terms, opinion expressions, and correlated sentiment polarities. As a newly proposed task, ASTE depicts the complete sentiment picture from different perspectives to better facilitate real-world applications. Unfortunately, several major challenges, such as the overlapping issue and long-distance dependency, have not been addressed effectively by the existing ASTE methods, which limits the performance of the task. In this article, we present an innovative encoder-decoder framework for end-to-end ASTE. Specifically, the ASTE task is first modeled as an unordered triplet set prediction problem, which is satisfied with a nonautoregressive decoding paradigm with a pointer network. Second, a novel high-order aggregation mechanism is proposed for fully integrating the underlying interactions between the overlapping structure of aspect and opinion terms. Third, a bipartite matching loss is introduced for facilitating the training of our nonautoregressive system. Experimental results on benchmark datasets show that our proposed framework significantly outperforms the state-of-the-art methods. Further analysis demonstrates the advantages of the proposed framework in handling the overlapping issue, relieving long-distance dependency and decoding efficiency.

Albumin pre-opsonized membrane-active iPep nanomedicine potentiates chemo to immunotherapy of cancer.

Chemotherapy-conjugated immunotherapy in clinical oncology conceptually resembles the combined effects of cytoreduction and immunostimulation in membrane targeted cell killings mediated by pore-forming proteins or host defense peptides. Of the similar concept, targeting cancer cell membrane using membrane active peptides is a hopeful therapeutic modality but had long been hindered from in vivo application. Here we report an enabling strategy of pre-opsonizing a membrane penetrating Ir-complexed octa-arginine peptide (iPep) with serum albumin via intrinsic amphipathicity-driven bimodal interactions into nanoparticles (NP). We found that NP triggered stress-mediated 4T1 cell oncosis which induced potent immunological activation, surpassing several well-known immunogenic medicines. Vested with albumin-enhanced in vivo tumor targeting specificity and pharmacokinetic properties, NP showed combined chemo to immunotherapies of s. c. tumors in mice, with decreased percentages of MDSC, Treg, M2-like macrophage and improved infiltration of CTLs in tumor site, caused complete regression of 4T1 and CT26 tumors, outperforming clinical medicines. In a challenging orthotopic breast cancer model, boost i. v. injections of NP acted as in situ tumor vaccine that drastically enhanced 4T1-specific cellular and humoral immunities to reverse disease progression. Thus, with combined effects of direct cytoreduction, immune activation and tumor vaccine, iPep-NP presents the promise and potential of a new modality of cancer medicine.

Direct Seawater Electrolysis: From Catalyst Design to Device Applications.

Direct seawater electrolysis (DSE) for hydrogen production, using earth-abundant seawater as the feedstock and renewable electricity as the driving source, paves a new opportunity for flexible energy conversion/storage and smooths the volatility of renewable energy. Unfortunately, the complex environments of seawater impose significant challenges on the design of DSE catalysts, and the practical performance of many current DSE catalysts remains unsatisfactory on the device level. However, many studies predominantly concentrate on the development of electrocatalysts for DSE without giving due consideration to the specific devices. To mitigate this gap, the most recent progress (mainly published within the year 2020-2023) of DSE electrocatalysts and devices are systematically evaluated. By discussing key bottlenecks, corresponding mitigation strategies, and various device designs and applications, the tremendous challenges in addressing the trade-off among activity, stability, and selectivity for DSE electrocatalysts by a single shot are emphasized. In addition, the rational design of the DSE electrocatalysts needs to align with the specific device configuration, which is more effective than attempting to comprehensively enhance all catalytic parameters. This work, featuring the first review of this kind to consider rational catalyst design in the framework of DSE devices, will facilitate practical DSE development.

A Three-Dimensional Vibration Theory for Ultralight Cellular Sandwich Plates Subjected to Linearly Varying In-Plane Distributed Loads.

Thin structural elements such as large-scale covering plates of aerospace protection structures and vertical stabilizers of aircraft are strongly influenced by gravity (and/or acceleration); thus, exploring how the mechanical behaviors of such structures are affected by gravitational field is necessary. Built upon a zigzag displacement model, this study establishes a three-dimensional vibration theory for ultralight cellular-cored sandwich plates subjected to linearly varying in-plane distributed loads (due to, e.g., hyper gravity or acceleration), with the cross-section rotation angle induced by face sheet shearing accounted for. For selected boundary conditions, the theory enables quantifying the influence of core type (e.g., close-celled metal foams, triangular corrugated metal plates, and metal hexagonal honeycombs) on fundamental frequencies of the sandwich plates. For validation, three-dimensional finite element simulations are carried out, with good agreement achieved between theoretical predictions and simulation results. The validated theory is subsequently employed to evaluate how the geometric parameters of metal sandwich core and the mixture of metal cores and composite face sheets influence the fundamental frequencies. Triangular corrugated sandwich plate possesses the highest fundamental frequency, irrespective of boundary conditions. For each type of sandwich plate considered, the presence of in-plane distributed loads significantly affects its fundamental frequencies and modal shapes.

Metal Oxide-Supported Metal Catalysts for Electrocatalytic Oxygen Reduction Reaction: Characterization Methods, Modulation Strategies, and Recent Progress.

The sluggish kinetics of the oxygen reduction reaction (ORR) with complex multielectron transfer steps significantly limits the large-scale application of electrochemical energy devices, including metal-air batteries and fuel cells. Recent years witnessed the development of metal oxide-supported metal catalysts (MOSMCs), covering single atoms, clusters, and nanoparticles. As alternatives to conventional carbon-dispersed metal catalysts, MOSMCs are gaining increasing interest due to their unique electronic configuration and potentially high corrosion resistance. By engineering the metal oxide substrate, supported metal, and their interactions, MOSMCs can be facilely modulated. Significant progress has been made in advancing MOSMCs for ORR, and their further development warrants advanced characterization methods to better understand MOSMCs and precise modulation strategies to boost their functionalities. In this regard, a comprehensive review of MOSMCs for ORR is still lacking despite this fast-developing field. To eliminate this gap, advanced characterization methods are introduced for clarifying MOSMCs experimentally and theoretically, discuss critical methods of boosting their intrinsic activities and number of active sites, and systematically overview the status of MOSMCs based on different metal oxide substrates for ORR. By conveying methods, research status, critical challenges, and perspectives, this review will rationally promote the design of MOSMCs for electrochemical energy devices.

Luminescent iridium(III) complexes as novel protein staining agents.

This article reports a new class of luminescent metal complexes, biscyclometalated iridium(III) complexes with an ancillary bathophenanthroline disulfonate ligand, for staining protein bands that are separated by electrophoresis. The performances of these novel staining agents have been studied in comparison with tris(bathophenanthroline disulfonate) ruthenium(II) tetrasodium salt (i.e. RuBPS) using a commercially available imaging system. The staining agents showed different limits of detection, linear dynamic ranges, and protein-to-protein variations. The overall performances of all three stains were found to be better than or equivalent to RuBPS under the experimental conditions.

Optimizing Attention for Sequence Modeling via Reinforcement Learning.

Attention has been shown highly effective for modeling sequences, capturing the more informative parts in learning a deep representation. However, recent studies show that the attention values do not always coincide with intuition in tasks, such as machine translation and sentiment classification. In this study, we consider using deep reinforcement learning to automatically optimize attention distribution during the minimization of end task training losses. With more sufficient environment states, iterative actions are taken to adjust attention weights so that more informative words receive more attention automatically. Results on different tasks and different attention networks demonstrate that our model is of great effectiveness in improving the end task performances, yielding more reasonable attention distribution. The more in-depth analysis further reveals that our retrofitting method can help to bring explainability for baseline attention.