An automatic analysis and quality assurance method for lymphocyte subset identification.

OBJECTIVES: Lymphocyte subsets are the predictors of disease diagnosis, treatment, and prognosis. Determination of lymphocyte subsets is usually carried out by flow cytometry. Despite recent advances in flow cytometry analysis, most flow cytometry data can be challenging with manual gating, which is labor-intensive, time-consuming, and error-prone. This study aimed to develop an automated method to identify lymphocyte subsets. METHODS: We propose a knowledge-driven combined with data-driven method which can gate automatically to achieve subset identification. To improve accuracy and stability, we have implemented a Loop Adjustment Gating to optimize the gating result of the lymphocyte population. Furthermore, we have incorporated an anomaly detection mechanism to issue warnings for samples that might not have been successfully analyzed, ensuring the quality of the results. RESULTS: The evaluation showed a 99.2 % correlation between our method results and manual analysis with a dataset of 2,000 individual cases from lymphocyte subset assays. Our proposed method attained 97.7 % accuracy for all cases and 100 % for the high-confidence cases. With our automated method, 99.1 % of manual labor can be saved when reviewing only the low-confidence cases, while the average turnaround time required is only 29 s, reducing by 83.7 %. CONCLUSIONS: Our proposed method can achieve high accuracy in flow cytometry data from lymphocyte subset assays. Additionally, it can save manual labor and reduce the turnaround time, making it have the potential for application in the laboratory.

Construction and effect evaluation of prediction model for red blood cell transfusion requirement in cesarean section based on artificial intelligence.

OBJECTIVES: This study intends to build an artificial intelligence model for obstetric cesarean section surgery to evaluate the intraoperative blood transfusion volume before operation, and compare the model prediction results with the actual results to evaluate the accuracy of the artificial intelligence prediction model for intraoperative red blood cell transfusion in obstetrics. The advantages and disadvantages of intraoperative blood demand and identification of high-risk groups for blood transfusion provide data support and improvement suggestions for the realization of accurate blood management of obstetric cesarean section patients during the perioperative period. METHODS: Using a machine learning algorithm, an intraoperative blood transfusion prediction model was trained. The differences between the predicted results and the actual results were compared by means of blood transfusion or not, blood transfusion volume, and blood transfusion volume targeting postoperative hemoglobin (Hb). RESULTS: Area under curve of the model is 0.89. The accuracy of the model for blood transfusion was 96.85%. The statistical standard for the accuracy of the model blood transfusion volume is the calculation of 1U absolute error, the accuracy rate is 86.56%, and the accuracy rate of the blood transfusion population is 45.00%. In the simulation prediction results, 93.67% of the predicted and actual cases in no blood transfusion surgery; 63.45% of the same predicted blood transfusion in blood transfusion surgery, and only 20.00% of the blood transfusion volume is the same. CONCLUSIONS: In conclusion, this study used machine learning algorithm to process, analyze and predict the results of a large sample of cesarean section clinical data, and found that the important predictors of blood transfusion during cesarean section included preoperative RBC, surgical method, the site of surgery, coagulation-related indicators, and other factors. At the same time, it was found that the overall accuracy of the AI model was higher than actual blood using. Although the prediction of blood transfusion volume was not well matched with the actual blood using, the model provided a perspective of preoperative identification of high blood transfusion risks. The results can provide good auxiliary decision support for preoperative evaluation of obstetric cesarean section, and then promote the realization of accurate perioperative blood management for obstetric cesarean section patients.

Auxiliary signal-guided knowledge encoder-decoder for medical report generation.

Medical reports have significant clinical value to radiologists and specialists, especially during a pandemic like COVID. However, beyond the common difficulties faced in the natural image captioning, medical report generation specifically requires the model to describe a medical image with a fine-grained and semantic-coherence paragraph that should satisfy both medical commonsense and logic. Previous works generally extract the global image features and attempt to generate a paragraph that is similar to referenced reports; however, this approach has two limitations. Firstly, the regions of primary interest to radiologists are usually located in a small area of the global image, meaning that the remainder parts of the image could be considered as irrelevant noise in the training procedure. Secondly, there are many similar sentences used in each medical report to describe the normal regions of the image, which causes serious data bias. This deviation is likely to teach models to generate these inessential sentences on a regular basis. To address these problems, we propose an Auxiliary Signal-Guided Knowledge Encoder-Decoder (ASGK) to mimic radiologists' working patterns. Specifically, the auxiliary patches are explored to expand the widely used visual patch features before fed to the Transformer encoder, while the external linguistic signals help the decoder better master prior knowledge during the pre-training process. Our approach performs well on common benchmarks, including CX-CHR, IU X-Ray, and COVID-19 CT Report dataset (COV-CTR), demonstrating combining auxiliary signals with transformer architecture can bring a significant improvement in terms of medical report generation. The experimental results confirm that auxiliary signals driven Transformer-based models are with solid capabilities to outperform previous approaches on both medical terminology classification and paragraph generation metrics.

Establishment of a Convenient System for the Culture and Study of Perineurium Barrier In Vitro.

The perineurium serves as a selective, metabolically active diffusion barrier in the peripheral nervous system, which is composed of perineurial cells joined together by tight junctions (TJs). Not only are these junctions known to play an essential role in maintaining cellular polarity and tissue integrity, but also limit the paracellular diffusion of certain molecules and ions, whereas loss of TJs barrier function is imperative for tumour growth, invasion and metastasis. Hence, a detailed study on the barrier function of perineurial cells may provide insights into the molecular mechanism of perineural invasion (PNI). In this study, we aimed to develop an efficient procedure for the establishment of perineurial cell lines as a tool for investigating the physiology and pathophysiology of the peripheral nerve barriers. Herein, the isolation, expansion, characterization and maintenance of perineurial cell lines under favourable conditions are presented. Furthermore, the analysis of the phenotypic features of these perineurial cells as well as the barrier function for the study of PNI are described. Such techniques may provide a valuable means for the functional and molecular investigation of perineurial cells, and in particular may elucidate the pathogenesis and progression of PNI, and other peripheral nerve disorders.

Identification of the Prognostic Value and Clinical Significance of Interferon Regulatory Factors (IRFs) in Colon Adenocarcinoma.

BACKGROUND Colon adenocarcinoma (COAD) is one of the most common malignant tumors and has high incidence and mortality rates. The interferon regulatory factor (IRF) family is known as a key transcription factor in the IFN signaling pathway and cellular immunity. This research explored the relationship between the IRF family and COAD through use of bioinformatics technology. MATERIAL AND METHODS Using the UALCAN and GEPIA databases, we analyzed the transcription and prognostic value of IRFs in COAD, and GSCALite was used in cancer genomics analysis. TIMER, LinkedOmics, and Metascape were used to assess the potential function of IRFs in COAD. RESULTS The transcription levels of IRF3 were elevated in COAD tissues, while IRF2/4/6 were downregulated compared with normal patients in subgroup analyses of race, age, weight, sex, nodal metastasis, individual cancer stages, TP53 mutation status, and histological subtypes. IRF3 and IRF7 in COAD were significantly associated with a poor prognosis. Drug sensitivity analysis revealed that the expression level of IRF2/4/8 was negatively associated with drug resistance. A significant correlation was found between the IRF family and immune cell infiltration. Moreover, enrichment analysis revealed that the IRFs were associated with response to tumor necrosis factor, transcription misregulation in cancer, and JAK-STAT signaling pathway. We also identified several kinase and miRNA targets of the IRF family in COAD. CONCLUSIONS We identified IRF3 and IRF7 as prognostic biomarkers in COAD, and the IRF family was associated with immune cell infiltration and gene regulation networks, providing additional evidence showing the significant role of the IRF family in COAD.

Deep learning-based subdivision approach for large scale macromolecules structure recovery from electron cryo tomograms.

MOTIVATION: Cellular Electron CryoTomography (CECT) enables 3D visualization of cellular organization at near-native state and in sub-molecular resolution, making it a powerful tool for analyzing structures of macromolecular complexes and their spatial organizations inside single cells. However, high degree of structural complexity together with practical imaging limitations makes the systematic de novo discovery of structures within cells challenging. It would likely require averaging and classifying millions of subtomograms potentially containing hundreds of highly heterogeneous structural classes. Although it is no longer difficult to acquire CECT data containing such amount of subtomograms due to advances in data acquisition automation, existing computational approaches have very limited scalability or discrimination ability, making them incapable of processing such amount of data. RESULTS: To complement existing approaches, in this article we propose a new approach for subdividing subtomograms into smaller but relatively homogeneous subsets. The structures in these subsets can then be separately recovered using existing computation intensive methods. Our approach is based on supervised structural feature extraction using deep learning, in combination with unsupervised clustering and reference-free classification. Our experiments show that, compared with existing unsupervised rotation invariant feature and pose-normalization based approaches, our new approach achieves significant improvements in both discrimination ability and scalability. More importantly, our new approach is able to discover new structural classes and recover structures that do not exist in training data. AVAILABILITY AND IMPLEMENTATION: Source code freely available at http://www.cs.cmu.edu/∼mxu1/software . CONTACT: mxu1@cs.cmu.edu. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Insights into the Dissociation Process and Binding Pattern of the BRCT7/8-PHF8 Complex.

DNA topoisomerase 2-binding protein 1 (Topbp1) plays a crucial role in activating the ataxia-telangiectasia mutated and rad3-related (ATR) complex to initiate DNA damage repair responses. For this process to occur, it is necessary for PHF8 to dissociate from Topbp1. Topbp1 binds to the acidic patch sequence (APS) of PHF8 through its C-terminal BRCT7/8 domain, and disrupting this interaction could be a promising strategy for cancer treatment. To investigate the dissociation process and binding pattern of BRCT7/8-PHF8, we employed enhanced sampling techniques, such as steered molecular dynamics (SMD) simulations and accelerated molecular dynamics (aMD) simulations, along with self-organizing maps (SOM) and time-resolved force distribution analysis (TRFDA) methodologies. Our results demonstrate that the dissociation of PHF8 from BRCT7/8 starts from the N-terminus, leading to the unfolding of the N-terminal helix. Additionally, we identified critical residues that play a pivotal role in this dissociation process. These findings provide valuable insights into the disassociation of PHF8 from BRCT7/8, which could potentially guide the development of novel drugs targeting Topbp1 for cancer therapy.

Unveiling dissociation mechanisms and binding patterns in the UHRF1-DPPA3 complex via multi-replica molecular dynamics simulations.

CONTEXT: Ubiquitin-like with PHD and RING finger domain containing protein 1 (UHRF1) is responsible for preserving the stability of genomic methylation through the recruitment of DNA methyltransferase 1 (DNMT1). However, the interaction between Developmental pluripotency associated 3 (DPPA3) and the pre-PHD-PHD (PPHD) domain of UHRF1 hinders the nuclear localization of UHRF1. This disruption has implications for potential cancer treatment strategies. Drugs that mimic the binding pattern between DPPA3 and PPHD could offer a promising approach to cancer treatment. Our study reveals that DPPA3 undergoes dissociation from the C-terminal through three different modes of helix unfolding. Furthermore, we have identified key residue pairs involved in this dissociation process and potential drug-targeting residues. These findings offer valuable insights into the dissociation mechanism of DPPA3 from PPHD and have the potential to inform the design of novel drugs targeting UHRF1 for cancer therapy. METHODS: To comprehend the dissociation process and binding patterns of PPHD-DPPA3, we employed enhanced sampling techniques, including steered molecular dynamics (SMD) and conventional molecular dynamics (cMD). Additionally, we utilized self-organizing maps (SOM) and time-resolved force distribution analysis (TRFDA) methodologies. The Gromacs software was used for performing molecular dynamics simulations, and the AMBER FF14SB force field was applied to the protein.

DNA Family: Boosting Weight-Sharing NAS With Block-Wise Supervisions.

Neural Architecture Search (NAS), aiming at automatically designing neural architectures by machines, has been considered a key step toward automatic machine learning. One notable NAS branch is the weight-sharing NAS, which significantly improves search efficiency and allows NAS algorithms to run on ordinary computers. Despite receiving high expectations, this category of methods suffers from low search effectiveness. By employing a generalization boundedness tool, we demonstrate that the devil behind this drawback is the untrustworthy architecture rating with the oversized search space of the possible architectures. Addressing this problem, we modularize a large search space into blocks with small search spaces and develop a family of models with the distilling neural architecture (DNA) techniques. These proposed models, namely a DNA family, are capable of resolving multiple dilemmas of the weight-sharing NAS, such as scalability, efficiency, and multi-modal compatibility. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using heuristic algorithms. Moreover, under a certain computational complexity constraint, our method can seek architectures with different depths and widths. Extensive experimental evaluations show that our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively. Additionally, we provide in-depth empirical analysis and insights into neural architecture ratings.

Correctable Landmark Discovery Via Large Models for Vision-Language Navigation.

Vision-Language Navigation (VLN) requires the agent to follow language instructions to reach a target position. A key factor for successful navigation is to align the landmarks implied in the instruction with diverse visual observations. However, previous VLN agents fail to perform accurate modality alignment especially in unexplored scenes, since they learn from limited navigation data and lack sufficient open-world alignment knowledge. In this work, we propose a new VLN paradigm, called COrrectable LaNdmark DiScOvery via Large ModEls (CONSOLE). In CONSOLE, we cast VLN as an open-world sequential landmark discovery problem, by introducing a novel correctable landmark discovery scheme based on two large models ChatGPT and CLIP. Specifically, we use ChatGPT to provide rich open-world landmark cooccurrence commonsense, and conduct CLIP-driven landmark discovery based on these commonsense priors. To mitigate the noise in the priors due to the lack of visual constraints, we introduce a learnable cooccurrence scoring module, which corrects the importance of each cooccurrence according to actual observations for accurate landmark discovery. We further design an observation enhancement strategy for an elegant combination of our framework with different VLN agents, where we utilize the corrected landmark features to obtain enhanced observation features for action decision. Extensive experimental results on multiple popular VLN benchmarks (R2R, REVERIE, R4R, RxR) show the significant superiority of CONSOLE over strong baselines. Especially, our CONSOLE establishes the new state-of-the-art results on R2R and R4R in unseen scenarios.

Point-Guided Contrastive Learning for Monocular 3-D Object Detection.

3-D object detection is a fundamental task in the context of autonomous driving. In the literature, cheap monocular image-based methods show a significant performance drop compared to the expensive LiDAR and stereo-images-based algorithms. In this article, we aim to close this performance gap by bridging the representation capability between 2-D and 3-D domains. We propose a novel monocular 3-D object detection model using self-supervised learning and auxiliary learning, resorting to mimicking the representations over 3-D point clouds. Specifically, given a 2-D region proposal and the corresponding instance point cloud, we supervise the feature activation from our image-based convolution network to mimic the latent feature of a point-based neural network at the training stage. While state-of-the-art (SOTA) monocular 3-D detection algorithms typically convert images to pseudo-LiDAR with depth estimation and regress 3-D detection with LiDAR-based methods, our approach seeks the power of the 2-D neural network straightforwardly and essentially enhances the 2-D module capability with latent spatial-aware representations by contrastive learning. We empirically validate the performance improvement from the feature mimicking the KITTI and ApolloScape datasets and achieve the SOTA performance on the KITTI and ApolloScape leaderboard.

DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Vision Transformers.

Dynamic networks have shown their promising capability in reducing theoretical computation complexity by adapting their architectures to the input during inference. However, their practical runtime usually lags behind the theoretical acceleration due to inefficient sparsity. In this paper, we explore a hardware-efficient dynamic inference regime, named dynamic weight slicing, that can generalized well on multiple dimensions in both CNNs and transformers (e.g. kernel size, embedding dimension, number of heads, etc.). Instead of adaptively selecting important weight elements in a sparse way, we pre-define dense weight slices with different importance level by nested residual learning. During inference, weights are progressively sliced beginning with the most important elements to less important ones to achieve different model capacity for inputs with diverse difficulty levels. Based on this conception, we present DS-CNN++ and DS-ViT++, by carefully designing the double headed dynamic gate and the overall network architecture. We further propose dynamic idle slicing to address the drastic reduction of embedding dimension in DS-ViT++. To ensure sub-network generality and routing fairness, we propose a disentangled two-stage optimization scheme. In Stage I, in-place bootstrapping (IB) and multi-view consistency (MvCo) are proposed to stablize and improve the training of DS-CNN++ and DS-ViT++ supernet, respectively. In Stage II, sandwich gate sparsification (SGS) is proposed to assist the gate training. Extensive experiments on 4 datasets and 3 different network architectures demonstrate our methods consistently outperform the state-of-the-art static and dynamic model compression methods by a large margin (up to 6.6%). Typically, we achieves 2-4× computation reduction and up to 61.5% real-world acceleration on MobileNet, ResNet-50 and Vision Transformer, with minimal accuracy drops on ImageNet. Code release: https://github.com/changlin31/DS-Net.

Knowledge Fusion Distillation: Improving Distillation with Multi-scale Attention Mechanisms.

The success of deep learning has brought breakthroughs in many fields. However, the increased performance of deep learning models is often accompanied by an increase in their depth and width, which conflicts with the storage, energy consumption, and computational power of edge devices. Knowledge distillation, as an effective model compression method, can transfer knowledge from complex teacher models to student models. Self-distillation is a special type of knowledge distillation, which does not to require a pre-trained teacher model. However, existing self-distillation methods rarely consider how to effectively use the early features of the model. Furthermore, most self-distillation methods use features from the deepest layers of the network to guide the training of the branches of the network, which we find is not the optimal choice. In this paper, we found that the feature maps obtained by early feature fusion do not serve as a good teacher to guide their own training. Based on this, we propose a selective feature fusion module and further obtain a new self-distillation method, knowledge fusion distillation. Extensive experiments on three datasets have demonstrated that our method has comparable performance to state-of-the-art distillation methods. In addition, the performance of the network can be further enhanced when fused features are integrated into the network.

Towards Causality-Aware Inferring: A Sequential Discriminative Approach for Medical Diagnosis.

Medical diagnosis assistant (MDA) aims to build an interactive diagnostic agent to sequentially inquire about symptoms for discriminating diseases. However, since the dialogue records for building a patient simulator are collected passively, the collected records might be deteriorated by some task-unrelated biases, such as the preference of the collectors. These biases might hinder the diagnostic agent to capture transportable knowledge from the simulator. This work identifies and resolves two representative non-causal biases, i.e., (i) default-answer bias and (ii) distributional inquiry bias. Specifically, Bias (i) originates from the patient simulator which tries to answer the unrecorded inquiries with some biased default answers. To eliminate this bias and improve upon a well-known causal inference technique, i.e., propensity score matching, we propose a novel propensity latent matching in building a patient simulator to effectively answer unrecorded inquiries; Bias (ii) inherently comes along with the passively collected data that the agent might learn by remembering what to inquire within the training data while not able to generalize to the out-of-distribution cases. To this end, we propose a progressive assurance agent, which includes the dual processes accounting for symptom inquiry and disease diagnosis respectively. The diagnosis process pictures the patient mentally and probabilistically by intervention to eliminate the effect of the inquiry behavior. And the inquiry process is driven by the diagnosis process to inquire about symptoms to enhance the diagnostic confidence which alters as the patient distribution changes. In this cooperative manner, our proposed agent can improve upon the out-of-distribution generalization significantly. Extensive experiments demonstrate that our framework achieves new state-of-the-art performance and possesses the advantage of transportability.

Discourse-Aware Graph Networks for Textual Logical Reasoning.

Textual logical reasoning, especially question-answering (QA) tasks with logical reasoning, requires awareness of particular logical structures. The passage-level logical relations represent entailment or contradiction between propositional units (e.g., a concluding sentence). However, such structures are unexplored as current QA systems focus on entity-based relations. In this work, we propose logic structural-constraint modeling to solve the logical reasoning QA and introduce discourse-aware graph networks (DAGNs). The networks first construct logic graphs leveraging in-line discourse connectives and generic logic theories, then learn logic representations by end-to-end evolving the logic relations with an edge-reasoning mechanism and updating the graph features. This pipeline is applied to a general encoder, whose fundamental features are joined with the high-level logic features for answer prediction. Experiments on three textual logical reasoning datasets demonstrate the reasonability of the logical structures built in DAGNs and the effectiveness of the learned logic features. Moreover, zero-shot transfer results show the features' generality to unseen logical texts.

Template-Based Contrastive Distillation Pretraining for Math Word Problem Solving.

Since math word problem (MWP) solving aims to transform natural language problem description into executable solution equations, an MWP solver needs to not only comprehend the real-world narrative described in the problem text but also identify the relationships among the quantifiers and variables implied in the problem and maps them into a reasonable solution equation logic. Recently, although deep learning models have made great progress in MWPs, they ignore the grounding equation logic implied by the problem text. Besides, as we all know, pretrained language models (PLM) have a wealth of knowledge and high-quality semantic representations, which may help solve MWPs, but they have not been explored in the MWP-solving task. To harvest the equation logic and real-world knowledge, we propose a template-based contrastive distillation pretraining (TCDP) approach based on a PLM-based encoder to incorporate mathematical logic knowledge by multiview contrastive learning while retaining rich real-world knowledge and high-quality semantic representation via knowledge distillation. We named the pretrained PLM-based encoder by our approach as MathEncoder. Specifically, the mathematical logic is first summarized by clustering the symbolic solution templates among MWPs and then injected into the deployed PLM-based encoder by conducting supervised contrastive learning based on the symbolic solution templates, which can represent the underlying solving logic in the problems. Meanwhile, the rich knowledge and high-quality semantic representation are retained by distilling them from a well-trained PLM-based teacher encoder into our MathEncoder. To validate the effectiveness of our pretrained MathEncoder, we construct a new solver named MathSolver by replacing the GRU-based encoder with our pretrained MathEncoder in GTS, which is a state-of-the-art MWP solver. The experimental results demonstrate that our method can carry a solver's understanding ability of MWPs to a new stage by outperforming existing state-of-the-art methods on two widely adopted benchmarks Math23K and CM17K. Code will be available at https://github.com/QinJinghui/tcdp.

Microstructure and Biomechanical Properties in Selective Laser Melting of Porous Metal Implants.

Two kinds of porous structure design strategies, ring-support (RS) and column-support (CS), are proposed for human implants. The accurate design of porosity is realized by adjusting the pore characteristics, such as strut diameter, pore diameter, and unit size. Porous specimens with porosity of 50%, 60%, 70%, and 80% were prepared by selective laser melting. The three-dimensional pore structure is basically consistent with the design characteristics, and the measured porosity is slightly lower than design value. The microstructure, microhardness, and friction and wear properties of the samples were studied. The results show that the performance along the scanning orientation is slightly better than that along the forming orientation. The compression and dynamic elastic modulus of porous specimens with different structures and porosities were analyzed. The CS porous with 60-80% porosity has suitable compressive strength and elastic modulus, which is close to that of human tissue, and effectively avoids the stress shielding phenomenon.

Dynamic Support Network for Few-Shot Class Incremental Learning.

Few-shot class-incremental learning (FSCIL) is challenged by catastrophically forgetting old classes and over-fitting new classes. Revealed by our analyses, the problems are caused by feature distribution crumbling, which leads to class confusion when continuously embedding few samples to a fixed feature space. In this study, we propose a Dynamic Support Network (DSN), which refers to an adaptively updating network with compressive node expansion to "support" the feature space. In each training session, DSN tentatively expands network nodes to enlarge feature representation capacity for incremental classes. It then dynamically compresses the expanded network by node self-activation to pursue compact feature representation, which alleviates over-fitting. Simultaneously, DSN selectively recalls old class distributions during incremental learning to support feature distributions and avoid confusion between classes. DSN with compressive node expansion and class distribution recalling provides a systematic solution for the problems of catastrophic forgetting and overfitting. Experiments on CUB, CIFAR-100, and miniImage datasets show that DSN significantly improves upon the baseline approach, achieving new state-of-the-arts.

Entity-Graph Enhanced Cross-Modal Pretraining for Instance-Level Product Retrieval.

Our goal in this research is to study a more realistic environment in which we can conduct weakly-supervised multi-modal instance-level product retrieval for fine-grained product categories. We first contribute the Product1M datasets and define two real practical instance-level retrieval tasks that enable evaluations on price comparison and personalized recommendations. For both instance-level tasks, accurately identifying the intended product target mentioned in visual-linguistic data and mitigating the impact of irrelevant content are quite challenging. To address this, we devise a more effective cross-modal pretraining model capable of adaptively incorporating key concept information from multi-modal data. This is accomplished by utilizing an entity graph, where nodes represented entities and edges denoted the similarity relations between them. Specifically, a novel Entity-Graph Enhanced Cross-Modal Pretraining (EGE-CMP) model is proposed for instance-level commodity retrieval, which explicitly injects entity knowledge in both node-based and subgraph-based ways into the multi-modal networks via a self-supervised hybrid-stream transformer. This could reduce the confusion between different object contents, thereby effectively guiding the network to focus on entities with real semantics. Experimental results sufficiently verify the efficacy and generalizability of our EGE-CMP, outperforming several SOTA cross-modal baselines like CLIP Radford et al. 2021, UNITER Chen et al. 2020 and CAPTURE Zhan et al. 2021.

Fine-Grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection.

Inspired by the success of vision-language methods (VLMs) in zero-shot classification, recent works attempt to extend this line of work into object detection by leveraging the localization ability of pretrained VLMs and generating pseudolabels for unseen classes in a self-training manner. However, since the current VLMs are usually pretrained with aligning sentence embedding with global image embedding, the direct use of them lacks fine-grained alignment for object instances, which is the core of detection. In this article, we propose a simple but effective fine-grained visual-text prompt-driven self-training paradigm for open-vocabulary detection (VTP-OVD) that introduces a fine-grained visual-text prompt adapting stage to enhance the current self-training paradigm with a more powerful fine-grained alignment. During the adapting stage, we enable VLM to obtain fine-grained alignment using learnable text prompts to resolve an auxiliary dense pixelwise prediction task. Furthermore, we propose a visual prompt module to provide the prior task information (i.e., the categories need to be predicted) for the vision branch to better adapt the pretrained VLM to the downstream tasks. Experiments show that our method achieves the state-of-the-art performance for open-vocabulary object detection, e.g., 31.5% mAP on unseen classes of COCO.